[TIMOB-24986] Investigate the skipped tests on Windows
| GitHub Issue | n/a |
|---|---|
| Type | Sub-task |
| Priority | None |
| Status | Closed |
| Resolution | Done |
| Resolution Date | 2017-07-25T12:42:26.000+0000 |
| Affected Version/s | n/a |
| Fix Version/s | n/a |
| Components | n/a |
| Labels | n/a |
| Reporter | Ewan Harris |
| Assignee | Ewan Harris |
| Created | 2017-07-20T12:45:38.000+0000 |
| Updated | 2017-07-25T12:42:31.000+0000 |
Description
There are around 500 skipped tests on Windows, the total number of tests is only around 1000 so half of them being skipped is not good and they almost definitely do not have ticket associated with them if they're skipped for bugs, I know I have skipped some of these in frustration when merging the suites.
Review the attached skipped list and then run through the tests, reviewing what can be unskipped as it now works, filing tickets as needed.
I don't think we'd be able to achieve a code coverage output so in my opinion we should have a mechanism where we require a percentage of tests to be run to allow us to measure this, while it's not a great metric it is something to aim for and stop the bulk skipping of tests
Ran through all but Ti.UI, here are my notes
Note that seems to run fine means the app doesn't crash, it does not mean the tests all pass :) Titanium.Analytics has 1 skipped tests - filterEvents() is not implemented on Windows Titanium.API has 1 skipped tests - timestamp() is not implemented on Windows and Android Titanium.App has 5 skipped tests - disableNetworkActivityIndicator is an iOS only thing - getDisableNetworkActivityIndicator is an iOS only thing - forceSplashAsSnapshot is an iOS only thing - getForceSplashAsSnapshot is an iOS only thing - searchQuery is an iOS only thing (also should be moved to an Ti.App.iOS) Titanium.App.Windows.BackgroundService has 6 skipped tests - Meh Titanium.Blob has 3 skipped tests - apiName unskipped - constructed from File.read() - It fails for some reason, unskipped though as it doesn't kill the app - constructed from image, accessing props on the blob seems to just kill the app :) Global has 1 skipped tests - global no idea why Titanium.Contacts has 27 skipped tests - Was skipped in frustration - should(queriedPerson).not.be.null; kills the app so skipped person add/remove. Other than that, all good in the hood Titanium.Contacts.Person has 32 skipped tests - Was skipped in frustration, unskipped, looks good? Titanium.Filesystem has 20 skipped tests - Suite was skipped, unskipped and runs fine with failures Titanium.Filesystem.File has 41 skipped tests - Suite was skipped, unskipped ad seems to run fine with failures, need to revisit when reduced Titanium.Filesystem.FileStream has 7 skipped tests - Suite was skipped, unskipped, runs fine with failures, but there are logs after the tests have finished? Titanium.Geolocation has 18 skipped tests - Prompting on mobile device family I think breaks the test run, we Sneed to decide how to handle these situations Titanium.Locale has 3 skipped tests - No crashes, just errors, unskipped them all Titanium.Map has 30 skipped tests - Seems to run fine, reminds me we need to split this out from core Titanium.Media.AudioPlayer has 10 skipped tests - Seems to run fine, unskipped! Titanium.Media.Sound has 10 skipped tests - Seems to run fine, unskipped! Titanium.Network.HTTPClient has 8 skipped tests - Network tests are inherently flakey due to environmental issues, I propose we remove these from unit tests or make it not dependent on an external network connection Titanium.Platform has 28 skipped tests - Looks to run fine, unskipped! Titanium.Platform.DisplayCaps has 14 skipped tests - Looks to run fine, unskipped! Titanium has 18 skipped tests - Looks to run fine, unskipped! Titanium.UI.Button has 1 skipped tests - image(Blob) crashes the app, other than that runs fine Titanium.UI has 147 skipped tests - All constants :/, might be a pita Titanium.UI.ImageView has 14 skipped tests - Just in a bad place right now, keeping skipped Titanium.UI.Label has 1 skipped tests - Runs fine, unskipped Titanium.UI.Layout has 16 skipped tests - Some tests in this suite (as stated in comments) seem kinda funky - TIMOB-23372 #10 crashes the app, skipped other than that runs fine Titanium.UI.ListView has 3 skipped tests - ListItem.properties is iOS only, other than that skipped Titanium.UI.ScrollView has 5 skipped tests - scrollType is android only - maxZoomScale is iOS only - minZoomScale is ios only - overScrollMode is android only Titanium.UI.Switch has 1 skipped tests - apiname unskipped Titanium.UI.TableView has 1 skipped tests - unskipped runs fine Titanium.UI.TextField has 2 skipped tests - unskipped runs fine Titanium.UI.WebView has 1 skipped tests - unskipped runs fine Titanium.UI.Window has 2 skipped tests - unskipped, runs fine Titanium.Utils has 1 skipped tests - unskipped, runs fine Titanium.XML has 12 skipped tests - unskipped, runs fineCurrent list of skipped tests that are need to be investigated, this means that either they crash the app, are noted as questionable tests in the code or are flaky (HTTPClient)
Considering this done