[ALOY-1476] Travis does not test against known good generated code because it targets Linux only

GitHub Issue	n/a
Type	Story
Priority	High
Status	Closed
Resolution	Fixed
Resolution Date	2016-03-28T16:25:35.000+0000
Affected Version/s	alloy 1.8.2
Fix Version/s	alloy 1.8.3, Release 5.2.2
Components	Testing
Labels	n/a
Reporter	Fokke Zandbergen
Assignee	Fokke Zandbergen
Created	2016-03-28T11:42:10.000+0000
Updated	2016-04-05T23:12:49.000+0000

Description

While working on ALOY-1474 and local tests I ran into 355 failures to match against known good generated code, while Travis didn't fail at all. After submitting https://github.com/appcelerator/alloy/pull/775 I noticed Travis logs don't mention the tests against the known good generated code at all:

[PASS] alloy compile advanced/device_query --> preparing test app
[PASS] alloy compile advanced/device_query [android] --> compiles without critical error
[PASS] alloy compile advanced/device_query [android] --> leaves no compiler directives in generated code
[PASS] alloy compile advanced/device_query [android] --> has no undefined style entries
[PASS] alloy compile advanced/device_query [ios] --> compiles without critical error
[PASS] alloy compile advanced/device_query [ios] --> leaves no compiler directives in generated code
[PASS] alloy compile advanced/device_query [ios] --> has no undefined style entries
[PASS] alloy compile advanced/device_query [mobileweb] --> compiles without critical error
[PASS] alloy compile advanced/device_query [mobileweb] --> leaves no compiler directives in generated code
[PASS] alloy compile advanced/device_query [mobileweb] --> has no undefined style entries

while locally it fails:

[PASS] alloy compile advanced/device_query --> preparing test app
[PASS] alloy compile advanced/device_query [android] --> compiles without critical error
[PASS] alloy compile advanced/device_query [android] --> leaves no compiler directives in generated code
[PASS] alloy compile advanced/device_query [android] --> has no undefined style entries
[PASS] alloy compile advanced/device_query [android] --> generated a alloy/controllers/index.js file
[FAIL] alloy compile advanced/device_query [android] --> matches known good generated code for alloy/controllers/index.js
[PASS] alloy compile advanced/device_query [ios] --> compiles without critical error
[PASS] alloy compile advanced/device_query [ios] --> leaves no compiler directives in generated code
[PASS] alloy compile advanced/device_query [ios] --> has no undefined style entries
[PASS] alloy compile advanced/device_query [ios] --> generated a alloy/controllers/index.js file
[FAIL] alloy compile advanced/device_query [ios] --> matches known good generated code for alloy/controllers/index.js
[PASS] alloy compile advanced/device_query [mobileweb] --> compiles without critical error
[PASS] alloy compile advanced/device_query [mobileweb] --> leaves no compiler directives in generated code
[PASS] alloy compile advanced/device_query [mobileweb] --> has no undefined style entries
[PASS] alloy compile advanced/device_query [mobileweb] --> generated a alloy/controllers/index.js file
[FAIL] alloy compile advanced/device_query [mobileweb] --> matches known good generated code for alloy/controllers/index.js

I also saw this test only runs on darwin: https://github.com/appcelerator/alloy/blob/master/test/specs/compile.js#L173 We don't specify an OS: https://github.com/appcelerator/alloy/blob/master/.travis.yml Which seems to make Travis to only test on Linux: https://travis-ci.org/appcelerator/alloy/builds/118954721 Unfortunately enabling osx on Travis fails: https://travis-ci.org/appcelerator/alloy/jobs/118956715 Because of: https://github.com/travis-ci/travis-ci/issues/2311 Which can be workaround like: https://github.com/travis-ci/travis-ci/issues/2311#issuecomment-171180704 Which I did: https://github.com/appcelerator/alloy/pull/775/files#diff-354f30a63fb0907d4ad57269548329e3L1 And now Travis is failing 355 tests on OSX just like local: https://travis-ci.org/appcelerator/alloy/builds/118959856 I've checked all 355 failures and all were expected because of tests that were not updated for tickets like ALOY-1355, ALOY-1237 and ALOY-1335, so generated new known good code for all.

Comments

Fokke Zandbergen 2016-03-28

Included this fix in https://github.com/appcelerator/alloy/pull/775 for ALOY-1475
Feon Sua Xin Miao 2016-03-28

PR merged.
Fokke Zandbergen 2016-03-28

[~cng] we are already at 1.8.2 - this would be 1.8.3

JSON Source