[TIMOB-27046] Testing: differentiate between multiple test report results
GitHub Issue | n/a |
---|---|
Type | Story |
Priority | None |
Status | Open |
Resolution | Unresolved |
Affected Version/s | n/a |
Fix Version/s | n/a |
Components | n/a |
Labels | n/a |
Reporter | Christopher Williams |
Assignee | Unknown |
Created | 2019-04-30T14:24:39.000+0000 |
Updated | 2019-04-30T14:24:39.000+0000 |
Description
In our mocha unit test suite for the SDK we use the platform and "target" to prefix the test results (i.e. "ios" or "android.device") so we can tell what specific os/target combination may have failed a test.
We now have a setup with a build node having 3 separate physics android devices attached and our build runs on all 3 simultaneously.
We have two issues:
1. While we get the results from all three devices for a given test, we have no way od knowing which one may have failed a test because the report does not use any unique prefix to differentiate
2. It's possible (though I have not tested this yet) that the "first one wins" - i.e. that whatever device runs to completion first effectively ends the recording of results for the others.
We should look into modifying how our test suite runs so that we can dynamically report some data on the target device to use for prefixing the reports (i.e. OS version, device name, etc).
Additionally, we should consider using a unique id to label each test result/log so we can associate the test results to a particular device.
The obvious possible workaround here is to *not* run on all the devices in parallel on the same machine but to run them in series, or to break them out to separate build nodes so they could be run in parallel but to tag them with specific labels so we ensure we run on one of each major OS version we want.
No comments