{ "id": "173563", "key": "TIMOB-27046", "fields": { "issuetype": { "id": "7", "description": "gh.issue.story.desc", "name": "Story", "subtask": false }, "project": { "id": "10153", "key": "TIMOB", "name": "Titanium SDK/CLI", "projectCategory": { "id": "10100", "description": "Titanium and related SDKs used in application development", "name": "Client" } }, "fixVersions": [], "resolution": null, "resolutiondate": null, "created": "2019-04-30T14:24:39.000+0000", "priority": { "name": "None", "id": "6" }, "labels": [], "versions": [], "issuelinks": [], "assignee": null, "updated": "2019-04-30T14:24:39.000+0000", "status": { "description": "The issue is open and ready for the assignee to start work on it.", "name": "Open", "id": "1", "statusCategory": { "id": 2, "key": "new", "colorName": "blue-gray", "name": "To Do" } }, "components": [], "description": "In our mocha unit test suite for the SDK we use the platform and \"target\" to prefix the test results (i.e. \"ios\" or \"android.device\") so we can tell what specific os/target combination may have failed a test.\r\n\r\nWe now have a setup with a build node having 3 separate physics android devices attached and our build runs on all 3 simultaneously.\r\n\r\nWe have two issues:\r\n1. While we get the results from all three devices for a given test, we have no way od knowing which one may have failed a test because the report does not use any unique prefix to differentiate\r\n2. It's possible (though I have not tested this yet) that the \"first one wins\" - i.e. that whatever device runs to completion first effectively ends the recording of results for the others.\r\n\r\nWe should look into modifying how our test suite runs so that we can dynamically report some data on the target device to use for prefixing the reports (i.e. OS version, device name, etc).\r\n\r\nAdditionally, we should consider using a unique id to label each test result/log so we can associate the test results to a particular device.\r\n\r\nThe obvious possible workaround here is to *not* run on all the devices in parallel on the same machine but to run them in series, or to break them out to separate build nodes so they could be run in parallel but to tag them with specific labels so we ensure we run on one of each major OS version we want.", "attachment": [], "flagged": false, "summary": "Testing: differentiate between multiple test report results", "creator": { "name": "cwilliams", "key": "cwilliams", "displayName": "Christopher Williams", "active": true, "timeZone": "America/New_York" }, "subtasks": [], "reporter": { "name": "cwilliams", "key": "cwilliams", "displayName": "Christopher Williams", "active": true, "timeZone": "America/New_York" }, "environment": null, "comment": { "comments": [], "maxResults": 0, "total": 0, "startAt": 0 } } }