[TIMOB-27744] Improve the speed of SDK installs
| GitHub Issue | n/a |
|---|---|
| Type | Improvement |
| Priority | None |
| Status | Open |
| Resolution | Unresolved |
| Affected Version/s | n/a |
| Fix Version/s | n/a |
| Components | CLI |
| Labels | cb-tooling |
| Reporter | Christopher Williams |
| Assignee | Eric Merriman |
| Created | 2020-01-31T21:27:44.000+0000 |
| Updated | 2020-02-06T16:23:05.000+0000 |
Description
Our CLI seems to be extra slow in terms of installing SDKs. Particularly, I believe the extracting portion is much slower than anticipated. I suspect our custom progressive's bar/unzip implementation may be introducing a significant performance penalty. (we forked the npm package 'progress' long ago and are using 30 ms timeouts to spit out progress, but also introduced a setTimeout in the unzip loop to let it render!)
I don't have exact timings, but ideally we'd just delegate to the
unzip command on Linux/macOS to do the work - particularly when we we're extracting without progress bars.
There's some unix pipe magic going on here for linux: https://medium.com/@takanori.ishikawa/how-to-show-progress-bar-while-unzipping-tons-of-files-7b1ea7a84b01
Note that I intentionally avoided using node-app's unzip in our SDK build scripts when we unzip native module zips for this exact reason... (and hand-rolled some code to just call unzip)
You are correct, the extraction is slow. We used to use the system unzip command, but that didn't exist on Windows and so we actually shipped the 7-Zip binary for Windows. Ultimately, there was a discussion to remove 7-Zip and use a pure JavaScript solution, namely adm-zip to extract and archiver to create. This simplified distribution and unlike spawning unzip, we can report extraction progress. The idea is people would be better waiting a bit longer just knowing it was working. Nowadays, we use yauzl (https://www.npmjs.com/package/yauzl) instead of adm-zip. I doubt it is as fast as the unzip command, but I doubt it's slower than adm-zip. In any case, you should be using our titaniumlib (https://github.com/appcelerator/titaniumlib) to install SDKs. It supports GA releases, branches, CI builds, arbitrary URLs, local zip files, you name it.
Well, I'm using our titanium CLI. Wed have to update it to use titaniumlib. Anyways, even if we decide *not* to use a native unzip for performance, we could have gains by ding a streaming unzip int he common case we have to download and unzip - i.e. don't download it all and then unzip, but unzip in a stream so we download and extract in parallel.
No can do. A zip file has a central directory is located at the end of the file. Each file entry does have a header, but it would be considered bad practice to use the filename, attributes, etc from the headers. So, you need to download the entire file. Once downloaded, the unzip library streams the decompressed bytes back to disk per file. For more info, please look at https://www.npmjs.com/package/yauzl#no-streaming-unzip-api. I'd be curious to see some benchmarks between what node-appc uses (adm-zip) and what titaniumlib uses (yauzl).
Side note, I was just updating my Android SDK and I noticed they download the entire zip file before extracting. This seems to be a common practice.