[TIMOB-8640] CLI: Evaluate using node.js in build scripts
| GitHub Issue | n/a |
|---|---|
| Type | Sub-task |
| Priority | Medium |
| Status | Closed |
| Resolution | Fixed |
| Resolution Date | 2012-04-13T12:35:13.000+0000 |
| Affected Version/s | n/a |
| Fix Version/s | Release 2.1.0, Sprint 2012-08 |
| Components | Tooling |
| Labels | core |
| Reporter | Stephen Tramer |
| Assignee | Stephen Tramer |
| Created | 2012-04-09T17:16:48.000+0000 |
| Updated | 2012-06-15T16:29:55.000+0000 |
Description
We need to investigate what it would take to migrate build infrastructure from python to node.js (or another alternative, ideally a solution that allows us to use JS end-to-end.) We need to figure out issues like:
* Distribution (footprint, installation/distribution, etc.)
* API assessment
* Supporting infrastructure
* IPC (for studio and drillbit/simulator integration)
Here are the results of the initial investigation:
Footprint
~7.5MB (base install, no npm packages, from brew on OS X)Distribution:
Windows: MSI, chocolatey * From the README on git: "Windows builds are not yet satisfactorily stable." * Can distribute the same way we do python now (complete install 'encapsulated' in Studio or pulled and installed from MSI, which is called out to by a plugin) OS X: brew, port, pkg installer, source Linux: deb, apt, RPM, yum, pacman, source For OS X and linux, either we would need brew, port, or a package manager (identified by command availability?) installed (to facilitate updating.) An advantage of doing such is that we could start bundling mobile releases (and modules?) as repos for package management tools (brew would be especially easy.) Another alternative is to simply lock a node version that gets distributed with Studio only, so that we can be guaranteed a consistent API, although this will take some work to periodically upgrade. Rapid releases and unstable development make working with node (esp. on Windows) much more difficult; we'd need a way to autoupdate (does the MSI install or OS X PKG do this? package managers certainly can)Supported required featuresets
* Support natively: ** Async sockets/HTTP ** Child process fork with I/O redirect ** 'assert' (can replace drillbit asserts on server-side, possibly) ** Filesystem access / pathing tools * Support from npm: ** mysql (Studio registry interaction) ** xcode (!!! - but last 63 days ago, no repo listed on npm) ** *NOTE:* No direct adb or other android support (internal project?) * Other robust npm support: ** colors ** commander (robust command-line invoke module) ** underscore (ruby-esque features)Advantages:
* End-to-end javascript * CLI can serve as a drillbit driver and command-line debug interface * Makes it easier to get non-PE involved in build modifications (i.e. CS/CE can deliver custom build solutions with minimal extra JS training) * Because node acts as a server, this may give us some distributed build or other automation capabilities * Eclipse as node debug driver: https://github.com/joyent/node/wiki/Using-Eclipse-as-Node-Applications-DebuggerAlternatives:
The primary concern for Node at this point is good Windows support, but because Microsoft has selected node.js as part of the Azure Cloud infrastructure, it's likely that node will rapidly improve in this area (possibly becoming robust long before we're ready to release a new CLI.) Basically the only real alternative we have at this point is another robust scripting language, likely either Python or Ruby. The primary disadvantage for either of these is good async server support (which both have, as external modules/plugins) which we might want for automation, debugging, etc. The other primary disadvantages would be the lack of end-to-end JS and additional training (or involvement of Core PE) for delivering custom build solutions that customers may require.We recommend against node as the primary driver for the new CLI, due to the lack of robustness, rapid speed of the project, and existence of suitable alternatives.
Overview
I, along with the help of Bryan Hughes, have performed our own technical analysis of node.js and found it to be viable solution for command line based tools. We identified a number of key requirements, some of which are driven by the needs of Mobile Web. * Distributable * Synchronous and asynchronous * File I/O * File/Directory management * Subprocesses * Logging * Image resizing * JSON parsing/serializing * Text encoding * Templates * Zip file creation/extraction * Web server w/ proxy support * Package system w/ dependency support * XML parsingDistributable
We are of the belief that any dependencies of Titanium Mobile should be distributed with or downloaded by Titanium Mobile. This includes the runtime that the CLI system depends on. node.js would be easy to bundle with Titanium Mobile. It's a single executable binary file that clocks in at ~7MB on Mac OS X, ~10MB on Linux, and ~5MB on Windows. All libraries node.js uses are statically linked into the executable. In addition to the node executable, node.js we would bundle npm for package management. npm is just under 3MB. npm is distributed as a node module which would be just one of the potentially many modules we may wish to distribute with Titanium Mobile. Both node.js and npm are licensed under the MIT license and are compatible with Titanium Mobile's license. We find that node.js to be suitable for distribution with Titanium Mobile.Synchronous and Asynchronous
Build systems generally perform tasks serially, one at a time. Build times can be reduced by processing tasks in parallel. However a specific task is usually a series of synchronous tasks. Most of node.js's built-in file system functions have both synchronous and asynchronous versions. There are a handful of operations that are only asynchronous where we'd need a way to synchronize them. There are three basic strategies for making asynchronous operations synchronous.C++ addon
File system synchronization
Async task framework
C++ Addon
node.js supports C++ "addons" that plug directly into V8. If we need synchronous HTTP requests or subprocesses, we can custom build a C++ solution. Pros: * We can do literally anything * C++ code is very fast Cons: * Requires C++ experience * Modules are homebrewed and not battle tested * We would need to distribute binaries for each platformFile system synchronization
There is a hack where we can block node.js until an asynchronous process completes. The hack consists of the following: * Synchronously write a small file containing JavaScript code that performs the async only function * Asynchronously execute another node process that runs that small JavaScript file * In a while(true) loop, synchronously read a temp file that marks the completion of the asynchronous node subprocess The synchronous file access forces node.js to context switch allowing the kernel to schedule time to the node subprocess. Pros: * Pure JavaScript solution * Synchronous subprocess implementation already exists in shell.js module Cons: * Requires at least two temp files * Continuous synchronous file reads in while loop may affect performance of subprocessAsync Task Framework
Perhaps the best strategy is to go with the flow of node.js. If node is async, then a framework that embraces node's asynchronous concepts would allow us to adapt to both sync and async tasks. This is not a new concept. In fact, it closely resembles the same process in Mobile Web's AMD loader. Bryan has built a prototype to demonstrate a solution. The idea is to define multiple tasks that would run asynchronously. Tasks can depend on the completion of other tasks. When a task completes, it simply fires the finished() or error() callback. ~~~ var fs = require("fs"), Task = require("task"), t = new Task; t.add("resize images", function(finished, error) { // resize some images finished(); }); t.add("copy images", ["resize images"], function(finished, error) { // copy some images cp("source", "dest") ? finished() : error(); }); t.run(); ~~~ Pros: * Pure JavaScript solution * Gracefully handles async and sync functions Cons: * Just requires a little different thinkingSynchronous and Asynchronous Conclusion
After comparing the 3 solutions, we recommend using a async task framework, namely the one Bryan has started building. Only in special situations should we resort to C++ addons. File system synchronization is a hack and should only be considered as a fallback for a C++ addon. We find that node.js to be suitable for both synchronous and asynchronous tasks with a little help from a module/addon.Package system w/ dependency support
We need the ability to bundle up files, binaries, documentation, and examples into distributable packages. The npm (node.js package manager) provides a means to installing packages and their dependencies. Packages can contain multiple modules and C++ addons. In the top-level directory of the package is a manifest file called "package.json". This file describes various meta data for the package including dependencies. When a package is downloaded, it's dependencies are fetched and installed. npm is flexible and allows us to host our own packages. Packages can be installed a couple of ways including by specifying a URL to a tarball or from a git repository. We find that package with dependency support via npm to be suitable for Titanium Mobile.File I/O
node.js has a file system module containing various functions for manipulating files. The easiest interface for reading files is the readFile() and readFileSync() functions. Similarly, writing files is writeFile() and writeFileSync(). ~~~ var fs = require("fs"); var passwd = fs.readFileSync("/etc/passwd"); ~~~ The file system module also offers generic read() and write() functions for writing to any sort of file descriptor. We find that the file system functions for file I/O to be suitable for Titanium Mobile.File/Directory Manipulation
The file system module also provides functions for virtually every file or directory operation Titanium Mobile would need such as creating, listing, copying, and deleting directories. ~~~ var fs = require("fs"); fs.readdirSync("/etc").forEach(function(file) { console.log(file); }); fs.mkdirSync("/tmp/titanium"); fs.rmdirSync("/tmp/titanium"); ~~~ Moving a file or directory will require a helper function that would copy a source to a destination, then delete the source. There is an excellent node module called [shell.js](https://github.com/arturadib/shelljs) that defines a number of functions that mimic Unix command names including cd(), ls(), mkdir(), pwd(). One interesting function is exec() for which support synchronous subprocesses. We find that the file system functions for file manipulation to be suitable for Titanium Mobile.Subprocesses
Subprocesses are either started using node's exec() or spawn() functions. Unfortunately, these are both asynchronous. To make these synchronous, we would need to use one of the methods described in the Synchronous and Asynchronous section. exec() is capable of running a subprocess and when the subprocess completes, a callback is fired with the stdout and stderr. exec() doesn't support interactive execution and thus has no way of passing data to the subprocess over stdin. spawn() on the other hand is capable of passing data over stdin to the subprocess as well as receiving data from stdout and stderr while the subprocess is running. Based on a quick survey of all subprocess calls in the current Titanium Mobile build scripts, it appears that exec() would be sufficient for nearly all scenarios. We find that subprocess support to be suitable (and in some cases with the help of a module/addon) for Titanium Mobile.JSON parsing/serializing
node.js has built-in support for both parsing and serializing JSON data. ~~~ var data = { a:1 }; var str = JSON.stringify(data); console.log(JSON.parse(str)); ~~~ We find that JSON support in node.js to be suitable for Titanium Mobile.Logging
node.js has built-in functions logging messages to the terminal (console). The console functions support automatically stringifying objects and arrays. While log() and info() calls output to stdout, warn() and error() output to stderr. A more sophisticated logger that could ignore log messages below a specific log level is desired. Additionally, it might be beneficial to support logging to various targets (both file system and network based). Since node.js's logging mechanisms aren't capable of doing sophisticated logging, we either have to use one of the two dozen loggers in npm or write our own. Writing our own logger is trivial. Loggers in npm should be evaluated and if there isn't a perfect solution, we can use it as a starting point for our own logger. It's worth mentioning that it's possible to render color text in the terminal which might be appealing for the Titanium Mobile CLI. We find that there are sufficient logging modules or a custom logger can be written that would be suitable for Titanium Mobile.Image resizing
node.js does not have any built-in functions to resize images, however there is an ImageMagick module that we can use. Somewhat related, there is a node.js module called [node-png](https://github.com/pkrumins/node-png) that we can use to optimize PNG images. We find that are suitable image resizing solutions for Titanium Mobile.Text encoding
node.js supports "ascii", "utf8", "ucs2", "base64", "binary", and "hex" encodings via the built-in Buffer object. node.js does not have support for recoding ISO-8859-1 text, though there is a module that wraps iconv that does (*nix only). The algorithm to convert ISO-8859-1 text to UTF-8 and back should be relatively trivial to implement. Since the vast majority of the text and code will be in UTF-8, we find node.js to be suitable for encoding text for Titanium Mobile.Templates
There are number of places where we need a template engine that will do simple variable substitution as well as scrub text for HTML entities and trim strings. While node.js doesn't have a built-in template engine, there are several in npm that would work. A module such as [handlebars.js](https://github.com/wycats/handlebars.js) would get the job done. We find that there are many viable template engine modules that would be suitable for Titanium Mobile.Web server w/ proxy support
node.js ships with an HTTP server that is capable of both http and https connections. Furthermore, there are a number of web server modules for node.js that implement missing features such as 404 handling and mime types. Setting up a proxy server on top of node.js is trivial. One thing that is not easy to support is video streaming. In some web browsers, they expect the video content to be chunked encoded and sent over the wire as HTTP 206 partial content chunks. node.js also has a module called [Socket.IO](http://socket.io/) that can be used to build server pushed messages to the web browser. This feature will be key for debug tools needed for Mobile Web. We find that web server including proxy support in node.js to be suitable for Titanium Mobile.Zip file creation/extraction
node.js has built-in gzip support. This is not the same as zip file support, but is used to compress the individual files in a zip archive. Using the [adm-zip](https://github.com/cthackers/adm-zip) module, we can support creating and extracting zip files. This module can list all the entries in the zip file, update the zip file, and more. It is not known at this time if adm-zip is capable of producing Android APK-compatible zip files. There are other zip file modules available that might be able to support Android APK-compatible zip files. Aside from not knowing precisely if APK files are supported, we find that zip file support via a module is suitable for Titanium Mobile.XML parsing
node.js does not have any built-in support for parsing XML, however there are a number of modules that can parse using either a SAX or DOM parser. The [node-o3-xml](https://github.com/ajaxorg/node-o3-xml) module also includes support for XPath and namespaces. Note: node-03-xml only works on Windows via Cygwin, so probably not a great solution. Several XML parsers leverage libxml2 which will most likely be a platform specific dependency we would need to manage. There are some XML parsers that are pure JavaScript implementations that would work for parsing files such as the tiapp.xml or timodule.xml files. We find that there are viable XML parsing solutions suitable for Titanium Mobile.Conclusion
node.js proves to be a very suitable candidate for use in Titanium Mobile. In general, here are a few pros and cons: Pros: * Active and healthy community * JavaScript language, something we all know * Very fast * Easily distributable * Support for native C++ addons * Fast paced development cycle Cons: * Asynchronous programming requires different thinking * Compared to other alternatives, node is much younger * Fast paced development cycle I have not found any evidence that node.js is any more or less unstable or not robust than any other alternative. Based on our experience and research, we have concluded that node.js is a viable solution for the CLI and build tools. Regardless of the selected technology choice, we plan to use node.js for all Mobile Web's build related processes, server-side reference implementations, and possibly AST parsing.I have put my async framework up on github at https://github.com/bryan-m-hughes/node-taskman. It isn't complete yet, but should be soon.
Regarding zip files: APK compliant zip files is not a requirement because the Android CLI tools can create them for you. Since APK compliant zip files are only required for Android, and since you can't build an Android app without the SDK installed, this support is guaranteed to always be available. Read http://developer.android.com/guide/developing/building/building-cmdline.html for more information.