[DAEMON-230] Stream stringified JSON data over IPC and response instead of JSON.stringify()

GitHub Issuen/a
Resolution Date2018-02-14T23:57:37.000+0000
Affected Version/sn/a
Fix Version/sn/a
Componentsappcd-core, appcd-plugin
ReporterChris Barber
AssigneeChris Barber


When a plugin sends a JS object response over IPC or a service writes a JS object response over the WebSocket or HTTP response, the JS object is serialized using JSON.stringify(). However, JSON.stringify() eats up a ton of memory when the JS object is large. It would be much more efficient to stringify while writing to the stream. There are several npm packages that will stringify a stream, but they are geared towards each write being a small object which they just call JSON.stringify() on it. What we need is transforming stream that does stringifies the JS object on the fly. Stringify is not a complex algorithm, but would need to figure out how to handle Node specific objects like Buffer.


  1. Chris Barber 2018-02-14 As good as an idea this is, it's not possible. :( When a plugin sends a response over IPC to the parent daemon process, it is calling process.send() which handles the serialization for us and there's no "streaming" taking place. When the daemon sends a response over a WebSocket, the ws library only has a simple ws.send() function and no stream interface. Internally it will write the data to a socket which is a stream interface, but it's abstracted by wrapping the data in a frame. It's worth noting that WebSocket responses are encoded with msgpack. msgpack supports streaming, so if ws should ever support streaming, we could use something like https://www.npmjs.com/package/bfj or https://www.npmjs.com/package/big-json to stringify the response and then stream it through msgpack as we write the bytes over the wire.

JSON Source