[DAEMON-230] Stream stringified JSON data over IPC and response instead of JSON.stringify()
When a plugin sends a JS object response over IPC or a service writes a JS object response over the WebSocket or HTTP response, the JS object is serialized using
JSON.stringify() eats up a ton of memory when the JS object is large. It would be much more efficient to stringify while writing to the stream.
There are several npm packages that will stringify a stream, but they are geared towards each write being a small object which they just call
JSON.stringify() on it.
What we need is transforming stream that does stringifies the JS object on the fly. Stringify is not a complex algorithm, but would need to figure out how to handle Node specific objects like
- Chris Barber 2018-02-14
As good as an idea this is, it's not possible. :( When a plugin sends a response over IPC to the parent daemon process, it is calling
process.send()which handles the serialization for us and there's no "streaming" taking place. When the daemon sends a response over a WebSocket, the
wslibrary only has a simple
ws.send()function and no stream interface. Internally it will write the data to a socket which is a stream interface, but it's abstracted by wrapping the data in a frame. It's worth noting that WebSocket responses are encoded with msgpack. msgpack supports streaming, so if
wsshould ever support streaming, we could use something like https://www.npmjs.com/package/bfj or https://www.npmjs.com/package/big-json to stringify the response and then stream it through msgpack as we write the bytes over the wire.