[DAEMON-230] Stream stringified JSON data over IPC and response instead of JSON.stringify()
GitHub Issue | n/a |
---|---|
Type | Improvement |
Priority | Critical |
Status | Resolved |
Resolution | Invalid |
Resolution Date | 2018-02-14T23:57:37.000+0000 |
Affected Version/s | n/a |
Fix Version/s | n/a |
Components | appcd-core, appcd-plugin |
Labels | n/a |
Reporter | Chris Barber |
Assignee | Chris Barber |
Created | 2018-02-02T16:42:06.000+0000 |
Updated | 2018-02-14T23:57:41.000+0000 |
Description
When a plugin sends a JS object response over IPC or a service writes a JS object response over the WebSocket or HTTP response, the JS object is serialized using
JSON.stringify()
. However, JSON.stringify()
eats up a ton of memory when the JS object is large. It would be much more efficient to stringify while writing to the stream.
There are several npm packages that will stringify a stream, but they are geared towards each write being a small object which they just call JSON.stringify()
on it.
What we need is transforming stream that does stringifies the JS object on the fly. Stringify is not a complex algorithm, but would need to figure out how to handle Node specific objects like Buffer
.
As good as an idea this is, it's not possible. :( When a plugin sends a response over IPC to the parent daemon process, it is calling
process.send()
which handles the serialization for us and there's no "streaming" taking place. When the daemon sends a response over a WebSocket, thews
library only has a simplews.send()
function and no stream interface. Internally it will write the data to a socket which is a stream interface, but it's abstracted by wrapping the data in a frame. It's worth noting that WebSocket responses are encoded with msgpack. msgpack supports streaming, so ifws
should ever support streaming, we could use something like https://www.npmjs.com/package/bfj or https://www.npmjs.com/package/big-json to stringify the response and then stream it through msgpack as we write the bytes over the wire.