[DAEMON-230] Stream stringified JSON data over IPC and response instead of JSON.stringify()
| GitHub Issue | n/a |
|---|---|
| Type | Improvement |
| Priority | Critical |
| Status | Resolved |
| Resolution | Invalid |
| Resolution Date | 2018-02-14T23:57:37.000+0000 |
| Affected Version/s | n/a |
| Fix Version/s | n/a |
| Components | appcd-core, appcd-plugin |
| Labels | n/a |
| Reporter | Chris Barber |
| Assignee | Chris Barber |
| Created | 2018-02-02T16:42:06.000+0000 |
| Updated | 2018-02-14T23:57:41.000+0000 |
Description
When a plugin sends a JS object response over IPC or a service writes a JS object response over the WebSocket or HTTP response, the JS object is serialized using
JSON.stringify(). However, JSON.stringify() eats up a ton of memory when the JS object is large. It would be much more efficient to stringify while writing to the stream.
There are several npm packages that will stringify a stream, but they are geared towards each write being a small object which they just call JSON.stringify() on it.
What we need is transforming stream that does stringifies the JS object on the fly. Stringify is not a complex algorithm, but would need to figure out how to handle Node specific objects like Buffer.
As good as an idea this is, it's not possible. :( When a plugin sends a response over IPC to the parent daemon process, it is calling
process.send()which handles the serialization for us and there's no "streaming" taking place. When the daemon sends a response over a WebSocket, thewslibrary only has a simplews.send()function and no stream interface. Internally it will write the data to a socket which is a stream interface, but it's abstracted by wrapping the data in a frame. It's worth noting that WebSocket responses are encoded with msgpack. msgpack supports streaming, so ifwsshould ever support streaming, we could use something like https://www.npmjs.com/package/bfj or https://www.npmjs.com/package/big-json to stringify the response and then stream it through msgpack as we write the bytes over the wire.