Titanium JIRA Archive
Titanium SDK/CLI (TIMOB)

[TIMOB-6849] iOS: Application hangs when trying to parse LARGE set of data using JSON.parse()

GitHub Issuen/a
TypeBug
PriorityHigh
StatusClosed
ResolutionInvalid
Resolution Date2012-03-27T13:47:32.000+0000
Affected Version/sRelease 1.8.0.1
Fix Version/sn/a
ComponentsiOS
Labelsn/a
ReporterFrancisco Antonio Duran Ramirez
AssigneeStephen Tramer
Created2011-12-22T13:04:21.000+0000
Updated2014-06-19T12:43:41.000+0000

Description

Problem

Application hangs on parsing Very Lage amount data.

Reproducible steps:

1. Run the code below 2. See the console Notice that Titanium still stucks on JSON.parse().

Expected behavior:

Customer wants to parse his information.

Tested with the next specs:

Simulator Titanium Studio, build: 1.0.8.201112191627 Titanium SDK: 1.8.0.1.v20111220190134 , 1.8.0.1.RC3, 1.7.5 & 1.9.0.v20111221123134 iOS SDK: 5.0 

[INFO] Application started
[INFO] i13399/1.0 (1.8.0.1.v20111220190134.f3575f8)
[INFO] LOGIN REQUEST WAS SENT: http://mobilisticstorage.cloudapp.net/login/spoad@acorndistributors.com/c1AxMjM0
[INFO] LOGIN RESPONSE TEXT: "53c0f5e4-d9ff-47e8-9196-95f378786e43"
[INFO] LOGIN RESPONSE OBJECT: 53c0f5e4-d9ff-47e8-9196-95f378786e43
[INFO] GET DATA REQUEST WAS SENT: http://mobilisticstorage.cloudapp.net/dashboard/salesperson?apikey=53c0f5e4-d9ff-47e8-9196-95f378786e43
[INFO] GET DATA RESPONSE TEXT LENGTH: 5439224


[INFO] Application started
[INFO] i13399/1.0 (1.8.0.1.RC3.c903964)
[INFO] LOGIN REQUEST WAS SENT: http://mobilisticstorage.cloudapp.net/login/spoad@acorndistributors.com/c1AxMjM0
[INFO] LOGIN RESPONSE TEXT: "53c0f5e4-d9ff-47e8-9196-95f378786e43"
[INFO] LOGIN RESPONSE OBJECT: 53c0f5e4-d9ff-47e8-9196-95f378786e43
[INFO] GET DATA REQUEST WAS SENT: http://mobilisticstorage.cloudapp.net/dashboard/salesperson?apikey=53c0f5e4-d9ff-47e8-9196-95f378786e43
[INFO] GET DATA RESPONSE TEXT LENGTH: 5439224


[INFO] Application started
[INFO] i13399/1.0 (1.7.5.ab20af7)
[INFO] LOGIN REQUEST WAS SENT: http://mobilisticstorage.cloudapp.net/login/spoad@acorndistributors.com/c1AxMjM0
[INFO] LOGIN RESPONSE TEXT: "53c0f5e4-d9ff-47e8-9196-95f378786e43"
[INFO] LOGIN RESPONSE OBJECT: 53c0f5e4-d9ff-47e8-9196-95f378786e43
[INFO] GET DATA REQUEST WAS SENT: http://mobilisticstorage.cloudapp.net/dashboard/salesperson?apikey=53c0f5e4-d9ff-47e8-9196-95f378786e43
[INFO] GET DATA RESPONSE TEXT LENGTH: 5439224

Helpdesk

APP-547691

Comments

  1. Stephen Tramer 2011-12-22

    This is an incoming set of 5MB of data - even parsing this on a desktop machine is slow (even with very, very fast specialized JSON parsers), and for mobile devices, it will be even slower. We can upgrade our JSON parser, but for data of this size, it will not keep the application from "hanging" while it processes a payload this large. The reason for this is that JSON.parse() is meant to be serialized and synchronous - and Javascript is a serialized and synchronous language. We do have plans to offer threading support in a future release, which will make it possible to perform tasks like JSON deserialization as a background task, JSON is generally intended to be used for relatively lightweight datagrams (certainly under 5MB). I'm unsure what the intention behind such a large payload is, but there are some other options for processing: * Retrieve specific data as requested via SQL queries * Cache information locally after an initial retrieval * Split data into smaller payloads that can be processed on an "as-needed" basis If this data must be processed in a complete chunk and cannot be divided in any way, the following is recommended: * Present an activity indicator in a window before JSON begins loading, and lock UI activity * Parse the JSON * Hide the activity indicator and unlock UI This prevents your app from appearing unresponsive while the data is processed.
  2. Alexander Konovkov 2011-12-24

    Stephen, actually it parses such large data ok, but it fails to iterate through it with four indexOf function calls on each iteration.
  3. Stephen Tramer 2011-12-27

    Alexander - Sorry, I missed part of the initial bug report. This could be an issue having nothing to do with JSON data but instead the ability to iterate over the parsed object due to its unusually large size. We will be taking a closer look at the problem.
  4. Stephen Tramer 2011-12-28

    Deleted an earlier comment with bad source code. The following works: 
       var jsonStr = '{"foo":"bar"}';
   var jsonObj = {"foo":"bar"};
   
   var xhr = Ti.Network.createHTTPClient();
   xhr.onload = function(e) {
   	Ti.API.info("BEFORE PARSE: "+jsonStr);
   //	var obj = null;
   	var obj = JSON.parse(jsonStr);
   	Ti.API.info("AFTER PARSE: "+obj);
   };
   Ti.API.info("obj->str: "+JSON.stringify(jsonObj));
   Ti.API.info("Outside onload: "+JSON.parse(jsonStr));
   xhr.open('GET', 'http://appcelerator.com');
   xhr.send();
    Note that if we change the jsonStr to be a bad JSON value (such as "{foo:bar}") then we throw an exception in the callback which is not properly handled. This leads me to believe two things: * The 5.2 MB JSON contains an error * We need to properly log exceptions during all callbacks and display them in a reasonable fashion
  5. Stephen Tramer 2011-12-28

    I'm currently running jsonlint on the information dump to see if it contains any errors... and it's been running for 40 minutes and still isn't done. I'm going to run it through a specialized parsing/validation tool, but do not expect it to run any faster. This is on a 2GHz Intel Core i7; significantly faster than any mobile device. This ticket may be marked as INVALID based on the results of a specialized parser run. It's also worth noting that we would have to upgrade to a new version of JSCore with completely rewritten JSON processing internals to make any speed improvements.
  6. Stephen Tramer 2011-12-28

    After running for over an hour, jsonlint produced the following error: 
       Error: Parse error on line 1:
   ...iseTotal":21.7111000
   -----------------------^
   Expecting '}', ','
    Either it ran out of buffer space (possible) or there is an actual JSON error.
  7. Stephen Tramer 2011-12-28

    Resolving as invalid as per issues with the JSON payload and amount of processing time required to handle it under any circumstances. It is worth noting that most libraries which ship with a test for a "large" JSON payload consider such a payload to be around 9kb.
  8. Thomas Huelbert 2012-01-23

    closing based on Stevens comments

JSON Source