[TIMOB-27875] iOS: Crash on TiUIListItem.m line 579
GitHub Issue | n/a |
---|---|
Type | Bug |
Priority | High |
Status | Closed |
Resolution | Fixed |
Resolution Date | 2021-09-01T21:34:30.000+0000 |
Affected Version/s | n/a |
Fix Version/s | Release 10.1.0 |
Components | iOS |
Labels | n/a |
Reporter | Jonas Funk Johannessen |
Assignee | Joshua Quick |
Created | 2019-11-20T12:08:57.000+0000 |
Updated | 2021-09-01T21:34:30.000+0000 |
Description
After updating from 8.1.0.GA to 8.2.1.GA we are seeing a new crash related to Ti.UI.ListItem rapidly growing in production. The crash is happening random places in the app where we are using ListViews.
And we haven't touched our code between SDK upgrades.
TiUIListItem.m line 579
-[TiUIListItem recordChangeValue:forKeyPath:withBlock:]
I have attached crashlog. Hope it's sufficient, for you guys to know what is the cause for this crash.
UPDATE: Crash is still happening with 8.3.0.GA in production.
Attachments
File | Date | Size |
---|---|---|
2019-11-25_09-11-05.9647_+0100-8a6b6a0da02deef725569687eeacbbc3777553b3.crash | 2019-11-27T12:07:16.000+0000 | 107180 |
2019-11-26_23-04-00.5542_+0100-5bbf81fd8edf5da6194d0a075792dfc5738c7e4f.crash | 2019-11-28T19:37:21.000+0000 | 110618 |
2019-11-27_23-57-00.5129_+0100-b0817bdf7536e44836abbdd5d4ed923f88a751ac.crash | 2019-11-28T19:37:21.000+0000 | 119139 |
crash-log-20210805.log | 2021-08-05T07:28:20.000+0000 | 116269 |
Screenshot 2021-04-13 at 20.08.47.png | 2021-04-13T18:08:54.000+0000 | 253625 |
TitaniumKit.xcframework.zip | 2021-07-30T01:25:17.000+0000 | 3986473 |
Any updates on this one? Are we the only ones seeing this?
Hello [~jonasfunk], Would you please share us a couple more crash reports by which we can find the exact issue? Thanks
I've attached two more crash logs.
Do you need anymore from me? Will happily help in any way I can.
We are still investigating on your log and trying to find the root cause. Hope we will back to you soon with some update. Thanks
Is there any way we temporarily patch the code to avoid the crash? Tried inserting try catch in both javascript and TiUIListItem.m with no luck.
[~jonasfunk] are you able to reproduce this on a development build at all? I have a rough idea what's going from the crash logs but having a reproducible test case would really help to collect valuable additional info to properly debug this. I wasn't able to reproduce this yet. Anything else that might help to pin this down? A specific list view that causes the crash? What views are used inside the list view rows that cause the crash?
Thanks for responding to this issue. We cannot recreate crashes consistently. We rarely experience it when we test which corresponds well with what we see in Crashlytics. Approx. 10% of our users experience a crash over a 7 day period, which amounts to approx 10.000 users. Furthermore we can also see, that it is not necessarily long sessions (memory build up of some sort) that trigger the crash. Crashes occur on both short and long sessions. We've run the app on Titanium since 2011, and almost never had any issues on iOS. Until recently. We used to have issues with Android, but now that's turned upside down. With 8.x our Android app is now almost crash free, while iOS is another story. To give you an idea of what our app does: It's a marketplace app where ListViews are used extensively throughout the app. Through Crashlytics we can see the views that are generating crashes - all with Ti.UI.Listviews. * *The listing details view*. Shows listing details and public comments. * *The private comments view.* Shows private comments - opened from listing details view. * *The activities view*. All buy/sell/comment notifications are listed here. This is the only view we've seen crash in development. Apart from that, we have a view that is used to show a grid of listings - also a Ti.UI.Listview. This is a widget and used many places in the app. This view +never+ crashes. This view is used in eg.: * *The listings overview* * *The listings of a single user* * *Favorites* * *Search results* There is no apparent difference between the views used in our grid ListView, and the ListViews that are crashing. They all have View, Label and an implementation of SDWebimage to handle image display/caching. Come to think of it – what differs between the views is, that even though our listings grid has ListItem height set to Ti.UI.SIZE, our listings never change in size. All the crashing ListViews have dynamic ListItem height, and their height change. Could that be the culprit? I've read that Ti.UI.SIZE on ListItems is discouraged, but our ListItems must have dynamic heights in order to show eg. comments of different lengths. If you wish, I will gladly send you samples of our code.
Thanks for all the details! Code samples would indeed be a tremendous help! Please feel free to send me anything you can share to [mailto:jvennemann@axway.com]. Especially the list item template setup and anything else related to interaction with the list view that may cause it to refresh, like events or data reloads. BTW, the crash seems to appear when the list view is (re)drawing rows, so repeatedly scrolling up and down through the list views should provoke a crash more likely. Maybe this can help you to pin down a view to consistently reproduce the crash.
Ok, I will prepare something for you. We have never seen the app crash while scrolling, but more instantly when loading a view. Will two consequtive calls to setItems trigger the same flow as scrolling?
Ok interesting. That scrolling provokes the crash was just an assumption from my side after checking the crash logs, so i may be wrong there. Hit me up as soon as you have some example code ready.
I've done some more digging into this issue, and I am almost certain that the crash happens when updating our activities view. I have sent screen recordings of the app together with sample code of that view directly to you. Looking at our Crashlytics log, I can se the that all crashes happen when the user at some point in her session has opened that view from bottom tab bar. When activities view is opened in a session, that view is continuously updated in the background to reflect the current state of notifications. To decide if notifications needs to be updated, we poll an endpoint which gives us the latest update timestamp of notifications. This is done every 30 sec., or whenever an action should initialise an update eg. marking a notification as read. If notifications needs to be loaded, we fetch the entire notification feed and set the parsed result by calling setItems on the ListView section. Realising that our activities view could be the reason for our crashes, I created a stress test targeted setItems method of the ListView in activities view. This is what I did:
* testData02 is a snapshot of notification data taken from a user directly after that user experienced a crash. * To simulate an update the first two notifications in testData02 has been removed in testData01. * The test is initialised immediately in the activities controller. * About 20 ListItems is set on each update. In production ListItem count is max. 100. * All other events that would normally trigger notification updates are disabled. * SetItem is called after postlayout on ListView. *Running this test actually makes it much easier to reproduce the crash.* While testing (on iPhone 11 device) I discovered the following: Staying on the activities view: – Sometimes the app can complete the procedure (video1). – Sometimes the app crashes immediately upon opening activities view, as we've seen happen in development and for users. Navigating away from the view and opening item detail views, sometimes just once(video2), sometimes multiple times(video3), will almost always crash the app - still with the same crash message regarding TiUIListItem. This is our observations - what do you make of it? In our next release, we will try to make sure activities ListView is not updated in the background, but only when in focus. This should at least take some of the crashes, but I would expect activities to still crash upon opening the view.
Update: We've sent a version to one of our very active users, where we did not update activities in background + added some more logging. The app still crashes, also when activities view is not opened, so my assumption regarding that activities view was the one that caused crashes was wrong. Seems like its a generic ListView issue. Have you looked further into the issue?
Any update?
@Jan Vennemann The silence is killing me.
[~jonasfunk] sorry for the long silence. I was caught up in other projects and unfortunately didn't have time to further look into the issue yet. Would it be possible for you to provide us access to your complete code base, including the stress test you mentioned in a previous comment that makes it easier to reproduce the issue? We didn't receive any similar reports for crashes like this and also were not able to reproduce it. Having a complete project that reliably reproduces the project will allow us to debug this issue right at our core.
Thanks for replying Jan! I would love to send you our complete project, but to spare you from having to go through a lot of really old and ugly code, I will try to create a minimal app, with some of our code, that can reproduce the issue. If that doesn't work, I will send you the entire project. I'll be in touch.
I've sent you an email Jan with a link to a minimal app reproducing the issue. Let me know if you have any questions.
[~jonasfunk] thanks a ton for your time to create the test case, greatly appreciated! I was able to reproduce the crash. Also send this to [~vijaysingh] so he can take a look as well. We'll let you know when we have any updates. BTW: Nice usage of TypeScript in your app! (y)
[~jvennemann] Thank you. Really happy to hear, you were able to reproduce the crash with the test project. As for the TypeScript part, this just makes are huge difference for us in regards to eg. getting strong typing and catching errors at compile time. [Mathias Lorenzen](https://github.com/ffMathy) assisted us in setting up Typescript for our project. If you see anything you like in how we have applied TypeScript in an Alloy context, you are welcome to get inspiration :).
Any update on this? We are upgrading from Ti SDK 8.1 to 8.3.1 and also have a lot of ListViews in our app, and are now seeing a ton of crashes at the exact same spot [TiUIListItem recordChangeValue:forKeyPath:withBlock:] - please advise.
[~btknorr] We are working on this. I'll update you here as soon as I get anything. Meanwhile can you let me know - 1. If the crash is not happening on SDK 8.1.0.GA? 2. Is it crashing for 8.2.0.GA and 9.0.0.GA as well ? 3. Do you have any small test case, which you can share with me . Thanks!
I believe it is happening on 8.1.0.GA, but just not as often. We have not tried 8.2 or 9.0. I can look into putting together a test case. Crash seems to be caused by Labels...here is a stack trace: 0 com.apple.CoreText 0x0000000106823f85 OTL::GPOS::ApplyPairPosAccelerated(OTL::Lookup const&, TGlyphIterator&) const + 1663 1 com.apple.CoreText 0x0000000106843c05 OTL::GPOS::ApplyLookups(TRunGlue&, int, OTL::GlyphLookups&, unsigned short) const + 561 2 com.apple.CoreText 0x00000001068435c6 TOpenTypePositioningEngine::PositionRuns(SyncState&, KerningStatus&) + 1392 3 com.apple.CoreText 0x00000001067a4fdc TKerningEngine::PositionGlyphs(TLine&, TCharStream const&, signed char) + 438 4 com.apple.CoreText 0x00000001067b16eb TTypesetter::FinishLayout(std::__1::tuple
Here is another stack trace that seems to happen often: 0 libobjc.A.dylib 0x0000000106595785 objc_msgSend + 5 1 com.apple.Foundation 0x000000010556634c hashProbe + 516 2 com.apple.Foundation 0x0000000105566592 -[NSConcreteHashTable getItem:] + 52 3 com.apple.UIFoundation 0x000000011c19a911 +[NSAttributeDictionary newWithDictionary:] + 115 4 com.apple.UIFoundation 0x000000011c1cb903 -[_NSCachedAttributedString initWithString:attributes:] + 107 5 com.apple.UIFoundation 0x000000011c1cddb9 __NSStringDrawingEngine + 5701 6 com.apple.UIFoundation 0x000000011c1cc74a -[NSString(NSExtendedStringDrawing) boundingRectWithSize:options:attributes:context:] + 187 7 com.apple.UIKitCore 0x000000011a8dd086 -[UILabel _textRectForBounds:limitedToNumberOfLines:includingShadow:] + 1456 8 com.apple.UIKitCore 0x000000011a8dc8b7 -[UILabel textRectForBounds:limitedToNumberOfLines:] + 50 9 com.apple.UIKitCore 0x000000011a8e1de6 -[UILabel _intrinsicSizeWithinSize:] + 384 10 com.starterstep.broadcast 0x00000001033221ca -[TiUILabel contentWidthForWidth:] + 106 (TiUILabel.m:86) 11 com.starterstep.broadcast 0x00000001033219a6 -[TiUILabelProxy contentWidthForWidth:] + 70 (TiUILabelProxy.m:17) 12 com.appcelerator.TitaniumKit 0x000000010648c690 -[TiViewProxy autoWidthForSize:] + 100 13 com.appcelerator.TitaniumKit 0x000000010648ce9b -[TiViewProxy minimumParentWidthForSize:] + 535 14 com.appcelerator.TitaniumKit 0x000000010648c873 -[TiViewProxy autoWidthForSize:] + 583 15 com.appcelerator.TitaniumKit 0x000000010648ce9b -[TiViewProxy minimumParentWidthForSize:] + 535 16 com.appcelerator.TitaniumKit 0x000000010648c873 -[TiViewProxy autoWidthForSize:] + 583 17 com.appcelerator.TitaniumKit 0x000000010648ce9b -[TiViewProxy minimumParentWidthForSize:] + 535 18 com.appcelerator.TitaniumKit 0x000000010649229f -[TiViewProxy computeChildSandbox:withBounds:] + 154 19 com.appcelerator.TitaniumKit 0x000000010648cb6a -[TiViewProxy autoHeightForSize:] + 527 20 com.appcelerator.TitaniumKit 0x00000001064927f8 -[TiViewProxy computeChildSandbox:withBounds:] + 1523 21 com.appcelerator.TitaniumKit 0x000000010648cb6a -[TiViewProxy autoHeightForSize:] + 527 22 com.appcelerator.TitaniumKit 0x000000010648d309 -[TiViewProxy minimumParentHeightForSize:] + 580 23 com.appcelerator.TitaniumKit 0x000000010648cb9c -[TiViewProxy autoHeightForSize:] + 577 24 com.appcelerator.TitaniumKit 0x000000010648d309 -[TiViewProxy minimumParentHeightForSize:] + 580 25 com.starterstep.broadcast 0x000000010331727c -[TiUIListView tableView:heightForRowAtIndexPath:] + 1260 (TiUIListView.m:1867) 26 com.apple.UIKitCore 0x000000011a6cab80 -[UITableView _dataSourceHeightForRowAtIndexPath:] + 130 27 com.apple.UIKitCore 0x000000011a6dac37 __66-[UISectionRowData refreshWithSection:tableView:tableViewRowData:]_block_invoke + 369 28 com.apple.UIKitCore 0x000000011a6da1ed -[UISectionRowData refreshWithSection:tableView:tableViewRowData:] + 3280 29 com.apple.UIKitCore 0x000000011a6ddeeb -[UITableViewRowData numberOfRows] + 67 30 com.apple.UIKitCore 0x000000011a699320 -[UITableView noteNumberOfRowsChanged] + 117 31 com.apple.UIKitCore 0x000000011a697d9b -[UITableView reloadData] + 1426 32 com.starterstep.broadcast 0x000000010330cd6c -[TiUIListView updateSearchResults:] + 92 (TiUIListView.m:558) 33 com.starterstep.broadcast 0x000000010344ad2e __38-[TiUIListSectionProxy insertItemsAt:]_block_invoke + 302 (TiUIListSectionProxy.m:210) 34 com.starterstep.broadcast 0x0000000103355988 -[TiUIListViewProxy dispatchBlock:] + 168 (TiUIListViewProxy.m:130) 35 com.starterstep.broadcast 0x000000010344ab63 -[TiUIListSectionProxy insertItemsAt:] + 1203 (TiUIListSectionProxy.m:216) 36 com.apple.CoreFoundation 0x0000000108fd938c __invoking___ + 140 37 com.apple.CoreFoundation 0x0000000108fd649f -[NSInvocation invoke] + 319 38 com.appcelerator.TitaniumKit 0x00000001064ab06b -[KrollMethod call:] + 1385 39 com.appcelerator.TitaniumKit 0x00000001064aa29f __KrollCallAsFunction_block_invoke + 30 40 com.appcelerator.TitaniumKit 0x00000001064aa0b9 KrollCallAsFunction + 425 41 com.apple.JavaScriptCore 0x00000001039788ec JSC::JSCallbackObject
[~jonasfunk] I am still working on this. Can you confirm if crash was not there in SDK 8.1.0.GA? And did you try on 9.0.1.GA or 9.0.2.GA ? Is it having similar crash frequency?
[~vijaysingh] I'm glad to hear you are working on this issue. I've revisited our release history and compared that to versions where the crash occurs, and I can confirm, that the crash was *not* in 8.1.0.GA. We have not yet an app in production with SDK 9, so I'm afraid I'm not able to give you any info about that. Have you by any chance tested 8.1.0.GA with the test project I provided?
This is regression from [PR #11028](https://github.com/appcelerator/titanium_mobile/pull/11028) introduced in SDK 8.2.1. I tried to test by reverting the PR and crash is no more. Probably we have to change fix of TIMOB-27207. cc [~jvennemann] Edit - After reverting the changes, sometimes I see crash as mentioned in AC-6396.
[~vijaysingh] So nice that you found issue. We're crossing our fingers that this can be fixed soon. As a quick fix, and since we had no issues before 8.2.1, will we be able to revert the changes made in [PR #11028](https://github.com/appcelerator/titanium_mobile/pull/11028) without breaking things? As I understand the PR should fix proxy memory build up, but we would rather have with this, than the crashes we are seeing :)
[~jonasfunk] 1. Reverting the PR should not break the things. 2. But sometimes I can see another crash as mentioned in comment, which was reported by you :). If that issue was not there in your app with SDK 8.1.0.GA, I think you should be fine.
[~jonasfunk] quick update: We have narrowed it down to the usage of
Ti.UI.AttributedString
together with aTi.UI.Label
in the ListView. Sometimes the attributed string proxy gets released right before it will be assigned to the label which is causing the crash. We are now doing more investigation to see why this is happening in a list view only, and will start working on a fix as soon as we found the root issue.[~jvennemann] Thank for the update! Great that you found out, that
Ti.UI.AttributedString
causes the crash. Looking at our project, that actually makes a lot of sense, as our most used view is aTi.UI.ListView
displaying items for sale. This view doesn't useTi.UI.AttributedString
and never crashes.[~jvennemann] Can you give us some insight into what challenges you are up against? 9.2.0 seems pretty far away :)
[~jonasfunk], the problem is that it seems totally random. Sometimes the attributed strings are retained just fine, sometimes they are wrongly GC'd right before usage in the view which causes the crash. This seemingly random behavior makes it very hard to debug and figure out what's causing the issue. I tried to reproduce the issue with forced garbage collections to provoke the crash in a consistent way, but unfortunately had no luck yet. I also had a very big project to finish for the 9.1.0 release so the time i was able to spend on this issue was limited as well. I understand that this issue is critical to you so If a solution is found before 9.2.0, we will find a way to provide you with a backport for the 9.1.x line.
Thanks for the update [~jvennemann]. I understand the challenges – hope you will have more time available to work on this issue in the coming period. We will appreciate if you can backport a possible solution to us. (y)
[~spulipakkam], Done.
I take the latest change in priority means a solution is further away. We will consider removing attributed strings from our listviews instead of waiting for a fix in the SDK.
This is still a very severe issue. Can someone from the team address this? Happy to provide symbolicated Xcode crash logs for this!
Can this please please be fixed? I am willing to pay anyone (core member or community) who manages to fix this bug.
Okay, so we should really get this fixed. I did some research and compared the changes between 8.2.0 and 8.2.1 (where the crash first appeared). I noticed some Garbage-collection related changes [here](https://github.com/appcelerator/titanium_mobile/compare/8.2.0...8.2.1#diff-acebe3f8a6a64a734c2a557093c0831f1ec331f1e9dab3c4fa169914c19d17faR647) which could be the root issue, because the actual crash also happens in the TiProxy class. So how does it end up there? I checked the stack trace over and over again. It always starts in the "cellForRowAtIndexPath:" delegate, confirming that Jan said regarding the quick redrawing of the cells causing it. From there, it calls the "setDataItem:" selector, which runs into the "default" switch-case statement ([here](https://github.com/appcelerator/titanium_mobile/blame/master/iphone/Classes/TiUIListItem.m#L487)), where it needs to do some reproxying. BUT: Compared to other parts of the SDK where "setReproxying:" is called (in the Ti.UI.View and Ti.UI.TableViewRow), it does not call "transferProxy:", which could lead to a memory leak causing this crash. Does this make sense? It would be great if someone could take this and help finding the final solution. It's causing really severe crashes on production that seem all random, but they all are related to this issue. *EDIT*: It has to be this one. Here is a most recent stack:
It crashes right after the "forgetKeylessKrollObject:" call and the change in 8.2.1 changed the behavior of the "noteKeylessKrollObject:" selector.
[~hknoechel], I see a different issue that might be the root cause. Have a look at the KrollObject code below. https://github.com/appcelerator/titanium_mobile/blob/master/iphone/TitaniumKit/TitaniumKit/Sources/Kroll/KrollObject.m#L927-L973 The
noteKeylessKrollObject
andforgetKeylessKrollObject
methods call theTiStringCreateWithPointerValue()
function to generate a unique property name based on a native object's pointer/memory-address. The *BIG* issue here is that it's casting a 64-bit pointer to an(int)
which is 32-bit. This means there's a chance that we are wrongly generating the same name for 2 different pointers if they exceed the 32-bit bounds (ie: name collision), which would cause a wild-pointer crash the 2nd time theforgetKeylessKrollObject
method gets called on a property withe name collision. The solution of course would be to changeTiStringCreateWithPointerValue()
to accept a 64-bit argument. _(Even if this is not the root cause of the crash, we should make this specific change anyways because casting a 64-bit pointer to a 32-bit value is clearly the wrong thing to do.)_ *Side Note:* Ideally, we should use a JS "Symbol" to create a unique property in order to avoid name collision, but JavaScriptCore only supports it on iOS 13 and higher. https://developer.apple.com/documentation/javascriptcore/jsvalue/3042804-valuewithnewsymbolfromdescriptio?language=objcPR (master): https://github.com/appcelerator/titanium_mobile/pull/12987 [~hknoechel], the above PR should theoretically fix the issue, but note that I'm unable to reproduce the issue. So, this is a guess. Would you mind trying it out? Or is it hard to reproduce on your end too? (I can see this being a rare random issue.)
Hi Josh! Super interesting change. You can btw reproduce it in our project (I sent you some more details via Slack). I will test it today! Regarding unique keys: Why not use a UUID (via
[NSUUID UUID]
), which is pretty save to use, even for larger collections of 1M+ data sets *Update*: It is *not* crashing for me, wow. Can someone else from this ticket confirm? To ease testing, I uploaded a ready-to-go SDK here (based on the latest 10.0.1 build + the commit from Josh). Are there any side issues that could occur? The patch seems pretty stable. Steps to test: 1. Download and extract the zip file [from here](https://www.dropbox.com/s/ox93xiqma3zrzhy/mobilesdk-10.0.2-osx.zip?dl=1) 2. Copy the/mobilesdk-10.0.2-osx/mobilesdk/osx/10.0.2
folder to~/Library/Application Support/Titanium/mobilesdk/osx/10.0.2
3. Change the SDK version in your tiapp.xml to 10.0.2 4. Run your app! *Update 2*: Sorry, it still crashes. It has to be something else :([~hknoechel] Are you able to obtain a crash log after applying Josh's PR, curious if the stack is different? *Update*: I've looked at the changes from 8.1.0 to 8.2.0 and reverted a GC related change that is related to
TiProxy replaceValue
, try using this TitaniumKit with the SDK you linked above (it includes Josh's change too). [^TitaniumKit.xcframework.zip]Was it the change from Jan back then? Because that definitely fixed another GC crash back then that was even more severe because it affected all proxies (and was harder to reproduce). Critical topic. Regarding the crash: It's indeed a different one and indeed related to the
replaceValue:
selector:So far, we did not find any crashes, but the other one from SDK 8.x was also harder to reproduce, so I'm not sure if that one may cause issues again now.
[~hknoechel] Thanks for the log, btw I didn't fully revert Jan's change. Just these two lines: https://github.com/appcelerator/titanium_mobile/blob/master/iphone/TitaniumKit/TitaniumKit/Sources/API/TiProxy.m#L639 https://github.com/appcelerator/titanium_mobile/blob/master/iphone/TitaniumKit/TitaniumKit/Sources/API/TiProxy.m#L660
[~hknoechel] Curious if you've had chance to test this more, have you been able to reproduce this crash with both Josh's and my patch?
I just got feedback from the team and it still crashes, although later than before. Attaching logs asap
Okay, so the crash looks differently. New logs are attached! *EDIT*: It's the same one as described in AC-6396
My PR has been merged. I'm thinking we should close this ticket since that PR addresses this ticket's original stack trace. For the other crash with a different stack trace, I think that should be a separate ticket.