Problem
Titanium tooling will not build a project that has more than 65536 total method calls
Currently android has a limit on the number of methods permitted in a single dex file. (65536 - who thought a short int should be used here???)
As an example, the latest google-play-services.jar from 3.4.1 has 23600.
For basic S3 support from aws 2.0, a module uses 26800.
By the time you build an empty titanium app you are left with:
[ERROR] trouble writing output: Too many method references: 68927; max is 65536.
Solution
The solution is to use the --multi-dex option in calling dexer.
The problem will be that --multi-dex cant just be dropped in as it requires a different output. A decision will need to be made on how to modify this.buildBinClassesDex.
Example (~2952)
dexArgs = [
'-Xmx' + this.dxMaxMemory,
'-XX:-UseGCOverheadLimit',
'-Djava.ext.dirs=' + this.androidInfo.sdk.platformTools.path,
'-jar', this.androidInfo.sdk.dx,
'--dex', '--multi-dex',
'--output=' + this.buildBinClassesDex,
this.buildBinClassesDir,
path.join(this.platformPath, 'lib', 'titanium-verify.jar')
In this case, if this.buildBinClassesDex points to a directory (lets say this.buildBinDir) we end up with our 2 .dex files
-rw-r--r-- 1 stephenfeather 8276656 Nov 21 16:53 classes.dex
-rw-r--r-- 1 stephenfeather 633200 Nov 21 16:53 classes2.dex
Around like 4029 we can add a hack to push the other .dex file into our package. This would need to be coded to scan the directory for all *.dex files and add them to the bundle.
dest.append(fs.createReadStream(path.join(this.buildBinDir, 'classes.dex')), { name: 'classes.dex' });
dest.append(fs.createReadStream(path.join(this.buildBinDir, 'classes2.dex')), { name: 'classes2.dex' });
The problem however, is that the app will not run on a device because we would need to tell the application loader that this is a multidex package which will require another ticket.
https://developer.android.com/reference/android/support/multidex/MultiDexApplication.html
[WARN] dalvikvm: Unable to resolve superclass of Lorg/appcelerator/titanium/TiBaseActivity; (802)
[WARN] dalvikvm: Link of class 'Lorg/appcelerator/titanium/TiBaseActivity;' failed
[WARN] dalvikvm: Unable to resolve superclass of Lorg/appcelerator/titanium/TiLaunchActivity; (7431)
[WARN] dalvikvm: Link of class 'Lorg/appcelerator/titanium/TiLaunchActivity;' failed
[WARN] dalvikvm: Unable to resolve superclass of Lorg/appcelerator/titanium/TiRootActivity; (7445)
[WARN] dalvikvm: Link of class 'Lorg/appcelerator/titanium/TiRootActivity;' failed
[WARN] dalvikvm: VFY: unable to
[WARN] dalvikvm: VFY: Ljava/lang/Object; is not instance of Landroid/app/Activity;
[WARN] dalvikvm: VFY: bad arg 1 (into Landroid/app/Activity;)
[WARN] dalvikvm: VFY: rejecting call to Lorg/appcelerator/titanium/TiVerify;.<init> (Landroid/app/Activity;Lorg/appcelerator/titanium/TiApplication;)V
[WARN] dalvikvm: VFY: rejecting opcode 0x70 at 0x0002
[WARN] dalvikvm: VFY: rejected Lcom/stephenfeather/test/classic/TestingClassicApplication;.verifyCustomModules (Lorg/appcelerator/titanium/TiRootActivity;)V
[WARN] dalvikvm: Verifier rejected class Lcom/stephenfeather/test/classic/TestingClassicApplication;
[WARN] dalvikvm: Class init failed in newInstance call (Lcom/stephenfeather/test/classic/TestingClassicApplication;)
code snippets are from _build.js
To those curious about why this is an issue. The map module includes Google Play services (by necessity). Google Play services added 12K+ methods from one version of the library to another. Yes, 12 *thousand* methods.
Unless I'm reading this wrong, Multidex is brought to pre-L platforms via the support library... with caveats: https://developer.android.com/tools/building/multidex.html#limitations
* The application will need to be derived from MultiDexApplication (http://developer.android.com/reference/android/support/multidex/MultiDexApplication.html) OR attachBaseContext will need to be overridden and a call for multidex.install() made.
public abstract class TiApplication extends MultiDexApplication implements KrollApplication { .. }[~hpham] Any thoughts ?
Any progress here?
@Hieu, while this addresses the immediate issue around multi-dex, we should likely keep a separate ticket for implementing it once 5.0 is the floor for Android (or in case a user only wishes to serve 5.0+ devices)
Verified the fix for the modules by using SDK 4.1.1.v20150816214206. The modules build fine on pre & post android 5.0. Also built kitchensink by adding these 3 modules & did not find any issues. Verification pending with master branch. Environment: Appc Studio : 4.2.0.201508062204 Ti SDK : 4.1.1.v20150816214206 Ti CLI : 4.1.4 Alloy : 1.6.2 MAC Yosemite : 10.10.4 Appc NPM : 4.1.0 Appc CLI : 4.2.0-54 Node: v0.10.37 Samsung Galaxy S4 - Android 4.4.2 Nexus 6 - Android 5.0.1 Nexus 5 - Android 5.1.1
Verified the fix with both 4_1_X & master (5.0.0) SDK. Closing. Environment: Appc Studio : 4.2.0.201508062204 Ti SDK : 5.0.0.v2015081914042, 4.1.1.v20150816214206 Ti CLI : 4.1.4 Alloy : 1.6.2 MAC Yosemite : 10.10.4 Appc NPM : 4.1.0 Appc CLI : 5.0.0-3 Node: v0.10.37 Nexus 5 - Android 5.1.1 Node : v0.10.37 Samsung Galaxy S4 - Android 4.4.4 Nexus 5 - Android 5.1.1
I'm sorry, but NO, the original issue has NOT been fixed. The original problem, namely **Titanium tooling will not build a project that has more than 65536 total method calls** is still there, no matter how tightly engineering closes its eyes. Sliding the issue off onto a new ticket, then hijacking this one to bring in hacked up google-play jars as a patch job is disingenuous. If you viewed your modules as the source of a different issue, then a ticket should have been opened for those. This ticket was raised to address the FACT, which remains as of the time of this comment, that without multi-dex support, serious developers will run into development issues caused by a limitation within Appcelerator Titanium when they try to scale their apps, to include additional 3rd party libraries. My piece said, I'm washing my hands of this.
I agree with Stephen -- any serious app will quickly run into this problem. Our WRAL News app has 16 android modules embedded. Four of them use google-play-services.jar, which is the big hog. It's hard to avoid using google play services these days. Mapping, push, analytics, and ads all use it. Not only is it a PITA to keep the versions of google-play-services.jar in sync across all these modules (I'm maintaining "hacked" versions of ti.map and ti.cloudpush just to use a more current jar), now we have to deal with the bloat in this library causing this dexer problem. My workaround was to unpack google-play-services.jar and remove chunks of the library that I'm not using (games, drive, vision, wallet, wearable). That seemed to get rid of about 10K references, which should buy me enough headroom for now.
There's a good script to re-pack the play services jar based on a configuration file at: https://gist.github.com/dextorer/a32cad7819b7f272239b. Still, multidex must be supported in android/cli/commands/_build.js
All, to confirm, --multi-dex does not work for Android SDKs < 5.0 without us adding yet another backwards compatibility library (https://developer.android.com/tools/support-library/features.html#multidex). Are you all comfortable with this only working for 5.0+ devices? Or do we need to enable it for 4.0 as well? See https://developer.android.com/tools/building/multidex.html for more info.
That's a tricky question. If you *need* multidex, then presumably, you're in a situation where you *can't* build without it. So if it only supported Android 5.0+, you wouldn't be able to build your app for 4.0. If you *did* implement it for 5.0+ only, it would probably be a long time before we could use it -- we'd have to wait until we could drop support for Android 4.x. Given how slowly android users adopt new versions, it might be 3 years before Android 4.x usage is low enough for us to drop support. I think the more immediate pain is all centered around google play services. The google play services JAR is responsible for a massive amount of references in apps, and almost every serious android app is going to use google play services. One solution is to use stripped-down versions of google play services, or maybe the API-specific jar files. But there are so many android modules using google play services, each with possibly different versions of the library, that all of this becomes very difficult to do. I really think that Appcelerator should drive a strategy for android developers to make the most efficient use of google play services. We need uniformity in the version of the JAR that is supported by appcelerator-provided modules *and* community-provided modules (like ti.dfp, ti.google-analytics, etc.). And maybe the Appcelerator tooling would let you pick google play services APIs from a menu to build a minimal JAR, and then use that custom JAR instead of the JARs shipped with the various modules. I think this would work safely if all the modules were on the same (up-to-date) version of google play services, *and* the user makes the right API choices in this proposed tool. Of course, it would be even better if the tool could detect which APIs were in use in the various modules and automatically built a minimal google play services JAR.
[~jpriebe] thank you for the comment. Did you view the work we did on TIMOB-19348? We honestly CANNOT use the whole SDK from Google Play Services as isand standardize on that. That alone has 30-40K methods and they keep adding to it all the time.
Support for SDK<5 is a must have. Splitting the play services jar as a general solution is problematic because of native modules that rely on different combinations of it. One solution would be to add multidex to the build script (including the additional support jar - which is small enough), and condition it on a ti config varialble (e.g., _ti config set android.multidex true_) The proper solution would be to move to a gradle build, with the google defined split libraries, but this of course is a LOT of work.
@Ingo: I understand that using the entire google play services jar is becoming problematic. I appreciate how you've been using individual APIs. That's good stuff. @grebulon: I understand that native modules rely on different combinations. But I'm not sure what I was saying is in conflict with either one of your statements. But perhaps my idea of stripping down the full JAR is not as good as just using the individual API jars. So -- if we could work towards version standardization for all modules that use google play services, then the modules could ship the individual API jars that they need, and they would all get combined by the build process into one big happy binary. I think it would be easy enough to get module authors to use the individual API jars. If Appcelerator issued an official statement with that guidance, I think module authors would get on board. As the maintainer of ti.dfp, I know I would, and if the titanium-google-analytics maintainer doesn't want to, I would be happy to fork that module and make the change. But I think any reasonable dev would comply. The bigger challenge is dealing with version mismatches of the jars. Somehow, we need to package the module with different "flavors" (same module source code, but linked against different versions of the google play services library). Then the build process could examine the modules, find the highest GPS version that is shared by each module, and it would use that one. Of course, this puts burden on the module author to link against a number of old GPS versions. Not exactly convenient, but I don't know of a better way to handle this. This could get impractical, since there's not a great way to get old versions of google play services. I've been maintaining a blog post with the version history here: [http://www.smorgasbork.com/2015/01/05/google-play-services-sdk-version-history/], but AFAIK, google doesn't provide an "official" way to get old libraries. Not only that, but for good interoperability, you'd probably want each module to be linked against all the google play services versions that have been released in the past year or two. That way, each module author can go 4 or 5 months without an update and know that all the other modules out there would still be including a build with the same GPS version. Google's released 7 versions in the past year alone. So you might have to link against 15 versions to cover a two-year period... I'd love to hear better solutions. But right now, there are only two ways a developer can build an android app with Titanium today: - use *only* GPS-linked modules provided by appcelerator directly (to guarantee version compatibility) - *OR* use a mix of appcelerator and open source modules, which requires you to get your hands dirty and start tearing apart the modules, replacing JARs, re-linking, etc. It's a nightmare right now, and it's only going to get worse without a real solution from Appcelerator.
Hey guys, just wanted to chime in here on this and see if we had any thoughts on when we might have a fix for this? It's becoming a pretty big problem and is actually a show-stopper for us at the moment that we are limited to that small number of methods. I know it's been a year since this was posted, but it's still very much an issue. Right now we have: 1) An Image Compression module 2) Facebook (Connect) 3) Crash Reporting (Crittercism) 4) Analytics (Google) 5) Push Notifications (Localytics) 6) A Drawer Layout module With just those 6 modules (which really don't seem that out of the ordinary to have in an app) we are 40-ish methods over the limit and have to strip one out to build it. Without too much knowledge of the Android OS, I really can't do too much to help out unfortunately, but will gladly try to dig in where I can.
Hi. My app stopped with this problem on 5.1.0 update. Anyone working on this? Regards.
[~flaviolacer] What error did it give you?
Hi, trouble writing output: Too many method references: 68004; max is 65536. [ERROR] : You may try using --multi-dex option.
I got the same problem. The FACT issue is not fixed: the method limit is still there! My application has modules with: chat, map, third party sign in, push.
trouble writing output: Too many method references: 67181; max is 65536. [ERROR] You may try using --multi-dex option.Hey guys same thing here. I wanted to know if there is any way we can get this bumped.
Greetings, We are encountering this issue in 5.1.2.GA. Is there a current workaround or a way to enable multidex?
(duplicate)
I'm having the same problem in 5.2.0.GA Failed to run dexer: [ERROR] : [ERROR] : trouble writing output: Too many method references: 66795; max is 65536. [ERROR] : You may try using --multi-dex option.
[~amurcia] This should be looked into. A workaround is to slim down the modules. Are there modules that include google play services jar that could be replaced? This would reduce the method count and possibly help compile it.
What is the progress on this? Seems a pretty critical issue for most serious app developers. This ticket is already open 1,5 years!
I am also bumping into this issue, after trying to compile with SDK 5.4.0. ;-[
Is there anyone or any team out there we can pay to add this functionality? It seems as though it's clearly not going to be worked on (as you said @rene, it's been 1.5 years). Maybe we should start a Gofundme page and hopefully hire someone with more Android technical know-how to implement this :)
https://github.com/appcelerator/titanium_mobile/pull/8095
Updated comment in the PR to mention that the PR does work for anything that is above Android L. Anything below that requires a different implementation. Also we need to note that there are some limitations as mentioned in the docs [https://developer.android.com/studio/build/multidex.html](https://developer.android.com/studio/build/multidex.html). {quote} The installation of .dex files during startup onto a device's data partition is complex and can result in Application Not Responding (ANR) errors if the secondary dex files are large. In this case, you should apply code shrinking techniques with ProGuard to minimize the size of dex files and remove unused portions of code. Applications that use multidex may not start on devices that run versions of the platform earlier than Android 4.0 (API level 14) due to a Dalvik linearAlloc bug (Issue 22586). If you are targeting API levels earlier than 14, make sure to perform testing with these versions of the platform as your application can have issues at startup or when particular groups of classes are loaded. Code shrinking can reduce or possibly eliminate these potential issues. Applications using a multidex configuration that make very large memory allocation requests may crash during run time due to a Dalvik linearAlloc limit (Issue 78035). The allocation limit was increased in Android 4.0 (API level 14), but apps may still run into this limit on Android versions prior to Android 5.0 (API level 21). There are complex requirements regarding what classes are needed in the primary dex file when executing in the Dalvik runtime. The Android build tooling updates handle the Android requirements, but it is possible that other included libraries have additional dependency requirements including the use of introspection or invocation of Java methods from native code. Some libraries may not be able to be used until the multidex build tools are updated to allow you to specify classes that must be included in the primary dex file. {quote}
For test case, please add modules to the Titanium app in 5.4.0 till it fails with
Too many method references: 68927; max is 65536.Then switch to test with 6.0.0. It should pass as multidex is in use.Verified the fix. Checked by using https://github.com/ashcoding/ReachTheLimit module. It fails with 5.5.1.GA but works fine with 6.0.0 which has the fix. Closing. Please reopen if you find the issue is reproducible after the fix. Environment: Appc Studio : 4.8.0.201609292239 Ti SDK : 6.0.0.v20161002235150 Ti CLI : 5.0.10 Alloy : 1.9.2 MAC El Capitan : 10.11.6 Appc NPM : 4.2.8-7 Appc CLI : 6.0.0-55 Node: 4.4.4 Nexus 5 - Android 6.0.1
How do we achieve multi-dex support in titanium for android versions < 5.0? I tried changing the android application name, i.e.: