Blog

Flexing Xamarin.Android’s Build to Work Around Android’s 65K Method Limitation

Graham Murray / Friday, February 6, 2015

I’ve you’ve ever used a few large java libraries in an Android application you may have run into an odd limitation. When the android build process compiles your Java code/classes to work with the Dalvik Virtual Machine, it creates .dex files. During this process you will get interesting errors if your application and all the libraries that it includes result in over 65,536 methods being referenced in the dex file. It will be something like this:

trouble writing output: Too many method references

Given that including just the Google Play Services library alone will result in over 20,000 method references in the dex, if you are using a few other reasonably sized libraries you can very quickly get in a lot of trouble.

Google provides some information about how to avoid the limitation once you reach it: https://developer.android.com/tools/building/multidex.html

(More discussion here: https://medium.com/@rotxed/dex-skys-the-limit-no-65k-methods-is-28e6cb40cf71)

However, if you are using Xamarin.Android, and are including large native Java libraries (very easy to do if taking a dependency on Google Play Services), the guidance doesn’t seem to be especially clear.

One of the recommendations Google makes is to run ProGuard to strip out portions of your application that aren’t going to be directly used at runtime. However getting this tool to run during the Xamarin.Android build process seems to be less straightforward than against a vanilla Android application. Some work has been done by the community to try and smooth the process over, but it seems some work is needed if you want to run it against Google Play Services and the latest SDK version.

So, that really leaves us with the multidex solution described in the articles. The high level view of that process is:

  • Have your application build multiple dex files instead of just one, individual dex files will stay under the limitation threshold.
  • When your application loads, either be running Android L, or call the multidex support library to cause the rest of the dex files to be loaded during initialization.

Pulling this off with traditional Android tooling is somewhat straightforward, because much of the build tooling has been updated to make it a simple option to enable multidex. However, it doesn’t seem Xamarin has created any simple way to enable this yet when using Xamarin.Android.

Well, I’ve given it a go anyway.

Warning: I haven’t been able to verify with Xamarin yet that there is no roadblock to using this approach. I haven’t run into an issue yet, but have only done cursory testing. Proceed with caution. Additionally I’ve inferred a lot of things about how their targets files work, so I could be stepping on things that shouldn’t be stepped on. Nevertheless, here’s an attempt to reconcile what Google describes you should do with Xamarin’s build process for Xamarin.Android.

  1. First of all you need to include the multidex support library in your Xamarin.Android application. Create a folder called Jars in your android project and add a link to: [Android SDK Root]extras\android\support\multidex\library\libs\android-support-multidex.jar (remember to add as link)
  2. Set the build action on that jar file to: AndroidJavaLibrary
  3. Create a new project in your solution, and call it MultiDexTasks. We need to create a few custom MSBuild tasks in order to modify a few interactions with the native Android tooling that Xamarin doesn’t make customizable enough yet.
  4. Add these framework references to the project: Microsoft.Build, Microsoft.Build.Framework, Microsoft.Build.Tasks.v4.0, Microsoft.Build.Utilities.v4.0.
  5. Add a file reference to Xamarin’s android build tasks assembly to the project. This is usually located at: C:\Program Files (x86)\MSBuild\Xamarin\Android\Xamarin.Android.Build.Tasks.dll

Ok, there’s the basic skeleton. Now you should add a new class to the MultiDexTasks project:

using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
using Xamarin.Android.Tasks;

namespace MultiDexTasks
{
    public class MultiDexCompileToDalvik
        : CompileToDalvik
    {
        [Required]
        public string DexesOutputDirectory { get; set; }

        protected override string GenerateCommandLineCommands()
        {
            var baseString = base.GenerateCommandLineCommands();
            var newString = baseString;

            var outputMatch = new Regex("--output=");
            var match = outputMatch.Match(baseString);
            if (match.Length > 0)
            {
                var startIndex = match.Index;
                var eqIndex = baseString.IndexOf('=', startIndex);
                var inQuoted = false;
                int endIndex = startIndex;
                int i = 0;
                for (i = eqIndex; i < baseString.Length; i++)
                {
                    var prevChar = baseString[i - 1];
                    var currChar = baseString[i];

                    if (currChar == '"')
                    {
                        if (prevChar != '\\')
                        {
                            if (inQuoted)
                            {
                                break;
                            }
                            else
                            {
                                inQuoted = true;
                            }
                        }
                    }
                    if (currChar == ' ' & !inQuoted)
                    {
                        i--;
                        break;
                    }
                }

                endIndex = i;

                var toReplace = baseString.Substring(startIndex, (endIndex - startIndex) + 1);
                Log.LogMessage("will replace: " + toReplace);
                var commands = GetExtraCommands();
                Log.LogMessage("replacing with: " + commands);
                newString = newString.Replace(toReplace, commands);
            }

            return newString;
        }

        private string GetExtraCommands()
        {
            var output = DexesOutputDirectory;
            CommandLineBuilder b = new CommandLineBuilder();
            b.AppendSwitchIfNotNull("--output=", output);
            b.AppendSwitch("--multi-dex");
            return b.ToString() + " ";
        }
    }
}

Because Xamarin doesn’t let us fully customize the area where they invoke the command that generates the main dex file for the application, this creates an MSBuild task that inherits from the one that Xamarin defines and will modify the output file so that it is not a single file, but a directory, and it will also make sure to specify the --multi-dex parameter, which informs the dex command to generate multiple dex files.

Next, because Xamarin only expects to put one dex file into the apk when it comes time to build the application package, we need to create a task that can help inject additional dex files into the application package. So create another file:

using Microsoft.Build.Framework;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.IO.Compression;

namespace MultiDexTasks
{
    public class AddOtherDexFilesToApk
        : Microsoft.Build.Utilities.Task
    {
        [Required]
        public ITaskItem ApkFile { get; set; }

        [Required]
        public ITaskItem[] Files { get; set; }

        public override bool Execute()
        {
            var fileName = ApkFile.ItemSpec;
            Log.LogMessage("updating apk: " + fileName);

            using (FileStream zipToOpen = new FileStream(fileName, FileMode.Open))
            {
                using (ZipArchive archive = new ZipArchive(zipToOpen, ZipArchiveMode.Update))
                {
                    foreach (var file in Files)
                    {

                        var existingEntry = archive.GetEntry(file.GetMetadata("Filename") + file.GetMetadata("Extension"));
                        if (existingEntry != null)
                        {
                            existingEntry.Delete();
                        }

                        Log.LogMessage("adding file to apk: " + file.ItemSpec);
                        ZipArchiveEntry fileEntry = archive.CreateEntry(file.GetMetadata("Filename") + file.GetMetadata("Extension"));
                        
                        using (StreamWriter writer = new StreamWriter(fileEntry.Open()))
                        {
                            using (StreamReader reader = new StreamReader(file.ItemSpec))
                            {
                                reader.BaseStream.CopyTo(writer.BaseStream);
                            }
                        }
                    }                   
                }
            }

            return true;
        }
    }
}

This will just open up the specified apk file (which is a zip file), and dump additional files into it.

Now, you can create a targets file, called EnableMultiDex.targets that will be imported into your project in order to invoke these extra tasks at the right time:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <UsingTask TaskName="MultiDexTasks.MultiDexCompileToDalvik" AssemblyFile="MultiDexTasks.dll" />
  <UsingTask TaskName="MultiDexTasks.AddOtherDexFilesToApk" AssemblyFile="MultiDexTasks.dll" />

  <ItemGroup>
    <MultiDexOutputs Include="$(IntermediateOutputPath)android\bin\*.dex" />
  </ItemGroup>

  <ItemGroup Condition="'@(MultiDexOutputs)' == ''">
    <MultiDexOutputs Include="$(IntermediateOutputPath)android\bin\dummy.dex" />
  </ItemGroup>
  
  <Target Name="_CompileDex"
  DependsOnTargets="_FindCompiledJavaFiles;_GetMonoPlatformJarPath;_GetAdditionalResourcesFromAssemblies;_GetLibraryImports"
  Inputs="$(MSBuildAllProjects);$(MonoPlatformJarPath);@(_CompiledJavaFiles);@(AndroidJavaSource);@(AndroidJavaLibrary);@(AndroidExternalJavaLibrary);@(ExtractedJarImports);@(_AdditionalJavaLibraryReferences)"
  Outputs="%(MultiDexOutputs.FullPath)">
    <!-- Compile java code to dalvik -->
    <MultiDexCompileToDalvik
      DxJarPath="$(DxJarPath)"
      JavaToolPath="$(JavaToolPath)"
      JavaMaximumHeapSize="$(JavaMaximumHeapSize)"
      JavaOptions="$(JavaOptions)"
      ClassesOutputDirectory="$(IntermediateOutputPath)android\bin\classes"
      DexesOutputDirectory="$(IntermediateOutputPath)android\bin"
      MonoPlatformJarPath="$(MonoPlatformJarPath)"
      JavaSourceFiles="@(AndroidJavaSource)"
      JavaLibraries="@(AndroidJavaLibrary)"
      ExternalJavaLibraries="@(AndroidExternalJavaLibrary)"
      LibraryProjectJars="@(ExtractedJarImports)"
      DoNotPackageJavaLibraries="@(_ResolvedDoNotPackageAttributes)"
      ToolPath="$(DxToolPath)"
      ToolExe="$(DxToolExe)"
      UseDx="$(UseDx)"
      AdditionalJavaLibraryReferences="@(_AdditionalJavaLibraryReferences)"
	/>
    <ItemGroup>
      <DexOutputs Include="$(IntermediateOutputPath)android\bin\*.dex" />
    </ItemGroup>
    
    <Message Text="@(DexOutputs)" />
    <Touch Files="%(DexOutputs.FullPath)" />
  </Target>

  <Target Name="AddAdditionalDexes"
          AfterTargets="_PrepareBuildApk"
          Inputs="$(MSBuildAllProjects);@(_ResolvedUserAssemblies);@(_ShrunkFrameworkAssemblies);@(AndroidNativeLibrary);@(MultiDexOutputs)"
          Outputs="$(IntermediateOutputPath)android\bin\packaged_resources">
    <ItemGroup>
      <OtherDexes Include="$(IntermediateOutputPath)android\bin\*.dex" Exclude="$(IntermediateOutputPath)android\bin\classes.dex" />
    </ItemGroup>
    <AddOtherDexFilesToApk ApkFile="$(IntermediateOutputPath)android\bin\packaged_resources"
                           Files="@(OtherDexes)" />
  </Target>
          
  
</Project>

This basically overrides the existing Xamarin.Android target that compiles the dex file and calls our customized version of the dex compilation task, which will generate the multiple dex files.

It also causes additional dex files to be injected into one of the intermediate apk files that gets built, so that all downstream final versions of the apk file should wind up with all the required dex files. There is probably a better place to inject these, but this way required far less overriding of Xamarin’s existing targets and so should be slightly less fragile if they move stuff around…..maybe.

Next, you should set the build output directory for the MultiDexTasks project to be ..\..\Tasks (depending on your project structure you may want to adjust this path, goal is to put Tasks folder at project root level), and make sure that your EnableMultiDex.targets file is set to always copy to the output directory. This puts the task dll and the targets file in an easier to access place to reference from your other project.

Next unload your Xamarin.Android project that needs to get around the limitation. You should now be able to select edit from the right click menu. You need to add this import:

<Import Project="..\..\Tasks\EnableMultiDex.targets" />

You may have to adjust that relative path depending on your project structure. Critically, though, you must add this import AFTER Xamarin.Android’s target file is imported:

<Import Project="$(MSBuildExtensionsPath)\Xamarin\Android\Xamarin.Android.CSharp.targets" />
<Import Project="..\..\Tasks\EnableMultiDex.targets" />

Now you can reload the project. The last thing you need to do is to do is to change the application type so that Android knows that you have this multiple dex file scenario, and will cause the other dex files to be loaded.

In order to do this open up your android manifiest at Properties => AndroidManifest.xml and add this attribute to the application tag:

android:name="android.support.multidex.MultiDexApplication"

If this is the first edit you have made to your manifest the application tag should probably look like this now:

<application android:name="android.support.multidex.MultiDexApplication"></application>

You may want to add a project dependency on MultiDexTasks to your Xamarin.Android project since you now have an indirect reference to the tasks and target file it produces. Keep in mind, MSBuild likes to lock assemblies on disk that it loads tasks from, so if you are getting errors that MultiDexTasks.dll can’t be written, you may have to kill any lingering MSBuild.exe processes.

Anyway, now you can build and run the application again.

If you are still getting a build error, make sure you have set Project => Properties => Android Options => Advanced => Java Max Heap Size: 1G

If you have been hitting the 64K method limitation, you probably also need more Java heap space during the build, otherwise you will get out of memory exceptions.

Caveat: Though the application builds, I haven’t extracted any feedback from Xamarin as of yet as to whether they expect any integration issues with Xamarin.Android when combining use with this multidex functionality. They could decide that the above is a bad idea. This should get you up and running though, if this is a blocking issue for you, and I’d be interested to hear what sort of issues you run into. If you think you know a cuter way of sliding these build changes into Xamarin’s process with less violence, I’d love to hear about it in the comments.

Best of luck,

Graham