For as nasty as the hack is, it still boils down to twiddling with a few numbers.
Also, inlining is probably easier said than done. It's not enough to inline the methods, you need to make them entirely disappear, and ISTR reading within the last few days somebody else commenting that Java still has to keep the metadata about the methods around (which is the problematic part, not the methods themselves, if I'm reading this right) because reflection may demand them. You'd need something more sophisticated to do it at the source level, and with an imperative language with unrestricted side effects, while that is certainly possible, it's also very much easier said than done. That's not a weekend hack either.
I'm not sure inlining is that difficult. Java bytecode is a relatively straightforward language is it not? Especially if it doesn't have to be for all cases, which, if you're looking at eliminating small methods, it does not have to be.
Assuming Java has something like the Cecil library for MSIL, it should be a fairly straightforward task. Although, as you point out, so is fixing up a number.
In order to elide the out-of-line method the compiler must statically prove that it is impossible call outside the visibility of its optimization scope.
I'm not familiar with Java compilers but I'd hazard a guess that to exploit this optimization (if it were possible at all) they would probably have to significantly rework their code.
It becomes hard when the app uses reflection. When you want to remove some method foo, you not only have to make sure that it isn't referenced by some code, but also make sure that no code is passing the string "foo" to reflection API. This may be hard for many apps and is undecidable in general.
Actually, even in plain, reflection-less java it may be quite complicated because subclass methods can be invoked by superclass-typed variables. For every method invocation you need to find all possible subclasses whose objects can be assigned to variable on which the method is invoked. You need to know which subclasses are ever passed as superclass/interface arguments to methods. You need to know which subclasses are assigned to superclass/interface member/static variables. You need to know which subclasses are stored in superclass collections. And so on.
The compiler would have to see all code that is linked with the app, including the Android runtime that calls into the app's code in various ways.
In Java land these kinds of fancy optimizations usually happen in the JVM JIT at runtime. Would this help the size of those tables in Android's Dalvik/dexopt setup? In any case you can't ship a custom version of Dalvik with your app.
You've got class foo, with methods "DoBar" which calls "DoBazStep1(), DoBazStep2(), SomeHelper.CalcPosition()".
You use a Java bytecode toolkit and identify leaf functions, small functions, whatever the criteria needs to be. Then you check every callsite for those functions, and if found, remove the call, and inline the code, doing fixup on locals and parameters. Now, "DoBar" has a lot more bytecode in it, as the bodies of DoBazStep1 and the others are now contained in it. The small functions are completely removed.
This process can be done totally offline, on the compiled Java code. The runtime just sees that you wrote one big function, versus a bunch of small ones.
The only real work is making sure you got everything, and perhaps being clever with stack local allocations. As the FB team says they moved to a new style with lots of small functions, I'm assuming they aren't doing all sorts of metaprogramming and inheritance, but have just split things up at the function level.
Also, inlining is probably easier said than done. It's not enough to inline the methods, you need to make them entirely disappear, and ISTR reading within the last few days somebody else commenting that Java still has to keep the metadata about the methods around (which is the problematic part, not the methods themselves, if I'm reading this right) because reflection may demand them. You'd need something more sophisticated to do it at the source level, and with an imperative language with unrestricted side effects, while that is certainly possible, it's also very much easier said than done. That's not a weekend hack either.