This writeup aims to fully explain what JEP 238: Multi-Release JAR Files are capable of, how they can be used, and also which problems can not be addressed by using them (by design, and that's fine).

This post was meant to be combined with another another one, being a "Guide for Reactive Streams library authors wanting to adopt j.u.c.Flow without waiting a decade", however the post grew too long and I'll post that one shortly as a follow-up.

This guide is mostly from the standpoint of a library author (big or small), and guided by practical experience in getting to understand mr-jars for future use in Akka Streams. The entire feature though is mostly intended for library authors, because end-users usually know which runtime they'll run their software (it depends of course how you distribute your app, sure ).

Basics: Using JDK9 features in implementations of classes

This JEP introduces a special way for the classloader to handle classes and resources depending on which version of the JDK is running the code.

To enable this feature the MANIFEST.MF must contain the following entry:

Multi-Release: true  

In real-life this would be added to the manifest by your build tool. For example when using sbt-multi-release-jar (more details later), it'll just work™ and add this entry to the jar's manifest file.

Since previous runtimes (e.g. Java 8) do not know the meaning of that setting they'll silently ignore it – which is desired behaviour in this case.

However, if running on Java 9, the runtime would pick up this setting and include some specific paths in its classpath resolution. Specifically, if a class is present in the /META-INF/versions/9 the classloader would pick it up automatically, as-if it was a normally defined class in the root of the jar file. Also, in case you'd have a normal class of the same name and one in /Meta-INF/versions/9, it will pick the 9-one instead of the "normal" class present in the root directory of the jar. This way we can “swap” implementations at runtime for ones that use new JDK9 features (and JDK10 in the future in META-INF/versions/10/).

This feature is very much designed for one prime use-case: allowing libraries to migrate away from Unsafe and use the new VarHandles APIs – or similar with other now deprecated APIs (like things from sun packages).

The Good: Using JDK9 features in implementations of classes

As explained above, the prime goal is to be able to "swap" implementations which depended on to-be-removed APIs like sun.misc.Unsafe or some other classes in sun packages–rest in peace, good guy Sun ;_;.

This works fine and will allow libraries to more smoothly migrate to using VarHandles or other APIs, without having to break compatibility with JDK 8.

We welcome this feature in Akka as it will enable us to slowly make use of new concurrency and network utilities without having to break JDK 8 compatibility for our users. While some may argue that "it's a hack", I'd argue that it's a good one–for users of our software and the Java ecosystem as a whole.

For end-users who know their target runtime this is a non-feature – pretty much like the entire JEP I think actually (that’s fine).

The (at-first) Suprising: JDK tools "support" is a bit tricky

As this is a JDK feature, the rest of the platform of course has to have ways of handling these jars as well. The naive intuitive meaning of "support" though here can be a bit misleading as you'll find that the same "magically swaps things depending on runtime" semantics are not preserved across the other tools with regards to this feature (which after some thought, all have a good reason–but still may catch you off guard if you're not aware of them).

Let's dive into some of the specific examples of other JDK tools and how they deal with mr-jars.

javap does not perform the class "swap" automatically

One might expect javap to perform the exact same kind of class resolution logic as the runtime does, however this is not the case. If you’d have the following:

$ find . -name JavaFlowAkkaSinkBuilderSupport.class
./akka/stream/javadsl/JavaFlowAkkaSinkBuilderSupport.class
./META-INF/versions/9/akka/stream/javadsl/JavaFlowAkkaSinkBuilderSupport.class

And ask javap for the JavaFlowAkkaSinkBuilderSupport class:

$ javap -cp . akka.stream.javadsl.JavaFlowAkkaSinkBuilderSupport 

Compiled from "JavaFlowAkkaSinkBuilderSupport.java" # "normal"

public interface akka.stream.javadsl.JavaFlowAkkaSinkBuilderSupport { ... }  

It will return the "normal" one. So it acts differently than the Java runtime. The rationale here is that javap is not a runtime tool and just prints whichever class you point it to in a nice format. I found this surprising at first, but I can sympathize with the choice made here, since it’s not really a “runtime tool” I guess, and the core feature that we need from mr-jars is enabling the runtime to use "the new stuff."

At the same time though, this behaviour may confuse developers who in presence of some issue want to inspect a class because they’d suspect a bug in it, and were not aware that that class actually was a multi-release class, hidden in versions/9javap would show them the "ok class", while the the bug actually was in the “9” version.

Perhaps I’m over worrying, and not many people do this, it would be nice though to have some indication from javap that “hey, multi-release version of this class found too!”. In any case, the correct way to inspect the versions/9 class is to use the path inside the jar file explicitly, like this:

$ javap jar:file:akka-stream\_2.12-2.5-SNAPSHOT.jar\!/META-INF/versions/9/akka/stream/javadsl/JavaFlowAkkaSinkBuilderSupport.class # versions/9

I’m not aware of a way of achieving the same thing with the -cp option, feel free to educate me if there is one.

There even was a ticket and option to make javap more aware of multi-release jars by allowing it to specify -multi-release 9 however this feature was dropped and instead one should specifically ask for the specific class one wants to analyse using the jar path syntax.

See also JDK-8153652 Update javap to be multi-release jar aware which explains the rationale for this decision.

javac compiles against the "normal" .class, so API of the “versions/9/” class should match

This makes complete sense when you think about it, but can catch you as a surprise when looking at mr-jar and what it COULD technically allow. Again, after some thought, I have to agree with the JDK team's decision about semantics here, which are:

When using a library released as mr-jar javac compiles against the "normal" class. It will not take into account the /META-INF/versions/9 classes because those are only a runtime concern, and javac does not care about runtime. It is given a source file, a classpath, and compiles against it. Its classpath logic can not assume “oh yeah, this will compile, because the META-INF/versions/9/ version would actually make it compile, even if the ‘normal’ version doesn’t!” If it had such logic then we basically allowed code to not compile on JDK 8 but compile on JDK 9, which would be pretty confusing. As such, the javac compiler takes the "usual" route which is to simply only look at the "normal" classes.

It does not however look at the jdk9 version of a class at all, so we can not know if we would not get runtime errors when running on jdk9, because for example we attempt to call a method that does not exist in the jdk9 version of that class! This means that library authors have to go the extra stretch to make sure the two versions of a class are completely binary compatible.

In other words: if we’re going to swap out a class that users directly call methods on - the API of the "normal" and “swapped” classes MUST be identical -or- binary compatible in a specific fashion.

The reason I'm explaining this case in depth is because, while working on Reactive Streams the solution for adopting java.util.concurrent.Flow would a few times be tounted as "oh yeah, we'll just mr-jar it and expose the new interfaces! – sadly this is not an option, the APIs of the swapped classes have to be the same, which leads to the next point:

The Mixed: Can’t introduce APIs for consumption by only JDK9 users

I call this a "mixed bad" because it’s a completely understandable state of affairs, however it also is something that some library authors (myself including for a while, until I've read up and thought about it some more) were intending to utilise.

The (practical) Downside: Need to duplicate entire classes

The one downside that mr-jars bring us is that the scope on which they operate is entire class fiels, so it's not easy to just say "if jdk9 { compiles only on jdk9 } else { jdk8 version }. This in practice leads to some duplication of potentially nasty code. This is because most of the use-cases I have in mind for this feature require the "nasty code" to be inside the same class that has other functionality (because local field accessing etc), such as an ActorCell etc. This means that library authors may be forced to duplicate some nasty concurrency heavy code into 2 files and maintain those in tandem, even if only a smaller part of code was meant to be "swapped".

Could this have been done differently? Perhaps, though the thought of going into advanced scope analysis in the javac compiler for it to notice "oh yeah, it's guarded by an if jdk9" are pretty unpleasant as well. Going C pre-processor style would also not be pleasant. So summing up we've simply selected one of the pains, and will be able to bite the bullet to provide value for our users, I think.

Conclusion: The practical usefulness of mr-jars

The practical usefulness of mr-jars is pretty limited. And perhaps that's for the better for all of us, since we won't get into completely crazy versioning maddness. It serves a single purpose, which is getting rid of implementations which use old, deprecated or sub-optimal APIs which in a following Java version have a replacement.

For end-users, "leafs" or application developers this feature is pretty much a non-issue, as you usually know what runtime you're running on - at least in the age of web and cloud based applications. It could be useful in shipping one jar for desktop or embedded apps, though that's not really the main battleground nowadays (except "edge", but there you control your vms most likely too anyway).

What if I really want to put JDK9 types in my API?

As side note, it is possible to use the jdk9 toolchain to emit jdk8 compatible bytecode, which means that one can compile using JDK9 targeting JDK8 and thourgh rigorous testing make sure that's actually valid and all good. Thanks to that such jar can be made to contain classes that refer and expose JDK9-only types, such as java.util.concurrent.Flow without forcing users to run using JDK9. They'd simply be unable to compile and/or use these specific classes when using prior versions of Java.

This is a road we've taken in Akka Streams's adoption of j.u.c.Flow and while it does carry quite some weight (on increasing the testing matrix mostly), it allows users to use the "new stuff" if they can, without having to release specific JDK9+ artifacts. Note that this technique does not utilise mr-jars, but plain old version dancing and the good old javac -target option.

Build tools support for mr-jars

sbt: Supported, via sbt-multi-release-jar plugin

I implemented the sbt-multi-release-jar plugin, which makes the simple use case that many of us need to tackle right now about "replace class that uses now forbidden things" by JDK9 specific implementation as easy as putting such classes in src/main/scala-jdk9 (or src/main/java-jdk9). Some edge cases remain (and I'd certainly welcome contributions and other forms of help), but the plugin provides a working and testable starting point ready for implementors to use.

Maven: Possible, with manual setup and some squinting

For Maven there's the Building Multi-Release JARs with Maven by Gunnar Morling, and you can have a look at mine example project putting it to action in ktoso/akka-streams-jdk9-strawman-mvn (though the actual example is not meaningful in there).

Gradle: Likely possible, did not find a guide or plugin though yet (I think?)

I'm currently not aware of a guide or built-in support or plugin in Gradle, please let me know in the comments if someone writes this and I'll happily add the link here for greater exposure.