Modularity is Hard. Let’s do a Jigsaw.
Mar 02
Java 8, The Well-Grounded Java Developer 12 Comments
Don’t forget: The 50% discount on the Early Access edition of our book expires tomorrow!
(Before we get underway, I want to make the point that all of this is based on my current understanding of the positions of both the OSGi folk and the current plans for Jigsaw. I’m not an expert in either, so there may well be egregious lies in this post. Informed comment from people who are an expert in either technology would be very welcome.)
Technologists don’t possess crystal balls. Early releases of technology platforms often have problems in them that are not ironed out (or even discovered) until much later.
One example of this is the Java platform’s lack of generalised modularisation and dependency management capability in the core. Sure, there are jar files, and as a basic means of shipping around units of functionality which are larger than a class, they’re OK.
However, when dealing with larger systems, the model of “just ship around a bunch of jars and manually manage the dependencies” starts to break down. What’s needed is a way to automatically manage the dependencies between functional units, via some metadata which is shipped within each unit.
This is where systems like Apache Ivy and Maven come from – they run outside of the core platform and automatically manage a repository of jar files. Build and runtime dependencies are then much easier to manage – all the jars are in one place. However, note that the platform doesn’t provide any support for these platforms out of the box – a classpath still has to be managed for start scripts, etc.
What would be better is if the platform provided for automatic management of dependencies in the core. Both of the mostly widely used systems broadly agree about which relationships between functional units need to be expressible, so there’s no major disagreement about what needs to be expressed.
The simplest cases are that a unit needs to be able to say that it requires:
- a particular version of a certain other unit
- a particular version or anything higher
- any version within a given range.
Dependencies should also be able to be marked as mandatory or optional, and different isolated contexts within the same JVM should be able to have different, incompatible versions of the same unit.
This much is basically agreed upon. Where the different viewpoints diverge is over what constitutes a functional unit, and at what stage of platform startup the dependency management and modularity functionality should kick in. The two big schools of thought are the OSGi view and the Project Jigsaw view.
OSGi
- Unit of sharing should be the package. This reduces conceptual burden on developers to learn a new model and is simpler.
- Dependency management should start once the platform is fully bootstrapped. This allows the benefits to be brought to any compliant installation, and doesn’t require changes to the core platform.
Project Jigsaw
- Unit of sharing should be a new concept – the module. This is “larger” than a package, and packages can also be split between modules.
- Changes should be made to the core, and the platform itself should be aware of module dependencies as early as possible. This is more work, and requires more co-ordination between implementations, but has potentially has big wins (quicker startup time, smaller footprint).
Which should you choose for your application? Well, Jigsaw is still being developed, and won’t land as part of Java 7 – it’s targeting Java 8. OSGi is a production technology, used in a surprising number of places (if you’ve ever used Eclipse, you’ve used OSGi technology, you just probably weren’t aware of it). However, it is still maturing – there are gaps around tooling, and there are gaps in the standard which has led to different approaches between vendors, with regard to e.g. bundle deployment.
If you want to learn a new technology, then Jigsaw is still very new, and a moving target, whereas OSGi is here today. On the other hand, unless you have a clear benefit from deploying something like this today, it may be preferable to live with the pain of Ivy/Maven and wait for the space to mature.
Finally, as a modest proposal, let’s talk about deprecation.
Java has always been bad at deprecation. We’ve known that Thread#stop and friends were the wrong thing to do for a very long time, but they’re still there, hanging on as deprecated methods. In the current setup it seems impossible to do anything to remove them. With a wide-ranging modularity solution, such as Jigsaw, in place – could we produce a release of a core module which removed the likes of Thread#stop and Object#finalize ?
RSS
Twitter
Email
Facebook
Join our Early Access Program, read chapters now!
Mar 02, 2011 @ 11:42:51
“Unit of deployment should be the package”: this is incorrect, the unit of deployment is the bundle, which is our term for a module. Physically, a bundle is a JAR file with some additional headers in its manifest.
Java packages are the unit of *sharing*: bundles export packages and import packages (just like Java code!). Jigsaw confuses these two concepts, i.e. you always depend on another named module rather than the packages offered by that module. In OSGi we regard this as a mistake: you should not bind to the arbitrary wrapper around the functionality that you want to use.
Note that OSGi also has a module-level dependency mechanism (known as Require-Bundle), but its use is discouraged. This also allows us to split packages across modules, but no matter what module system you choose — or even if you use plain old JARs/classpath — splitting packages is a terrible mistake.
I accept that Jigsaw’s approach is perhaps necessary for the special case of modularising the JRE, because it is a pathologically tangled mess of dependencies, which cannot be refactored without breaking all the production Java code out there. This goes to your final point about deprecation: at what point should we accept the need to make incompatible changes to the platform and libraries?
Mar 02, 2011 @ 12:06:04
Thanks Neil. Changed to sharing.
Your comment is the absolute crux of the difference between world views, so thank you for providing such a clear articulation of it.
I don’t think I agree that splitting packages is always a mistake, and my own personal view is to lean towards the view expressed by Mark Reinhold & co – that an additional level of indirection (which is how I see modules) is necessary – that packages are too low-level.
Have you seen some of the slides for the presentations that Mark and Tom Marble gave at FOSDEM? There was some really interesting stuff in there. Not only that, from a purely Free Software perspective, it became clear to me that making this all play nicely with Linux package managers is very important.
Martijn and I attended some talks in the Mono room, and while some of the things I saw in there shocked me (e.g. the primitiveness of their GC) the one t hing they did emphasize was their good relationships with upstream packagers. Compared to the state we find ourselves in with Java, they seemed to be mostly doing it right.
Anyway, sorry I couldn’t make it to the event the other night & let’s try to catch up at a pub event soon.
Mar 02, 2011 @ 14:57:09
Hi Ben (quick note: I wasn’t sure whether you or Martijn were the author of this blog post… you should add a by-line to your posts!)
I’ll certainly look at the presentations by Mark and Tom, could you help me out with a link? I must admit that I’m skeptical about this need since it hasn’t arisen in ten years’ of experience with modular Java applications using OSGi… and I’m even more skeptical that it could not have been incorporated into OSGi rather than inventing an entirely new module system from scratch. But I’m still open minded
I agree that playing nicely with package managers is a good feature, and credit to Mark for identifying this. There is nothing to stop somebody from doing this for OSGi as well, but nobody (AFAIK) has put in the effort yet. Also there has been a certain reluctance because it feels like moving away from Java’s WORA promise — previously you could just download a single JAR file for all platforms, whereas native packagers are obviously OS-specific. Then again if there was an automated way to transform a bundle into a .deb and an .rpm etc then this would seem to offer the best of both worlds.
Mar 02, 2011 @ 16:22:01
Hi Neil,
I think there is actually a little byline just under the title – but maybe it’s too small to be easily spotted
.
Tom’s slides are linked from here: http://wiki.debian.org/Java/DevJam/2011/Fosdem/JavaSpeakers#TomMarble
Mark doesn’t seem to have uploaded his.
I definitely think that automated production of platform-specific bundles is the way to go, and I think that it can help to straddle the divide between “applications” and “libraries”.
Partly this is because I’m slightly concerned by the continued rise of Mono as the way that more and more Linux desktop apps are being written – I think it’s the wrong thing, and I’d much rather that those people were developing Java desktop apps instead.
Mar 02, 2011 @ 20:40:49
Hi Neil,
This was indeed Ben’s post. I checked it over quickly and still missed the sharing aspect, appreciate you picking that up! I’ll be posting the discussion summary from our chat at IBM on the LJC blog later on this week.
Cheers,
Martijn
Mar 03, 2011 @ 16:07:02
@Ben: come on you just added that byline didn’t you… I looked very carefully before complaining
Thanks for the link to Tom’s slides. Unfortunately I can’t really see anything in them that refers to this higher level of abstraction that you talk about. I suppose you had to be there.
This is another of my beefs with Jigsaw: information about it is released piecemeal, informally and in a format this is difficult to reference later, e.g. a conference talk. If only it were backed by a good, transparent JSR!
Mar 03, 2011 @ 21:52:24
Hi Neil,
Hopefully that JSR is coming soon, we’ll be chatting to Mark et al about it over the coming year. As a developer who kind of sits in the middle of a lot of these debates, I just want to see as much inter-operability as possible. Choice is good, unnecessary fragmentation is bad
(That’s my opinion, I don’t claim to speak for Ben).
Hope you caught my other OSGi post – feel free to tear holes in it
Cheers,
Martijn
Apr 03, 2011 @ 16:49:20
Ben,
Interesting comment…
“What would be better is if the platform provided for automatic management of dependencies in the core.”
This is exactly the approach we’ve taken with Nimble and the Service Fabric. Both are OSGi centric – but the Nimble resolver is actually a generic dependency resolution engine. See http://www.paremus.com/products/products.html if you are interested in more detail.
Personally, I’m not at all convinced by JigSaw. From my perspective – OSGi is gaining traction in the enterprise, there are some great build tool projects like Neil’s BNDtool.
Meanwhile all I’ve found on JigSaw are some vague slides.
Best Wishes
Richard
Apr 04, 2011 @ 07:58:12
Hi Richard,
Thanks for your comment. From my perspective I haven’t seen too many projects in the enterprises I’ve worked with which were using OSGi – problems such as the lack of unified deployment standards, concerns about complexity and vendor lock-in are holding it back.
Martijn wrote a good blog post about this subject as well: http://martijnverburg.blogspot.com/2011/03/osgi-are-we-there-yet.html
As to Jigsaw, give the guys a break – the project leads are a bit tied up with delivering JDK 7 right now
Once 7 is out, I think we’ll see Jigsaw move forward quickly. The other part is that there are several strands (Jigsaw, OSGi, Linux packaging) which it would be much better to bring together.
The other point is that Jigsaw does things which OSGi can’t do. Without altering the VM and the early stages of classloading, the base JDK can’t be modularized – but there are plenty of use cases where we want to load a subset of the JDK.
Apr 03, 2011 @ 16:54:30
Also…
I would wager that modularising an application will be no simpler with JigSaw.
In his new book Kirk Koernschild demonstrates modularising a Java application without any modularity framework!
Upon completion he then translates his JARs to OSGi bundles with almost zero effort. Dependencies and structure exist whichever metadata system you use to describe them.
Best Wishes
Richard
Jul 29, 2012 @ 03:23:38
This almost feels that this is becoming more of a question regarding where we should be drawing a box around our class diagram. Coming from from the spring-dm/virgo world, bundles seemed like a natural fit provided that package hierarchy was well formed and the naming system was consistent to the root package’s namespace (which in the spring OSGI bundle repository made for a well formalised naming convention on maven dependencies and worked spectacularly). But these days with the polyglot jvm author none of these things can be insured making the osgi model much harder to work with (unless you have a repository maintainer which insures this type of formalisation; which they can’t as it should be language agonistic). So the ideal is lost and virtually impossible to maintain in the OSGI realm, making Jigsaw (if any) a more realistic option…
But on a more important note. I think the real interesting area is how they compare with respect to classloading. I’m going to investigate this now for JIGSAW but the OSGI models classloading hierarchy is really what makes for a well defined component model. Now unless the jvm is going to include something similar to that provided by http://zeroturnaround.com I can’t help but think that JIGSAW’s upcoming java8 release will be nothing more than dynamically loading modules via curl (and jdk module partials, which I imagine will also come for free when using osgi on a supported jvm verison).
Jul 29, 2012 @ 03:29:06
ensures, ensures!