An introduction


This article describes Mistral, an opensource, high level imaging engine aimed at professional imaging applications which:

  • allows to use under an unified API a number of existing imaging engines (such as Java2D, JAI and others);
  • implements platform-specific optimizations;
  • can be easily extended;
  • includes an advanced image renderer;
  • has extensive support for distributed processing;
  • supports metadata extraction (in progress).
Feedback to this article can be posted to the Mistral forum.

Introduction

In more than ten years Java has colonized a lot of application fields, ranging from the cellular phone to the data center and beyond. In the latest years there has also been a "renaissance" in the desktop segment, both thanks to new rich APIs such as SwingLabs and to the general fact that the Virtual Machine and Swing itself have gotten faster, removing a lot of showstopping problems that haunted their early versions. Furthermore, rich platforms such as NetBeans RCP or Eclipse RCP now provide a lot of tools for the design of complex and modular desktop applications.

Focusing on the desktop, one of the application fields that have been only marginally targeted is imaging - I'm talking of the end-user, desktop kind of imaging, as in some scientific and technologic fields Java2D, JAI (Java Advanced Imaging) and ImageJ have been used for years.

But it looks like things are changing also in this scenario. More or less one year ago, Lightzone, a commercial application targeted at photographers, has appeared; Jasper Potts has recently blogged about a nice desktop application used for setting up photos for printing albums; Jasper himself has worked at Xerto that has produced Imagery, announced some months ago and that now seems to be near the beta stage; even though is not a full-fledged imaging software, JAlbum is one of the most used tools for creating an publishing gallery. Also worth mentioning is JH Labs, an open source application; and Dirk Hennig has his own project named Quippix. All of these projects have been done in Java and this means that probably the technology is now mature also in this field.

I've been working on opensource imaging since 2003, even though for some time I wasn't clearly committed in delivering a real product - I've rather worked on prototypes that I used to try some technologies I didn't knew. I've taken the final decision of going towards a real product only recently. Today the project, called blueMarine, is near to its first Release Candidate (compulsive downloaders, beware: there are some early eaccess builds already available, but they aren't quality tested and in most cases they will fail just when installing - I'd suggest you to go to the site and rather subscribe to the news feed and wait for the incoming RC1; if instead you're interested in joining the project, contact me following the links on the website).

As the project has grown, I've started to spin-off some sub-projects which covers specific parts, since I thought that they could be useful to others. One of the APIs at the core of blueMarine is Mistral, an imaging engine with all the features for a professional imaging application.

State of the art of Java imaging

Before talking of Mistral, I think it's a good thing to look at the current scenario about imaging APIs.
  • The core API for imaging is Java2D, which is part of the JRE. Java2D deals with BufferedImage, provides some operation for image manipulation - and BufferedImage is fully compatible with Swing, so it can be directly drawn on a GUI component.
  • JAI is the Java Advanced Imaging API and is a step forward: not only it provides a much richer set of manipulation primitives (such as Fourier Transforms and Histograms, support for Region of Interests, etc...), but it also comes with its own image class, PlanarImage, which has support for "tiling". Tiling is a memory allocation strategy by which only the currently used portions of the image are kept in memory. The JAI attitude towards efficient handling of large images is also demonstrated by the "deferred execution" feature: when you apply a sequence of operations to an image, they are not executed at once, but only when the API understands that the application wants a result (e.g. it wants to draw the image somewhere). In this way execution can be optimized as only the strictly required subset of tiles is manipulated. This attitude towards very large images is explained by the fact that NASA - who has contributed the design of the API - is using JAI for the management of astronomic images that tend to be very large. JAI is not part of the JDK but it's available as a standard extension; and PlanarImage is somewhat a close relative of BufferedImage (but must be handled in a rather different way if you want to use tiling).
  • ImageJ is a public domain imaging engine developed by the National Institutes of Health. While it is not as sophisticated as JAI, it has some popularity in the scientific world, specially for medical image processing. It comes with a desktop application for image manipulation, with the same name (I didn't cited it before since it's not targeted at the desktop casual end-user). ImageJ models images with its own class.
  • JHLabs provides a set of filter classes orientated to photo manipulation, with similar effects as Photoshop. JHLabs works on regular BufferedImages.
  • JIU is the latest API I've been aware of (just a few days ago) and provides another set of image manipulation classes. I don't know much about it since I was be pointed to JIU just a few days ago. I could be wrong but I think that JIU models images with its own class.
So, indeed it looks like there's a lot of stuff. While blueMarine editing attitude is presently only aimed to RAW camera formats conversion (that require a very special set of filters), I plan to integrate a full-fledged editor sooner or later. Even though JAI is by far my favourite imaging API, as blueMarine has been designed as a plugin-expandable core I would also like to have people to use it with their own library. How to achieve this?

A meta imaging engine

Mistral is a "meta imaging engine", in the sense that it doesn't include its own imaging code, but use other APIs. It acts as an abstract imaging layer, as the programmer only sees Mistral classes, while the actual implementation is pluggable. This means that you can use the very same Mistral code and have Java2D, or JAI, or JHLabs, or JIU to do the work (of course, supposing that you're done something which is covered by all these APIs). Furthermore, you are not constrained in using a single implementation, but you can use multiple ones at different stages of the elaboration. For instance, you can start with Java2D (by default Mistral always starts with Java2D when an image is loaded from a file, since the JDK has a very rich and puggable image I/O facility), perform some operations with Java2D, then use some JAI features, later JHLabs and so on. This makes a lot of sense when the different APIs share the same image representation (such as, for instance, Java2D and JHLabs) or where the conversion of the image representation is fast (such as for Java2D and JAI when no tiling is used). For instance, a typical imaging application operation is creating thumbnails from the original files, which can be done by using Java2D; but if you need a Fourier Transform, you can immediately switch to JAI.

The core of Mistral is the EditableImage class, an opaque wrapper against the actual image representation. It exposes a few methods for reading an image from an external source or creating it from the scratch, performing operations on it and getting the most common attributes such as the size, the number of colors, and so on. Here it is an example of code:
EditableImage image = EditableImage.create(new ReadOp(file, 0));
image.execute(new CropOp(10, 10, 600, 400));
Histogram histogram = image.execute(new HistogramOp()).getHistogram();
This sequence loads an image from a file, crops it, and computes the histogram on the cropped image. ReadOp, CropOp and HistogramOp are classes representing the three operations; they are simple Java beans that only hold the required parameters (and eventually the result if it's not an image), but not the implementation code. The code is instead contained in some specific CropXXXOp and HistogramXXXOp classes which are specific to the API used (for instance, CropJ2DOp, HistogramJ2DOp - and CropJAIOp, HistogramJAIOp, and so on). The mapping between the generic and the specialized operations is stored in so called "implementation factories". There's one implementation factory for every supported API and they are registered into the runtime with the following code:
ImplementationFactoryRegistry registry = ImplementationFactoryRegistry.getInstance();
registry.registerFactory(new ImplementationFactoryJ2D());
registry.registerFactory(new ImplementationFactoryJAI());
registry.registerFactory(new ImplementationFactoryImageJ());
(in the real case, ImplementationFactoryJ2D is always registered by default, while the others have to be explicitly registered by the programmer). When an execute() operation is performed, Mistral searches the implementation in the current implementation factory (always Java2D at the beginning); if it can't find it, other factories are searched and the image representation is eventually converted.



Referring to the above histogram example, the whole sequence is executed using Java2D since all the three operations are defined in its factory. Somebody could prefer to use the JAI version of HistogramOp (which BTW is faster and much accurate). The programmer can take the control and choose the proper API for each operation, unregistering the operation from all the other factories. For instance:
ImplementationFactoryJ2D.getInstance().unregister(HistogramOp.class);
makes sure that the implementation from Java2D is not used (note that the previous sample code for computing the histogram doesn't change, but now the histogram is computed by means of JAI).
Last but not least, new operations can be defined and added to a factory by registering them at runtime.
It is to be pointed out that this abstraction layer doesn't have a significant performance cost since finding an implementation is just a matter of a few hashtable lookups, a much faster operation than the average image manipulation.

Added value of the abstraction layer

If you need this imaging API interchangeability you are already appreciating the abstraction layer of Mistral; if you don't, consider another bunch of clear advantages of it:
  • Simplified APIs. A lot of image manipulation can be performed in different ways, often a trade-off between performance and quality. For instance, there are many ways of resizing an image, and the better quality ones involve more than a single API call. For instance, the following is a typical error from many Java2D and JAI programmers that want to resize an image: there are two available operations, AffineTransform and Scale, that "seem" to do the desired thing; but often the result quality is not good for many people and the API are blamed for it. Indeed, good quality resizing requires a more complex approach which is well described in this post and the linked document, while APIs are usually "atomic" in the sense that they do only one thing at a time. This holds true for resizing as well as other complex image manipulations. In this perspective, Mistral helps the programmer as its operations are not necessarily atomic. For instance, there is SizeOp with the algorithm described in the above linked documents and everything is simple as:
    image.execute(new SizeOp(0.5));
    Generally speaking, the easy way in which the programmer can define his own custom operations allows to implement a consistent implementation strategy for complex operations and to stick with it in all the application. This usually also leads to a reduced number of lines of code in the application.
  • Performance optimizations. In spite of the WORA concept of Java, some Java2D/JAI operations have dramatically different performance on different operating systems (this is partly due to bugs). For instance, you can scale an image using an AffineTransform or drawing the source image over the target image using the Swing Graphics object: the former approach is faster or slower depending on the operating system where you run the test. Furthermore, Java2D allows different layouts for the raster (PixelInterleavedSampleModel, SinglePixelPackedSampleModel, and so on). Some are faster, some are quicker, again it depends on the operating system and the differences can be wild.
    One of the basic best practices in Swing is to convert images into a "compatible image", that "has a layout and color model that is closest to this native device configuration and can therefore be optimally blitted to this device" (from the javadoc of GraphicsConfiguration). Mistral takes care of this thanks to the OptimizeOp() operation. The difference in manipulating the original image versus the optimized one can be tenth of seconds versus minutes. Here it is a table of test results (from Mistral v0.9.2.256 unit tests) - of course the tests have been run on the same hardware, a Mac Mini Core Duo with a triple boot, so results from the different operating systems are comparable (the higher the number the worst performance):

    Test Quality
        Mac OS X Linux Ubuntu   Win XP  
    J2DUtils.ScaleWithAffineTransform
    INTERMEDIATE 1214328 1169925 1236672
    J2DUtils.ScaleWithAffineTransform (opt) INTERMEDIATE 1859 FAILS  1921
    J2DUtils.ScaleWithDrawImage INTERMEDIATE 1844
    1044457  1095032
    J2DUtils.ScaleWithDrawImage (opt)
    INTERMEDIATE 3991 3230  2062
    ScaleJ2DOp INTERMEDIATE  1755 1161022 1220625
    ScaleJ2DOp (opt) INTERMEDIATE  1818 2950 1906
    OptimizeJ2DOp INTERMEDIATE  5146 4053 4062
    OptimizeJ2DOp (opt) INTERMEDIATE  1663 2940  1734
    RotateJ2DOp INTERMEDIATE 24805 4040127 4300516
    RotateJ2DOp (opt)
    INTERMEDIATE 57435 28843 18188

  • Profiling. Mistral automatically collects the execution timing for each operation and make them available for dumping in the log files, aggregated by operation type. This avoid cluttering the application code with explicit timestamping code.
  • Parallel / Distributed Processing. Mistral has also extensive support for distributed processing (including distributed profiling). I'm not going to talk about this feature now: you can read a previous introductory blog post. More information will be released as the implementation of this area is complete - at present time there's still some work to do about the distributed image cache - the important point is that the bitmaps are distributed transparently, behind the EditableImage scenes: so the same Mistral code, written following a few best practices, is able to run in very different parallel contexts. For instance, some preliminary prototypes have been able to run the very same Mistral code on multi-core computers, local clusters and the Sun Grid. If you are interested in this topic, track my blog or come to Jazoon  where Emmanuele Sordini (Mistral co-author) and I will have a speech (please note: the speech is about parallel processing design in general and will not be about Mistral on the whole, but about its distributed computing design patterns).

The Renderer

A central feature of any desktop imaging application is the image renderer: a component able to show images at various scales, quickly, possibly integrating with other tools visual cues (such as selection tools and so on). Thus it's no surprise that Mistral comes with its own image renderer able to work with EditableImages. The design of such a component is apparently easy: after all an image can be directly drawn over a Graphics object.

There are some issues indeed, that Mistral rendering component EditableImageRenderer deals with:
  • Performance. As we said before, scaling is one of those troubled operations, as its speed can dramatically change in different scenarios. There are also other potential problems: for instance, some performance bugs must be worked around (e.g.  it's extremely slow to resize images with certain non-sRGB color profiles). Furthermore, a professional image renderer should be also able to rotate an image and this is another source of performance troubles. Last but not least, there are some specialized Java2D image classes, such as VolatileImage, which offer a substantial performance boost and sometimes even hardware acceleration. EditableImageRenderer takes care of this and offers some approaches for trading off memory with speed, for instance keeping optimized copies of the image in memory.
  • Flexibility. Many image visualization components use scroll bars for dealing with very large images - after all scroll bars have been invented for this. Nevertheless, this approach is not necessarily the best for an imaging tool. Modern applications usually prefer to use the mouse drag to pan the image and mouse click to toggle between zoomed mode and 1:1 rendering. Also, zooming can be controlled by specialized buttons or by mouse wheel. EditableImageRenderer allows to implement all of these approaches by attaching specialized (and extendable) controllers. 
  • Previewing editing. Users want to preview in real time the editing changes they apply - for instance a contrast or a white point adjustment. For performance reasons, it's not feasible to apply these changes in real time to the whole image and redisplay it: the preview transformation - when possible - must be instead directly applied to the scaled image in the renderer. For this purpose a special PreviewSettings class can be applied to rendering.
  • Support for different imaging APIs. The actual rendering of the image is delegated to a PaintOp, which can be implemented in different ways by the underlying adapter as previously explained. For instance, the JAI adapter could take advantage of tiling for large images and take into memory only the portion of the image that should be shown. Other existing image rendering components usually require a preliminary conversion to a BufferedImage, which is not compatible with tiling.
The renderer has been designed in a modular way as all the classes that control the interaction with the user are external controllers. In this way, for instance, it has been possible to implement an AnimatedZoomController which applies a smooth animation when the scale is changed; the programmer can add his own as well.
Some JavaWebStart demos for the image renderer are available.

Status

Mistral, at present time, is in beta stage. The design requires only a few changes before the final version, most of the things described in this article are ready and  stable (there are just a few platform-specific performance issues). The two parts that now require extensive development are: operation coverage (only a subset of Java2D and JAI features are covered - up to now Mistral development has been driven by the needs of some projects that rely on it) and testing (there is basic automated testing, but it's not enough). Only adapters for Java2D and JAI exist; there's a non working skeleton for ImageJ. Some old workarounds, written at the old times of JDK 1.4, could probably be removed as the relevant JDK bugs could have been fixed (Mistral targets JDK 1.5).
Work is already scheduled for integrating metadata extraction capability (including native maker notes and XMP) - taking advantage of both the standard Java capabilities and external libraries. Mistral is able to work with "RAW format files" thanks to the Image I/O jrawio plugin.
While most of this article has been focused on desktop applications, Mistral can be used on the server side as well. Its various features are implemented in separated JARs, so you can include in your classpath only what you need and keep the footprint low.

Future plans

The evolution of Mistral is strictly related to the feedback we will have from the community (the current plans are constrained by the requirements of a few opensource and paid projects that rely upon it). The ideal target is to have Mistral to become something similar as CoreImage is for Mac OS X. At the moment there are two improvement areas we are thinking of:
  • providing a component for generation and maintainance of thumbnails;
  • creating an adapter for Mac OS X CoreImage Image Units, in order to take full advantage of Mac OS X imaging facilities.
The former activity is already scheduled (indeed there's already working code in blueMarine that just has to be moved and refactored); the latter would start only if programmers with specific CoreImage skills will join the project.

Feedback to this article can be posted to the Mistral forum.

Further reading