Performance of Java Compilers: An Empirical Study @ JDJ

How fast should our Java code be to be considered fast? After all, speed is a relative concept. I'll compare the results of CPU performance for the following JVMs: Sun's J2SE 1.4.1, 1.4.0, 1.3.1, and Jikes. These results can be used to make a number of educated decisions such as choosing a JVM, deciding on algorithmic designs, and selecting the right method from the API. They provide an overall assessment of performance that's not custom related since the code used is quite common and drawn directly from Sun's Java APIs.

This article studies the Java APIs for an extra boost in performance. It's not a new idea, and is often referred to as micro-benchmarking (MBM). However, a systematic and thorough performance analysis at that level is still missing. Herein, I'll address performance as speed, measured in wall clock time. I'll cover various Java Virtual Machines (JVMs), and show that the results differ significantly. A study of memory consumption is also warranted, but it will not be addressed here; for such an analysis, visit www.marmanis.com. See the Resources section for references to performance studies.

When dealing with performance, one of the major difficulties is the many "scales" or layers that are usually involved in Java applications, especially in enterprise Java applications. I categorize performance problems based on their scale:

1. System architecture
2. Algorithm selection
3. Code implementation
4. System configuration
5. System infrastructure

Only categories 1, 2, and 3 are directly related to the Java programming language. Problems in category 1 can be dealt with or, even better, prevented, by the proper use of J2EE Blueprints, the Java version of design patterns for enterprise applications (see http://java.sun.com/blueprints/enterprise and www.theserverside.com/patterns/index.jsp). Categories 4 and 5 involve handling and testing components that may be irrelevant to Java per se. These categories are given in order of decreasing importance. Experience shows that you will get most of your performance increase from improvements in categories 1 and 2, regardless of how you measure performance! Nevertheless, if you want to squeeze as much performance as possible out of your infrastructure, it's worth knowing what the performance of your fundamental APIs is. This part of performance is aptly called micro-performance and it belongs to category 3.

Micro-Performance
This article addresses the performance of Java applications with respect to the underlying JVM, and is based only on the standard API classes and algorithms that are implemented by them. We do this in order to establish results that are widespread in their applicability. No matter what code you write, you can decompose it into parts that can be studied as individual units. The performance of the whole is equal to the performance of its parts plus the overhead of the interaction between the parts. The simplest parts that you can decompose are the classes that are offered by the Java API. Thus, knowing how well these parts perform can be crucial to the overall performance of your product.