Apache Karaf 2.2.5 Garbage Collection with IBM Java 6 on AIX 7.1

I've been experimenting with the Garbage Collection (GC) policies in IBM Java 6 on AIX 7.1 while using Apache Karaf 2.2.5, and thought I'd share some tips and links for anyone else following in my footsteps.

If you're wondering why one would play with the GC policy of the underlying jvm Karaf runs upon it stems from wanting to get a better idea of the throughput of applications and the overall system, and the pause times that are caused by garbage collection (wondering what "pause time" is? It's the time spent by the jvm cleaning up, or freeing, heap memory when an object can't be created using the available heap memory).

The version of Java I was testing has three options for garbage collection; gencon, optthruput, and subpool. Gencon is a combination of concurrent and generational GC, it's goal is to minimize the amount of time spent in any GC pause. Optthruput is the default GC policy, and is designed to deliver high through put at the cost of occasional GC pauses. Subpool is aimed at large SMP systems, and attempts to deliver better performance on those systems. Unfortunately the system I have access too is too small to try out subpool with any meaningful results, as such I kept to gencon and optthruput policies.

To setup Karaf to use specified garbage collection policies edit the KARAF_HOME/bin/karaf file as follows:

# grep for the setup defaults routine:
setupDefaults(){

# in the aix section try adding the highlighted text:
DEFAULT_JAVA_OPTS="-verbose:gc 
                   -Xverbosegclog:$KARAF_DATA/gc-log.txt 
                   -Xgcpolicy:gencon 
                   -Xverify:none -Xlp $DEFAULT_JAVA_OPTS"

When you start Karaf using the above Java options you will enable verbose garbage collection logging, sending generated data to KARAF_DATA/gc-log.txt. You can use this log file to review GC statistics as you tweak jvm options. The -Xgcpolicy flag is where you select the GC policy. When you change this value do not be surprised if you do not see this changed reflected in the Karaf info command's memory section. Use the generated verbose GC logs to perform your analysis.

Trying out gencon and the default optthruput policies I can see very different runtime behaviors in regards to frequency of GC and durations. Gencon ran more often, but for short periods of time, while optthruput ran infrequently but took longer when called. Given different application loads the characterization of GC runs could become important in providing optimal performance, the determination of "optimal" however is very subjective and will vary depending on the goals of your deployment.

Of course, changing the GC policy isn't the end of the story. There are a lot of parameters that can be tweaked which affect the size of the jvm heap in use, and the thresholds different GC policies use. I would recommend when testing to tweak one value at a time, then assessing it's impact. The online documentation (see below) will steer you in the right direction.

IBM JDK 6 Garbage Collection Links:

Overview of Java 6 Memory Management. Read this to learn about the garbage collector and allocator.

Detailed description of garbage collection. Dive deep into garbage collection.

Garbage Collector diagnostics. How to diagnose garbage collection. A must read.

Specifying garbage collection policy. Part of the running Java applications section in the IBM User Guide for Java v6 on 64-bit AIX.

Garbage Collector command-line options. List of all the garbage collector options (here there be dragons, careful tweaking these values).

Verbose garbage collection logging. Examples of how to analyze garbage collection logs.