Testing Parallel Junit testing in Karaf

Recently a new revision of surefire-maven-plugin was released with the new feature of being able to take advantage of JUnit 4.7 parallel testing capabilities by simply configuring the plugin to enable the parallel tests. Seeing that we use the surefire plugin in Karaf I decided to take a quick look to see if we could benefit from this feature.

Test platform: Dell T5400, 2 * Xeon E5410 @ 2.33Ghz, 4 GB Ram,
SCSi Disk. Ubuntu 8.04 (2.6.24-28-generic, amd64),
Maven 2.0.11 and JDK 1.5.0_16. Karaf source version was apache-karaf-1.99.0-SNAPSHOT, rev 961352 (trunk).

Pre-conditions: An initial build of Karaf was performed on the system to eliminate need for maven to download artifacts.

Test: Execute "mvn verify" and record successful build report. For each test case I'll alter the parallel build configuration.

Before attempting to gather metrics on the new plugin's performance with Karaf, I tested the current surefire plugin.

Surefire version 2.3.1

[INFO] BUILD SUCCESSFUL
[INFO] -----------------------------------------
[INFO] Total time: 3 minutes 57 seconds
[INFO] Finished at: Wed Jul 07 13:38:05 NDT 2010
[INFO] Final Memory: 95M/1022M
[INFO] -----------------------------------------


Surefire version 2.5

No Parallel Confguration

[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------
[INFO] Total time: 4 minutes 16 seconds
[INFO] Finished at: Wed Jul 07 13:47:49 NDT 2010
[INFO] Final Memory: 103M/988M
[INFO] ------------------------------------------

Parallel Classes
Thread Count == Default

Erratic test failures or hangs :(

Parallel Methods
Thread Count == Default

[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------
[INFO] Total time: 4 minutes 10 seconds
[INFO] Finished at: Wed Jul 07 13:43:01 NDT 2010
[INFO] Final Memory: 98M/976M
[INFO] ------------------------------------------

Parallel Methods
Thread Count == 1

[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------
[INFO] Total time: 4 minutes 17 seconds
[INFO] Finished at: Wed Jul 07 13:54:15 NDT 2010
[INFO] Final Memory: 102M/1020M
[INFO] ------------------------------------------

Parallel Methods
Thread Count == 2

[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------
[INFO] Total time: 4 minutes 32 seconds
[INFO] Finished at: Wed Jul 07 14:03:39 NDT 2010
[INFO] Final Memory: 98M/966M
[INFO] ------------------------------------------
 
Parallel Methods
Thread Count == 4

[INFO] BUILD SUCCESSFUL
[INFO] -----------------------------------------
[INFO] Total time: 4 minutes 37 seconds
[INFO] Finished at: Wed Jul 07 14:09:22 NDT 2010
[INFO] Final Memory: 99M/1025M
[INFO] -----------------------------------------

Parallel Methods
Thread Count == 8 (equal to number of real cores on test system)

[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------
[INFO] Total time: 4 minutes 35 seconds
[INFO] Finished at: Wed Jul 07 14:14:28 NDT 2010
[INFO] Final Memory: 96M/1077M
[INFO] ------------------------------------------

Parallel Methods
Thread Count == Unlimited

[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------
[INFO] Total time: 3 minutes 43 seconds
[INFO] Finished at: Wed Jul 07 13:32:50 NDT 2010
[INFO] Final Memory: 99M/1054M
[INFO] ------------------------------------------



* Note: It took several runs to have a clean test result with unlimited threads.

Results: Only with unlimited threads enable did I observe an increase in performance of 14 seconds. The failures and hangs observed when using class parallelism could be a target for improvement. Considering these results I'm not inclined to enable parallel tests in Karaf at this time, we will have to revisit this issue on a later revision.