Tuesday, October 28, 2014

Want to see how your Karaf container is performing? Try the ktop command.

Aetos Integration Platform
I've been working on a utility command for Apache Karaf based containers that will display JVM usage metrics in a manner similar to the Linux Top command. The result has been the Aetos ktop command.

Aetos is Savoir Technologies' Integration Platform - essentially a custom stack of raw Apache projects that makes using Apache Karaf easy for developing and running production large scale enterprise solutions.
JVM vital statics in a Karaf console - awesome!

The Aetos ktop command provides a Top like display of vital JVM metrics, including:

  • Basic platform details.
  • JVM Uptime.
  • JVM Thread counts.
  • Garbage Collector Stats.
  • ClassLoader Stats.
  • JVM Memory Stats, and
  • Periodically updated top threads by CPU usage. 

The ktop command allows users to specify how many threads they'd like displayed, and the information update interval. Use the --help option to read the command's usage information.

Sounds cool, I'd like to try it out on my Karaf system!

We've published a MileStone 1 release to Maven Central:

Source tag link:

If you're using an Apache Karaf 3.0.x based system (such as OpenDaylight Helium or Aetos 3.0.2), you can install MileStone 1 using the following Karaf console command:

install -s mvn:com.savoirtech.karaf.commands/ktop/0.1.0.M1

Feedback is welcome! Please submit any ideas, enhancements, bugs to the project issue tracker:

Nice! I'd like to have the command on a different Karaf platform...

There are three branches of the Aetos ktop project at this time, tracking Apache Karaf 2.3.x,  3.0.x, and 4.0.x lines. At current, users will have to clone the source repo, checkout the branch appropriate to their Karaf deployment, and build the code locally before installing in their container. Please use the following instructions to get ktop running on your container:

How do I get the code?

git clone https://github.com/savoirtech/ktop.git
cd ktop
git checkout k23x | k30x | master

The k23x branch is maintained for Karaf 2.3.x format, k30x for Karaf 3.0.x format, and master will track Karaf 4 style commands.

How do I build and install ktop into my container?

To build, invoke:

mvn install

To install in Karaf, invoke from Karaf console:

install -s mvn:com.savoirtech.karaf.commands/ktop/version-SNAPSHOT

How do I use ktop once installed?

To execute command on Karaf, invoke:


To exit ktop, press control + c

The code is under the GNU license at the moment, as per a project requirement -- we will have it under the ASL in the future. In the mean time if you're looking for a way to monitor your Karaf container from your console window, give Aetos ktop a try!

Wednesday, October 15, 2014

OpenDaylight Helium on Raspberry Pi first performance benchmark results!

Last week I posted about getting OpenDaylight Helium release to run on my Raspberry Pi. This week I've spent some time setting up the platform for some simple benchmarks.

The TL; DR version is that I was able to observe approximately 170 flows/second! **

Now for the longer story...

To benchmark the system three things had to come together: Helium running on Karaf, the wcbench utility running on a separate host, and the overall platform being stable enough to execute enough tests to produce statistically relevant data sets.

In my previous post, I outlined how I was able to tweak Helium into running on RPi. To these alterations I added one more tuning parameter: Xss200k. This reduces the startup size of Java threads, making it easier on the JVM and RPi to provide resources. See my github repo for sample Helium scripts.

Setting up the wcbench tool required standing up a host machine, in my case a Fedora Core 20 VM, and configuring its scripts to point to the RPi. Given the RPi's limit resources I tuned the number of switches, mac addresses, and time to run each test down. See my github repo for sample wcbench configuration.
Ready for testing!
Using the test tool would also require installing a few packages onto the Helium deployment. Through trial and error I discovered it was easiest on the RPi if I installed as many smaller features first before attempting top level targets. See 'Setting up Helium for testing' on my github page main read me file.

Unfortunately this initial setup resulted in a failure to collect useful output :(

The trouble was that even though OpenDaylight Helium could boot up and install packages, it couldn't handle the heavy traffic. The system would freeze up, becoming unresponsive. Memory was completely consumed.

Looking at the base system's 700MHz CPU, and memory specs, something would need to change:
This is where the above '**' comes into play. Up to now I've been using a relatively default vanilla installation of Rasbian on the RPi. To obtain useful benchmark results I would have to provide OpenDaylight with more resources to do its thing. Hence I proceeded to the net, and found performance tweaks for RPi. See my github repo for a full list of the configuration changes I made to my RPi.

The final result was a base system with a 850MHz CPU and the following memory specs:
An extra 40MB free ram, and 255MB of swap space!
With my platform tweaks applied, the system booted up much quicker. I executed the loop_wcbench script with the -l and -t5 option for 25 data sets. See results.csv for full results. Unfortunately, the scripts were not able to dial into the instance to grab detailed system information, so I made the adjacent screen capture of the top command during the test.

After running the benchmark tool with 10 simulated switches and 10 MAC addresses, I attempted increasing the load to 100 switches and 100 MAC addresses - the system promptly froze.

There are probably more system tweaks that could be applied to the RPi to free up more memory, and  additional JVM tuning to make operating Helium more performant. I've published all of my tunings, and configurations to a git repo so that others may use my experiments as a starting point.

If you do decide to dive in and try out Helium on RPi, please let us all know about your results in the comments below.

Wednesday, October 8, 2014

OpenDaylight Helium Release on Raspberry Pi!

The question we've all been asking has finally been answered - yes, OpenDaylight Helium Release can run on Raspberry Pi!

If you're not familiar with Raspberry Pi it's a small, inexpensive computer which at its heart runs a 700 MHz Low Power ARM1176JZ-F Applications ProcessorCPU with 512MB ram.

My build utilizes a Debian based Linux distro, Raspbian, and an Arm compatible Oracle JDK build.

Out of the box I had to make three small modifications to the scripts found in the OpenDaylight distributions' bin folder.

  • The set environment script (setenv) had JVM parameters for PermGen and Max Memory set higher than the RPi could support. I set these values to 340MB for PermGen, and 400MB for Max Memory. These are not optimal settings, but will work well enough to get things going.
  • The karaf script used to start the container also required a memory setting to be adjusted from 512m down to 340m (just grep the script code for 512 and replace). 
  • Finally, in the karaf script the JVM "server" flag was not recognized by the Java installation on my RPi. Editing this out of the script allowed Karaf to start up happily.

Once I had the OpenDaylight console banner displayed, I executed the console info command to see system environment information:
Now we're cooking with fire! Time to start making some toast...

Awesome! The Apache Karaf container environment OpenDaylight Helium uses appears to be stable. Next step was to try to deploy the OpenDaylight Toaster demo... which failed on not being able to find SSHD 0.12.0 :(

This turned out to be a minor inconvenience. A simple invocation of install -s mvn:org.apache.sshd/apache-sshd/0.12.0, and an edit to ODL_HOME/etc/config.properties to include a JDK 1.8 entry for osgi.ee and the ODL Toaster was installable, and running!
Helium on RPi can make toast!
I used jconsole to manipulate the makeToast and clearToastsMade operations exposed via JMX on the controller to verify their correct operation - using log:display I could view each operation's logging events.

RPi CPU Stats
This is an very impressive improvement over the previous OpenDaylight release which, despite my best efforts, I could not get to boot on RPi to a stable console out of the box. It's amazing how well OpenDaylight has taken to scaling from small PCs like the RPi to serious big iron servers we find in datacenters.

Its important to note that for my ODL on RPi test I've only installed the base ODL Helium distribution, SSHD 0.12.0, and performed a feature install on odl-toaster target. The Karaf based features mechanism took care of downloading, installing, and configuring all the plumbing required to get the demo running.

So, now that we know we can get OpenDaylight Helium running on an RPi, what should we program it to do? I'm looking forward to reading your ideas in the comments below :)

Wednesday, September 24, 2014

Apache Karaf Cookbook Launch Party!

Last night I held a book launch event for Apache Karaf Cookbook. I wish my co-authors would have been able to be onsite for the event, however travel and other commitments made that impractical.

The night consisted of a short speech on the release of our latest book, then transitioned into two hours of guided Wine, Oils, and Vins tastings. So much fun!

I've collected below some photos from the evening. I'd like to thank the staff of O'Leva Oils & Vins Tasting Room for their excellent service and presentations.

O'Leva Oils & Vins Tasting Room (Venue):

Our Books:

Launch Event:

Garlic Crostinis with Brie and Garlic oil and Fig Balsamic.
Beef meatballs with Basil olive oil bruschetta.
Lettuce wraps with Walnut Oil and Chocolate Balsamic.
Vanilla Bean ice cream with Blood Orange olive oil and Espresso Balsamic.
Mingling time in between wine tastings.

Guided tasting of Wine, Oils, and Vins 
Campo Viejo, Toasted Head, and Lucky Stones.
Red Esc and a multitude of Oils & Vins samplers.

I'd like to thank all of my guests for making it out for the night, and sharing in the excitement of having the Apache Karaf Cookbook reach publication.

Tuesday, September 23, 2014

Please vote for our talks to be accepted to ConFoo 2015

Heath Kesler and I have submitted talks to ConFoo 2015 for consideration, and we need your votes to help make our proposals get noticed!

ConFoo. February 18 - February 20, 2015 | Montreal, CanadaEasy RESTful services using Apache Camel and CXF

"In this session attendees will learn how simple RESTful services are to implement and interact with using Camel Integration Engine and the CXF Framework. In as little as a few lines of code, a developer can implement a fully functioning, testable CXFRS endpoint. Learn how to implement thread pooling and asynchronous processing on the service while maintaining a flexible and scalable architecture that can grow with demand."

ConFoo. February 18 - February 20, 2015 | Montreal, CanadaHow OpenDaylight learned to breathe in Apache Karaf.

How OpenDaylight learned to breathe in Apache Karaf. OpenDaylight is an open platform for network programming to enable SDN and NFV. Apache Karaf is a small, light weight server side OSGi runtime environment. In this session I will share our experiences in introducing OpenDaylight to Apache Karaf, and the subsequent large scale, multi-project migration to Apache Karaf’s OSGi environment.

ConFoo. February 18 - February 20, 2015 | Montreal, CanadaApache Karaf in the Trenches

Apache Karaf is a small, light weight server side OSGi runtime environment, that has been gaining in popularity in the Open Source community. The shift towards using OSGi has introduced new challenges to architects, developers, and operators in building and deploying these systems. In this session I will share some of my experiences in developing, deploying, and maintaining Karaf based software systems among Global 2000 companies.

Please register and vote on the ConFoo 2015 website.

Sunday, September 21, 2014

Apache Karaf 2.3.8 and 2.4.0 Released!

The Apache Karaf team are pleased to announce the availability of Apache Karaf 2.3.8 and 2.4.0!

Apache Karaf 2.3.8 is an update patch for Apache Karaf 2.3.7, containing several bug fixes, and dependency updates. We have introduced support of !$ to recall the last command argument, and improved the command description for osgi:shutdown. It is recommended that you review our release notes before updating your deployments.

Apache Karaf 2.4.0 release represents a new branch of Apache Karaf development. It includes the latest Aries and Pax libraries, Apache Pom 14, add support for role-based JMX authorization, includes heap dumps when we create debugging data from create dump command, blueprint-web feature, support for JAAS groups, and includes Pax CDI feature. There are several hundred items currently in this RC's release notes, so you'll want to spend some time discovering all the hidden gems.

As always, I'd like to thank the community for coming together to help make the Karaf release process one of the best in all of open source software.

Sunday, September 14, 2014

New Apache Karaf book and up coming release candidates

Apache Karaf Cookbook is now available from Packt Publishing!

This past winter Johan, Heath, Achim, and I started our latest adventure writing Apache Karaf Cookbook for Packt Publishing. We're happy to announce that the book is finally available for order via Packt's website.
We've taken care to cram as many hints, tips, and experiences into its pages as we could.

Taking a look at our table of contents you'll find recipes for:

  • Apache Karaf for system builders
  • Turning Karaf into a Smart Router with Apache Camel
  • Model Karaf into a Message Broker with Apache ActiveMQ
  • Transform Karaf into a JSP host with Pax-web
  • Distribute your Karaf applications across a clustered container with Apache Karaf Cellar
  • Transmute Karaf into a web service container with Apache CXF
  • Turbo-charge your Karaf applications’ persistence layer scalability with Apache Cassandra
  • Manage Big Data on Karaf with Apache Hadoop
  • Integration test with Pax Exam (bonus chapter!)

We hope the community benefits from our experiences, we've poured a lot of ourselves into this title. For more publications on Karaf please see our Instant OSGi Starter, and Learning Apache Karaf.

Up coming release candidates - Apache Karaf 2.3.8, 2.4.0, and 3.0.2!

Apache Karaf 2.3.8:

This patch update will contain an essential update to Felix FileInstall, we hope to have this patch available for vote ASAP.

Apache Karaf 2.4.0:

This release candidate represents a new branch of Apache Karaf development. It shall include the latest Aries and Pax libraries, Apache Pom 13, add support for role-based JMX authorization, includes heap dumps when we create debugging data from create dump command, blueprint-web feature, support for JAAS groups, and includes Pax CDI feature. There are several hundred items currently in this RC's release notes, so you'll want to spend some time discovering all the hidden gems.

Apache Karaf 3.0.2:

Our second patch update to the 3.0 line, this release candidate will contain vital bug fixes, and essential library updates.