A Brief History of Apache Karaf

Everyone once in a while a question comes up in discussion of Apache Karaf regarding its origins. Well to help answer this question I've written up this brief history of Karaf so that I may reference it.

In the beginning...

Apache Karaf can trace it's origins back to the Apache Servicemix project's Kernel. As early as 2008 the first blog posts regarding its development can be observed. The first GA release was on September 18th, 2008. The original release notes can be viewed here. Subsequent releases soon followed, introducing new features.

The Servicemix Kernel was off to a great start, eventually however by spring 2009 the project grew to the point that it was thought best to leave Servicemix and join it to the Apache Felix community. It was felt that this would allow awareness to grow about the project, and to expand the Kernel community. At this time the Servicemix Kernel project was renamed to "Karaf". For those of you interested about how the project became called 'Karaf' here is the relevant excerpt from the discussion:

"A carafe is a small container used for serving wine and other drinks (http://en.wikipedia.org/wiki/Carafe). In similarity to the name the Kernel allows applications to be more easily handled, and improves their characteristics (much like a bottle of wine left to breath in a decanter) :)"

So why the respelling? It was felt that 'Karaf' would be an easier search target, and it would fill the relatively empty 'K' project name listings at Apache ;)

Once established in the Apache Felix community Karaf again started to grow and mature, once again coming to the point that it was felt that another growth spurt was required - becoming a top level Apache project. This occurred mid 2010. Since then Apache Karaf has released it's first GA as 2.0, and several point releases and patches. The community is now busy readying the next major point release, and preparing for its future 3.0.

My Guest Lecture experience at Memorial University

I just got back from providing my first guest lecture to an under graduate class of Computer Science 3716 Software Methodologies at Memorial University. It was a fun experience, which I am supposed to re-iterate next semester (Winter 2011).

Memorial University Engineering Building

My talk touched on various subjects including a quick Apache Way introduction, a brief description of the projects I've worked on, my views on technology, and project management. We attempted to keep the talk focused towards entry level development in St John's, but did expand things to include working on the international level in large scale projects. Particular empathises was given to working on open source projects such as Apache Servicemix, Felix, and Karaf.

Unfortunately there is never enough time to dive as deeply into technology issues as I'd like there to be, but I hope that the students enjoyed the talk and took away something useful :)

Working from home - experiments

I'd like to draw some attention to a great Apache Karaf post on Ade Trenaman's blog. If you have any interest in Karaf or Servicemix then I'd suggest reading it :)

As this post was intended for talking about experiments, I wanted to post a quick pic of a pair of Dell 1850 1U servers I picked up for trying out new bits of code in Karaf/Servicemix. Fun times driving around town with a pair of 1Us in the trunk of my car - thanks Jon for lending hand saving these machines from the dump.

The following pics are for the benefit of my team mates that ask from time to time what I'm making for lunch - here was today's kitchen experiment:

The dish is composed of Tandoori Chicken, Madras Chick Peas & Lentils, and Buttered Chicken Basmati rice served atop fresh Na'an bread.

Apache Camel podcast on The Basement Coders

The Basement Coders recently posted a podcast on Apache Camel. The talk is a pretty good introduction to the concepts behind Camel and is worth listening to if your just becoming familiar with the project. They also cover some of the reasons why it's a good idea to use Camel in Servicemix :)

Karaf is now a top level Apache project!

It was barely a little more than a year ago that I was announcing that the Servicemix Kernel sub-project had moved to Felix and renamed to "Karaf". Time has moved on and development has been proceeding at an amazing pace, this growth has been so pronounced that we have requested to have Karaf recognized as its own top level project - this request was successfully granted :)

Now that the process has begun to establish Apache Karaf as a top level project, I thought it would be good to start publishing a few links so that Karaf developers & users could find its new home.

Karaf home page:
http://karaf.apache.org/

Karaf issue tracker:
https://issues.apache.org/jira/browse/KARAF

Karaf source code (anonymous check out from Karaf trunk):
http://svn.apache.org/repos/asf/karaf/trunk/

Karaf hudson build status:
http://hudson.zones.apache.org/hudson/job/Karaf/

Karaf mailing lists:
http://karaf.apache.org/mailing-lists.html

Karaf contributing guide:
http://karaf.apache.org/contributing.html

As a small personal note, with the establishment of Karaf as a top level project I have been included as a committer and the Karaf PMC :) 

Apache ServiceMix 4.2.0 Released!

Hi All,

Just wanted to pass along the official announcement from the ServiceMix team:

The ServiceMix team is pleased to announce the availability of ServiceMix 4.2.0. This is a major release of the OSGi based ServiceMix 4.2.0 and contains many bug fixes as well as new features. For more information, please see the release notes.

Setting up SMX4 failover with Oracle

In the tradition of posting articles regarding Servicemix 4 failover here is yet another blog entry. The functionality described below is currently available as a patch to Karaf, a future release should include this feature.

The following system.properties configuration will allow your Karaf based SMX4 instance to use an Oracle JDBC lock:

karaf.lock=true
karaf.lock.class=org.apache.felix.karaf.main.OracleJDBCLock
karaf.lock.jdbc.url=jdbc:oracle:thin:@hostname:1521:XE
karaf.lock.jdbc.driver=oracle.jdbc.OracleDriver
karaf.lock.jdbc.user=user
karaf.lock.jdbc.password=password
karaf.lock.jdbc.table=KARAF_LOCK
karaf.lock.jdbc.clustername=karaf
karaf.lock.jdbc.timeout=30

Please note that the Oracle JDBC driver will have to be added to your classpath, this can be easily done by coping the Oracle JDBC jar into the SMX/lib folder. Also note that the karaf.lock.jdbc.url requires an active SID, this means you must manually create a database instance before using this particular lock.

Setting up SMX4 failover with MySQL

This post is an update to my previous post on setting up SMX4 with high availability in mind.

Since the transition to Apache Felix Karaf the support for more database locking table back ends has expanded. Specifically you can now use Apache Derby, MySQL, or a generic JDBC connector. The Derby and Generic JDBC locks make use of the 'FOR UPDATE' feature common to many databases. The MySQL lock however makes use of the ' LOCK TABLES' directive, this is due to the 'FOR UPDATE' command not returning a ResultSet on more recent versions of MySQL (see some discussions on this here).

The following system.properties configuration will allow your Karaf based SMX4 instance to use a MySQL JDBC lock:

karaf.lock=true
karaf.lock.level=20
karaf.lock.delay=1000
karaf.lock.class=org.apache.felix.karaf.main.MySQLJDBCLock
karaf.lock.jdbc.url=jdbc:mysql://address:port/dbname
karaf.lock.jdbc.driver=com.mysql.jdbc.Driver
karaf.lock.jdbc.user=root
karaf.lock.jdbc.password=karaf
karaf.lock.jdbc.table=KARAF_LOCK
karaf.lock.jdbc.clustername=mycluster
karaf.lock.jdbc.timeout=30

Please note that the MySQL JDBC driver will have to be made available on the classpath.

Getting to know VisualVM

Recently I had to refactor a feature that I had introduced to the old Servicemix Kernel, now Felix Karaf project. The issue was relatively minor to resolve, however testing it would require profiling memory and CPU usage over a period of time (to avoid an eventual out of memory error).

There are multiple tools out there that would allow me to do this sort of monitoring, for this particular task I chose to try out VisualVM as it was already bundled with my JDK 1.6 install. VisualVM is a visual tool that integrates several commandline JDK tools and lightweight profiling capabilities for the Java SE platform.

To begin my refactoring task I first setup Karaf with the configuration required to reproduce the out of memory condition. No special configuration or instrumentation of Karaf itself was required for using the profiler. Starting Karaf, using JDK 1.6, I was able to attach to the running process via the VisualVM interface. Once connected to the process I could view the live heap usage, threading, permgen, and loaded classes, further I could inspect the parameters fed to the JVM at start up to confirm the environment was as I configured. To view memory and CPU usage of Karaf, I selected the profiler display tab. Adjusting the profiler settings to only monitor org.apache.felix classes, I could quickly identify the offending set of method calls that contained the source of my error (high CPU usage, and multiple instances accumulating in memory).

VisualVM displaying monitor view of Felix Karaf

Resolving the error didn't require much work except to ensure a few resources were being properly freed, and removal of an unneeded for-loop. To test out my modifications I redeployed Karaf, and began reviewing its run time characteristics via VisualVM. This time the trouble methods were not being called as often, and not creating memory leaks, while still providing the same functionality the original code was intended to provide.

As a simple profiler I can recommend VisualVM as a quick, convenient way to observe your Java applications. It will be interesting to see what features they integrate into the tool in future releases.

Apache Servicemix Kernel subproject moves to Apache Felix

So the big news in the Apache Servicemix project, after its recent 4.0 release, is the move of the Kernel subproject over to Apache Felix. This change came about for several reasons which were discussed on Guillaume Nodet's blog. After some discussion on the mail lists, the name "Karaf" was accepted for the subproject. As annouced by Guillaume, the vote to move the subproject was successful and the work to make the move began.

Now that the process has begun to establish Apache Felix Karaf, I thought it would be good to start publishing a few links so that Kernel developers & users could find its new home.

Karaf Overview:
http://felix.apache.org/site/apache-felix-karaf.html

Karaf Users' Guide:
http://felix.apache.org/site/karaf-users-guide.html

Karaf issue tracker (see Karaf component):
http://issues.apache.org/jira/browse/Felix

Karaf source code (anonymous check out from Felix trunk):
http://svn.apache.org/repos/asf/felix/trunk/karaf/

Felix project contributing guide:
http://felix.apache.org/site/contributing.html

FUSE ESB 4.1 Released!

FUSE ESB 4.1 is now available at FUSESource.

FUSE ESB 4 is an enterprise version of Apache ServiceMix 4 – the most popular standards-based open source ESB. FUSE ESB 4 continues to provide support for widely adopted integration standards like JBI 1.0 and JMS while also ensuring support for the latest emerging standards like OSGi. The new FUSE ESB 4 provides a single platform that makes it easy for developers to implement the integration patterns they need with the programming model of their choice

Get your copy here:
http://fusesource.com/products/enterprise-servicemix4/

Release Notes:
http://fusesource.com/wiki/display/ProdInfo/FUSE+ESB+v4.1+Release+Notes

Apache ServiceMix 4.0.0 Released!

I'm pleased to echo the following ServiceMix announcement:

"The Apache ServiceMix team is pleased to annouce the release of Apache ServiceMix 4.0.

Download links and detailed release notes are available at
http://servicemix.apache.org/smx4/servicemix-400.html

Apache ServiceMix 4.0 is the first release of our OSGi based integration platform. It includes two major components:

• Apache ServiceMix Kernel 1.1.0 : an OSGi runtime with a lot of extra features (SSH connectivity,provisioning enhancements, Spring integration, ...)
• Apache ServiceMix NMR 1.0.0 : an OSGi based NMR and JBI container, which also comes with a new clustering engine ready for enterprise deployment

In addition, ServiceMix 4.0 also ships with enhanced ActiveMQ, Camel and CXF integration as well as a whole set of examples to let you leverage this functionality. We also have out-of-the-box support for deploying and running web applications, so they can run together with everything else in the same container."

Apache ServiceMix NMR 1.0.0 Released!

Hi All, Just wanted to echo the ServiceMix NMR 1.0.0 Release announcement.

"The Apache ServiceMix team is pleased to announce the release of Apache ServiceMix NMR 1.0.0.

Download links and detailed release notes are available at
http://servicemix.apache.org/nmr/servicemix-nmr-100.html

Apache ServiceMix NMR 1.0.0 is the first release of our JBI container based on OSGi. This distribution is a minimal JBI container, if you are looking for a more features distribution, we'll welcome you to have a look at Apache ServiceMix 4.0.0 instead, which includes all the JBI components provided by ServiceMix."

Once you've downloaded the new NMR kit be sure to check out the new examples. Expect to read more about them in a future post.

Apache ServiceMix Kernel 1.1.0 Released!

The Apache ServiceMix Team is pleased to announce the release of Apache ServiceMix Kernel 1.1.0.

The Apache ServiceMix Kernel 1.1.0 release brings a lot of new features, enhancements, and bug fixes:

• Remote connection using SSH protocol.
• Provisioning enhancements: versioning / hot deployment of features.
• New commands, including OSGi related commands for the Configuration Admin and Package Admin services.
• Improved spring integration: upgrade to spring 2.5.6 and spring-dm 1.2.0-m2, the osgi/list command now displays spring applications status.
• Container level locking for master / slave deployments.
• Support for JAXP 1.4 on all platforms.
• Improved JMX support for managing the OSGi framework and features.

Note that the commands syntax has changed due to the upgrade to thelatest gshell version.

This release, with the detailed release notes, is available at:
http://servicemix.apache.org/kernel/servicemix-kernel-110.html

I'm an Apache Servicemix committer!

I first started using Servicemix in the fall of 2006 while it was still in Apache Incubator. My first few patches began appearing soon after on the project issue tracker in the winter of 2007. I submitted these patches initially just for fun and continued submitting the patches while I worked on other projects. As of the summer of 2008 I had the opportunity to make Servicemix my main focus, and its been a fun ride ever since.

This is my first committer status on an Apache project so this will be a new learning experience. I have been a long time supporter of open source and look forward to the adventures ahead.

Cheers!

Adding container level locking to your H.A. Servicemix 4 deployment.

As an improvement to the High Availability features being implemented in the latest versions of the Servicemix 4 Kernel the concept of container level locking has been introduced.

The Container Level locking mechanism allows bundles to be loaded into slave kernel instances in order to provide faster failover performance (when a slave instance becomes the master it will have fewer bundles to load before starting operation). The Container Level refers to the starting priority assigned to each bundle in the OSGI container. These start levels are specified in $SERVICEMIX_HOME/etc/startup.properties, in the format jar.name=level. The core system bundles have levels below 50, where as user bundles have levels greater than 50.

Level: 1 Behavior: A 'cold' standby instance. Core bundles are not loaded into container. Slaves will wait until lock acquired to start server.

Level: <50 Behavior: A 'hot' standby instance. Core bundles are loaded into the container. Slaves will wait until lock acquired to start user level bundles. The console will be accessible for each slave instance at this level.

Level: >50 Behavior: This setting is Not recommended as user bundles will be started.

Container Level locking is supported in both currently supported failover deployment schemes;

• Simple Lock File
• JDBC Locking

To make use of this capability the following must be set on each system in the master/slave setup:

• $SERVICEMIX_HOME/etc/system.properties file updated to include the below entries in addition to other configuration entries to enable H.A.
  

servicemix.lock.level=50
servicemix.lock.delay=10

For more information on how to use the High Availability features supported with Servicemix 4 please visit the users guide.

Setting up Servicemix 4 with H.A. in mind.

Looking to setup Servicemix 4 with High Availability in mind? Then follow this guide to setting up a JDBC Master Slave deployment (Requires Servicemix 4 Kernel 1.1.0-SNAPSHOT or higher).

What is a JDBC Master Slave deployment?

A JDBC Master Slave deployment is a collection of Servicemix instances configured to act as one logical instance (which I refer to as a cluster). Under this High Availability model one instance node provides services (Master) while all other instance nodes (Slaves) wait by the side lines for the node to release its JDBC lock. When the JDBC lock is not held by any node then the first node to obtain the lock becomes active and will fully start its instance of Servicemix.

In this deployment scenario the JDBC connection to a locking table hosted on a database server becomes a point of failure to the cluster, as such it is highly recommended that the database also be provided in a highly available manner. A lock monitor is implement in Servicemix to ensure that as a master node the loss of its lock will force a graceful shutdown of the node.

The above figure depicts three servers each hosting an instance of servicemix in a JDBC Master Slave setup. The DB server hosts an Apache Derby database instance. The instance labeled "Master" has obtained the lock on the database table, the Servicemix instances on servers B and C are left waiting to obtain the lock.

What happens during the initial start of the cluster?

When Servicemix nodes are started they will read the Servicemix lock configuration and attempt to connect to the specified database. The first cluster node to establish connection to the locking table will become the master, the remaining instances will wait, retrying their connection periodically.

What happens when a master fails?

In the event that the master node experiences a failure it will release its hold on the locking table, this will allow awaiting slave nodes to attempt access to the lock. Once the master node has failed it will require a manual restart, upon which it will join the slave node pool.

What if a database failure occurs?

In the event of database failure the current master node will detect the loss of connection and shutdown. The slave nodes will continue to await connection to the database. The former master node will require manual restart. This feature was developed to ensure that a temporary loss of lock would not allow two master nodes to operate at one time.

What happens when an instance restarts?

When a node is started as part of a running cluster it will attempt to make connection with the database to access the lock, failing to obtain the lock it will retry periodically.

How to configure Servicemix 4 JDBC Master Slave:

To setup the JDBC Master Slave deployment you will need to configure the following on each node to be included in the cluster:

• Classpath updated to include JDBC driver.
• $SERVICEMIX_HOME/etc/system.properties file updated to reflect the below entries.

servicemix.lock=true
servicemix.lock.class=org.apache.servicemix.kernel.main.DefaultJDBCLock
servicemix.lock.jdbc.url=jdbc:derby://dbserver:1527/sample
servicemix.lock.jdbc.driver=org.apache.derby.jdbc.ClientDriver
servicemix.lock.jdbc.user=user
servicemix.lock.jdbc.password=password
servicemix.lock.jdbc.table=SERVICEMIX_LOCK
servicemix.lock.jdbc.clustername=smx4
servicemix.lock.jdbc.timeout=30

Note:

• Will fail if jdbc driver is not on classpath.
• The database name "sample" will be created if it does not exist on the db.
• First SMX4 instance to acquire the locking table is the master instance.
• If connection to the DB is lost then the master instance will attempt to gracefully shutdown, allowing a slave instance to become master when the DB service is restored. The former master will require manual restart.

Happy New Year!

Happy New Year!

This past year has been very busy, and 2009 looks like it will be more so. I'm going to continue posting about AIX development issues however I'm going to focus more upon running ServiceMix on the AIX platform. You can also expect to see more Mac related posts and my observations on open source.

Cheers,

MiR: November 2008

This month my routine has been changed up a little by refocusing upon ServiceMix 3 issues and internal infrastructure projects (more CPI work, again this always benefits ServiceMix as it frees up developer time to focus on features and improvements).

On a completely different topic, this past month has been fun as a Mac user/fan - the new Macbooks are sweet, convincing more of my friends to finally make the switch. We don't have a local Apple store here in Newfoundland so I get to help friends in their transition to Mac OS X - a favour I happily oblige :)

Meeting the new Macbook.

Cheers,

MiR: October 2008

I've settled into a routine this month that consists of daily building the latest SMX4 Kernel, NMR, and Features code from SVN and checking for any AIX specific errors. While nightly continuous integration testing works great for automating the discovery & identification of issues I find that this manual approach has been quite fruitful (When no issues are found I just move on to other JIRAs in the SMX4 projects). 

Common AIX issues I've seen this month include updates to the classpath and JVM security settings to work around IBM JDK differences in comparison to Sun, and some defensive coding around annotations.

Cheers,