Setting up SMX4 failover with MySQL

This post is an update to my previous post on setting up SMX4 with high availability in mind.

Since the transition to Apache Felix Karaf the support for more database locking table back ends has expanded. Specifically you can now use Apache Derby, MySQL, or a generic JDBC connector. The Derby and Generic JDBC locks make use of the 'FOR UPDATE' feature common to many databases. The MySQL lock however makes use of the ' LOCK TABLES' directive, this is due to the 'FOR UPDATE' command not returning a ResultSet on more recent versions of MySQL (see some discussions on this here).

The following system.properties configuration will allow your Karaf based SMX4 instance to use a MySQL JDBC lock:

karaf.lock=true
karaf.lock.level=20
karaf.lock.delay=1000
karaf.lock.class=org.apache.felix.karaf.main.MySQLJDBCLock
karaf.lock.jdbc.url=jdbc:mysql://address:port/dbname
karaf.lock.jdbc.driver=com.mysql.jdbc.Driver
karaf.lock.jdbc.user=root
karaf.lock.jdbc.password=karaf
karaf.lock.jdbc.table=KARAF_LOCK
karaf.lock.jdbc.clustername=mycluster
karaf.lock.jdbc.timeout=30

Please note that the MySQL JDBC driver will have to be made available on the classpath.

XHarness 1.0-beta-4 Released

XHarness version 1.0-beta-4 has just been released. It includes a couple of bug fixes and enhancements that have accumulated over the last few months, most notably the long overdue support for Ant 1.7.

Please see the Changelog on the XHarness website http://xharness.codehaus.org for more details.

You can download XHarness 1.0-beta-4 from the download page, alternatively if your using maven just update your poms.

Finding Bugs with FindBugs

FindBugs is a static analysis tool developed and maintained by the University of Maryland. The tool can find bugs in Java code, and present a listing of issues detected to developers. The project maintains a list of bug patterns they detect when performing an analysis of a code base here.  

As an experiment to begin learning about FindBugs, I ran the tool against a recent build of Apache Felix Karaf. In figure 1 you can see the listing of issues flagged.

Figure 1: FindBugs analysis screen.

Reviewing the flagged issues, not all of them are bugs but suggestions on performance or alternative implementation practice. The remainder of issues are actual problems, luckily many of these can often be remedied in a few minutes. One such issue was a minor file descriptor leak discovered in Karaf's Main class. Resolving the issue only required closing an IO stream after use. After modifying the code, and testing to ensure nothing accidentally broke, I re-ran FindBugs; the issue list was reduced by one :) Having made a small improvement to the code base I took the time to open a minor issue under Felix Karaf issue tracker and submitted a patch. 

Using tools such as FindBugs is only one part of the process of developing software, many issues will not be caught with out proper unit and system testing, and feed back from users in the field. As a second set of eyes to help catch programming omissions and in maintaining good coding practices I feel that the time spent with FindBugs is well worth the effort.

Getting to know VisualVM

Recently I had to refactor a feature that I had introduced to the old Servicemix Kernel, now Felix Karaf project. The issue was relatively minor to resolve, however testing it would require profiling memory and CPU usage over a period of time (to avoid an eventual out of memory error).

There are multiple tools out there that would allow me to do this sort of monitoring, for this particular task I chose to try out VisualVM as it was already bundled with my JDK 1.6 install. VisualVM is a visual tool that integrates several commandline JDK tools and lightweight profiling capabilities for the Java SE platform.

To begin my refactoring task I first setup Karaf with the configuration required to reproduce the out of memory condition. No special configuration or instrumentation of Karaf itself was required for using the profiler. Starting Karaf, using JDK 1.6, I was able to attach to the running process via the VisualVM interface. Once connected to the process I could view the live heap usage, threading, permgen, and loaded classes, further I could inspect the parameters fed to the JVM at start up to confirm the environment was as I configured. To view memory and CPU usage of Karaf, I selected the profiler display tab. Adjusting the profiler settings to only monitor org.apache.felix classes, I could quickly identify the offending set of method calls that contained the source of my error (high CPU usage, and multiple instances accumulating in memory).

VisualVM displaying monitor view of Felix Karaf

Resolving the error didn't require much work except to ensure a few resources were being properly freed, and removal of an unneeded for-loop. To test out my modifications I redeployed Karaf, and began reviewing its run time characteristics via VisualVM. This time the trouble methods were not being called as often, and not creating memory leaks, while still providing the same functionality the original code was intended to provide.

As a simple profiler I can recommend VisualVM as a quick, convenient way to observe your Java applications. It will be interesting to see what features they integrate into the tool in future releases.

Apache Servicemix Kernel subproject moves to Apache Felix

So the big news in the Apache Servicemix project, after its recent 4.0 release, is the move of the Kernel subproject over to Apache Felix. This change came about for several reasons which were discussed on Guillaume Nodet's blog. After some discussion on the mail lists, the name "Karaf" was accepted for the subproject. As annouced by Guillaume, the vote to move the subproject was successful and the work to make the move began.

Now that the process has begun to establish Apache Felix Karaf, I thought it would be good to start publishing a few links so that Kernel developers & users could find its new home.

Karaf Overview:
http://felix.apache.org/site/apache-felix-karaf.html

Karaf Users' Guide:
http://felix.apache.org/site/karaf-users-guide.html

Karaf issue tracker (see Karaf component):
http://issues.apache.org/jira/browse/Felix

Karaf source code (anonymous check out from Felix trunk):
http://svn.apache.org/repos/asf/felix/trunk/karaf/

Felix project contributing guide:
http://felix.apache.org/site/contributing.html

FUSE ESB 4.1 Released!

FUSE ESB 4.1 is now available at FUSESource.

FUSE ESB 4 is an enterprise version of Apache ServiceMix 4 – the most popular standards-based open source ESB. FUSE ESB 4 continues to provide support for widely adopted integration standards like JBI 1.0 and JMS while also ensuring support for the latest emerging standards like OSGi. The new FUSE ESB 4 provides a single platform that makes it easy for developers to implement the integration patterns they need with the programming model of their choice

Get your copy here:
http://fusesource.com/products/enterprise-servicemix4/

Release Notes:
http://fusesource.com/wiki/display/ProdInfo/FUSE+ESB+v4.1+Release+Notes

Apache ServiceMix 4.0.0 Released!

I'm pleased to echo the following ServiceMix announcement:

"The Apache ServiceMix team is pleased to annouce the release of Apache ServiceMix 4.0.

Download links and detailed release notes are available at
http://servicemix.apache.org/smx4/servicemix-400.html

Apache ServiceMix 4.0 is the first release of our OSGi based integration platform. It includes two major components:

• Apache ServiceMix Kernel 1.1.0 : an OSGi runtime with a lot of extra features (SSH connectivity,provisioning enhancements, Spring integration, ...)
• Apache ServiceMix NMR 1.0.0 : an OSGi based NMR and JBI container, which also comes with a new clustering engine ready for enterprise deployment

In addition, ServiceMix 4.0 also ships with enhanced ActiveMQ, Camel and CXF integration as well as a whole set of examples to let you leverage this functionality. We also have out-of-the-box support for deploying and running web applications, so they can run together with everything else in the same container."

Apache ServiceMix NMR 1.0.0 Released!

Hi All, Just wanted to echo the ServiceMix NMR 1.0.0 Release announcement.

"The Apache ServiceMix team is pleased to announce the release of Apache ServiceMix NMR 1.0.0.

Download links and detailed release notes are available at
http://servicemix.apache.org/nmr/servicemix-nmr-100.html

Apache ServiceMix NMR 1.0.0 is the first release of our JBI container based on OSGi. This distribution is a minimal JBI container, if you are looking for a more features distribution, we'll welcome you to have a look at Apache ServiceMix 4.0.0 instead, which includes all the JBI components provided by ServiceMix."

Once you've downloaded the new NMR kit be sure to check out the new examples. Expect to read more about them in a future post.

Apache ServiceMix Kernel 1.1.0 Released!

The Apache ServiceMix Team is pleased to announce the release of Apache ServiceMix Kernel 1.1.0.

The Apache ServiceMix Kernel 1.1.0 release brings a lot of new features, enhancements, and bug fixes:

• Remote connection using SSH protocol.
• Provisioning enhancements: versioning / hot deployment of features.
• New commands, including OSGi related commands for the Configuration Admin and Package Admin services.
• Improved spring integration: upgrade to spring 2.5.6 and spring-dm 1.2.0-m2, the osgi/list command now displays spring applications status.
• Container level locking for master / slave deployments.
• Support for JAXP 1.4 on all platforms.
• Improved JMX support for managing the OSGi framework and features.

Note that the commands syntax has changed due to the upgrade to thelatest gshell version.

This release, with the detailed release notes, is available at:
http://servicemix.apache.org/kernel/servicemix-kernel-110.html

I'm an Apache Servicemix committer!

I first started using Servicemix in the fall of 2006 while it was still in Apache Incubator. My first few patches began appearing soon after on the project issue tracker in the winter of 2007. I submitted these patches initially just for fun and continued submitting the patches while I worked on other projects. As of the summer of 2008 I had the opportunity to make Servicemix my main focus, and its been a fun ride ever since.

This is my first committer status on an Apache project so this will be a new learning experience. I have been a long time supporter of open source and look forward to the adventures ahead.

Cheers!

Adding container level locking to your H.A. Servicemix 4 deployment.

As an improvement to the High Availability features being implemented in the latest versions of the Servicemix 4 Kernel the concept of container level locking has been introduced.

The Container Level locking mechanism allows bundles to be loaded into slave kernel instances in order to provide faster failover performance (when a slave instance becomes the master it will have fewer bundles to load before starting operation). The Container Level refers to the starting priority assigned to each bundle in the OSGI container. These start levels are specified in $SERVICEMIX_HOME/etc/startup.properties, in the format jar.name=level. The core system bundles have levels below 50, where as user bundles have levels greater than 50.

Level: 1 Behavior: A 'cold' standby instance. Core bundles are not loaded into container. Slaves will wait until lock acquired to start server.

Level: <50 Behavior: A 'hot' standby instance. Core bundles are loaded into the container. Slaves will wait until lock acquired to start user level bundles. The console will be accessible for each slave instance at this level.

Level: >50 Behavior: This setting is Not recommended as user bundles will be started.

Container Level locking is supported in both currently supported failover deployment schemes;

• Simple Lock File
• JDBC Locking

To make use of this capability the following must be set on each system in the master/slave setup:

• $SERVICEMIX_HOME/etc/system.properties file updated to include the below entries in addition to other configuration entries to enable H.A.
  

servicemix.lock.level=50
servicemix.lock.delay=10

For more information on how to use the High Availability features supported with Servicemix 4 please visit the users guide.

Setting up Servicemix 4 with H.A. in mind.

Looking to setup Servicemix 4 with High Availability in mind? Then follow this guide to setting up a JDBC Master Slave deployment (Requires Servicemix 4 Kernel 1.1.0-SNAPSHOT or higher).

What is a JDBC Master Slave deployment?

A JDBC Master Slave deployment is a collection of Servicemix instances configured to act as one logical instance (which I refer to as a cluster). Under this High Availability model one instance node provides services (Master) while all other instance nodes (Slaves) wait by the side lines for the node to release its JDBC lock. When the JDBC lock is not held by any node then the first node to obtain the lock becomes active and will fully start its instance of Servicemix.

In this deployment scenario the JDBC connection to a locking table hosted on a database server becomes a point of failure to the cluster, as such it is highly recommended that the database also be provided in a highly available manner. A lock monitor is implement in Servicemix to ensure that as a master node the loss of its lock will force a graceful shutdown of the node.

The above figure depicts three servers each hosting an instance of servicemix in a JDBC Master Slave setup. The DB server hosts an Apache Derby database instance. The instance labeled "Master" has obtained the lock on the database table, the Servicemix instances on servers B and C are left waiting to obtain the lock.

What happens during the initial start of the cluster?

When Servicemix nodes are started they will read the Servicemix lock configuration and attempt to connect to the specified database. The first cluster node to establish connection to the locking table will become the master, the remaining instances will wait, retrying their connection periodically.

What happens when a master fails?

In the event that the master node experiences a failure it will release its hold on the locking table, this will allow awaiting slave nodes to attempt access to the lock. Once the master node has failed it will require a manual restart, upon which it will join the slave node pool.

What if a database failure occurs?

In the event of database failure the current master node will detect the loss of connection and shutdown. The slave nodes will continue to await connection to the database. The former master node will require manual restart. This feature was developed to ensure that a temporary loss of lock would not allow two master nodes to operate at one time.

What happens when an instance restarts?

When a node is started as part of a running cluster it will attempt to make connection with the database to access the lock, failing to obtain the lock it will retry periodically.

How to configure Servicemix 4 JDBC Master Slave:

To setup the JDBC Master Slave deployment you will need to configure the following on each node to be included in the cluster:

• Classpath updated to include JDBC driver.
• $SERVICEMIX_HOME/etc/system.properties file updated to reflect the below entries.

servicemix.lock=true
servicemix.lock.class=org.apache.servicemix.kernel.main.DefaultJDBCLock
servicemix.lock.jdbc.url=jdbc:derby://dbserver:1527/sample
servicemix.lock.jdbc.driver=org.apache.derby.jdbc.ClientDriver
servicemix.lock.jdbc.user=user
servicemix.lock.jdbc.password=password
servicemix.lock.jdbc.table=SERVICEMIX_LOCK
servicemix.lock.jdbc.clustername=smx4
servicemix.lock.jdbc.timeout=30

Note:

• Will fail if jdbc driver is not on classpath.
• The database name "sample" will be created if it does not exist on the db.
• First SMX4 instance to acquire the locking table is the master instance.
• If connection to the DB is lost then the master instance will attempt to gracefully shutdown, allowing a slave instance to become master when the DB service is restored. The former master will require manual restart.