Showing posts with label open-source. Show all posts
Showing posts with label open-source. Show all posts

Sunday, January 15, 2012

Writing a Mule JMX Agent

A client of ours uses Mule ESB mainly as a mediator component that will throttle, translate and call external parties. Some of these parties are however not so reliable when it comes to response times. There were several utilities in use to check what the response time was of a specific third party. All of these mechanisms could only provide a rough estimate however. Since Mule was the system executing the call to the third party, we found that these kind of statistics should come out of Mule instead. Next question was: how to maintain these and get them out of Mule? To expose statistics, JMX seems a logical choice.

Mule ESB comes with a number of JMX Agents: http://www.mulesoft.org/documentation/display/MULE3USER/JMX+Management. These give some great insight in the system. At our client this info is already being used to display statistics in Zabbix, not only from Mule but also from HornetQ JMS. Zabbix is used to monitor the Average service execution times, JMS Queue depth, memory usage, CPU usage and so on.
These agents however do not expose the information that we want so we we need to write our own.

But first we need to collect the statistics For this we created a statistics object that will keep track of the number of calls, minimum, maximum and average duration and also the average of the last 500 calls. This last metric allows us to have an average which does show peaks over time. To capture the call and add the call info to our statistics object we used AOP:
@Around("execThirdPartyCall")
public Object execute(ProceedingJoinPoint pjp) throws Throwable{
   long timeBefore = System.currentTimeMillis();
   Object result = pjp.proceed();
   try{
      getStatistics(pjp).addMuleCallInfo(
         System.currentTimeMillis() - timeBefore);
   }catch(Exception e){
      LOGGER.error("Error adding MuleCallInfo to Statistics Object.",e);
   }
   return result;
}
This statistics object than keeps all this info but does not calculate all the metrics: min, max, avg, avg500. Because adding the call info is on the call stack, we want to keep these methods as short as possible. The actual calculation is done when the info is requested via an MBean.

So now we come to the main part of this post: how to create an MBean and register it to the Mule MBeanServer. I've worked with Spring and MBeans before and I do like the annotation driven mechanisms that Spring offers for auto registrating your MBeans.
Mule however does not use annotations but a method that is more reflection based. The main class in Mule that provides this logic is org.mule.module.management.agent.ClassloaderSwitchingMBeanWrapper. This class needs an interface for introspection and a concrete class in order to create an MBean instance.

The first thing you do is create an interface that will define the MBean attributes and methods. In the example there are 4 read-only attributes. The JMX_PREFIX will be used later on to define the JMX objectName:
public interface StatisticsMBean {
   String DEFAULT_JMX_PREFIX = "type=Thirdparty.Statistics,name=";
   long getHttpCallLast500Average();
   long getHttpCallLast500Minimum();
   long getHttpCallLast500Maximum();
   int getTotalNumberOfCalls();
}
Second, you need to define an implementation of this interface:
public final class StatisticsService implements StatisticsMBean {
    private Statistics Statistics;

    public StatisticsService(Statistics statistics) {
        this.statistics = statistics;
    }
    @Override
    public long getHttpCallLast500Average() {
        return statistics.getAvgLast500HttpCall();
    }

    @Override
    public long getHttpCallLast500Minimum() {
        return statistics.getMinLast500HttpCall();
    }

    @Override
    public long getHttpCallLast500Maximum() {
        return statistics.getMaxLast500HttpCall();
    }

    @Override
    public int getTotalNumberOfCalls() {
        return statistics.getTotalCallsMuleCall();
    }
}
Third you need to create the actual agent. A few things to note about the code below:
  • the fields are missing and also some methods that are left empty anyway
  • only the registerMbean part is in here, if you have some fancy hot deploy setup, you'll need to provide an unregisterMBean part
  • a MuleContextListener is used to make sure that Spring has finished initializing before doing any work
  • the Statistics object below is an Enum, because we keep statistics for multiple third parties

public final class StatisticsAgent extends AbstractAgent {

   @Override
   public void initialise() throws InitialisationException {
      if (initialized.get()) {
         return;
      }
      //get mbeanserver
      if (mBeanServer == null) {
         mBeanServer = ManagementFactory.getPlatformMBeanServer();
      }
      if (mBeanServer == null) {
         throw new InitialisationException(
             ManagementMessages.cannotLocateOrCreateServer(), this);
      }
      try {
         // We need to register all the services once the server has initialised
         muleContext.registerListener(new MuleContextStartedListener());
      } catch (NotificationException e) {
         throw new InitialisationException(e, this);
      }
      initialized.compareAndSet(false, true);
   }

   protected class MuleContextStartedListener implements
       MuleContextNotificationListener {
      public void onNotification(MuleContextNotification notification) {
         if (notification.getAction() == MuleContextNotification.CONTEXT_STARTED) {
            try {
               registerMBeans();
            } catch (Exception e) {
               throw new MuleRuntimeException(
                   CoreMessages.objectFailedToInitialise("MBeans"), e);
            }
         }
      }
   }

   private void registerMBeans() throws MalformedObjectNameException, 
      NotCompliantMBeanException, InstanceAlreadyExistsException, MBeanRegistrationException {
      Statistics[] statisticsAr = Statistics.values();
      for (Statistics statistics : statisticsAr) {
         ObjectName on = jmxSupport.getObjectName(
                           String.format("%s:%s",
                           jmxSupport.getDomainName(muleContext, false),
                           StatisticsMBean.DEFAULT_JMX_PREFIX + statistics.name())
                          );
         StatisticsService statisticsService = new StatisticsService(statistics);
         ClassloaderSwitchingMBeanWrapper mBean = new ClassloaderSwitchingMBeanWrapper(
                           statisticsService,
                           StatisticsMBean.class, 
                           muleContext.getExecutionClassLoader()
                           );
         logger.debug("Registering StatisticsAgent with name: " + on);
         mBeanServer.registerMBean(mBean, on);
      }
   }
}

As a last step you need to add your agent to the Mule configuration as follows:
<mule>
   <custom-agent class="management.StatisticsAgent" name="statistics-agent"/>
</mule>

And that's it, once you start up Mule, you will see your custom agent appearing and your MBeans are available via JMX.

That is how our operations guys can now provide a nice screen with the call durations over time:


Author: Jeroen Verellen

Wednesday, February 23, 2011

JMS speed test: ActiveMQ vs HornetQ

A while ago I was asked by a client to evaluate different open source JMS providers. The ultimate goal was to set up a highly available messaging system that can manage high throughput.

Me and my colleagues dug deep in our memories and on the internet to find all of the open source JMS providers. And there are plenty. In the end we looked at the following:
  • ActiveMQ
  • OpenMQ
  • RabbitMQ
  • OpenJMS
  • HornetQ
Besides the requirements of high availability and throughput there were some other requirements:
  • must be JMS 1.1 compliant
  • must be easy to set up and administer
  • vibrant community for support
Because of these reasons we quickly abandoned OpenJMS, which seems to have stopped evolving somewhere in 2006.
RabbitMQ is not JMS compliant, which we really need.
OpenMQ was dropped a little later, since in the first performance tests we found that it was noticably slower than ActiveMQ and HornetQ.

In a next phase we did quite extensive load tests on HornetQ and ActiveMQ, which I'll summarize below.

For our tests we used the following setup:
3 similar machines with 2 quad cores, 8GB of RAM, RHEL 5, Java Hotspot VM 64 bit (1.6.0_21-b06). 2 machines were used for hosting the JMS providers. One machine contains a master, the other a backup instance. The third machine is used for generating load.

The load generation is done using the Sonic test harness. This framework allowed us to generate load on the the JMS providers with different producers and consumers of the JMS queues.

These are some of the results we got (based on size of message, whether messages are persisted, if a transaction is used, number of concurrent producers and receivers):


As you can see both JMS providers are about equal when you look at non peristent messaging.BUT when looking at persistent messaging, HornetQ is just amazing. The throughput you get there is just mindblowing. This has without a doubt to do with the Asynchronous IO feature of HornetQ. This is only available on *NIX based systems but it is well worth it when you are looking for a performant open source JMS provider.

There are other sources of comparisons like these, but they were in our opinion either biased or did not give us enough insight for our situation:
Authored by: Jeroen