Showing posts with label JMX. Show all posts
Showing posts with label JMX. Show all posts

Sunday, January 15, 2012

Writing a Mule JMX Agent

A client of ours uses Mule ESB mainly as a mediator component that will throttle, translate and call external parties. Some of these parties are however not so reliable when it comes to response times. There were several utilities in use to check what the response time was of a specific third party. All of these mechanisms could only provide a rough estimate however. Since Mule was the system executing the call to the third party, we found that these kind of statistics should come out of Mule instead. Next question was: how to maintain these and get them out of Mule? To expose statistics, JMX seems a logical choice.

Mule ESB comes with a number of JMX Agents: http://www.mulesoft.org/documentation/display/MULE3USER/JMX+Management. These give some great insight in the system. At our client this info is already being used to display statistics in Zabbix, not only from Mule but also from HornetQ JMS. Zabbix is used to monitor the Average service execution times, JMS Queue depth, memory usage, CPU usage and so on.
These agents however do not expose the information that we want so we we need to write our own.

But first we need to collect the statistics For this we created a statistics object that will keep track of the number of calls, minimum, maximum and average duration and also the average of the last 500 calls. This last metric allows us to have an average which does show peaks over time. To capture the call and add the call info to our statistics object we used AOP:
@Around("execThirdPartyCall")
public Object execute(ProceedingJoinPoint pjp) throws Throwable{
   long timeBefore = System.currentTimeMillis();
   Object result = pjp.proceed();
   try{
      getStatistics(pjp).addMuleCallInfo(
         System.currentTimeMillis() - timeBefore);
   }catch(Exception e){
      LOGGER.error("Error adding MuleCallInfo to Statistics Object.",e);
   }
   return result;
}
This statistics object than keeps all this info but does not calculate all the metrics: min, max, avg, avg500. Because adding the call info is on the call stack, we want to keep these methods as short as possible. The actual calculation is done when the info is requested via an MBean.

So now we come to the main part of this post: how to create an MBean and register it to the Mule MBeanServer. I've worked with Spring and MBeans before and I do like the annotation driven mechanisms that Spring offers for auto registrating your MBeans.
Mule however does not use annotations but a method that is more reflection based. The main class in Mule that provides this logic is org.mule.module.management.agent.ClassloaderSwitchingMBeanWrapper. This class needs an interface for introspection and a concrete class in order to create an MBean instance.

The first thing you do is create an interface that will define the MBean attributes and methods. In the example there are 4 read-only attributes. The JMX_PREFIX will be used later on to define the JMX objectName:
public interface StatisticsMBean {
   String DEFAULT_JMX_PREFIX = "type=Thirdparty.Statistics,name=";
   long getHttpCallLast500Average();
   long getHttpCallLast500Minimum();
   long getHttpCallLast500Maximum();
   int getTotalNumberOfCalls();
}
Second, you need to define an implementation of this interface:
public final class StatisticsService implements StatisticsMBean {
    private Statistics Statistics;

    public StatisticsService(Statistics statistics) {
        this.statistics = statistics;
    }
    @Override
    public long getHttpCallLast500Average() {
        return statistics.getAvgLast500HttpCall();
    }

    @Override
    public long getHttpCallLast500Minimum() {
        return statistics.getMinLast500HttpCall();
    }

    @Override
    public long getHttpCallLast500Maximum() {
        return statistics.getMaxLast500HttpCall();
    }

    @Override
    public int getTotalNumberOfCalls() {
        return statistics.getTotalCallsMuleCall();
    }
}
Third you need to create the actual agent. A few things to note about the code below:
  • the fields are missing and also some methods that are left empty anyway
  • only the registerMbean part is in here, if you have some fancy hot deploy setup, you'll need to provide an unregisterMBean part
  • a MuleContextListener is used to make sure that Spring has finished initializing before doing any work
  • the Statistics object below is an Enum, because we keep statistics for multiple third parties

public final class StatisticsAgent extends AbstractAgent {

   @Override
   public void initialise() throws InitialisationException {
      if (initialized.get()) {
         return;
      }
      //get mbeanserver
      if (mBeanServer == null) {
         mBeanServer = ManagementFactory.getPlatformMBeanServer();
      }
      if (mBeanServer == null) {
         throw new InitialisationException(
             ManagementMessages.cannotLocateOrCreateServer(), this);
      }
      try {
         // We need to register all the services once the server has initialised
         muleContext.registerListener(new MuleContextStartedListener());
      } catch (NotificationException e) {
         throw new InitialisationException(e, this);
      }
      initialized.compareAndSet(false, true);
   }

   protected class MuleContextStartedListener implements
       MuleContextNotificationListener {
      public void onNotification(MuleContextNotification notification) {
         if (notification.getAction() == MuleContextNotification.CONTEXT_STARTED) {
            try {
               registerMBeans();
            } catch (Exception e) {
               throw new MuleRuntimeException(
                   CoreMessages.objectFailedToInitialise("MBeans"), e);
            }
         }
      }
   }

   private void registerMBeans() throws MalformedObjectNameException, 
      NotCompliantMBeanException, InstanceAlreadyExistsException, MBeanRegistrationException {
      Statistics[] statisticsAr = Statistics.values();
      for (Statistics statistics : statisticsAr) {
         ObjectName on = jmxSupport.getObjectName(
                           String.format("%s:%s",
                           jmxSupport.getDomainName(muleContext, false),
                           StatisticsMBean.DEFAULT_JMX_PREFIX + statistics.name())
                          );
         StatisticsService statisticsService = new StatisticsService(statistics);
         ClassloaderSwitchingMBeanWrapper mBean = new ClassloaderSwitchingMBeanWrapper(
                           statisticsService,
                           StatisticsMBean.class, 
                           muleContext.getExecutionClassLoader()
                           );
         logger.debug("Registering StatisticsAgent with name: " + on);
         mBeanServer.registerMBean(mBean, on);
      }
   }
}

As a last step you need to add your agent to the Mule configuration as follows:
<mule>
   <custom-agent class="management.StatisticsAgent" name="statistics-agent"/>
</mule>

And that's it, once you start up Mule, you will see your custom agent appearing and your MBeans are available via JMX.

That is how our operations guys can now provide a nice screen with the call durations over time:


Author: Jeroen Verellen

Friday, December 10, 2010

Command Line JMX + SSL

The tool I use for command line JMX and which I think really works well is http://wiki.cyclopsgroup.org/jmxterm

You can download and start the single jar as shown below. In this example I'm connecting to a HornetQ instance.
[jeroen@homesrv~]# java -jar jmxterm-1.0-alpha-4-uber.jar -l service:jmx:rmi:///jndi/rmi://192.168.1.3:10995/jmxrmi
Welcome to JMX terminal. Type "help" for available commands.
$>
Next you can fetch a list of all the available beans (not showing all the beans):
$>beans #domain = JMImplementation: JMImplementation:type=MBeanServerDelegate #domain = com.sun.management: com.sun.management:type=HotSpotDiagnostic #domain = java.lang: java.lang:name=Code Cache,type=MemoryPool ... #domain = java.util.logging: java.util.logging:type=Logging #domain = org.hornetq: org.hornetq:address="hornetq.notifications",module=Core,name="notif.fb6d27d5-02ac-11e0-ae9b-1cc1de6fb76e",type=Queue ... org.hornetq:module=JMS,name="TEST.IN",type=Queue org.hornetq:module=JMS,type=Server
To see the available attributes and operations on a specific MBean you can use the info command:
$>info -b org.hornetq:module=JMS,name="TEST.IN",type=Queue #mbean = org.hornetq:module=JMS,name="TEST.IN",type=Queue #class name = org.hornetq.jms.management.impl.JMSQueueControlImpl # attributes %0 - Address (java.lang.String, r) %1 - ConsumerCount (int, r) %2 - DeadLetterAddress (java.lang.String, rw) %3 - DeliveringCount (int, r) %4 - ExpiryAddress (java.lang.String, rw) %5 - JNDIBindings ([Ljava.lang.String;, r) %6 - MessageCount (int, r) ... %12 - Temporary (boolean, r) # operations %0 - void addJNDI(java.lang.String jndiBinding) %1 - boolean changeMessagePriority(java.lang.String messageID,int newPriority) %2 - int changeMessagesPriority(java.lang.String filter,int newPriority) %3 - int countMessages(java.lang.String filter) ... %20 - int sendMessagesToDeadLetterAddress(java.lang.String filter) #there's no notifications
To get the actual value of an MBean attribute, use the get command
$>get -b org.hornetq:module=JMS,name="TEST.IN",type=Queue MessageCount
#mbean = org.hornetq:module=JMS,name="TEST.IN",type=Queue:
MessageCount = 0;
To invoke an operation use the run command:
$>run -b org.hornetq:module=JMS,name="TEST.IN",type=Queue resetMessageCounter
#calling operation resetMessageCounter of mbean org.hornetq:module=JMS,name="TEST.IN",type=Queue
#operation returns:
null

$>get -b org.hornetq:module=JMS,name="TEST.IN",type=Queue MessageCount
#mbean = org.hornetq:module=JMS,name="TEST.IN",type=Queue:
MessageCount = 0;
As you can see the tool is very easy to use. BUT, since we are talking about Java Management Extensions which let you control, in this case, a HornetQ server, I think some security is in order.
The JMX agent on the JVM is by default started with TLS/SSL and username + password authentication. More information on how to set this up is on: http://download.oracle.com/javase/1.5.0/docs/guide/management/agent.html#auth. In the example above, security was switched off, which is easy to do by setting the following properties:
com.sun.management.jmxremote.authenticate=false
com.sun.management.jmxremote.ssl=false

Now to enable TLS/SSL and username + password on the JMX agent set the following properties, first 2 were set before as well:
com.sun.management.jmxremote com.sun.management.jmxremote.port=10999 com.sun.management.jmxremote.password.file=$HOME/jmxremote.password javax.net.ssl.keyStore=hornetq-srv1-keystore.jks javax.net.ssl.keyStorePassword=changeit
The jmxremote.password is following the template under java.home/jre/lib/management/jmxremote.password.template

On the client side, in this case jmxterm, you need to set the following properties:

javax.net.ssl.trustStore=jmxclient-truststore.jks
javax.net.ssl.trustStorePassword=changeit
This is all following the java specs and should work, but when we start with the following command
[jeroen@homesrv~]# java -Djavax.net.ssl.trustStore=jmxclient-truststore.jks -Djavax.net.ssl.trustStorePassword=changeit -jar jmxterm-1.0-alpha-4-uber.jar -l service:jmx:rmi:///jndi/rmi://192.168.1.3:10999/jmxrmi
The following error pops up:
constituent[0]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/jmxterm.jar
constituent[1]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/commons-beanutils.jar
constituent[2]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/commons-cli.jar
constituent[3]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/commons-collections.jar
constituent[4]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/commons-lang.jar
constituent[5]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/commons-logging.jar
constituent[6]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/commons-io.jar
constituent[7]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/jcli.jar
constituent[8]: jar:file:/root/jmxterm-1.0-alpha-4-uber.jar!/WORLDS-INF/lib/jline.jar
---------------------------------------------------
java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
javax.net.ssl.SSLKeyException: RSA premaster secret error
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:286)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2327)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:279)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
at org.cyclopsgroup.jmxterm.cc.SessionImpl.doConnect(SessionImpl.java:85)
at org.cyclopsgroup.jmxterm.cc.SessionImpl.connect(SessionImpl.java:49)
at org.cyclopsgroup.jmxterm.cc.CommandCenter.connect(CommandCenter.java:110)
at org.cyclopsgroup.jmxterm.boot.CliMain.execute(CliMain.java:139)
at org.cyclopsgroup.jmxterm.boot.CliMain.main(CliMain.java:48)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.classworlds.Launcher.launchStandard(Launcher.java:353)
at org.codehaus.classworlds.Launcher.launch(Launcher.java:264)
at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
at org.codehaus.classworlds.Launcher.main(Launcher.java:375)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.classworlds.uberjar.boot.Bootstrapper.bootstrap(Bootstrapper.java:209)
at org.codehaus.classworlds.uberjar.boot.Bootstrapper.main(Bootstrapper.java:116)
Caused by: javax.net.ssl.SSLKeyException: RSA premaster secret error
at com.sun.net.ssl.internal.ssl.RSAClientKeyExchange.(RSAClientKeyExchange.java:97)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:673)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:230)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:529)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:465)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:884)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1120)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:623)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:211)
... 25 more
Caused by: java.security.NoSuchAlgorithmException: SunTlsRsaPremasterSecret KeyGenerator not available
at javax.crypto.KeyGenerator.(DashoA13*..)
at javax.crypto.KeyGenerator.getInstance(DashoA13*..)
at com.sun.net.ssl.internal.ssl.JsseJce.getKeyGenerator(JsseJce.java:223)
at com.sun.net.ssl.internal.ssl.RSAClientKeyExchange.(RSAClientKeyExchange.java:89)
... 37 more
It looks like the uberjar is doing something fishy with the classloaders which make that the standard sun security provider is no longer available. Bummer ...

This is the only explanation I can think of. Because when connecting with jconsole, like below, I only need to enter the username/pwd. No other issues.
[jeroen@homesrv~]# jconsole -J-Djavax.net.ssl.trustStore=jmxclient-truststore.jks -J-Djavax.net.ssl.trustStorePassword=changeit service:jmx:rmi:///jndi/rmi://192.168.1.3:10995/jmxrmi
I guess for command line jmx with SSL I'll have to look elsewhere ...
If someone has ideas please post :-)


Authored by: Jeroen