Thursday, January 26, 2012

Running Oracle Weblogic Server on Windows: a shutdown class

Overview



Running Oracle Weblogic server on Windows can be done in two ways:

  •  You can start Oracle Weblogic Server from the command line. This is good for test purposes but not very likely to be used in a production environment. 
  • You can install Oracle Weblogic Server to run as a windows service using the java wrapper beasvc.exe (or beasvcX64.exe on windows 64-bit)
The problem with the last option is that, if you stop the service, the Java Virtual Machine (and thus the Oracle Weblogic Server) simply gets killed. This can result in corrupt files, database connections that aren’t closed or even transactions that are started but won’t be stopped. A possible workaround is to stop each Weblogic server in the domain from the admin console. This not only time consuming but also not the preferred way of working.
In order to provide a proper shutdown, the beasvc.exe has an option called “stopclass”. It specifies a java class with a stop method that will be invoked at the moment the windows service gets stopped. Although you can write any kind of java code in the shutdown class, we will focus on writing code for a decent shutdown for both admin servers and managed servers.
The shutdown class works as follows: 
  • It sends a request over HTTP to the Admin server 
  • The Admin server sends the shutdown request to the corresponding server.
This means that the admin server sends a request to itself in case of a shutdown. The shutdown request for a managed server passes first via the admin server back to the managed server.

 

Configuration

First of all the beasvc.exe service wrapper needs to be aware of the shutdown class. This can be done by adding the -stopclass option (for example: -stopclass: be.i8c.systemtools.weblogic.ServerStopper)
Secondly, two options need to be set on each server in the domain:
  • You have to enable tunneling. You do this by ticking “Enable Tunneling” under Domain > Environment > Servers > “Server” > Protocols > General
  • You have to ignore http sessions during shutdown, otherwise the server keeps on waiting for ever until no more http sessions are active. Enable the option “Ignore Sessions During Shutdown” under Domain > Environment > Servers > “Server” > Control > Start/Stop

 

Code

package be.i8c.systemtools.weblogic;

import java.io.File;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.Date;
import java.util.Iterator;
import java.util.Properties;
import java.util.Set;

import javax.naming.Context;

import weblogic.jndi.Environment;
import weblogic.management.MBeanHome;
import weblogic.management.runtime.ServerRuntimeMBean;
import weblogic.management.runtime.ServerStates;

public class ServerStopper
{
  public static void stop()
  {
    Properties prop = System.getProperties();
    try
    {
      MBeanHome home = null;
      //url of the Admin server
      String url = "";

      if (prop.getProperty("weblogic.management.server") != null)
      {
        // this property will be set automatically on a managed server to indicate the location of the admin server    
        url = prop.getProperty("weblogic.management.server");
      }
      else
      {
        // the admin.url is a property to indicate the location of the admin server on the admin server. It refers to itself, but is necessary to perform a proper shutdown. You can add it manually as a –D property when you install the server as a windows service. The name of the property can be any random name.
   url = prop.getProperty("admin.url");
      }
      // username and password to perform the shutdown operation
      username = "username";
      password = "password";
               
      ServerRuntimeMBean serverRuntime = null;
      Set mbeanSet = null;
      Iterator mbeanIterator = null;
      // Set ContextClassloader to prevent assertions
      URL[] urls = { new File("/").toURL() };
      Thread.currentThread().setContextClassLoader(new URLClassLoader(urls));
      Environment env = new Environment();
      env.setProviderUrl(url);
      env.setSecurityPrincipal(username);
      env.setSecurityCredentials(password);
      Context ctx = env.getInitialContext();
      home = (MBeanHome)ctx.lookup("weblogic.management.adminhome");
      mbeanSet = home.getMBeansByType("ServerRuntime");
      mbeanIterator = mbeanSet.iterator();
             
      while(mbeanIterator.hasNext())
      {
        serverRuntime = (ServerRuntimeMBean)mbeanIterator.next();
        try
        {
          // we have to add this check, since the iterator will list all the servers, both admin and managed. If we don’t add this check a shutdown of all the servers will be performed in case we only stop one server
          if (serverRuntime.getName().equalsIgnoreCase(prop.getProperty("weblogic.Name")))
       {
         if(serverRuntime.getState().equals(ServerStates.RUNNING))
         {
           // we perform a normal shut down in case the server is in the RUNNING state
           serverRuntime.shutdown();
         }
         else
         {
           //This is the same as a forced shutdown, but not the same as killing the JVM. We don’t do a normal shutdown in case the server is not in the RUNNING state.
 System.exit(10);
         }
       }
        }
        catch (Exception e)
        {
          // You can do nothing over here. In case the shutdown call will fail, windows will wait for the service to end, but that won’t happen. It will kill the service anyway after a while.
        }
      }
    }
    catch (Exception e)
    {
      // We get here in case the admin server couldn’t be reached
      System.exit(10);
    }
  }
}

Author: Dimitri Van Kerckhoven

Friday, January 20, 2012

How to: automatically remove LOCKFILE on server start (WINDOWS)

Have you ever had that annoying issue when your server started up and the LOCKFILE still existed in the webMethods folder?

I’ve created a small script for windows which tackles this issue.

set jobname=softwareAGwebMethodsIntegrationServer_8.0
set location=C:\SoftwareAG\IntegrationServer\LOCKFILE

for /F "tokens=3 delims=: " %%H in ('sc query "%jobname%" ^| findstr "        STATE"') do (
  if /I "%%H" NEQ "RUNNING" (
                IF EXIST %location% GOTO REMOVEFILEANDSTART
  )

)

:REMOVEFILEANDSTART
del %location%
net start "%jobname%"

Variable jobname is your servicename which you set to start automatic
To find this name open Services and double click the job name



The name entered behind Service name is the name you need to enter for this variable.
Variable “location” is the location of the LOCKFILE, enter the full path.

Explanation of the code :
set jobname=softwareAGwebMethodsIntegrationServer_8.0
set location=C:\SoftwareAG\IntegrationServer\LOCKFILE

Declaration of the variables.

for /F "tokens=3 delims=: " %%H in ('sc query "%jobname%" ^| findstr "        STATE"') do (
  if /I "%%H" NEQ "RUNNING" (
                IF EXIST %location% GOTO REMOVEFILEANDSTART
  )

)

Loop over the result of sc query "%jobname%"  and search for the word STATE
If the result of STATE is different from RUNNING then execute function REMOVEFILEANDSTART 

:REMOVEFILEANDSTART
del %location%
net start "%jobname%"

Delete the LOCKFILE and start the service.

To use this script, create a scheduled task which is triggered on server startup.

Author: Jeroen W.

Sunday, January 15, 2012

Writing a Mule JMX Agent

A client of ours uses Mule ESB mainly as a mediator component that will throttle, translate and call external parties. Some of these parties are however not so reliable when it comes to response times. There were several utilities in use to check what the response time was of a specific third party. All of these mechanisms could only provide a rough estimate however. Since Mule was the system executing the call to the third party, we found that these kind of statistics should come out of Mule instead. Next question was: how to maintain these and get them out of Mule? To expose statistics, JMX seems a logical choice.

Mule ESB comes with a number of JMX Agents: http://www.mulesoft.org/documentation/display/MULE3USER/JMX+Management. These give some great insight in the system. At our client this info is already being used to display statistics in Zabbix, not only from Mule but also from HornetQ JMS. Zabbix is used to monitor the Average service execution times, JMS Queue depth, memory usage, CPU usage and so on.
These agents however do not expose the information that we want so we we need to write our own.

But first we need to collect the statistics For this we created a statistics object that will keep track of the number of calls, minimum, maximum and average duration and also the average of the last 500 calls. This last metric allows us to have an average which does show peaks over time. To capture the call and add the call info to our statistics object we used AOP:
@Around("execThirdPartyCall")
public Object execute(ProceedingJoinPoint pjp) throws Throwable{
   long timeBefore = System.currentTimeMillis();
   Object result = pjp.proceed();
   try{
      getStatistics(pjp).addMuleCallInfo(
         System.currentTimeMillis() - timeBefore);
   }catch(Exception e){
      LOGGER.error("Error adding MuleCallInfo to Statistics Object.",e);
   }
   return result;
}
This statistics object than keeps all this info but does not calculate all the metrics: min, max, avg, avg500. Because adding the call info is on the call stack, we want to keep these methods as short as possible. The actual calculation is done when the info is requested via an MBean.

So now we come to the main part of this post: how to create an MBean and register it to the Mule MBeanServer. I've worked with Spring and MBeans before and I do like the annotation driven mechanisms that Spring offers for auto registrating your MBeans.
Mule however does not use annotations but a method that is more reflection based. The main class in Mule that provides this logic is org.mule.module.management.agent.ClassloaderSwitchingMBeanWrapper. This class needs an interface for introspection and a concrete class in order to create an MBean instance.

The first thing you do is create an interface that will define the MBean attributes and methods. In the example there are 4 read-only attributes. The JMX_PREFIX will be used later on to define the JMX objectName:
public interface StatisticsMBean {
   String DEFAULT_JMX_PREFIX = "type=Thirdparty.Statistics,name=";
   long getHttpCallLast500Average();
   long getHttpCallLast500Minimum();
   long getHttpCallLast500Maximum();
   int getTotalNumberOfCalls();
}
Second, you need to define an implementation of this interface:
public final class StatisticsService implements StatisticsMBean {
    private Statistics Statistics;

    public StatisticsService(Statistics statistics) {
        this.statistics = statistics;
    }
    @Override
    public long getHttpCallLast500Average() {
        return statistics.getAvgLast500HttpCall();
    }

    @Override
    public long getHttpCallLast500Minimum() {
        return statistics.getMinLast500HttpCall();
    }

    @Override
    public long getHttpCallLast500Maximum() {
        return statistics.getMaxLast500HttpCall();
    }

    @Override
    public int getTotalNumberOfCalls() {
        return statistics.getTotalCallsMuleCall();
    }
}
Third you need to create the actual agent. A few things to note about the code below:
  • the fields are missing and also some methods that are left empty anyway
  • only the registerMbean part is in here, if you have some fancy hot deploy setup, you'll need to provide an unregisterMBean part
  • a MuleContextListener is used to make sure that Spring has finished initializing before doing any work
  • the Statistics object below is an Enum, because we keep statistics for multiple third parties

public final class StatisticsAgent extends AbstractAgent {

   @Override
   public void initialise() throws InitialisationException {
      if (initialized.get()) {
         return;
      }
      //get mbeanserver
      if (mBeanServer == null) {
         mBeanServer = ManagementFactory.getPlatformMBeanServer();
      }
      if (mBeanServer == null) {
         throw new InitialisationException(
             ManagementMessages.cannotLocateOrCreateServer(), this);
      }
      try {
         // We need to register all the services once the server has initialised
         muleContext.registerListener(new MuleContextStartedListener());
      } catch (NotificationException e) {
         throw new InitialisationException(e, this);
      }
      initialized.compareAndSet(false, true);
   }

   protected class MuleContextStartedListener implements
       MuleContextNotificationListener {
      public void onNotification(MuleContextNotification notification) {
         if (notification.getAction() == MuleContextNotification.CONTEXT_STARTED) {
            try {
               registerMBeans();
            } catch (Exception e) {
               throw new MuleRuntimeException(
                   CoreMessages.objectFailedToInitialise("MBeans"), e);
            }
         }
      }
   }

   private void registerMBeans() throws MalformedObjectNameException, 
      NotCompliantMBeanException, InstanceAlreadyExistsException, MBeanRegistrationException {
      Statistics[] statisticsAr = Statistics.values();
      for (Statistics statistics : statisticsAr) {
         ObjectName on = jmxSupport.getObjectName(
                           String.format("%s:%s",
                           jmxSupport.getDomainName(muleContext, false),
                           StatisticsMBean.DEFAULT_JMX_PREFIX + statistics.name())
                          );
         StatisticsService statisticsService = new StatisticsService(statistics);
         ClassloaderSwitchingMBeanWrapper mBean = new ClassloaderSwitchingMBeanWrapper(
                           statisticsService,
                           StatisticsMBean.class, 
                           muleContext.getExecutionClassLoader()
                           );
         logger.debug("Registering StatisticsAgent with name: " + on);
         mBeanServer.registerMBean(mBean, on);
      }
   }
}

As a last step you need to add your agent to the Mule configuration as follows:
<mule>
   <custom-agent class="management.StatisticsAgent" name="statistics-agent"/>
</mule>

And that's it, once you start up Mule, you will see your custom agent appearing and your MBeans are available via JMX.

That is how our operations guys can now provide a nice screen with the call durations over time:


Author: Jeroen Verellen

Tuesday, January 10, 2012

XSLT transformations and smooks driven CSV parsing in JBoss ESB

In this last entry that completes the video series of our JBoss ESB introduction, we explore XSLT transformations and basic smooks parsing.
First up is the XSLT transformation:


Link: http://www.youtube.com/watch?v=vkd12IHJMGY

Parsing flat file into usable XML formats is a rather common requirement for integration platforms. Out of the box JBoss ESB comes with smooks.
The following video shows you how to parse a CSV file using smooks.

Link: http://www.youtube.com/watch?v=Rme4Fv-eqUo

Author: Alexander

Tuesday, January 3, 2012

webMethods Process performance analyzing

We all get in touch with different processes on daily basis. Most of the times there are tools available to monitor if a process ran successfully.  But in many cases the performance of these processes aren’t measured before they are deployed to the production environment.

We noticed the same problem with some of our customers. There were processes that should deal with a large amount of traffic. So it was critical that the performance was as good as possible.

How did we get started?          
  • We recognized different steps in the process
    • Validation
    • Xslt transformations
    • Database transactions
    • Queuing
  • We created SQL scripts to measure the time these process steps ran
    • Based on custom audit data from the customer
    • Based on the webMethods audit tables (service execution)
  • We analyzed the results with our custom excel template
·      
Performance Analyze:
We wanted to see if there were bottlenecks in our process and if the load had any effect on the process. 

We noticed that all the steps ran a bit longer on a heavy loaded server than on a dedicated one. This seemed to be normal. Abnormal behavior would be when a server started to freeze and the more instances ran, the longer it would take. The curve you can see below is a normal curve. The average runtime doesn’t start to be longer and longer over time.



We also noticed that there wasn’t  a normal curve in the queuing step. After some debugging we noticed that though the process was async.

The webMethods triggers were defined as serial.  The processing curve improved after changing these triggers.

Author: Pieter Van de Broeck