Monday, February 27, 2012

Troubleshooting a HTTPS TLSv1 handshake between Microsoft software and Webmethods 8

The problem

Microsoft software (biztalk, wfetch, IE,…) all have a problem when performing a HTTPS TLSv1 handshake to Webmethods 8.0.1 running on a JVM 1.6 server. The problem manifests itself as an error on the client side (biztalk):
“The underlying connection was closed: An unexpected error occurred on a send.”
Note: IE will actually catch this error and retry using SSLv3 so to the end-user it seems that IE works but it actually exhibits the same behavior.
Server specifications:

Webmethods: windows 2003, JSE 1.664 bit, v8.0.1
Biztalk: windows 2008

Recognizing the error

The first step is recognizing the error on our system so we can identify it when trying to reproduce it.

Webmethods

First step is setting all the logging to “TRACE” for thewebmethodsserver logger. The server log/error log does not show any indication that the failed connection took place.

JVM

The JVM offers a few system attributes to enable security debugging:

java.security.debug
javax.net.debug

Setting “java.security.debug=all” in the server.bat (and running from CLI) produces a stupendous amount of logging at startup which made it ratherunusable (after 10minutes it wasn’t even at the first startup log you expect in server.log), especially because I couldn’t redirect the output of the CLI to a file (for unknown reasons)
Setting “javax.net.debug=all”(or ssl)in the server.bat actually results in no additional output whatsoever.This may point to a custom webmethods implementation for HTTPS.

Wireshark

We installed wireshark on the server to capture the actual traffic going back and forth.We needed some additional configuration to actually make heads and tails from the encrypted traffic (source:http://support.citrix.com/article/CTX116557)

Go to edit > preferences > protocols > SSL
Add <ip-of-webmethods-server>,<https-port-of-webmethods-server>,TCP,c:/path/to/private-key.pem
- E.g. 10.0.0.1,443,TCP,c:\my-private-key.pem
- Use “;” to separate multiple entires and make sure there are no spaces after each comma
- Or in newer versions of wireshark:

Fill in an SSL debug file (e.g.c:\ssl-debug.log)
Restart wireshark and the debug file you configured should show something like:
- ssl_init found host entry<configured-entry>
- ssl_init addr‘<ip>' port '<port>' filename '<name>' password(only for p12 file) '(null)'
- Private key imported: KeyID<id>..
- ssl_init private key file<path>successfully loaded
If the key could not be imported (wrong format or the likes) you will see an error in the log file

In our case the private key was in DER format so we converted it to PEM using openssl:
$ openssl rsa -in private-key.der -out private-key.pem -outform PEM–inperform DER
Now that we can capture the HTTPS traffic,we add a filter to only see the traffic from the client:
ip.src == <client-ip> || ip.dst == <client-ip>
You can also add a port filter if necessary:
tcp.port == <local-port>
You can see the followingtrafficin wiresharkwhen the client attempts to connect:

Reproducing the error

Now that we can recognize the error on our system, we need to reproduce the error so we can narrow down the problem.

Firefox

Firefox 9.0.1 will perform the entire conversation in SSLv3 which presents no problem to the server:

Internet Explorer

Internet explorer would seem to work from the client perspective as the HTTPS connection issuccessful;however wireshark tells us a slightly different story:

As you can see, it actually tries a TLSv1 connection which fails in the same way as the BizTalk one did, but then simply retries using SSLv3 which is successful.

WFetch

Microsoft provides a tool to test HTTPS connections: WFetch. If we tell it to connect to the server using TLSv1 (labeled TLS 3.1 because SSL 3.1 == TLS 1.0)we get an error as you can see:

Wireshark shows exactly the same pattern as the BizTalk failed connection.
If you toggle the connect option to “https” it will actually work on windows XP because the initial handshake uses SSLv2:

If you toggle it to “https” on windows 2008(and presumably vista/7)it will try the initial handshake with TLSv1 and fail(note that SSLv2 is disabledin windows 2008 due to security concerns).

OpenSSL

With OpenSSL you can also emulate a client (using the s_client option) to test a server:
$openssl s_client -verify 6 -state -msg -showcerts -connect 10.32.54.5:4483 > ssl-output.txt
The output in “ssl-output.txt” shows us that this command uses SSL 2.0 for the initial handshakeand thenproceeds in TLSv1:

However you can force openssl to use tls1:
$openssl s_client -verify 6 -state -msg -showcerts-tls1-connect 10.32.54.5:4483 > ssl-output.txt
Surprisingly, the TLSv1 handshake works for OpenSSL:

The server shows us this:

Java Client

The following code will allow you tomake a rather low level SSL socket connection in java where you can choose the supported cypher suites and protocols:
publicstaticvoidmain(String...args)throwsUnknownHostException, IOException {
SSLSocketFactory factory = (SSLSocketFactory) SSLSocketFactory.getDefault();
SSLSocket socket = (SSLSocket) factory.createSocket("10.32.54.5", 4483);
socket.setEnabledProtocols(newString [] {"TLSv1"});
socket.setEnabledCipherSuites(newString [] {"TLS_RSA_WITH_AES_128_CBC_SHA"});
socket.startHandshake();

Writer writer =newOutputStreamWriter(socket.getOutputStream());
PrintWriter printer =newPrintWriter(writer);
printer.println("GET /invoke/wm.server:ping HTTP/1.1");
printer.println();
printer.flush();

System.out.println(StreamUtils.toString(socket.getInputStream()));

writer.close();
}
There does not seem to be a way to enable or set specific TLS extensionsfor the requestthough.

Analysis of the error

We surmise from the above that:

It is not just a bug in TLSv1 handshakes as OpenSSL works
All Microsoft based software seems to be impacted suggesting they use a central configuration which leads to this error

Analysing the “unexpected error” in wireshark shows us this:

TheRFC (http://www.ietf.org/rfc/rfc2246.txt)does not really shed any more light on the “unexpected message” error.

Comparing the request to the successful OpenSSL TLSv1 handshake, we notice that OpenSSL sends along 27 suggested cipher suites (a selection of them in the screenshot):

As opposed to theMicrosoftclientswhichsend a much shorter list:

None of these suits ismentioned in the JSE6 documentationhttp://docs.oracle.com/javase/6/docs/technotes/guides/security/SunProviders.html#SupportedCipherSuitesbut it may be that there are functional aliases that should work.

Fixing the error

Enable Renegotiation

More about the renegotiation bug can be found in the sources at the end of this document, but hereis a very short summary:

A renegotiation security error was discovered recently (http://www.oracle.com/technetwork/java/javase/documentation/tlsreadme2-176330.html) which triggered an update to both the JVM and to Webmethods to disable renegotiation by default, you can reenable them using the following system properties:
- JVM:sun.security.ssl.allowUnsafeRenegotiation=true
- Webmethods:
  - watt.ssl.iaik.clientAllowUnboundRenegotiate=true
  - watt.ssl.iaik.serverAllowUnboundRenegotiate=true
Renegotiation has to be enabled on both client and server to work

We enabled all renegotiation parameters on the server to no avail, the BizTalk client still failed.While we weren’t able to check if the client had renegotiation enabled, their TLS extensions would seem to indicate that they do:

However, even if enabling renegotiation was the fix, itdoes not negate the inherent security risk in doing so in a production environment.

Upgrade the JVM

Java 7 was released a little while ago and it contains a number of improvements for TLS communication:http://docs.oracle.com/javase/7/docs/technotes/guides/security/enhancements7.html
However Webmethods does not support it so we can’t run it on a production system.Additionally it seems that Webmethods bypasses the default implementation anyway so it may not impact the error.

Add support for the windows ciphers to the existing JVM

Just in case the lack of a common cipher is indeed the problem, we tried adding another security provider: bouncy castle. We addeditto the security file (jvm\w64_160\jre\lib\security\java.security):
#
# List of providers and their preference orders (see above):
#
security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=com.sun.net.ssl.internal.ssl.Provider
security.provider.4=com.sun.crypto.provider.SunJCE
security.provider.5=sun.security.jgss.SunProvider
security.provider.6=com.sun.security.sasl.Provider
security.provider.7=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.8=sun.security.smartcardio.SunPCSC
security.provider.9=org.bouncycastle.jce.provider.BouncyCastleProvider
And added it to the classpath in the server.bat filein the hopes that it would contain a working cipher. No luck there.

Add support for the JVM ciphers to windows

In the same vain, we looked into adding ciphers to the windows machine (as a client-side fix).There seems to be no easy way to add a cipher to windows though the process in C is explained here:http://msdn.microsoft.com/en-us/library/windows/desktop/bb870930%28v=vs.85%29.aspx

Install latest core patch in Webmethods

Although it would seem like the most logical step, installing a core patch for webmethods is also the most invasive solution which is why it is not at the top of our list.
Empower contains (at least) one reference to a handshake failure:https://empower.softwareag.com/sl24sec/SecuredServices/KCFullTextASP/viewing/view.asp?KEY=081375-10412598&DSN=PIVOTAL&DST=TCD&HL=1&QUERY=tls|1.0|handshake&SessionID=608289137
It actually points to a problemfor integration server 7.1.3 instead of 8.0.1, but it advises the installation of the latest core fix to solve the problem.
Installation of core fix 20 fixes the problem though we do still do not know exactly which part of the client handshake triggered the error.

Sources

An incomplete list of sources:

Author: Alexander Verbruggen

Wednesday, February 15, 2012

Invoking stored procedure from DataPower ESB

On a recent project, a few stored procedures had to be exposed as web services. Actually, the stored procedures already took an XML message as input and returned an XML message as output.

Initial plan was to write some .Net components to expose the stored procedures in the SQL Server 2008 database as web services. The web services themselves would then be exposed via a DataPower ESB.

But with the DataPower ESB already in place and being capable of directly invoking the stored procedure (see also), the .Net components and underlying Windows servers could be avoided.

A WSDL was created based on the XML input and output messages. That WSDL was the starting point to create a Web Service Proxy (WSP) on DataPower. Within a rule of the WSP policy, a Transform step (XSLT) is executed that will invoke the stored procedure using a DataPower specific extension function.

A standard Web Service Proxy will invoke a back-end web service. The rule of the Web Service Proxy immediately returns - wihout invoking any back-end web service - by setting the variable service/mpgw/skip-backside to 1.

Below the XSLT used to invoke the stored procedure using the dp:sql-execute function. We obtain the XML part from the soap:Body and pass that on to the stored procedure. The respone XML is wrapped in a SOAP Envelope.
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:dp="http://www.datapower.com/extensions"
xmlns:xxx="http://xxx.com/"
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
extension-element-prefixes="dp" >
<xsl:template match="/">
    <xsl:variable name="statement">exec stored_proc ?</xsl:variable>
    <xsl:variable name="result">
      <dp:sql-execute source="'DS_DB'" statement="$statement" >
        <arguments>
          <argument name="request_id" type="SQL_XML" mode="INPUT">
            <xsl:copy-of
             select="./soap:Envelope/soap:Body/xxx:Operation" />
          </argument>
        </arguments>
      </dp:sql-execute>
    </xsl:variable>
    <soap:Envelope>
      <soap:Body>
        <xsl:copy-of select="dp:parse($result)" />
      </soap:Body>
    </soap:Envelope>
</xsl:template>
</xsl:stylesheet>
Note on the last few lines: the XML response from the stored procedure was contained in a CDATA segment. With the utility function dp:parse() the XML content within the CDATA could be obtained.

Conclusion: one of the great features of Integration tools (ESB, EAI) is their built-in database connectivity. The database adapter of an ESB allows querying, updating and invoking of stored procedures. Database adapters typically also support polling (staging) tables for new or modified records.

Author: Guy

Thursday, January 26, 2012

Running Oracle Weblogic Server on Windows: a shutdown class

Overview

Running Oracle Weblogic server on Windows can be done in two ways:

You can start Oracle Weblogic Server from the command line. This is good for test purposes but not very likely to be used in a production environment.
You can install Oracle Weblogic Server to run as a windows service using the java wrapper beasvc.exe (or beasvcX64.exe on windows 64-bit)

The problem with the last option is that, if you stop the service, the Java Virtual Machine (and thus the Oracle Weblogic Server) simply gets killed. This can result in corrupt files, database connections that aren’t closed or even transactions that are started but won’t be stopped. A possible workaround is to stop each Weblogic server in the domain from the admin console. This not only time consuming but also not the preferred way of working.

In order to provide a proper shutdown, the beasvc.exe has an option called “stopclass”. It specifies a java class with a stop method that will be invoked at the moment the windows service gets stopped. Although you can write any kind of java code in the shutdown class, we will focus on writing code for a decent shutdown for both admin servers and managed servers.

The shutdown class works as follows:

It sends a request over HTTP to the Admin server
The Admin server sends the shutdown request to the corresponding server.

This means that the admin server sends a request to itself in case of a shutdown. The shutdown request for a managed server passes first via the admin server back to the managed server.

Configuration

First of all the beasvc.exe service wrapper needs to be aware of the shutdown class. This can be done by adding the -stopclass option (for example: -stopclass: be.i8c.systemtools.weblogic.ServerStopper)

Secondly, two options need to be set on each server in the domain:

You have to enable tunneling. You do this by ticking “Enable Tunneling” under Domain > Environment > Servers > “Server” > Protocols > General

You have to ignore http sessions during shutdown, otherwise the server keeps on waiting for ever until no more http sessions are active. Enable the option “Ignore Sessions During Shutdown” under Domain > Environment > Servers > “Server” > Control > Start/Stop

Code

package be.i8c.systemtools.weblogic;

import java.io.File;

import java.net.URL;

import java.net.URLClassLoader;

import java.util.Date;

import java.util.Iterator;

import java.util.Properties;

import java.util.Set;

import javax.naming.Context;

import weblogic.jndi.Environment;

import weblogic.management.MBeanHome;

import weblogic.management.runtime.ServerRuntimeMBean;

import weblogic.management.runtime.ServerStates;

public class ServerStopper

{

public static void stop()

{

Properties prop = System.getProperties();

try

{

MBeanHome home = null;

//url of the Admin server

String url = "";

if (prop.getProperty("weblogic.management.server") != null)

{

// this property will be set automatically on a managed server to indicate the location of the admin server

url = prop.getProperty("weblogic.management.server");

}

else

{

// the admin.url is a property to indicate the location of the admin server on the admin server. It refers to itself, but is necessary to perform a proper shutdown. You can add it manually as a –D property when you install the server as a windows service. The name of the property can be any random name.

url = prop.getProperty("admin.url");

}

// username and password to perform the shutdown operation

username = "username";

password = "password";

ServerRuntimeMBean serverRuntime = null;

Set mbeanSet = null;

Iterator mbeanIterator = null;

// Set ContextClassloader to prevent assertions

URL[] urls = { new File("/").toURL() };

Thread.currentThread().setContextClassLoader(new URLClassLoader(urls));

Environment env = new Environment();

env.setProviderUrl(url);

env.setSecurityPrincipal(username);

env.setSecurityCredentials(password);

Context ctx = env.getInitialContext();

home = (MBeanHome)ctx.lookup("weblogic.management.adminhome");

mbeanSet = home.getMBeansByType("ServerRuntime");

mbeanIterator = mbeanSet.iterator();

while(mbeanIterator.hasNext())

{

serverRuntime = (ServerRuntimeMBean)mbeanIterator.next();

try

{

// we have to add this check, since the iterator will list all the servers, both admin and managed. If we don’t add this check a shutdown of all the servers will be performed in case we only stop one server

if (serverRuntime.getName().equalsIgnoreCase(prop.getProperty("weblogic.Name")))

{

if(serverRuntime.getState().equals(ServerStates.RUNNING))

{

// we perform a normal shut down in case the server is in the RUNNING state

serverRuntime.shutdown();

}

else

{

//This is the same as a forced shutdown, but not the same as killing the JVM. We don’t do a normal shutdown in case the server is not in the RUNNING state.

System.exit(10);

}

catch (Exception e)

{

// You can do nothing over here. In case the shutdown call will fail, windows will wait for the service to end, but that won’t happen. It will kill the service anyway after a while.

}

catch (Exception e)

{

// We get here in case the admin server couldn’t be reached

System.exit(10);

}

Author: Dimitri Van Kerckhoven

Friday, January 20, 2012

How to: automatically remove LOCKFILE on server start (WINDOWS)

Have you ever had that annoying issue when your server started up and the LOCKFILE still existed in the webMethods folder?

I’ve created a small script for windows which tackles this issue.

set jobname=softwareAGwebMethodsIntegrationServer_8.0

set location=C:\SoftwareAG\IntegrationServer\LOCKFILE

for /F "tokens=3 delims=: " %%H in ('sc query "%jobname%" ^| findstr " STATE"') do (

if /I "%%H" NEQ "RUNNING" (

IF EXIST %location% GOTO REMOVEFILEANDSTART

)

:REMOVEFILEANDSTART

del %location%

net start "%jobname%"

Variable jobname is your servicename which you set to start automatic

To find this name open Services and double click the job name

The name entered behind Service name is the name you need to enter for this variable.

Variable “location” is the location of the LOCKFILE, enter the full path.

Explanation of the code :

set jobname=softwareAGwebMethodsIntegrationServer_8.0

set location=C:\SoftwareAG\IntegrationServer\LOCKFILE

Declaration of the variables.

for /F "tokens=3 delims=: " %%H in ('sc query "%jobname%" ^| findstr " STATE"') do (

if /I "%%H" NEQ "RUNNING" (

IF EXIST %location% GOTO REMOVEFILEANDSTART

)

Loop over the result of sc query "%jobname%" and search for the word STATE

If the result of STATE is different from RUNNING then execute function REMOVEFILEANDSTART

:REMOVEFILEANDSTART

del %location%

net start "%jobname%"

Delete the LOCKFILE and start the service.

To use this script, create a scheduled task which is triggered on server startup.

Author: Jeroen W.

Sunday, January 15, 2012

Writing a Mule JMX Agent

A client of ours uses Mule ESB mainly as a mediator component that will throttle, translate and call external parties. Some of these parties are however not so reliable when it comes to response times. There were several utilities in use to check what the response time was of a specific third party. All of these mechanisms could only provide a rough estimate however. Since Mule was the system executing the call to the third party, we found that these kind of statistics should come out of Mule instead. Next question was: how to maintain these and get them out of Mule? To expose statistics, JMX seems a logical choice.

Mule ESB comes with a number of JMX Agents: http://www.mulesoft.org/documentation/display/MULE3USER/JMX+Management. These give some great insight in the system. At our client this info is already being used to display statistics in Zabbix, not only from Mule but also from HornetQ JMS. Zabbix is used to monitor the Average service execution times, JMS Queue depth, memory usage, CPU usage and so on.
These agents however do not expose the information that we want so we we need to write our own.

But first we need to collect the statistics For this we created a statistics object that will keep track of the number of calls, minimum, maximum and average duration and also the average of the last 500 calls. This last metric allows us to have an average which does show peaks over time. To capture the call and add the call info to our statistics object we used AOP:

@Around("execThirdPartyCall")
public Object execute(ProceedingJoinPoint pjp) throws Throwable{
   long timeBefore = System.currentTimeMillis();
   Object result = pjp.proceed();
   try{
      getStatistics(pjp).addMuleCallInfo(
         System.currentTimeMillis() - timeBefore);
   }catch(Exception e){
      LOGGER.error("Error adding MuleCallInfo to Statistics Object.",e);
   }
   return result;
}

This statistics object than keeps all this info but does not calculate all the metrics: min, max, avg, avg500. Because adding the call info is on the call stack, we want to keep these methods as short as possible. The actual calculation is done when the info is requested via an MBean.

So now we come to the main part of this post: how to create an MBean and register it to the Mule MBeanServer. I've worked with Spring and MBeans before and I do like the annotation driven mechanisms that Spring offers for auto registrating your MBeans.
Mule however does not use annotations but a method that is more reflection based. The main class in Mule that provides this logic is org.mule.module.management.agent.ClassloaderSwitchingMBeanWrapper. This class needs an interface for introspection and a concrete class in order to create an MBean instance.

The first thing you do is create an interface that will define the MBean attributes and methods. In the example there are 4 read-only attributes. The JMX_PREFIX will be used later on to define the JMX objectName:

public interface StatisticsMBean {
   String DEFAULT_JMX_PREFIX = "type=Thirdparty.Statistics,name=";
   long getHttpCallLast500Average();
   long getHttpCallLast500Minimum();
   long getHttpCallLast500Maximum();
   int getTotalNumberOfCalls();
}

Second, you need to define an implementation of this interface:

public final class StatisticsService implements StatisticsMBean {
    private Statistics Statistics;

    public StatisticsService(Statistics statistics) {
        this.statistics = statistics;
    }
    @Override
    public long getHttpCallLast500Average() {
        return statistics.getAvgLast500HttpCall();
    }

    @Override
    public long getHttpCallLast500Minimum() {
        return statistics.getMinLast500HttpCall();
    }

    @Override
    public long getHttpCallLast500Maximum() {
        return statistics.getMaxLast500HttpCall();
    }

    @Override
    public int getTotalNumberOfCalls() {
        return statistics.getTotalCallsMuleCall();
    }
}

Third you need to create the actual agent. A few things to note about the code below:

the fields are missing and also some methods that are left empty anyway
only the registerMbean part is in here, if you have some fancy hot deploy setup, you'll need to provide an unregisterMBean part
a MuleContextListener is used to make sure that Spring has finished initializing before doing any work
the Statistics object below is an Enum, because we keep statistics for multiple third parties

public final class StatisticsAgent extends AbstractAgent {

   @Override
   public void initialise() throws InitialisationException {
      if (initialized.get()) {
         return;
      }
      //get mbeanserver
      if (mBeanServer == null) {
         mBeanServer = ManagementFactory.getPlatformMBeanServer();
      }
      if (mBeanServer == null) {
         throw new InitialisationException(
             ManagementMessages.cannotLocateOrCreateServer(), this);
      }
      try {
         // We need to register all the services once the server has initialised
         muleContext.registerListener(new MuleContextStartedListener());
      } catch (NotificationException e) {
         throw new InitialisationException(e, this);
      }
      initialized.compareAndSet(false, true);
   }

   protected class MuleContextStartedListener implements
       MuleContextNotificationListener {
      public void onNotification(MuleContextNotification notification) {
         if (notification.getAction() == MuleContextNotification.CONTEXT_STARTED) {
            try {
               registerMBeans();
            } catch (Exception e) {
               throw new MuleRuntimeException(
                   CoreMessages.objectFailedToInitialise("MBeans"), e);
            }
         }
      }
   }

   private void registerMBeans() throws MalformedObjectNameException, 
      NotCompliantMBeanException, InstanceAlreadyExistsException, MBeanRegistrationException {
      Statistics[] statisticsAr = Statistics.values();
      for (Statistics statistics : statisticsAr) {
         ObjectName on = jmxSupport.getObjectName(
                           String.format("%s:%s",
                           jmxSupport.getDomainName(muleContext, false),
                           StatisticsMBean.DEFAULT_JMX_PREFIX + statistics.name())
                          );
         StatisticsService statisticsService = new StatisticsService(statistics);
         ClassloaderSwitchingMBeanWrapper mBean = new ClassloaderSwitchingMBeanWrapper(
                           statisticsService,
                           StatisticsMBean.class, 
                           muleContext.getExecutionClassLoader()
                           );
         logger.debug("Registering StatisticsAgent with name: " + on);
         mBeanServer.registerMBean(mBean, on);
      }
   }
}

As a last step you need to add your agent to the Mule configuration as follows:

<mule>
   <custom-agent class="management.StatisticsAgent" name="statistics-agent"/>
</mule>

And that's it, once you start up Mule, you will see your custom agent appearing and your MBeans are available via JMX.

That is how our operations guys can now provide a nice screen with the call durations over time:

Author: Jeroen Verellen

Tuesday, January 10, 2012

XSLT transformations and smooks driven CSV parsing in JBoss ESB

In this last entry that completes the video series of our JBoss ESB introduction, we explore XSLT transformations and basic smooks parsing.
First up is the XSLT transformation:

Link: http://www.youtube.com/watch?v=vkd12IHJMGY

Parsing flat file into usable XML formats is a rather common requirement for integration platforms. Out of the box JBoss ESB comes with smooks.
The following video shows you how to parse a CSV file using smooks.

Link: http://www.youtube.com/watch?v=Rme4Fv-eqUo

Author: Alexander

Tuesday, January 3, 2012

webMethods Process performance analyzing

We all get in touch with different processes on daily basis. Most of the times there are tools available to monitor if a process ran successfully. But in many cases the performance of these processes aren’t measured before they are deployed to the production environment.

We noticed the same problem with some of our customers. There were processes that should deal with a large amount of traffic. So it was critical that the performance was as good as possible.

How did we get started?

We recognized different steps in the process

Validation
Xslt transformations
Database transactions
Queuing
…

We created SQL scripts to measure the time these process steps ran

Based on custom audit data from the customer
Based on the webMethods audit tables (service execution)

We analyzed the results with our custom excel template

Performance Analyze:

We wanted to see if there were bottlenecks in our process and if the load had any effect on the process.

We noticed that all the steps ran a bit longer on a heavy loaded server than on a dedicated one. This seemed to be normal. Abnormal behavior would be when a server started to freeze and the more instances ran, the longer it would take. The curve you can see below is a normal curve. The average runtime doesn’t start to be longer and longer over time.