Saturday, July 14, 2012

WS-Security mustUnderstand in webmethods

WS-Security mustUnderstand

Environment: webmethods 8.0.1

Problem

A client has asked us to enable ws-security when using a particular webservice he exposes (webmethods policy "Consumer policy for Username"). However when we call his webservice with the appropriate message level authentication, we receive the following soap fault:

Unprocessed 'mustUnderstand' header element: {http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd}Security

Analyzing the problem

Using wireshark/fiddler/.. you can inspect the soap that webmethods sends along and you see this:

...
<wsse:Security xmlns:wsse = "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
      xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:mustUnderstand="1"
      xmlns:wsu = "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
      <wsse:UsernameToken wsu:Id = "UsernameToken-123456">
            <wsse:Username>user</wsse:Username>
            <wsse:Password Type = "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">pass</wsse:Password>
      </wsse:UsernameToken>
</wsse:Security>
...

As you can see, the "mustUnderstand" header is indeed set. Transmitting the entire soap sans mustUnderstand via soapUI works. A quick google showed us that we are not the only ones experiencing this but no solutions were forthcoming. A succinct explanation of mustUnderstand can be found here.

The most likely cause of the issue is a bug in the client webservice implementation where he does interpret the security header but does not clear the mustUnderstand attribute which would result in the error we are seeing.

Solution

The simplest solution is of course for us not to send along the attribute but unfortunately we found no reference in the documentation regarding this particular functionality. Instead we opted to write a custom soap handler which strips the attribute when found. The webservice functionality has been subjected to a large overhaul in 8.0.1 so instead of extending javax.xml.rpc.handler.GenericHandler to add some functionality, you can work with actual services. This makes it much easier to test changes on the fly.

We create a handler service that implements the spec pub.soap.handler:handlerSpec:

The java service DgCommon.admin.utils:removeMustUnderstand uses the public org.w3c.dom api to strip the mustUnderstand:

IDataCursor cursor = pipeline.getCursor();
org.w3c.dom.Element element = (org.w3c.dom.Element) IDataUtil.get(cursor, "header");
element.removeAttribute("mustUnderstand");
IDataUtil.put(cursor, "header", element);
cursor.destroy();

This will however not work and give you a rather vague error:

com.wm.dom.DOMExceptionImpl: [ISC.133.8] Bundle:com.wm.dom.resources.DOMMessageBundle Key:133.8
      at com.wm.lang.xml.ElementNode.removeAttribute(ElementNode.java:606)
...

The exception is not entirely according to the spec so I assume this is an error in the webmethods implementation. Anyway, removing it with a fully qualified namespace does work:

element.removeAttributeNS("http://schemas.xmlsoap.org/soap/envelope/", "mustUnderstand");

You can register the handler using pub.soap.handler:registerWmConsumer (note that this must be re-registered on reboot, so do it in a startup service).

The last step is to actually add it to your list of handlers:

Author: Alexander Verbruggen

Sunday, June 24, 2012

Changing variables during runtime (BizTalk 2010)

For one of our customers I had to come up with a solution for the changing variables and different environment parameters.
For the changing send/receive location we use Deployment Framework for BizTalk, but there are some parameters that cannot be changed during runtime.

One of the proposals was too place the environment variables in the different config-files used by BizTalk. Yes BizTalk uses different config-files, we all know the config-files for 32-bit and 64-bit host instances. But not everyone knows that de isolated hosts and WCF use different ones.
This means that you have to maintain at least four files per server and we all know how quickly people forget things. This and the fact that altering config-files isn’t a best practice, we opted for the use of a database. The data can be changed at runtime and we can make an identical database with different data per environment that all the servers for that environment can access.

The following screenshot describes the table I’m using for this explanation. 

 

By including the application name as a column, we can reuse the same table, stored procedures and the generated schemas by the WCF-SQL adapter. This also eliminates the possibility of identical parameter names by different applications.  Depending on the expected records, you can index the table. We put an index on the ApplicationName together with ParameterName.
For retrieving the data, we're using the WCF-SQL adapter that calls one of two stored procedures. There are 2 stored procedures:
1. usp_GetParameterValueByName: it takes the application and parameter name as an input value and returns the value of that parameter

USE [EbtsServices]
GO
SET
ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[usp_GetParameterValueByName]
      @applicationName nvarchar(10),
      @parameterName nvarchar(10)
AS
BEGIN     
      SET NOCOUNT ON;
      SELECT ParameterValue
      FROM ApplicationParameters
      WHERE ApplicationName = @applicationName
      AND ParameterName = @parameterName;
END

     2. usp_GetParametersByApplicationName: it takes the application name as an input value and returns all the parameters associated with the application

USE [EbtsServices]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[usp_GetParametersByApplicationName]
      @applicationName nvarchar(10)
AS
BEGIN
      SET NOCOUNT ON;
      SELECT ParameterName , ParameterValue
      FROM ApplicationParameters
      WHERE ApplicationName = @applicationName;
END

Don’t forget to grant the correct BizTalk groups execute (with grant) permissions on the stored procedures.  Using WCF-SQL to generate the schemas results in the following three schemas: 
- dbo.xsd  
The first thing to do is to promote the ApplicationName en ParameterName element in the dbo.xsd so that we can alter them inside an orchestration.
This is a response message when using usp_GetParametersByApplication : 
 
 
As you can see, the response consists out of - depending on the number of parameters in the table - multiple StoredProcedureResultSet0 records, which have a ParameterName and ParameterValue element.  
Because BizTalk doesn’t let you promote elements that can occur multiple times, we had to use XPath expression to extract the value.

On the internet there are a lot of blogs that suggest that you count the number of repeated records and then loop x time to extract the parameter value by using the index in the XPath.  This approach has some disadvantages:
               -  you need to add three extra shapes to you orchestration (one expression to determine the count, one loop and one expression insight the loop the extract the values)
               -  It has an impact on performance because it needs to loop over all the StoredProcedureResultSet0, check if the parameter name is the one you expect and then extract the value, even if you don’t use them all.

So I searched for another solution. Just like with an collection it is possible to replace the index by a key value, in this case the parameter name. First I tried to figure out the correct xpath by trail and error. Eventually I went looking for an interesting tool. The lucky winner is Altova’s xml spy. 
I opened the response message and started to experiment until I found the correct xpath expression:
string(/*[local-name()='usp_GetAllParametersByApplicationNameResponse' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo']/*[local-name()='StoredProcedureResultSet0' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo']/*[local-name()='StoredProcedureResultSet0' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/usp_GetAllParametersByApplicationName'] /*[local-name()='ParameterName' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/usp_GetAllParametersByApplicationName' and .=’{ParameterName}’] /../*[local-name()='ParameterValue' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/usp_GetAllParametersByApplicationName'])"))

Now I just replace {ParameterName} with the parameter name (as written in the database) where I want to get the corresponding parameter value form. And then I had to place the expression inside string(), to get the value as a System.String object.
Still this wasn’t good enough for BizTalk, I got an object of type Microsoft.XLANGs.Core.Part+ArrayBasedXmlNodeList, not exactly what I needed. The solution is to use System.Convert.ToString.
The full command to retrieve the parameter value in an orchestration,  parameterValue is of type System.String. 
parameterValue = System.Convert.ToString(xpath(parametersResponse.Body , "string(/*[local-name()='usp_GetAllParametersByApplicationNameResponse' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo']/*[local-name()='StoredProcedureResultSet0' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo']/*[local-name()='StoredProcedureResultSet0' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/usp_GetAllParametersByApplicationName'] /*[local-name()='ParameterName' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/usp_GetAllParametersByApplicationName' and .= ’{ParameterName}’] /../*[local-name()='ParameterValue' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/usp_GetAllParametersByApplicationName'])"));

When using usp_GetParameterValueByName the xpath expression is slightly different:
 /*[local-name()='usp_GetParameterValueByNameResponse' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo']/*[local-name()='StoredProcedureResultSet0' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo']/*[local-name()='StoredProcedureResultSet0' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/sp_GetParameterValueByName']/*[local-name()='ParameterValue'])"))

It is still necessary to put the expression inside of string() and do a conversion to System.String.



author: Martijn

Tuesday, June 19, 2012

Common error when connecting to IBM Websphere MQ from ASP.NET


Problem

When connecting to an IBM WebSphere MQ queue using the WebSPhere MQ classes for .NET from within an ASP.NET application, you might run into the following error:

General Exception: The type initializer for 'IBM.WMQ.MQQueueManager' threw an exception.


When investigating the error a bit further we find the following error in the Windows EventViewer:

An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/1/ROOT/HealthCheckMQSeriesService
Process ID: 4972
Exception: System.Reflection.TargetInvocationException
Message: Exception has been thrown by the target of an invocation.
StackTrace:    at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
   at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache)
   at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipVisibilityChecks, Boolean skipCheckThis, Boolean fillCache)
   at System.Activator.CreateInstance(Type type, Boolean nonPublic)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture)
   at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)
   at IBM.WMQ.CommonServices.CreateCommonServices()
   at IBM.WMQ.CommonServices.TraceEntry(String objectId, UInt32 component, UInt32 method, Object[] parameters)
   at IBM.WMQ.MQQueueManager.Dispose(Boolean disposing)
   at IBM.WMQ.MQQueueManager.Finalize()
InnerException: System.DllNotFoundException
Message: Unable to load DLL 'amqxcs2.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)
StackTrace:    at IBM.WMQ.MQCommonServices.xcsInitialize(UInt16 scope, UInt16 attributes, Byte[] charName, Byte[] charPrefix, xcsHPOOL& hPool)
   at IBM.WMQ.MQCommonServices.Initialize()




Solution

The solution is very simple.
In IIS go to the advanced settings of  application pool your application is running under.
Be sure that the “Load User Profile” property is set to true.


Run your application again and you should notice that the error disappeared.

Author: Christophe Verschuere


Wednesday, May 2, 2012

Test Driven Development on Oracle Service Bus

As a Java developer I have learned to appreciate Test Driven Development (TDD). It helps you to deliver quality software solutions and it gives you confidence when you make changes later on. Your test code will help you to make sure that no regression arises when changing your production code.
An intro on TDD: http://www.agiledata.org/essays/tdd.html

When working with ESB products you work on a different abstraction layer. You are more focused on configuring a message flow, detailing how a specific type of message needs to be routed, transformed, filtered and delivered. You are not as close to the code as a java developer hence it is more difficult to test the complete flow. In addition to that, ESB products are very focused on connectivity to for example an Oracle Database or a SAP backend system. This makes it even more difficult to test because it requires these external systems to be available during testing and also to produce predictable results.

Oracle Service Bus (OSB) is such an ESB. I will not go into depth about the product here, but important to know is that it is very much oriented towards Web Services and XML. These standards will help us to test our integration logic.

In the below picture you can see the setup for our tests. The following steps happen during a test run:

  • A test request is read from a file and sent using a HTTP client to OSB
  • OSB runs the integration logic as it would for any request
  • The business service it invokes is configured with a test endpoint. An embedded HTTP server is used for this purpose.
  • The test endpoint can assert the incoming request, wait for a configured period of time and return a configured response.
  • OSB will return a response to the test client as well, which can then in its turn assert the response.

Java libraries used in this setup: JUnit, XMLUnit, Hamcrest, Jetty HTTP client and server.

Some of the code for executing this test:



First question to expect is of course: why the custom code and not SOAP UI?
Indeed you can use SOAPUI as SOAP client and SOAP mock server. The reasons for not doing this is because you have a lot more control. When I start up a test I can change the behaviour of the mock server:

  • I can choose to let it return different responses
  • I can let it wait so that OSB will not have a response in time; allows me to test the exception handling and retry mechanisms.

I can also assert the request that comes from OSB to make sure it is as expected. In case I have to call multiple business services I can have one service return a sucessfull response and have another one time out.
All this more complex testing is a lot harder, if not impossible, to configure using SOAP UI.

Author: Jeroen V

Friday, April 27, 2012

Microsoft Installer Custom Actions User Impersonation

Problem

When creating an installer with custom actions you might run into some security issues when executing it on a Windows Vista/7/2008 or later OS.
This is because custom actions will be executed in the context of the user running the Windows Installer Service being the SYSTEM user.
This behavior is enforced from Windows Vista on and can give you authorization problems for certain tasks you want to perform in your custom actions.

Solution

To overrule this default behavior and run the custom actions as the impersonated user that is executing the MSI we will have to flip the msidbCustomActionTypeNoImpersonate bit that is on by default.
In a Visual Studio Setup project it is however not possible to set this flag through the Properties Window.
To solve this we must create a post-build script that will flip this bit:
// CustomAction_Impersonate.js <msi-file>
// Performs a post-build fixup of an msi to change all deferred custom actions to Impersonate
// Constant values from Windows Installer
var msiOpenDatabaseModeTransact = 1;

var msiViewModifyInsert         = 1
var msiViewModifyUpdate         = 2
var msiViewModifyAssign         = 3
var msiViewModifyReplace        = 4
var msiViewModifyDelete         = 6

var msidbCustomActionTypeInScript       = 0x00000400;
var msidbCustomActionTypeNoImpersonate  = 0x00000800

if (WScript.Arguments.Length != 1)
{
       WScript.StdErr.WriteLine(WScript.ScriptName + " file");
       WScript.Quit(1);
}

var filespec = WScript.Arguments(0);
var installer = WScript.CreateObject("WindowsInstaller.Installer");
var database = installer.OpenDatabase(filespec, msiOpenDatabaseModeTransact);

var sql
var view
var record

try
{
       sql = "SELECT `Action`, `Type`, `Source`, `Target` FROM `CustomAction`";
       view = database.OpenView(sql);
       view.Execute();
       record = view.Fetch();
    //Loop through all the Custom Actions
       while (record)
       {
           if (record.IntegerData(2) & msidbCustomActionTypeInScript)
           {
               //We must flip the msidbCustomActionTypeNoImpersonate bit only for deferred custom actions
               record.IntegerData(2) = record.IntegerData(2) & ~msidbCustomActionTypeNoImpersonate;
              view.Modify(msiViewModifyReplace, record);
           }
        record = view.Fetch();
       }

       view.Close();
       database.Commit();
}
catch(e)
{
       WScript.StdErr.WriteLine(e);
       WScript.Quit(1);
}

This script file (CustomAction_Impersonate.js) must be placed in the same folder as your setup project (Setup.vdproj) and add the following PostBuildEvent in your setup project:
cscript.exe "$(ProjectDir)CustomAction_Impersonate.js" "$(BuiltOuputPath)"

Build and run your setup project and you will notice that all the custom actions will now run as the impersonated user that executes the MSI.

Author: Christophe

Monday, April 23, 2012

General Access Denied error (0x80070005) in Hyper-V Manager when starting VM

I ran into the following error when I tried to start a VM in Hyper-V Manager recently:



To solve this issue I had to grant Full control to the folder containing all the VM's files (Virtual Hard Disks, config,...) using the following command:

icacls <VM directory> /grant "NT VIRTUAL MACHINE\0BA74464-10AF-4709-AAE9-4C1B196C08ED":(OI)(CI)F /t

After executing this command you can verify if it was successful by checking the Security properties of the directory where the VM files are located through Windows Explorer. The user that corresponds with the Virtual machine ID in the error message should have Full control of the folder where the VM files are located.




Author: Kristof Lievens

Tuesday, April 17, 2012

Graceful shutdown of a webMethods Integration Server through custom developed java program

Out of the box a webMethods Integration Server is shutdown via the Administrator screen.

Following code snippet with allow you to gracefully shutdown a webMethods Integration Server through the usage of a custom developed java program.

Java code snippet :
import java.io.*;
import com.wm.app.b2b.client.*;
import com.wm.util.*;
import com.wm.data.*;

public class IntegrationServerShutdown
{
            public String hostName = null;
            public String port = null;
            public String userName = null;
            public String password = null;
            Context context = null;
            public boolean connected = false;
            public static void main(String[] args)
            {           String input = null;
                        //Create an instance of a b2bServer
                        IntegrationServerShutdown b2bServer = new IntegrationServerShutdown();

                        b2bServer.hostName=args[0];
                        b2bServer.port=args[1];
                        b2bServer.userName=args[2];
                        b2bServer.password=args[3];
                        System.out.println("");
                        b2bServer.connect();
                        if(!b2bServer.connected)
                                    System.exit(0);
                        b2bServer.shutdown();

            }
            public void connect()
            {           context = new Context();
                        try
                        {           context.connect(hostName + ":" + port, userName, password);
                                    System.out.println("Connected to " + hostName + ":" + port);
                                    connected = true;
                        }
                        catch(ServiceException e)
                        {           System.out.println("Could not connect to " + hostName + ":" + port);
                                    System.out.println(e.toString());
                                    connected = false;
                        }
            }
            public void disconnect()
            {           if(context != null)
                                    context.disconnect();
            }
            public void shutdown()
            {           try
                        {
                                    System.out.println("Shutting down IS Server... ");
                                    IData inputs = IDataFactory.create();
                                    IDataCursor myCursor = inputs.getCursor();
                                    myCursor.insertAfter("bounce", "no");
                                    myCursor.insertAfter("timeout", "0");
                                    myCursor.insertAfter("option", "force");
                                    context.invoke("wm.server.admin","shutdown",inputs);
                                    System.out.println("IS Server shutdown complete.");
                                    System.exit(0);
                        }
                        catch(ServiceException e)
                        {           System.out.println("IS Server shutdown failed.");
                                    System.out.println(e.toString());
                                    System.exit(1);
                        }
            }

}

Remarks :

  1. When compiling and running the java program make sure to include files wm-isclient.jar and mail.jar in your class path. File wm-isclient.jar can found in the “common/lib” folder of an Integration Server whereas file mail.jar is localized in the “ext” sub folder of the “common/lib” folder.
  2. Usage of the IntegrationServerShutdown program is IntegrationServerShutdown <IS hostname> <IS port> <IS account with administrative privileges> <IS account password>
  3. At this moment the java program is working with at least webMethods Integration Server v7.1.2 and v8.2. Specifications contained herein are potentially subject to change so use them at your own risk.
Author: Johan De Wulf