Best wishes!

For application integration, things look very promising according to Gartner: "organizations will spend 33% more on application integration in 2016 than they will in 2013".  That's the main message of Gartner's report "Predicts 2013: Application Integration".

 

Stephanie Mann of Techtarget reports on that same message being communicated by Gartner at their AADI conference, some interesting snippets from her article:

• "What you want me to say is that cloud APIs [application programming interfaces] are solving your problems, and that REST is the answer," Lheureux told his audience. "But we're not just automating the process and sending messages. We're looking at actually collaborating more at the process-execution level."
• "More than 50% of the cost of implementing new systems will be spent on integration in the next five years," Schulman said. "Our architectures are obsolete; the way we approach integration is obsolete; and the way we think about integration development is increasingly obsolete."

Best wishes for 2013!

WebMethods SalesForce Adapter

With the upcoming services in the cloud, integration with and in the cloud is a hot topic. One of these cloud services is SalesForce. WebMethods comes out of the box with a solution to connect Integration Server with the SalesForce platform. To do so it makes use of the SalesForce adapter.

Version 7.1.1 of the adapter allows us to connect to SF using the SOAP API version 23. It is not supported to make a connection using another version of the API. This will be possible on the future webMethods 9.0 release using CloudStreams.

The adapter allows us to perform the following operations, using webMethods services:

• Create
• Delete
• Query (only for queries on a single object)
• Retrieve (only for queries on a single object)
• Update
• Upsert
• Utility: getServerTimestamp, getUserInfo, resetPassword, sendEmail, sendMassEmail, setPassword

It is also possible to retrieve upsert and delete notifications from the adapter.

One of the biggest advantages of the adapter is transparency. You only have to enter the SF endpoint, a username and a password and the adapter will keep track of the login procedure, session management and session renewal. Clustering and connection pooling are possible since the adapter is incorporated in the adapter runtime framework.

Besides the advantages the adapter has some disadvantages. Not every SalesForce operation is supported. Non-supported operations are convertLead, emptyRecycleBin, merge, process, search and undelete.

It’s also not possible to call custom web services on SalesForce using the adapter. To do so you have to make use of the standard web service descriptor functionality. This means you need to connect manually and keep track of your session.

Complex queries are not possible. The adapter allows you to select a single object, and then you select the fields of the object you want to get as output. Selection criteria can be added too (some kind of where clause), but queries over multiple objects or sub queries are not possible.

There is also a version 8.2 of the SF adapter. It requires at least version 8.2 of the Integration Server but addresses some of the 7.1.1 shortcomings:

• convertLead is now a service in 8.2
• there is now a custom query service that allows you to type in any SOQL query
• all the services mentioned in the report can indeed be invoked using regular Web Services descriptors
• the 8.2 version also has a service that gives you the SF.com token, which can be used with the Web Service
• descriptors to invoke these services along with custom web services on SF.com

Setting up an intelligent load-balancer group in IBM Datapower

The Datapower appliances offer a lot of functionality, but not all configuration options have clear documentation. In this blog I’ll present a typical setup for an intelligent Load Balancer Group.   

What we want to achieve in this setup is an intelligent load balancer that can do fail-over, but that excludes the members which are known to be in 'down' state from these fail-over tries.
In case none of the members is considered ‘up’, we want an immediate error-response and no extra waiting times for retries to backends that will probably fail. One of the reasons for such an approach might be that we want to isolate problems with a specific backend and make sure our Datapower appliance is not affected by a huge number of open connections (e.g. due to to a very high load and a relatively large time-out value). If the amount of open connections keeps on rising we will eventually reach the boundaries of the appliance and start receiving out-of-memory errors...

First of all a short explanation on the different states of a member of the load balancer group:  
- ‘up’: this is the default state of a member of the load balancer when it is added, but this state can also be reached when a member that was ‘down’ succeeds a health check. 
- ‘down’: this state can be reached when a health check on this member failed or when the member is manually disabled. 
- ‘softdown’: aka dampened, when an actual request reaches a connection error, the state of the member that caused the error is set to ‘softdown’ for a period specified in the ‘damp time’.

Healthcheck
After adding the members to the load balancer group we are going to add a health check. The default health check sends an empty soap-request and validates the received response against the entered xpath. Since the default xpath is ‘/’ the health check will be valid if any xml-response is received.
 
One option needs some extra attention: the ‘Timeout’ value. This value represents the amount of seconds before the health check fails with a time out. However, since the health check request is done using a user agent object, in reality the time out of the user agent is the one that is used.

Main load balancer group settings

The options that need some extra clarification here are ‘Do Not Bypass Down State’ and ‘Try Every Server Before Failing’.
- ‘Do Not Bypass Down State’: This property should ensure that when all members of the load-balancing group are down and a new request for that group is made, no attempt will be made to connect. However you should note that the load balancer is only behaving like this when this property is set to ‘On’ AND the property ‘Try Every Server Before Failing’ is set to off.
- ‘Try Every Server Before Failing’: As stated above, this property overrules the ‘Do Not Bypass Down State’ when it is set to ‘On’. The Datapower help-file will tell you the following:
This text is a bit confusing, since in reality we observed the following behavior: not only dampened and healthy members, but also unhealthy members are tried. This can cause some serious problems in case of time-out errors on multiple members, since each member adds this time-out to the total execution time... When this property is set to ‘Off’, the behavior observed is that in case of a failed connection attempt, the next member in state ‘Up’ is tried.

For the intelligent load balancer that we want to create we need to set the property ‘Do Not Bypass Down State’ to ‘On’ and the property ‘Try Every Server Before Failing’ to ‘Off’ as can be seen in the screenshot below.
Conclusion
With only a few configuration steps we can create an intelligent load balancer. The load balancer actively executes health checks with SOAP-calls and maintains a list with the status of its members. When a connection error occurs during an actual call from a client it automatically retries to make a connection with one of the known healthy members. When no member is ‘Up’ the call will directly fail and therefore prevent an exponential rise of open connections in case of a high load.

Author: Tim

Getting started with NServiceBus, Request-Response example

In this blog post we explain some request-response example that we mentioned in the previous blogpost.

Below you can see the first steps to do, before we really can start with sending some messages.

  - Go to http://www.nservicebus.com à Downloads
  - Download the msi file under the “Get version 3.3.1 now” button.

 

   - Install the msi file on your system.
   -  Now you can find under “All programs > NServicebus v3.3.1” a Welcome to NServiceBus link.

You can click on the samples link to go to the samples. Or you can choose a sample below. We will talk about the “FullDuplex – RequestResponse” sample. Copy past the location of the sample and open it with visual studio. You will get the following solution.

The client is going to send a message to the server, the server will process the message and return it back to the client.

Let’s run the server and client application. You should see some screens like below.

Now we can exchange some messages between server and client. Let’s hit the “Enter” button in the Client Console. The client sends a message to the server with a Guid, which is received by the server and the server sends a message back to the client.

As you can see, this is not a real life scenario. What if the client is down? What if the server is down.

Let’s now start only the client application.

As you can see we send a message from the client to the server, but we don’t get some reaction back from the server because we didn’t start the server application. You don’t lose the message because NServicebus uses MSMQ and the message stays in the queue.

When we look in the MSMQ queue, you can see that there is a message.

Now we open the message, there you can find the DataID in the message. This ID is the same as on the Client application.

Now we can start the server application.

You can now see that the server picks up the message to process it. And send it back to the client.

I hope you enjoyed reading this blogpost.

Author: Sven Van den brande

What is NServiceBus

NServicebus is designed for collaboration between different services. It’s not just a replacement for WCF, BizTalk, ….. You can easily combine NServiceBus with BizTalk.

For BizTalk, the communication goes through one central box. We can call BizTalk a Broker style. All the information goes through BizTalk itself.

NServiceBus is something different. It’s more a peer-to-peer architecture that works on the message queues from Microsoft ( MSMQ ). So NServiceBus will do all the communication between systems with MSMQ. When you handle your message you still need to do the further communication.

When one of the “servers” go down because some hardware failure,... the communication between the other systems continues. This is some key feature for NServiceBus because you can keep sending messages. When the systems comes back online after the repair. It will get all the messages from the other systems, because they hold their  messages in the queues .

                                       

Now you know the basic difference between some other Integration tools. Like always a demo/example says more than thousands words.  So in the next blog post we will setup, the NServiceBus Environment and let you see some simple example about Request / Response.

Do you want to read more about NServiceBus, stay tuned!

Author: Sven Van den brande

Getting a glimpse of the Webmethods Integration Server OSGi-container.

After having followed an excellent presentation @Devoxx (Modular Architecture Today by Kirk Knoernschild), I decided to dive a bit into the OSGi container of SoftwareAG's Integration Server.

As documentation on this is scarce, the following information is not official and just bits and pieces I picked up, or investigated myself. The OSGi container was introduced in version 8.2.2, to cope with new components such as the Event server.

What SAG decided to do with the OSGi runtime, was to run the IS as one bundle and other components (such as the event server or common components) as other bundles. As runtime they choose the equinox implementation from Eclipse. This implementation comes with an easy to use console which can show us all the (active/loaded) bundles. This will be the goal of this article to get the console running and show us the bundles SAG is loading.

We're in luck: all we need to do is add "-console [port]" to the start-up of the OSGi-container. If you think you can find it in server.bat, you're half way there...this file now points to the OSGi-container, which you can find under [your_install_dir]\profiles\IS. We are interested in the start-up, so we will open "bin\runtime.bat". There you will find:

%JAVA_RUN% -Xbootclasspath/a:"%OSGI_CLASSPATH%" %JAVA_OPTS% %JAVA_SYSPROPS% -cp "%OSGI_FRAMEWORK_JAR%" org.eclipse.equinox.launcher.Main -configuration %OSGI_CONFIGURATION_AREA% %CMD_ARGS%

Just change it to:

%JAVA_RUN% -Xbootclasspath/a:"%OSGI_CLASSPATH%" %JAVA_OPTS% %JAVA_SYSPROPS% -cp "%OSGI_FRAMEWORK_JAR%" org.eclipse.equinox.launcher.Main -configuration %OSGI_CONFIGURATION_AREA% %CMD_ARGS% -console [port]

with [port], the port which you want to use to access the console.

Now just launch the server.bat. In your prompt, you should see something like: "Listening on port [port] ...". So lets now connect to the console, just open a telnet session to the port and your there:

telnet localhost [port]

Next type "ss" and you should get a list of all the bundles that are loaded:

Some useful commands are listed below:

start	Starts a bundle given an ID or symbolic name
stop	Stops a bundle given an ID or symbolic name
install	Adds a bundle given a URL for the current instance
uninstall	Removes a bundle given a URL for the current instance
update	Updates a bundle given a URL for the current instance
active	Lists all active bundles in the current instance
headers	List the headers for a bundle given an ID or symbolic name
ss	Lists a short status of all the bundles registered in the current instance
services <filter>	Lists services given the proper filter
diag	Runs diagnostics on a bundle given an ID or symbolic name

Hopefully this could prove helpful, especially when SAG extends its use of OSGi.

Author: Stefan

SAP NetWeaver Cloud Integration

During the TechEd conference, SAP has released more information about their new Netweaver Cloud Integration offering, that is the "Integration-As-As-Service" offering from SAP.

From this presentation we learn that Netweaver Cloud Integration is not "SAP PI in the cloud", but comes very close to it.  Netweaver Cloud Integration is fully compatible and integrated with the on-premise version of SAP PI. Same development, configuration and monitoring approach.

Also interesting is the blog entry of Prasanna Burri where he talks about using Netweaver Cloud Integration with SuccessFactors.  In his answers to questions, Prasanna mentions an availability date of Feb. 2013.

Author: Guy

IBM DataPower as ESX appliance

i8c does quite some work with IBM WebSphere DataPower.  And counts a number of experienced and certified DataPower developers and architects.  DataPower is great at securing and mediating web services.

Colleague Kim came up with a very interesting evolution: DataPower as a virtual appliance, so the XG45 gateway and XI52 integration appliance as ESX images.

This is similar to the CastIron offering of IBM that comes with on-premise hardware and virtual appliances.  Right now we have a CastIron instance running on our ESX server (i8c employees can deploy their images either on an ESX server or a Windows server running HyperV).

Note: together with Joris Verberckmoes (Lead Designer SOA Services @ Gdf-Suez) I'll be giving a presentation at the Belgian WebSphere User Group on Nov 27; topic is the use of DataPower @ Gdf-Suez

Author: Guy

Web Service incompatibilities

SOAP Web Services have lost quite some of their popularity: too complex, incompatibilities etc.  My answer is always that 1) SOAP just adds a very simple envelope around the request and response messages and 2) SOAP does work fine when you stick to the rules (a copy of a slide I use in my training classes):

Just recently I had encountered 2 nice examples of SOAP incompatibilities.
 

Cookies and SOAP

While investigating the web services API of a cloud SAAS application, encountered another example how things should not be done.  First of all it was not "stateless" but required the use of a login and logout operation. With security not based on standard HTTP basic authentication or WS-Security, but a proprietary scheme:

  <urn:credential>

    <urn:companyId>company-id</urn:companyId>

    <urn:username>user-name</urn:username>

    <urn:password>password</urn:password>

  </urn:credential>

But then came the surprise: the login operation returns a session handle which is actually a cookie! The cookie is to be passed as an HTTP header in each subsequent web service.  Had seen many ways to make web service implementations incompatible, but this is one for the top 5!  Obviously most web service clients require some hack to pass this cookie along the SOAP request.
 

Doc/literal with 2 parts

A more subtle challenge came recently by at a customer: the IBM DataPower ESB refused to import the WSDL file an Oracle product.  The web service used the document/literal style and one of the operations had a request message consisting of 2 parts.  So who was wrong and who was right: IBM or Oracle?

SOAP went through some growing pains in the beginning. The initial idea was an RPC mechanism whereby an operation could have multiple parameters. These parameters are passed as multiple parts in a request and response message. But with a better understanding of XML and XML schema's, the world move to a model whereby XML documents were passed. Microsoft introduced the document/literal wrapped style whereby the root contains the name of the operation.

<soap:Envelope xmlns:soap="http://www.w3.org/2001/12/soap-envelope">

  <soap:Body>

    <OperationName>

      actual XML document...

    </operationName>

  <soap:Body>

</soap:Envelope

</soap:Envelope>

So my initial response was, document/literal web services should only have one part and Oracle is wrong. But a colleague pointed to the fact that Oracle would not implement web services that violate the standards. And indeed, the IBM article clearly explains that a document/literal web service can have multiple parts in a message.

The WS-I Basic Profile was an initiative to sharpen the rules and states: "R2201 A document-literal binding in a DESCRIPTION MUST, in each of its soapbind:body element(s), have at most one part listed in the parts attribute, if the parts attribute is specified.". So the Oracle web service is not WS-I basic compliant but does not violate the SOAP/WSDL specifications.

Again a situation where one has to go for workaround, this time in the DataPower ESB. Had IBM implemented the specs correctly and/or Oracle stuck to the widely accepted ways-of-working and the WS-I Basic profile, everything would have worked smoothly.

 

Author: Guy

Live from Software AG Processworld 2012: Day 2

On day 2 of ProcessWorld I attended some sessions that were focusing on the 9.0 release. The first one was about a general introduction to webMethods 9.0. It started with a short overview of the new products: webMethods cloudStreams, webMethods Pulse, Command Central, ActiveTransfer, etc. After that they dived into the new functions of the existing components like IntegrationServer, Broker, Optimize, etc. The most important ones are:

For integration server:

• Support for webservice reliable messaging, SFTP and JSON. 
• Worry-free updates with maintenance mode support. You can put the integration server in maintenance mode to perform updates. 
• Local development. 
• BigXML and service caching thanks to the implementation of TerraCotta. 
• WebMethods Command Central Integration. 

For Trading Networks: 

• Community management & partner on-boarding 
• Integration of the new webMethods ActiveTransfer product. 

For BPM: 

• Improved architecture by OSGI enablement of the my webMethods Server. 
• Event-enabled BPM processes can feed webMethods Pulse, Business Events & Mashzone. 
• BPMN 2.0 support for sub-processes & debugging 

Optimize:

• Improved usability & management of KPI data. 
• Enhanced dashboards with MashZone. 
• Improved reliability. 
• WebMethods Command Central Integration 

CentraSite: 

• New business UI for occasional & non-technical users 
• Easy keyword search 
• Event-based automatic promotion across life-cycle states 
• Simplified infrastructure 
• WebMethods Command Central Integration 

Nirvana messaging: 
Something I forgot to mention in my previous blog post is that Nirvana also supports priority messaging and message throttling for bandwidth limitations.
 

Important to mention are the following system requirements for the whole webMethods 9.0 suite.

• There will be ipV6 support 
• Only 64-bit operating systems will be supported 
• The standard JDK is Java 7 
• The ability to define checkpoints & the rollback of deployments  

Another new webMethods product that I didn’t mention yesterday is webMethods ActiveTransfer for Managed File Transfer. There are two major components: ActiveTransfer server & ActiveTransfer proxy. The ActiveTransfer server needs to be installed on an IntegrationServer in the trusted zone of your IT infrastructure. The server part allows you to:

• Configure and manage all server instances centrally. It allows you to define IP banning rules and/or filename or folder restrictions. 
• Manage users and their access privileges by prioritizing file transfers, restrict file transfer actions and inheritance of user settings from templates. 
• Perform post-processing and alerting by the definition of file transfer criteria based triggers. Based on these triggers IntegrationServer services can be invoked or native scripts can be called. 
• Perform scheduled and event-driven transfers 
• Accelerate file transfers in high latency environments by definition of multiple active transfer tunnels. This allows file transfers speed up to 20 times faster. 
• Suspend and resume file transfers by the definition of checkpoints 
• Provide centralized monitoring to keep track of all transfers.

 

The ActiveTransfer proxy server part runs on a reverse gateway IntegrationServer in the DMZ and communicates with third parties.

Author: Dimitri Van Kerckhoven