Using SO-Aware with BizTalk 2010 and ESB 2.1 part 2 of 2

As a quick start, for those that don’t know, SO-Aware is a service metadata repository, very similar to UDDI, except without the complexity, all based on a REST-ful architecture. If you do know, you can skip to using SO-Aware with BizTalk 2010 and ESB 2.1 section. It supports registry of SOAP, REST, and OData based services, written using Microsoft’s WCF technology stack, as well and Java based web services. However, that’s not all. SO-Aware has five major capability buckets: Centralized Configuration, Service Testing, Dependency Modeling, Activity Monitoring and last but not least Service Cataloging.

Centralized Configuration allows you take full advantage of Microsoft’s WCF based services, by containing a central repository database for all WCF Configurations. This feature allows you to allows you to store and retrieve all information pertaining to WCF configurations. You can retrieve endpoints, bindings, behaviors, Url’s, security information, just to name a few. You can also dynamically change your bindings and configurations so that all you existing services can point to this central location for your configuration.

Service Testing allows you to test registered services. One derived idea about Service Cataloging and Centralized Configuration, is that if SO-Aware knows about your service and communication protocol, then testing becomes simple. It’s simple because if you registered any security, binding, message type format information about the service, then all SO-Aware needs to do is query this information and build the communication stack and messages to send to the service for testing. What better tool to test than the one that understands how to communicate with the Service.

Dependency Modeling allows you to build a diagram of service versions, and the dependencies of the service version. Thus if you needed to see which services depended each other, you have a view into this.

Activity Monitoring allows you see tracked events and aggregations about registered services, such which operations were invoked, and how many services were sent message over a period of time, and many other dimensions and measurements.

Service Cataloging allows you to store and retrieve custom metadata about Services, Service Versions, and environment details. Using these features, allow you to query the catalog for information about services, and service versions dynamically.

Which leads us into the next discussion, using SO-Aware with BizTalk 2010 and ESB 2.1. The first part of this post covered how SO-Aware can use its Service Cataloging, Centralized Configuration, Dependency Modeling, and Dynamic Resolution capabilities to enhance BizTalk server 2010. This post covers how we can use SO-Aware with BizTalk’s ESB 2.1toolkit.

What is BizTalk ESB 2.1?

The BizTalk ESB Toolkit is a collection of tools and libraries that extend BizTalk Server capabilities of supporting a loosely coupled and dynamic messaging architecture. It functions as middleware that provides tools for rapid mediation between services and their consumers. Enabling maximum flexibility at run time, the BizTalk ESB Toolkit simplifies loosely coupled composition of service endpoints and management of service interactions. The ESB Toolkit contains many components that allows BizTalk Server solutions to be agile in its execution. For example, BizTalk Server contains Adapters which are connectors into different systems and protocols. It also contains Maps, which are Xml stylesheet transformation documents that have been precompiled to transform documents from one format into another. There are Pipelines which are components that allow you to translate and convert one file into another such as a Adobe PDF document into a comma seperated text file. There are also Schemas which support validation of different messages, both xml and non xml. Lastly there are Orchestrations which are graphically modeled business processes that support long running transactions, compensation models, error handling, correlation, and custom logic flow.

Adapters in BizTalk are one of the key components for communicating with BizTalk Server. There are sending adapters and receiving adapters. There’s an adapter for practically every protocol that exists such as ftp, http, smtp, tcp, ftps (new in 2010), Msmq and others. There are application type of adapters that adhere to the rules and policies of Line of Business applications, such as Sharepoint, SAP, Oracle E-Business Suite, and other CRM applications. There are even adapters for databases such as SQL, Oracle, and DB2. When developing solutions with BizTalk Adapters are usually configured statically and there’s little room for agility. Thus BizTalk supports the notion of Dynamic adapters for sending data. These adapters are configured at runtime, which will yield great agile solutions, at the sacrifice of High performance. , naturally. Using Dynamic adapters requires knowledge of which properties must be set for a particular adapter, and custom code inside of Orchestrations and/or custom pipeline components. This is where some aspects of the ESB 2.1 toolkit come in. The ESB 2.1 toolkit contains Adapter Providers which can dynamically configure adapters for use in a agile solution. All the Adapter provider needs is the runtime configuration values.

The ESB toolkit utilizes a component called Resolvers which feed the Adapter provider the runtime configuration values so that the Adapter Provider can dynamically configure the Dynamic Adapter at runtime. Resolvers can not only discern which Adapter runtime configuration to use, but also which Map, Pipeline, Schema, and Orchestration. Resolvers are simply a .Net component which build up a dictionary collection of entries which then tell other ESB components what values to use for the various BizTalk Components mentioned earlier.

Using Resolvers dramatically changes the way architects and developers can build, design and implement BizTalk Server 2010 solutions. As a quick example, most designs of BizTalk prior to the ESB toolkit hard coded, and statically configured maps. A Map contains a Source schema associated with an xslt transformation, and a destination schema. If either the source, destination or transformation stylesheet changes, it can potentially render the map invalid and cause all kinds of errors. Using a more agile design allows architects to use different versions of maps at runtime, and even dynamically pre-compile a auto generated map all at runtime.

The ESB Toolkit not only contains Resolvers, but also Routing Services, Transformation Services, Itinerary Services, Exception Management services and a portal. With exception or the Error portal, Resolvers play an integral role, communicating with the Routing Services, Itinerary Services and Transformation Services. All this to say Resolvers are the key compoonent of the ESB toolkit. The ESB Toolkit comes with many Resolvers: Static, Business Rules, XPath, UDDI, WSMEX, and LDAP, all of which allow for the dynamic resolution of Maps, and adapter providers.

Which leads us to SO-Aware…. SO-Aware provides an ESB Resolver to allow you to dynamically resolve Adapter providers, Maps, schemas, pipelines, orchestrations, WCF Service: Bindings, Versions, Behaviors and other configurations. Inside the SDK, you can find samples on how to use the SO-Aware resolver which show how to resolve Adapter providers, WCF Service: Bindings, Versions, and Behaviors. The SO-Aware resolver and database can be easily extended to resolve Maps, schemas, pipelines and orchestrations.

Using the SO-Aware ESB Resolver

After installing the SO-Aware SDK, you can find the Solution and samples inside the C:\Program Files (x86)\Tellago Studios\SO-Aware\SDK\Samples\BizTalkESB\vs.net2k10\. You can install the SO-Aware resolver by running the setup on a BizTalk 2010 ESB 2.1 system. The Install can be found here: C:\Program Files (x86)\Tellago Studios\SO-Aware\SDK\Samples\BizTalkESB\vs.net2k10\Tellago.SOA.ESB.Extensions.Solution\Tellago.SOA.ESB.Extensions.Setup\Debug\setup.exe .

After installing the SO-Aware ESB Resolver, you can use the Resolver inside of Business Rules, Itineraries, Orchestrations, or pipeline components, basically anywhere resolvers can be used. Below is an example of how to use the SO-Aware resolver inside an Itinerary.

Steps.

1. Create a new BizTalk 2010 Itinerary inside a C# Library project

3. Add An On-Ramp, Off-Ramp, and 2 Itinerary Service steps into the Itinerary

4. Right click on the Resolver on one of the Itinerary Service Steps and add a new Resolver:

5. In the properties windows select the Tellago SO-Aware ESB Resolver

6. Name the resolver SoAware, and set the properties:

The SO-Aware Resolver contains these properties:

Endpoint Name: –This is the Name of the endpoint registered inside the SO-Aware Repository

Environment: This is the configuration category such as “Production” “Q&A” “Testing” that the service version is registered under.

Operation Name- This is the name of the operation of the Service version that is registered inside SO-Aware

Service Name: This is the name of the Service You would like to retrieve its configuration from.

SO-Aware URL – this is the URL location of where the SO-Aware Service Repository Service is installed.

Version – This is the version number of the Service registered inside SO-Aware.

7. Configure the remaining Itinerary steps, choosing Receive Port/Location, Dynamic Send Port for the Off Ramp, and adding the Itinerary connectors, completing the Itinerary design.

8. This itinerary is a simple Itinerary that resolves the People Finder version 1.0 GetAllRegisteredPeople Operation.

Note: The BizTalk 2009 Version of this Resolver is a more generic example, whereas the BizTalk 2010 version contains more functionality.

Attaching an ESB 2.0 Itinerary To WCF Message Header

Attaching an ESB 2.0 Itinerary To WCF Message Header

Yesterday I was building a unit test to test an Http Adapter provider ( http://esbextlibrary.codeplex.com ) that we needed for communication with a Http based endpoint; and before you go down the road of why use HTTP Adapter vs the WCF Adapter and all its endless possibilities, the HTTP Legacy adapter didn’t have issues with badly formed messages like the WCF Adapter did.

When building the unit test, I wanted to use the ESB WCF Generic on ramps, and the WCF Adapter to send a message into the ESB. For those that don’t know this, there is a sample test client that comes with the ESB Samples. This test client contains code to embed an Itinerary and send this itinerary using a SOAP header. The sample also contains code to send to an ASMX web service, and a WCF service. In my opinion the samples are fairly cumbersome and awkward in regards to sending an itinerary into the ESB. Also, keep in mind that this test client was built for ESB 1.0, and ASMX focused doesn’t really take WCF and it’s power into account.

Thus the reasoning for this post…. I wanted to show that there are other ways accomplish this same goal, however using WCF, and the WCF InboundHeaders promoted property.

Let’s say I have an itinerary that looks as simple as this:

This itinerary uses a custom Http Adapter provider through the Resolve_Http_Endpoint resolver configuration to send in the message using the BizTalk HTTP Adapter.

To embed this itinerary into a WCF message, I’ll need to export this itinerary as XML and get the Itinerary content as such:

Now that I have the itinerary content, I can use one of the Generic WCF Onramps, or create my own, which is actually what I did. To create your own WCF Generic on ramp:

Create a BizTalk Receive Location inside the ESB BizTalk Application using the BizTalk Admin Console.
Configure the Receive Location to use a WCF-Custom adapter. I choose to use the netTCP Binding with all the default settings. The Uri for this adapter was set to: net.tcp://localhost:8888/ReceiveMsg/service.svc
Configure the Receive Location to use the ItineraryReceivePassthrough pipeline with all the defaults.
Create a Dynamic Send port that matches the filter expressions you created inside the Itinerary. Mine were set to
1. … ServiceName=DTSTwoWaySyncResponse
2. … ServiceType=Messaging
3. … ServiceState=Pending
4. … IsRequestResponse=True

After I configured all the BizTalk receive and send ports, I needed the client code in my unit test project to invoke and send the test message to the WCF Generic on ramp, along with the attached itinerary as a Soap header (WCF Message Header).

To Configure a client with the message header I created a unit test with a method marked up with the [TestMethod] attribute and added the following code:

NetTcpBinding b = new NetTcpBinding ();

EndpointAddress epa = new EndpointAddress ( “net.tcp://localhost:8888/ReceiveMsg/service.svc” );

IBizTalkReceiveLocation proxy = ChannelFactory < IBizTalkReceiveLocation >.CreateChannel(b, epa);

Message msg = Message .CreateMessage( MessageVersion .Default, “SendToBizTalk” ,

XmlTextReader .Create(

new StringReader (

“<?xml version=\”1.0\” ?><TestMessage>Post To Http Adapter</TestMessage>” )));

Write_ItineraryTo_WCF_InboundHeaders( ref msg);

proxy.SendToBizTalk(msg);

The Method of interest here is the “ Write_ItineraryTo_WCF_InboundHeaders(ref msg);” line of code.

This method is outlined here:

private void Write_ItineraryTo_WCF_InboundHeaders( ref Message wcfMessage)

{

// code taken from msdn: http://technet.microsoft.com/en-us/library/bb246105%28BTS.20%29.aspx

//const string PropertiesToPromoteKey = “http://schemas.microsoft.com/BizTalk/2006/01/Adapters/WCF-properties/Promote”;

const string PropertiesToWriteKey = “http://schemas.microsoft.com/BizTalk/2006/01/Adapters/WCF-properties/WriteToContext” ;

//XmlQualifiedName propItinerary = new XmlQualifiedName(“Itinerary”, “http://schemas.microsoft.biztalk.practices.esb.com/itinerary”);

XmlQualifiedName writeItinerary = new XmlQualifiedName ( “Itinerary” , “http://schemas.microsoft.biztalk.practices.esb.com/itinerary” );

//Create a List of KeyValuePairs that indicate properties to be promoted to BizTalk message context.

//A Property Schema must be deployed and string values have a limit of 256 characters

//List<KeyValuePair<XmlQualifiedName, object>> promoteProps = new List<KeyValuePair<XmlQualifiedName, object>>();

//promoteProps.Add(new KeyValuePair<XmlQualifiedName, object>(propItinerary , HttpLegacyItinerary ));

//wcfMessage.Properties[PropertiesToPromoteKey] = promoteProps;

//Create a List of KeyValuePairs that indicate properties to be written to BizTalk message context

KeyValuePair < XmlQualifiedName , object > pair = new KeyValuePair < XmlQualifiedName , object >(writeItinerary,

HttpLegacyItinerary);

List < KeyValuePair < XmlQualifiedName , object >> writeProps = new List < KeyValuePair < XmlQualifiedName , object >>();

writeProps.Add(pair);

XmlSerializer ser = new XmlSerializer ( typeof ( Itinerary ), “http://schemas.microsoft.biztalk.practices.esb.com/itinerary” );

//XmlSerializerNamespaces ns = new XmlSerializerNamespaces();

//ns.Add(“itin”, “http://schemas.microsoft.biztalk.practices.esb.com/itinerary”);

Itinerary itin =

( Itinerary )ser.Deserialize( XmlDictionaryReader .Create( new StringReader (HttpLegacyItinerary)));

MessageHeader header = MessageHeader .CreateHeader( “Esb” , “” , itin, new myItinerary (itin, “i” , “http://schemas.microsoft.biztalk.practices.esb.com/itinerary” ));

// code for testing the Itinerary pipeline component…

//string str2 = header.ToString(); // .Replace(“\r\n”, “”);

//string pattern = “<[a-zA-Z0-9]:Itinerary.*?</[a-zA-Z0-9]:Itinerary>”;

//string str = Regex.Match(str2, pattern).Value;

wcfMessage.Headers.Add(header);

wcfMessage.Properties.Add(PropertiesToWriteKey, writeProps);

//wcfMessage.Properties[PropertiesToWriteKey] = writeProps;

}

This method takes the itinerary content as a string, and deserializes it into an actual Itinerary class from the ESB Itinerary OM v1.0 assembly. This class then used to create a WCF Message header using the MessageHeader.Create static method call.

There’s one major note here to pay attention to…

The ESB Itinerary pipeline Component that is used with the ItineraryReceivePassthrough pipeline retrieves the Itinerary from a SOAP 1.x message header which is written (not distinguished/nor promoted) into the BizTalk Message context by either the SOAP adapter, or the WCF Adapter. When using the SOAP legacy adapter in BizTalk, SOAP headers are automatically written to the message context, again I must stress, that they are written, not promoted, nor distinguished. This means that the values are not available to orchestrations, nor routing capabilities by default. To get these values promoted, then you have to deploy a property schema with the correct SOAP target namespace. For WCF adapters, however, to get the headers promoted or written you must follow this msdn article:

http://technet.microsoft.com/en-us/library/bb246105%28BTS.20%29.aspx

The component uses a regular expression in order to find the itinerary inside the Soap Headers content. The Regular expression is: string pattern = “<[a-zA-Z0-9]:Itinerary.*?</[a-zA-Z0-9]:Itinerary>”;

What this regular expression tells us is that we must take care to add a prefix qualifier with exactly 1 character to the Itinerary Message Header, otherwise, your itinerary will not be found. Also make sure that you don’t use the “ a” prefix qualifier, as this is already take by WCF and SOAP specifics. Thus any other value will do fine.

Well in order to do this I had to use the overridden MessageHeader.Create() static method, which supports the use of a custom XmlObjectSerializer. This will allow me to control the serialization of the ESB Itinerary so that I may achieve this functionality. This is outlined in the following code:

publicclassmyItinerary:XmlObjectSerializer

{

private Itinerary itin = null ;

private XmlSerializerNamespaces ns = null ;

private XmlSerializer ser = null ;

public myItinerary( Itinerary itin, string prefix, string nameSpace)

{

this .itin = itin;

ns = new XmlSerializerNamespaces ();

ns.Add(prefix, nameSpace);

ser = new XmlSerializer ( typeof ( Itinerary ), “http://schemas.microsoft.biztalk.practices.esb.com/itinerary” );

}

public override bool IsStartObject( XmlDictionaryReader reader)

{

return true ;

}

public override object ReadObject( XmlDictionaryReader reader, bool verifyObjectName)

{

return ser.Deserialize(reader);

}

public override void WriteEndObject( XmlDictionaryWriter writer)

{

}

public override void WriteObjectContent( XmlDictionaryWriter writer, object graph)

{

ser.Serialize(writer , this .itin, this .ns);

}

public override void WriteStartObject( XmlDictionaryWriter writer, object graph)

{

}

This custom XmlObjectSerializer class just uses the XmlSerializer behind the scenes, and a XmlSerializerNamespaces instance to add the prefix and namespace setting when serializing the Itinerary ESB OM v1.0 class back into the message stream with a single character prefix qualifier.

After all this was done, I was able to send the test message into the WCF Channel, with the attached WCF Message Header containing the ESB 2.0 strict Itinerary.

Case Closed!!!

Routing 2-way Synchronous response to endpoints other than the initiator

Routing 2-way Synchronous response to endpoints other than the initiator

Recently, in our implementation of an ESB 2.0 BizTalk 2009 solution, a requirement popped up where an endpoint initiates a request to a synchronous system, and the system responds back with a response. In typical development practices, this is referred to a request/response or client/server pattern. You may say to yourself, “big deal, this is easy…” In a normal solution, the response is routed back to the initiator of the request. Lo and behold, we came across a scenario, where the response needs to be routed to other endpoints based on fairly complex rules and processes; not the original caller (initiator), hence the purpose of this blog entry.

As a BizTalk Developer, you say the above is not a problem. Why? Because BizTalk can handle designs like this with ease, using orchestrations, static send ports with unique filter expressions, and even orchestrations with dynamic send ports. You can actually put the complex logic, rules and processes inside custom components, or the Business Rules Engine (BRE) within orchestrations or even pipeline components. However if we’re staying true to the Agile game, and we want to build a architecturally sound ESB, using the normal BizTalk approach doesn’t fit the bill, in other words, it won’t work. The reasons why are simple, we either loose Agility, gain invisibility, create a mystery black box with custom code, or have increasingly complex algorithms which are prone to break and miss the agile mark altogether. The goal here is to think about BizTalk 2009 and the ESB 2.0 toolkit as a set of capabilities and then model the capabilities to generate the result you want.

Question: So now what capabilities do we need?

Answer: We want the ability to design steps or process flow which accepts a message from an initiator, and then based off of some rules, dynamically determine the 2 way service, retrieve the response and through more rules dynamically send the response to another endpoint who is not the initiator.

Question: So which capabilities match BizTalk and ESB?

Answer: The capabilities to design steps and process flows are the Itineraries pattern of ESB. The BizTalk Business Rules Engine incorporates the complex logic. The ESB Resolvers assist to resolve logic, and endpoint addresses, such as the 2 way service to invoke, and the resulting endpoint to send the responses to. In addition to the resolvers, BizTalk provides WCF Adapters to send and receive the request/response and results.

Question: So how do we do this?

Answer: You model an itinerary in BizTalk 2009’s ESB 2.0 using the Itinerary designer that accepts a message from a caller (initiator), and then sends the request to a 2-way Synchronous Endpoint, which then blocks (correlates automatically for the response) and then sends the Response to a completely different endpoint based off of a Business rule.

Question: Is this easy?

Answer: Yep Follow these high level steps:

Create an Itinerary using the ESB Itinerary Designer.
Configure the Itinerary to use a 2-way Receive Port, even though you will use a 1-way receive port in production.
Configure the Itinerary to be a Request/Response Itinerary, which is to say verify the IsRequestResponse = true. (See the Itinerary Cache Post for more info on this… https://dgoins.wordpress.com/2010/02/22/overstanding-the-esb-itinerary-cache-2/)
Model the itinerary with with the call the the synchronous request/response endpoint.
Within your steps, use the BRE Resolver to determine maps, and endpoint addresses for the configuration of which synchronous endpoint to send the request to.
Optionally: Use the Service Broker shape to route to different endpoints based off of promoted properties from the response.
Lastly, use a Business Rule Resolver to dynamically determine the resulting endpoint configurations.

Below is an example of an Itinerary which uses Business Rules to invoke a Synchronous endpoint and routes the results to another endpoint which is not the initiator.

Explanation of Itinerary:

The above image can be read following the directions of the arrows, starting with the upper Left shape named: ReceiveMsg . This first shape signifies an ESB OnRamp, which equates to a BizTalk Physical receive port.

The next shape: MapToCanonical represents an ESB Messaging Step. Messaging steps are processes or components that are destined to execute inside of pipelines. These pipelines are configured with ESB based pipeline components such as the ESB Dispathcher, which understand the Itinerary step mechanism to execute ESB Itinerary steps. In the above example, the MapToCanonical step uses a Transform based ESB Step process to execute a BizTalk transformation (Map) to convert the message received from the OnRamp into a canonical message format. The map which executes is determined by the ESB Resolver: SetMapToCanonical . This resolver utilizes the BRE Resolver to execute and evaluate complex logic using the BizTalk Business Rules Engine (BRE) to choose the correct map for transformation.

The next step: RouteToEndPoint is another ESB messaging step, which utilizes the ESB Routing process to promote properties and publish the message to the BizTalk Message box. This step is configured to use the BRE Resolver named: GetURI . This resolver executes business logic and evaluate rules to determine the URI address, and Send adapter to send the request to.

The next step: PushToMsgBox, is an off ramp extender shape which gathers more promoted property information from the actual ESB OffRamp: SendToEndPoint (BizTalk Physical Send Port). Here’s an interesting point to note, remember that the ESB Itinerary pattern is just a model to design steps as in the order of execution, not the actual process such as an orchestration. As we define our steps, BizTalk needs to know where these ordered steps should execute. In other words, when we use “Messaging Steps” these obviously execute inside Messaging components of BizTalk, hence pipelines stages. However, for BizTalk we have two sides of the messaging system. BizTalk has a messaging stack before saving to the Message Box (Receive Pipelines), and a stack after the Message Box (Send Pipeline). In the design above, the PushToMsgBox off ramp extender is used to signify all steps after this will execute in the Send Handler (Send pipeline stages).

The next step after the off ramp extender is the MapToEndPoint step. This step, again is a messaging step which uses the BRE Resolver named: SetMapToEndPoint , will be executed after the message is published to the BizTalk Message box, and inside the send pipeline associated with the send port configured through Off Ramp: SendToEndPoint .

The next step is the SendToEndPoint ESB Off Ramp. This Off Ramp is configured to use a 2-way Solicit-Response dynamic send port. From a BizTalk perspective, the 2-way dynamic send port is configured to use a send pipeline which contains the ESB Dispatcher pipeline component, and the ESB Itinerary Cache pipeline component. The receive pipeline is configured to use the ESB Itinerary Cache, Xml Disassembler, and the ESB Dispatcher components respectively. The 2-way send port is also configured with the 4 basic ESB promoted properties within the Filter expression. Screen shots of the various properties and values are provided below.

The next step is MapResponseToCanonical . Notice here that the messaging step is set to run in the Receive Handler (Receive Pipeline) of the SendToEndPoint off ramp. This means that this step runs on the receiving side of the 2-way dynamic send port. This step is configured to use the ESB Transformation service, and the BRE Resolver to again determine which map to use for transformation. This step transforms the reponse to a canonical format to eventually publish back to the Message Box.

The next step is RouteResponseToEndPoint, just like step before, this step also is configured to execute in the Receive Handler of the SendToEndPoint off ramp. Which also means it will execute on the receiving side of the 2-way dynamic send port. This step is also configured to use the ESB Routing service and again the BRE Resolver to determine the Endpoint URI and adapter to use to send the final response.

The next step is PushResponseToMsgBox. This step is an off ramp extender which signifies that the remaining steps will be executed after the message is published to the Message Box, along with any other special properties that need to be promoted.

The next step is MapResponseToEndPoint, this step is a messaging step which again uses the ESB Routing Service and the BRE Resolver to determine the Uri address and BizTalk Send adapter settings to use for sending the response.

The last step is SendResponseToEndPoint, this is the Off Ramp which points to a simple Dynamic Send port for actually sending the response from BizTalk.

D Goins Insperience

Technological outformation for this day and age

Tag: ESB Itinerary

Using SO-Aware with BizTalk 2010 and ESB 2.1 part 2 of 2

Attaching an ESB 2.0 Itinerary To WCF Message Header

Routing 2-way Synchronous response to endpoints other than the initiator

Share this:

Share this:

Share this: