Monitoring a BizTalk WCF Receive Location with SO-Aware

As a quick start, for those that don’t know, SO-Aware is a service metadata repository, very similar to UDDI, except without the complexity, all based on a REST-ful architecture. If you do know, you can skip to Monitoring a BizTalk WCF Receive Location section. (UPDATE: you can also view a video on the webinar explaining and demoing this here.) It supports registry of SOAP, REST, and OData based services, written using Microsoft’s WCF technology stack, as well and Java based web services. However, that’s not all. SO-Aware has five major capability buckets: Centralized Configuration, Service Testing, Dependency Modeling, Service Cataloging and last but not least Activity Monitoring.

Centralized Configuration

Centralized Configuration allows you take full advantage of Microsoft’s WCF based services, by containing a central repository database for all WCF Configurations. This feature allows you to allows you to store and retrieve all information pertaining to WCF configurations. You can retrieve endpoints, bindings, behaviors, Url’s, security information, just to name a few. You can also dynamically change your bindings and configurations so that all you existing services can point to this central location for your configuration.

Service Testing

Service Testing allows you to test registered services. One derived idea about Service Cataloging and Centralized Configuration, is that if SO-Aware knows about your service and communication protocol, then testing becomes simple. It’s simple because if you registered any security, binding, message type format information about the service, then all SO-Aware needs to do is query this information and build the communication stack and messages to send to the service for testing. What better tool to test than the one that understands how to communicate with the Service.

Dependency Modeling

Dependency Modeling allows you to build a diagram of service versions, and the dependencies of the service version. Thus if you needed to see which services depended each other, you have a view into this.

Service Cataloging

Service Cataloging allows you to store and retrieve custom metadata about Services, Service Versions, and environment details. Using these features, allow you to query the catalog for information about services, and service versions dynamically.

Activity Monitoring

Activity Monitoring allows you see tracked events and aggregations about registered services, such which operations were invoked, and how many services were sent message over a period of time, and many other dimensions and measurements.

What is BizTalk Server?

BizTalk Server is one of Microsoft’s flagship products for System Integration, message brokering, and service bus designs. It comes packed with a collection of tools and libraries that extend normal middleware Server capabilities by supporting a loosely coupled and dynamic messaging architecture. It functions as middleware server product that provides rapid mediation between services, endpoints, line of business systems and their consumers. Enabling maximum flexibility at run time, BizTalk Server simplifies loosely coupled composition of service endpoints and management of service interactions. There are many components that allows BizTalk Server solutions to be agile in its execution. For example, BizTalk Server contains Adapters which are connectors into different systems and protocols. It also contains Maps, which are Xml stylesheet transformation documents that have been precompiled to transform documents from one format into another. There are Pipelines which are components that allow you to translate and convert one file into another such as a Adobe PDF document into a comma separated text file. There are also Schemas which support validation of different messages, both xml and non xml. Lastly there are Orchestrations which are graphically modeled business processes that support long running transactions, compensation models, error handling, correlation, and custom logic flow.

Adapters in BizTalk are one of the key components for communicating with BizTalk Server. There are sending adapters and receiving adapters. There’s an adapter for practically every protocol that exists such as ftp, http, smtp, tcp, ftps (new in 2010), Msmq and others. There are application type of adapters that adhere to the rules and policies of Line of Business applications, such as SharePoint, SAP, Oracle E-Business Suite, and other CRM applications. There are even adapters for databases such as SQL, Oracle, and DB2. When developing solutions with BizTalk Adapters are usually configured statically and there’s little room for agility. Thus BizTalk supports the notion of Dynamic adapters for sending data. These adapters are configured at runtime, which will yield great agile solutions, at the sacrifice of High performance. There are also Dynamic adapters which require turn static BizTalk Server solutions into dynamic ones, along the Microsoft ESB toolkit, which can make everything in the BizTalk arena agile.

BizTalk Server also contains Monitoring capabilities. Every message and process that is run within BizTalk Server can be monitored using its internal Tracking system. As well as using a more open consumer business process driven ones entitled BAM, business activity monitoring. BizTalk has the capability to automatically monitor ports, receive locations, pipelines, maps, validation attempts, business rules, orchestrations and adapters. Using BAM, architects and developers can create custom monitoring scopes, and key performance indicators, for tracking and monitoring specific business requirements and processes. This leads us to why SO-Aware and what benefits can we discover when using SO-Aware to monitor BizTalk.

Monitoring using SO-Aware

BizTalk Server contains WCF Adapters, which provide limitless range of possibilities for communication protocols, and access to various endpoint systems. What I’m trying to say here is that through the use of WCF Adapters, any and every system can potentially be accessed and communicated to. To support this claim, Microsoft released the Microsoft WCF LOB Adapter SDK. This SDK is a framework that makes it easy for developers to create “Adapters” or better put, wcf channels to connect any type of system. These channels can be used with BizTalk, and regular .Net Applications. To go along with the SDK, Microsoft released some Production ready, sample Adapters based off this framework, called the Microsoft BizTalk Adapter Pack . Here’s the interesting kicker… The BizTalk Monitoring architecture was not updated sufficiently to handle these new additions to the BizTalk arena.

To Monitor these WCF Adapters, you have a few options:

1. Use the WCF Message logging and Activity Tracing capabilities, which can log, messages, operations, actions, instances, events and everything related WCF services. To View these traces and logs by default, you need to use the WCF Trace Viewer application provided with the .Net Framework SDK. This is your standard WCF Tracing capability. It is not tied to BizTalk in any shape, form or fashion. One nice thing about using the WCF Message logging and tracing is the ability to control where the logs and traces are written, how much data is written, and the ability to view this information RAW as it’s being recorded. Of course each pro has a con associated with it. The downside is that to turn on WCF tracing requires you to modify the BizTalk configuration file, and it will record messages for ALL WCF Adapters – Send and Receive. Thus if you have and large amount of Adapters being used, your traces can become convoluted very quickly, and fill up your trace/log repository very fast. Along with this, the traces and logs may be a little cryptic by default, unless you create your own logger and format the results accordingly.
2. Use BAM WCF Interceptors to log custom KPI’s to BAM tables, and create a custom view to see the results. This second option uses the only real updated portion of BizTalk to handle these new WCF Adapter features: WCF Interceptors. The main issue here is that configuring the WCF Interceptor tracking profile (WCF Interceptor configuration file), is a very tedious and complex process. As of this writing there is no graphical User interface for configuring it, like the BAM model in Excel. Thus this process is prone to trial and error. It’s a very powerful enhancement, however just not feasible in the quick development turn around times needed in Today’s market. Let’s put it this way, it took me a month to get the very basics down, after having to deal with reverse polish notation, and Interceptor configuration files. However once you understand it, it too has a limitless range of possibilities. Thus for this option, I’d say a big *PRO*, with the downside of complexity, time to configure and brittleness.
3. Use the WCF Extension model through behaviors (which is what BAM WCF Interceptors is, an Endpoint Behavior) to create a custom Behavior to monitor anything and everything related to WCF Services/LOB Adapter channels. The third option, is the exact same as the second option, the only difference is, you must build your own tracking logic, message logging, tracing and etc. Thus the time taken to invest in this option will outweigh both the previous two option multiplied together, in the long run, you will have complete control over it’s complexity, and brittleness.

We at Tellago Studios, have made it very easy for you to monitor these WCF Adapters. We followed the 3^rd option of creating our own custom WCF Service Behavior. This behavior supports logging messages, actions, operations, and also providing basic metrics such as number of operations called, number of errors over a specific period of time. When you configure the usage of the behavior for a WCF Receive location, we track these details, and provide a meaningful user interface to see the results.

Inside the SO-Aware Portal, Clicking on the Monitoring tab reveals this content:

Figure 1: Monitoring BizTalk Receive Locations

Figure 2: Viewing a BizTalk Orchestration published as WCF Receive Location Service

Figure 3: Request/Response

Figure 4: Request Message Body and Header Details

 

So How Easy is it for you to monitor the BizTalk WCF Adapters?

Well the answer is as easy as adding a Service Behavior to the service. BizTalk Server WCF Adapters, provide two ways of adding a service behavior to a service depending on where the service is being hosted. BizTalk Server WCF Adapter Services can be hosted inside of the BizTalk Process: BTSNTSVC.exe, or inside an isolated host instance: IIS (w3wp.exe). These hosted services only apply to BizTalk Receive locations, as BizTalk WCF Send adapters don’t host services, they just create Channel factories for sending messages to other endpoints.

Basic Steps:

1. Install the SO-Aware Monitoring Behavior.
2. Register your WCF Receive Location inside SO-Aware; register the service name, and service version, pointing to the correct location of the WSDL
3. Turn on a tracking profile for the registered service version such as VerboseSoap

   

Figure 5: Registered BizTalk Orchestration published as a WCF Service Receive Location in SO-Aware

1. Verify the SO-Aware Windows Host is started and running:

   

Figure 6: SO-Aware Windows Host

 

1. Configure the Inprocess or Isolated Process Receive Location to use the SO-Aware Monitoring Behavior, and you’re done.

    

Installing the SO-Aware Monitoring Behavior

1. The monitoring behavior can be found in the SO-Aware SDK.
2. To install, GAC the Tellago.ServiceModel.Governance.Monitoring.Client, Version=1.1.0.0, Culture=neutral, PublicKeyToken=68f3f79a5464509d
   assembly; This assembly can be found inside the Reference Assemblies folder (C:\Program Files (x86)\Tellago Studios\SO-Aware\SDK\Reference Assemblies – by default).
3. Register this behavior inside the Machine Configuration file (Both 32bit and 64bit, if on a 64bit machine).
4. You can register the behavior with this syntax inside the <behaviorExtensions> section. (Note: this must be inside both 32bit and 64bit machine configuration files for 64bit systems).

 

Web.Config – Behavior Extension Registration

 

<add
name=“SOAwareServiceMonitoring“
type=“Tellago.ServiceModel.Governance.Monitoring.MessageTrackingBehaviorElement, Tellago.ServiceModel.Governance.Monitoring.Client, Version=1.1.0.0, Culture=neutral, PublicKeyToken=68f3f79a5464509d“ />

 

 

 

Adding the SO-Aware Monitoring Behavior To an InProcess BizTalk WCF Receive Location

1. Open up the BizTalk Administration Console, navigate to the registered WCF Receive Location. Change the WCF Adapter to use the WCF –Custom adapter.
2. Configure the WCF Custom adapter to use the same binding and settings as the previous WCF Receive location.
3. Click on the Behaviors and Add the SoAwareMonitoring Behavior.
4. Configuring the Behavior is simple, set the serviceVersion and soAwareConfigurationCategory properties.
5. The serviceVersion property is set to a registered Service Name appended with the “(1.0)” version number such as this:

   

Figure 7: Configuring the SO-Aware Service Monitoring Behavior

1. The soAwareConfigurationCategory property is the name of the environment, if any such as Production or QA.
2. Last, open up the BizTalk configuration file: BTSNTSVC.exe.config and add these two xml configuration entries:

 

BTSNTSVC.exe.config – Register Service Repository section inside Config Sections

 

<configSections >

<section
name=“serviceRepository“
type=“Tellago.ServiceModel.Governance.ServiceConfiguration.ServiceRepositoryConfigurationSection, Tellago.ServiceModel.Governance.ServiceConfiguration, Version=1.1.0.0, Culture=neutral, PublicKeyToken=68f3f79a5464509d“/>

</configSections>

 

 

 

BTSNTSVC.exe.config – Configure serviceRepository section to point to SO-Aware URL

 

<serviceRepository
url=“http://localhost:8090/So-Aware/ServiceRepository.svc“>

</serviceRepository>

 

 

 

1. Save and you’re done.

 

Adding the SO-Aware Monitoring Behavior To an Isolated Process such as IIS for a BizTalk Receive Location

1. Open up the web.config file for the IIS Virtual Directory hosting the BizTalk Receive Location.
2. Add these sections to your web config file:

 

web.config – Register Service Repository section inside Config Sections

 

<configSections >

<section
name=“serviceRepository“
type=“Tellago.ServiceModel.Governance.ServiceConfiguration.ServiceRepositoryConfigurationSection, Tellago.ServiceModel.Governance.ServiceConfiguration, Version=1.1.0.0, Culture=neutral, PublicKeyToken=68f3f79a5464509d“/>

</configSections>

 

 

 

web.config – Configure serviceRepository section to point to SO-Aware URL

 

<serviceRepository
url=“http://localhost:8090/So-Aware/ServiceRepository.svc“>

</serviceRepository>

 

 

 

1. Add the Behavior to the WCF Service:

 

web.config – Register Service Repository section inside Config Sections

 

<serviceBehaviors>

<behavior
name=“ServiceBehaviorConfiguration“>

<serviceDebug
httpHelpPageEnabled=“true“
httpsHelpPageEnabled=“false“
includeExceptionDetailInFaults=“true“ />

<serviceMetadata
httpGetEnabled=“true“
httpsGetEnabled=“false“ />

         <SOAwareServiceMonitoring
serviceVersion=“BtsOrchSample(1.0)“ />

</behavior>

</serviceBehaviors>

 

 

 

1. Configuring the Behavior is simple, set the serviceVersion and soAwareConfigurationCategory properties to the registered services inside SO-Aware.
2. Save and you’re done.

    

Using SO-Aware with BizTalk 2010 and ESB 2.1 part 1 of 2

As a quick start, for those that don’t know, SO-Aware is a service metadata repository, very similar to UDDI, except without the complexity, all based on a REST-ful architecture. If you do know, you can skip to using SO-Aware with BizTalk 2010 and ESB 2.1 section. It supports registry of SOAP, REST, and OData based services, written using Microsoft’s WCF technology stack, as well and Java based web services. However, that’s not all. SO-Aware has five major capability buckets: Centralized Configuration, Service Testing, Dependency Modeling, Activity Monitoring and last but not least Service Cataloging.

Centralized Configuration allows you take full advantage of Microsoft’s WCF based services, by containing a central repository database for all WCF Configurations. This feature allows you to allows you to store and retrieve all information pertaining to WCF configurations. You can retrieve endpoints, bindings, behaviors, Url’s, security information, just to name a few. You can also dynamically change your bindings and configurations so that all you existing services can point to this central location for your configuration.

 

Service Testing allows you to test registered services. One derived idea about Service Cataloging and Centralized Configuration, is that if SO-Aware knows about your service and communication protocol, then testing becomes simple. It’s simple because if you registered any security, binding, message type format information about the service, then all SO-Aware needs to do is query this information and build the communication stack and messages to send to the service for testing. What better tool to test than the one that understands how to communicate with the Service.

 

Dependency Modeling allows you to build a diagram of service versions, and the dependencies of the service version. Thus if you needed to see which services depended each other, you have a view into this.

 

Activity Monitoring allows you see tracked events and aggregations about registered services, such which operations were invoked, and how many services were sent message over a period of time, and many other dimensions and measurements.

 

Service Cataloging allows you to store and retrieve custom metadata about Services, Service Versions, and environment details. Using these features, allow you to query the catalog for information about services, and service versions dynamically.

 

Which leads us into the next discussion, using SO-Aware with BizTalk 2010 and ESB 2.1. First, for those that don’t know BizTalk and ESB 2.1, here’s a quick blurb from Microsoft’s web site on each technology:

  What is BizTalk?

BizTalk Server is Microsoft’s Integration and connectivity server solution. A mature product on its seventh release, BizTalk Server 2010 provides a solution that allows organizations to more easily connect disparate systems. Including over 25 multi-platform adapters and a robust messaging infrastructure, BizTalk Server provides connectivity between core systems both inside and outside your organization. In addition to integration functionality, BizTalk also provides strong durable messaging, a rules engine, EDI connectivity, Business Activity Monitoring (BAM), RFID capabilities and IBM Host/Mainframe connectivity.

What is BizTalk ESB 2.1?

The BizTalk ESB Toolkit is a collection of tools and libraries that extend BizTalk Server capabilities of supporting a loosely coupled and dynamic messaging architecture. It functions as middleware that provides tools for rapid mediation between services and their consumers. Enabling maximum flexibility at run time, the BizTalk ESB Toolkit simplifies loosely coupled composition of service endpoints and management of service interactions.

Using SO-Aware with BizTalk 2010

One of the main features of BizTalk Server is its adapter component design. BizTalk uses the Adapters to communicate to various systems, such as Line of Business systems, Databases, Mainframe Applications, and most notably Web Services. To be more specific on web services, BizTalk contains WCF adapters that facilitate communication to different types of web services: REST/ODATA and SOAP.  The more involved BizTalk becomes in integration solutions, the more WCF adapter, and Web services are used to compliment or run the entire solution.

Maintaining such as solution can easily become a maintenance nightmare. When a particular web service changes, or is versioned there are no tools that help keep track of this. Usually trying to remember all the configuration values are impossible, and the solution becomes brittle to the slightest change. SO-Aware easily alleviates this by Service Cataloging, Centralized Configuration, Dependency Modeling, and Dynamic Resolution.

With Service Cataloging, SO-Aware you can register the WCF Adapter configurations for Receive Locations (BizTalk as a Service Provider), and Web Service configurations that the WCF Adapters communicate with through Send Ports (BizTalk as a Service Consumer/client). Below is an example of what may be registered.

Centralized Configuration allows a BizTalk Architect to store configurations about WCF Adapter bindings (individual components that make up a complete communication channel for the WCF Adapter). Using the Centralized confguration of SO-Aware, you can store binding information, behavior information, extension information, security details, transport protocol information, line of business information and etc. We include binding and behavior templates that even provide a dynamic user interface  for configuring the commonly changed settings. This feature facilitates the use of BizTalk’s WCF LOB adapters such as SAP, Oracle and Oracle E-Business Suite, and SQL. Below is an example of registering the Oracle E-Business Suite LOB Adapter binding and  binding template.

With Dependency Modeling, you can quickly view which services depend on each other. A reason why this is important in BizTalk is simple. Say you have a business process flow that requires order information to be uploaded to two different web services in one atomic transaction, such as an Order Header Service and a Order Detail Service. This business process can be modeled using BizTalk’s Orchestration design. Below is an image depicting the business process flow.

When these services change later, who will remember that these services are dependent upon each other, and that this orchestration depends on both of these services? SO-Aware can easily depict this dependency:

Dynamic Resolution

Another option we have with BizTalk Server is using Dynamic Adapters. Dynamic adapters allow for runtime resolution of adapters such as WCF Adapters. This resolution can be performed in two BizTalk components: Orchestrations and pipeline components. Using SO-Aware, dynamic runtime resolution is very easy to do. SO-Aware yields a REST-ful interface to every aspect of its repository. This means that any component which supports invoking .Net code, such as BizTalk’s orchestrations and pipeline components do, will support querying SO-Aware for adapter configuration. Included in the SDK is an example of how to use a BizTalk Orchestration with Dynamic ports and adapters to implement a runtime resolution solution. Below is an example of code inside the BizTalk Orchestration of invoking dynamic runtime resolution.

In the above example, the ProcessOrderInformation Orchestration is dynamically resolving the OrderDetailPort  which is configured at runtime to send data to the OrderDetailService registered inside SO-Aware. At runtime, when BizTalk Receives an Order, it will split the order into a Header message and a Order Details message and send both messages to their respective services in one atomic transaction. The differenece being that the OrderDetailService’s binding, configuration, address, and transport protocol will be dynamically resolved using SO-Aware api calls. The Order Detail Service is registered inside the SO-Aware Repository and when queried will return the WCF service configuration details. Below are some depictions of the Order Detail Service SO-Aware registration.

 

Remember this example was using an Orchestration, however the same applies to BizTalk Pipeline components as well. In the next posting, I will discuss how SO-Aware can be utilized with ESB 2.1 and BizTalk 2010 together.