Day: November 2, 2010

Monitoring a BizTalk WCF Receive Location with SO-Aware

As a quick start, for those that don’t know, SO-Aware is a service metadata repository, very similar to UDDI, except without the complexity, all based on a REST-ful architecture. If you do know, you can skip to Monitoring a BizTalk WCF Receive Location section. (UPDATE: you can also view a video on the webinar explaining and demoing this here.) It supports registry of SOAP, REST, and OData based services, written using Microsoft’s WCF technology stack, as well and Java based web services. However, that’s not all. SO-Aware has five major capability buckets: Centralized Configuration, Service Testing, Dependency Modeling, Service Cataloging and last but not least Activity Monitoring.

Centralized Configuration

Centralized Configuration allows you take full advantage of Microsoft’s WCF based services, by containing a central repository database for all WCF Configurations. This feature allows you to allows you to store and retrieve all information pertaining to WCF configurations. You can retrieve endpoints, bindings, behaviors, Url’s, security information, just to name a few. You can also dynamically change your bindings and configurations so that all you existing services can point to this central location for your configuration.

Service Testing

Service Testing allows you to test registered services. One derived idea about Service Cataloging and Centralized Configuration, is that if SO-Aware knows about your service and communication protocol, then testing becomes simple. It’s simple because if you registered any security, binding, message type format information about the service, then all SO-Aware needs to do is query this information and build the communication stack and messages to send to the service for testing. What better tool to test than the one that understands how to communicate with the Service.

Dependency Modeling

Dependency Modeling allows you to build a diagram of service versions, and the dependencies of the service version. Thus if you needed to see which services depended each other, you have a view into this.

Service Cataloging

Service Cataloging allows you to store and retrieve custom metadata about Services, Service Versions, and environment details. Using these features, allow you to query the catalog for information about services, and service versions dynamically.

Activity Monitoring

Activity Monitoring allows you see tracked events and aggregations about registered services, such which operations were invoked, and how many services were sent message over a period of time, and many other dimensions and measurements.

What is BizTalk Server?

BizTalk Server is one of Microsoft’s flagship products for System Integration, message brokering, and service bus designs. It comes packed with a collection of tools and libraries that extend normal middleware Server capabilities by supporting a loosely coupled and dynamic messaging architecture. It functions as middleware server product that provides rapid mediation between services, endpoints, line of business systems and their consumers. Enabling maximum flexibility at run time, BizTalk Server simplifies loosely coupled composition of service endpoints and management of service interactions. There are many components that allows BizTalk Server solutions to be agile in its execution. For example, BizTalk Server contains Adapters which are connectors into different systems and protocols. It also contains Maps, which are Xml stylesheet transformation documents that have been precompiled to transform documents from one format into another. There are Pipelines which are components that allow you to translate and convert one file into another such as a Adobe PDF document into a comma separated text file. There are also Schemas which support validation of different messages, both xml and non xml. Lastly there are Orchestrations which are graphically modeled business processes that support long running transactions, compensation models, error handling, correlation, and custom logic flow.

Adapters in BizTalk are one of the key components for communicating with BizTalk Server. There are sending adapters and receiving adapters. There’s an adapter for practically every protocol that exists such as ftp, http, smtp, tcp, ftps (new in 2010), Msmq and others. There are application type of adapters that adhere to the rules and policies of Line of Business applications, such as SharePoint, SAP, Oracle E-Business Suite, and other CRM applications. There are even adapters for databases such as SQL, Oracle, and DB2. When developing solutions with BizTalk Adapters are usually configured statically and there’s little room for agility. Thus BizTalk supports the notion of Dynamic adapters for sending data. These adapters are configured at runtime, which will yield great agile solutions, at the sacrifice of High performance. There are also Dynamic adapters which require turn static BizTalk Server solutions into dynamic ones, along the Microsoft ESB toolkit, which can make everything in the BizTalk arena agile.

BizTalk Server also contains Monitoring capabilities. Every message and process that is run within BizTalk Server can be monitored using its internal Tracking system. As well as using a more open consumer business process driven ones entitled BAM, business activity monitoring. BizTalk has the capability to automatically monitor ports, receive locations, pipelines, maps, validation attempts, business rules, orchestrations and adapters. Using BAM, architects and developers can create custom monitoring scopes, and key performance indicators, for tracking and monitoring specific business requirements and processes. This leads us to why SO-Aware and what benefits can we discover when using SO-Aware to monitor BizTalk.

Monitoring using SO-Aware

BizTalk Server contains WCF Adapters, which provide limitless range of possibilities for communication protocols, and access to various endpoint systems. What I’m trying to say here is that through the use of WCF Adapters, any and every system can potentially be accessed and communicated to. To support this claim, Microsoft released the Microsoft WCF LOB Adapter SDK. This SDK is a framework that makes it easy for developers to create “Adapters” or better put, wcf channels to connect any type of system. These channels can be used with BizTalk, and regular .Net Applications. To go along with the SDK, Microsoft released some Production ready, sample Adapters based off this framework, called the Microsoft BizTalk Adapter Pack . Here’s the interesting kicker… The BizTalk Monitoring architecture was not updated sufficiently to handle these new additions to the BizTalk arena.

To Monitor these WCF Adapters, you have a few options:

  1. Use the WCF Message logging and Activity Tracing capabilities, which can log, messages, operations, actions, instances, events and everything related WCF services. To View these traces and logs by default, you need to use the WCF Trace Viewer application provided with the .Net Framework SDK. This is your standard WCF Tracing capability. It is not tied to BizTalk in any shape, form or fashion. One nice thing about using the WCF Message logging and tracing is the ability to control where the logs and traces are written, how much data is written, and the ability to view this information RAW as it’s being recorded. Of course each pro has a con associated with it. The downside is that to turn on WCF tracing requires you to modify the BizTalk configuration file, and it will record messages for ALL WCF Adapters – Send and Receive. Thus if you have and large amount of Adapters being used, your traces can become convoluted very quickly, and fill up your trace/log repository very fast. Along with this, the traces and logs may be a little cryptic by default, unless you create your own logger and format the results accordingly.
  2. Use BAM WCF Interceptors to log custom KPI’s to BAM tables, and create a custom view to see the results. This second option uses the only real updated portion of BizTalk to handle these new WCF Adapter features: WCF Interceptors. The main issue here is that configuring the WCF Interceptor tracking profile (WCF Interceptor configuration file), is a very tedious and complex process. As of this writing there is no graphical User interface for configuring it, like the BAM model in Excel. Thus this process is prone to trial and error. It’s a very powerful enhancement, however just not feasible in the quick development turn around times needed in Today’s market. Let’s put it this way, it took me a month to get the very basics down, after having to deal with reverse polish notation, and Interceptor configuration files. However once you understand it, it too has a limitless range of possibilities. Thus for this option, I’d say a big *PRO*, with the downside of complexity, time to configure and brittleness.
  3. Use the WCF Extension model through behaviors (which is what BAM WCF Interceptors is, an Endpoint Behavior) to create a custom Behavior to monitor anything and everything related to WCF Services/LOB Adapter channels. The third option, is the exact same as the second option, the only difference is, you must build your own tracking logic, message logging, tracing and etc. Thus the time taken to invest in this option will outweigh both the previous two option multiplied together, in the long run, you will have complete control over it’s complexity, and brittleness.

We at Tellago Studios, have made it very easy for you to monitor these WCF Adapters. We followed the 3rd option of creating our own custom WCF Service Behavior. This behavior supports logging messages, actions, operations, and also providing basic metrics such as number of operations called, number of errors over a specific period of time. When you configure the usage of the behavior for a WCF Receive location, we track these details, and provide a meaningful user interface to see the results.

Inside the SO-Aware Portal, Clicking on the Monitoring tab reveals this content:

Figure 1: Monitoring BizTalk Receive Locations

Figure 2: Viewing a BizTalk Orchestration published as WCF Receive Location Service

Figure 3: Request/Response

Figure 4: Request Message Body and Header Details

 

So How Easy is it for you to monitor the BizTalk WCF Adapters?

Well the answer is as easy as adding a Service Behavior to the service. BizTalk Server WCF Adapters, provide two ways of adding a service behavior to a service depending on where the service is being hosted. BizTalk Server WCF Adapter Services can be hosted inside of the BizTalk Process: BTSNTSVC.exe, or inside an isolated host instance: IIS (w3wp.exe). These hosted services only apply to BizTalk Receive locations, as BizTalk WCF Send adapters don’t host services, they just create Channel factories for sending messages to other endpoints.

Basic Steps:

  1. Install the SO-Aware Monitoring Behavior.
  2. Register your WCF Receive Location inside SO-Aware; register the service name, and service version, pointing to the correct location of the WSDL
  3. Turn on a tracking profile for the registered service version such as VerboseSoap

Figure 5: Registered BizTalk Orchestration published as a WCF Service Receive Location in SO-Aware

  1. Verify the SO-Aware Windows Host is started and running:

Figure 6: SO-Aware Windows Host

 

  1. Configure the Inprocess or Isolated Process Receive Location to use the SO-Aware Monitoring Behavior, and you’re done.

     

Installing the SO-Aware Monitoring Behavior

  1. The monitoring behavior can be found in the SO-Aware SDK.
  2. To install, GAC the Tellago.ServiceModel.Governance.Monitoring.Client, Version=1.1.0.0, Culture=neutral, PublicKeyToken=68f3f79a5464509d
    assembly; This assembly can be found inside the Reference Assemblies folder (C:\Program Files (x86)\Tellago Studios\SO-Aware\SDK\Reference Assemblies – by default).
  3. Register this behavior inside the Machine Configuration file (Both 32bit and 64bit, if on a 64bit machine).
  4. You can register the behavior with this syntax inside the <behaviorExtensions> section. (Note: this must be inside both 32bit and 64bit machine configuration files for 64bit systems).

 

Web.Config – Behavior Extension Registration

 

<add
name=“SOAwareServiceMonitoring“
type=“Tellago.ServiceModel.Governance.Monitoring.MessageTrackingBehaviorElement, Tellago.ServiceModel.Governance.Monitoring.Client, Version=1.1.0.0, Culture=neutral, PublicKeyToken=68f3f79a5464509d“ />

 

 

 

Adding the SO-Aware Monitoring Behavior To an InProcess BizTalk WCF Receive Location

  1. Open up the BizTalk Administration Console, navigate to the registered WCF Receive Location. Change the WCF Adapter to use the WCF –Custom adapter.
  2. Configure the WCF Custom adapter to use the same binding and settings as the previous WCF Receive location.
  3. Click on the Behaviors and Add the SoAwareMonitoring Behavior.
  4. Configuring the Behavior is simple, set the serviceVersion and soAwareConfigurationCategory properties.
  5. The serviceVersion property is set to a registered Service Name appended with the “(1.0)” version number such as this:

Figure 7: Configuring the SO-Aware Service Monitoring Behavior

  1. The soAwareConfigurationCategory property is the name of the environment, if any such as Production or QA.
  2. Last, open up the BizTalk configuration file: BTSNTSVC.exe.config and add these two xml configuration entries:

 

BTSNTSVC.exe.config – Register Service Repository section inside Config Sections

 

<configSections >

<section
name=“serviceRepository“
type=“Tellago.ServiceModel.Governance.ServiceConfiguration.ServiceRepositoryConfigurationSection, Tellago.ServiceModel.Governance.ServiceConfiguration, Version=1.1.0.0, Culture=neutral, PublicKeyToken=68f3f79a5464509d“/>

</configSections>

 

 

 

BTSNTSVC.exe.config – Configure serviceRepository section to point to SO-Aware URL

 

<serviceRepository
url=“http://localhost:8090/So-Aware/ServiceRepository.svc“>

</serviceRepository>

 

 

 

  1. Save and you’re done.

 

Adding the SO-Aware Monitoring Behavior To an Isolated Process such as IIS for a BizTalk Receive Location

  1. Open up the web.config file for the IIS Virtual Directory hosting the BizTalk Receive Location.
  2. Add these sections to your web config file:

 

web.config – Register Service Repository section inside Config Sections

 

<configSections >

<section
name=“serviceRepository“
type=“Tellago.ServiceModel.Governance.ServiceConfiguration.ServiceRepositoryConfigurationSection, Tellago.ServiceModel.Governance.ServiceConfiguration, Version=1.1.0.0, Culture=neutral, PublicKeyToken=68f3f79a5464509d“/>

</configSections>

 

 

 

web.config – Configure serviceRepository section to point to SO-Aware URL

 

<serviceRepository
url=“http://localhost:8090/So-Aware/ServiceRepository.svc“>

</serviceRepository>

 

 

 

  1. Add the Behavior to the WCF Service:

 

web.config – Register Service Repository section inside Config Sections

 

<serviceBehaviors>

<behavior
name=“ServiceBehaviorConfiguration“>

<serviceDebug
httpHelpPageEnabled=“true“
httpsHelpPageEnabled=“false“
includeExceptionDetailInFaults=“true“ />

<serviceMetadata
httpGetEnabled=“true“
httpsGetEnabled=“false“ />

         <SOAwareServiceMonitoring
serviceVersion=“BtsOrchSample(1.0)“ />

</behavior>

</serviceBehaviors>

 

 

 

  1. Configuring the Behavior is simple, set the serviceVersion and soAwareConfigurationCategory properties to the registered services inside SO-Aware.
  2. Save and you’re done.