Overstanding the SO-Aware Test Workbench Test Types (part 1: Unit & Simple Load Testing)


SO-Aware, allows you to create unit tests for WCF SOAP, and WCF REST based web services. These tests can be executed using the standard web based user interface that SO-Aware provides. The tests can also be executed using a PowerShell script. There is also a automated windows service that runs that can schedule a test to run every so often. The results of these tests are stored inside the SO-Aware repository.

After utilizing these testing features of SO-Aware, it became evident from our customers that there should be a Client Desktop form of this testing. Where a Client application can run these tests, and the results of these tests can be exported out into a format for inclusion into a reporting system like SQL Reporting services, where the results can be gathered and displayed using other tools.

The SO-Aware Test Workbench (TWB) is this client side application. TWB contains many different types of tests that a developer, architect, and IT administrator can use to stress test services in their enterprise. TWB, supports executing single unit tests, just like inside SO-Aware, Simple Load Tests, Linear Load Tests, Burst Load Tests, and SawTooth Load tests.

Each different type of test has a unique design pattern and purpose to allow you to determine different metrics and derive different information from the results. In these next series of blogs, I will explain what each test is designed to do, and explain in detail about each of the configurable parameters. I will also explain a few ways of how to interpret the results.

In this first posting, let’s discuss two tests: Unit Test and Simple Load testing. Before we delve into the specifics, I want to provide a baseline for comparison. If you want to do these same type of tests, I’d suggest you go and download SO-Aware and the SO-Aware Test Workbench now.

Just so that you know, all tests were performed on a VMWare virtual machine. The VM image was configured to have: 

  • 1 CPU with 8 cores (Intel i7 2GHz)
  • Running on a 256 GB SSD (100GB allocated to image on single drive: C:\)
  • RAM 4GB
  • Networking Shared
  • Windows Server 2008 R2 (SP1)
    • Web Role
    • AppFabric
    • Active Directory Domain Controller
    • File Application Server Role
  • SQL 2008 R2 (Installed and Running)
  • SharePoint 2010 (Installed and Running)
  • BizTalk Server 2010 (Installed not running)
  • VS.NET 2010 (SP1)

Services used in these tests are:

    1. SO-Aware Service Repository (REST registered endpoint)
    2. People Finder Service (AppFabric WCF Service as found in the SO-Aware SDK)

Now with all the logistical settings out the way… let’s continue on.

Unit Testing

The Unit testing feature of TWB allows you to run singular tests configured inside SO-Aware and export its results using PDF and Excel formats for inclusion into other reporting tools.

This test is designed to test the functionality of the service operation. It is a single test that allows you to obtain its execution time, success/fail status, and basic operation. B y right-clicking on any test, you can select Run Test and it will display the screen below with an option to change the registered message in SO-Aware, restore the original message and Run the single unit test.


Simple Load Test

The Simple Load test is designed to allow you to test the singular unit tests over and over again for a certain period of time. The concept here is that after you’ve run the unit test a few times, you should have a good feeling of how long that test runs. Now if you wanted to test that unit test with concurrency, that is… let’s say 10 of the same tests running at the same time for over 60 seconds, this is where the Simple Load test comes into play.

You can see the execution time, success/fail status and basic operation. Not only does the Simple load test allow you to test concurrency, you can also test redundancy to get an average execution time on the current hardware and software design, by executing it over and over again.

The parameters of the Simple Load test are:

Concurrency – how many threads to execute a single test on for concurrent processing

Delay (ms) – how long to wait for the concurrent tests to run. This value should be set high enough to allow the other concurrent test to load up and execute without affecting the previous tests. However setting this value to a lower value would simulate a constant (albeit linear) stress factor of concurrent tests that load on top of concurrent tests before the previous tests have had a chance to complete.

Duration (sec) – the number of seconds to continue running the concurrent tests.

Number of tests – the number of iterations to run the complete duration of concurrent tests.


In the above test, I executed the Get Services REST endpoint for a SO-Aware instance I have installed. Below are the first five results from the Excel exported results workbook.

Making sense out of the first five results in the table:






Thread Count





































Basically what this boils down to is this, up to the first second, we had 37 tests run concurrently, and out of all these tests, it took an average of .0089 seconds (89.81 ms) for all 37 tests to complete. This yields an average of 1 test every 2.42 milliseconds. During the 89ms, there was a max count of 10 threads that occurred during the processing of these tests. Another way of looking at it is that every 3.7 tests a new thread may have been spawned. – Ok Great…

Now within the next second (up to the 2nd Second), we had 1 more test get crammed into the process to total 38 test being run concurrently. It took 61 seconds, on average. This is a speed increase of 30%.

On the next second (up to the 3rd second), we had 4 more test get added totaling 42 tests. These test ran the quickest at a blazing 40ms. That’s an average of 1 test every 0.94 ms which is .000094 seconds. Remember 2GHz is 2 billion ticks per second, and depending on the CPU architecture (# of Cores/RISC/CISC etc) different sets of instructions may execute on 1 tick (clock cycle).

On the fourth and fifth seconds the median appears to have been reached ranging from 43 – 53 milliseconds easily of 38 -42 tests, while taking the median of those ranges yields 50ms per grouping of 40 tests.

Simple Load tests help you determine how fast your service/operation will run with the same operation running at the same time. It will also allow you to see the fastest and slowest times on a given hardware design. This test will also allow you to see how many threads will yield performance degradation for a given operation. Hopefully this is enough to get you started on executing Unit Tests and Simple Load tests using the TWB. In the next posting I will talk about Linear Load Tests, so start testing your services today!!!

Moveover Ubuntu… I’m on Gentoo now

For those interested…

I’m not sure that you all know this or not, however I’ve successfully been running a flavor of Unix for the past 2 years as a desktop system where I do all my development/testing/ and etc. I started with Ubuntu 9.10 and moved onto Ubuntu 10.04 then 10.10 and finally 11.04. I’ve presented at  Microsoft conferences, externally and internally, all with the Unix platforms running, however hidden from view. However after the major announcements made by Ubuntu within the last year, I’ve decided to bump up the game up another level.

After all the past calamities of Ubuntu changing their directions to be more in line with Canonical apps, and themes, I decided to bump my knowledge on linux up a notch. Ubuntu recently made the decision to switch it’s desktop engine/environment to Unity and ditch the gnome desktop engine, in favor of a more “Touch Based” type of GUI. I absolutely hated this decision, after using Unity in its infacy stages and then trying Ubuntu 11.04 in its second year stage, I’ll have to say that Unity that “U n I” don’t see eye to eye. Unity did not have a very good feel personally and I wanted to actually move to a more open platform that allowed me to choose exactly what I wanted from the start.

Enter the world of Gentoo. Gentoo is a linux distribution that from the start makes no default decisions on how and what your computer system runs. This is the interesting part. You load it from a liveCD, and you get this black screen with a prompt that leaves you wondering…. “OK Now What???” This is the beauty behind it. “OK Now what… !!!” you are in complete control of what you want your system to do. You just need to google it, and then add it and it’s running. It takes a while to get it the way you want it, but it’s so open that you can tell it to do just about anything upon boot.

I’m also not the only one using this too… If you’ve ever flown on a Delta plane across the US. You may have been on certain planes that have the wifi access, and a tv touch screen on the back of the chair head rest. If you’ve ever been so lucky to see these computers start up, and you read closely if a flavor of linux. Which if I remember the lines actually had the same gentoo splash screen: two “tux’s” penguins at the top of the screen.

The genearal steps I learned and tasks that I performed were:

  • Installed Gentoo
  • Downloaded linux kernel 2.6.36 r-8
  • Configured, Edited source, and compiled a new kernel
  • chrooted my old laptop (“This was the easy button…”)
  • Customized my boot up process
  • I must warn you. It took a whole weekend, google, and the #gentoo IRC chat room to get it, but once it was on, the speed, the usability, the freedom to do what you want amazed me. I’m now confident that if one of my “drivers” goes bad, or if something breaks, I know pretty much where to look and how to fix it. Linux- I’m not scared of you anymore!!!

    I’m sorry Microsoft, due to the lack of control on a Win 7 box, and the terrible slow performance each and everytime an windows update occurred, I’m afraid that I’m now totally converted over to the “DarkSide!!!” I’m now a ‘Nix head for life’ as some say. In my opinnion, the best things about this experience is i’ve learned so much about the platform, I’m *almost* as comfortable with it and it’s command line actions, as with the life long embedded – hit over your head with this is how it’s got to work Windows platform.

    But Don’t worry Microsoft, I still have to use Microsoft products, because you force me to. I still have to deal with developing inside your environements, mostly because you have the best dev environments. Thus I’m not moving off you anytime soon. I just wanted to let you know there’s a nice alternative to those who want to get their hands and feet wet, and actually know the underlying pieces of how their system works…!