Overstanding the SO-Aware Test Workbench Test Types (part 1: Unit & Simple Load Testing)


Introduction

SO-Aware, allows you to create unit tests for WCF SOAP, and WCF REST based web services. These tests can be executed using the standard web based user interface that SO-Aware provides. The tests can also be executed using a PowerShell script. There is also a automated windows service that runs that can schedule a test to run every so often. The results of these tests are stored inside the SO-Aware repository.

After utilizing these testing features of SO-Aware, it became evident from our customers that there should be a Client Desktop form of this testing. Where a Client application can run these tests, and the results of these tests can be exported out into a format for inclusion into a reporting system like SQL Reporting services, where the results can be gathered and displayed using other tools.

The SO-Aware Test Workbench (TWB) is this client side application. TWB contains many different types of tests that a developer, architect, and IT administrator can use to stress test services in their enterprise. TWB, supports executing single unit tests, just like inside SO-Aware, Simple Load Tests, Linear Load Tests, Burst Load Tests, and SawTooth Load tests.

Each different type of test has a unique design pattern and purpose to allow you to determine different metrics and derive different information from the results. In these next series of blogs, I will explain what each test is designed to do, and explain in detail about each of the configurable parameters. I will also explain a few ways of how to interpret the results.

In this first posting, let’s discuss two tests: Unit Test and Simple Load testing. Before we delve into the specifics, I want to provide a baseline for comparison. If you want to do these same type of tests, I’d suggest you go and download SO-Aware and the SO-Aware Test Workbench now.

Just so that you know, all tests were performed on a VMWare virtual machine. The VM image was configured to have: 

  • 1 CPU with 8 cores (Intel i7 2GHz)
  • Running on a 256 GB SSD (100GB allocated to image on single drive: C:\)
  • RAM 4GB
  • Networking Shared
  • Windows Server 2008 R2 (SP1)
    • Web Role
    • AppFabric
    • Active Directory Domain Controller
    • File Application Server Role
  • SQL 2008 R2 (Installed and Running)
  • SharePoint 2010 (Installed and Running)
  • BizTalk Server 2010 (Installed not running)
  • VS.NET 2010 (SP1)

Services used in these tests are:

    1. SO-Aware Service Repository (REST registered endpoint)
    2. People Finder Service (AppFabric WCF Service as found in the SO-Aware SDK)

Now with all the logistical settings out the way… let’s continue on.

Unit Testing

The Unit testing feature of TWB allows you to run singular tests configured inside SO-Aware and export its results using PDF and Excel formats for inclusion into other reporting tools.

This test is designed to test the functionality of the service operation. It is a single test that allows you to obtain its execution time, success/fail status, and basic operation. B y right-clicking on any test, you can select Run Test and it will display the screen below with an option to change the registered message in SO-Aware, restore the original message and Run the single unit test.

image

Simple Load Test

The Simple Load test is designed to allow you to test the singular unit tests over and over again for a certain period of time. The concept here is that after you’ve run the unit test a few times, you should have a good feeling of how long that test runs. Now if you wanted to test that unit test with concurrency, that is… let’s say 10 of the same tests running at the same time for over 60 seconds, this is where the Simple Load test comes into play.

You can see the execution time, success/fail status and basic operation. Not only does the Simple load test allow you to test concurrency, you can also test redundancy to get an average execution time on the current hardware and software design, by executing it over and over again.

The parameters of the Simple Load test are:

Concurrency – how many threads to execute a single test on for concurrent processing

Delay (ms) – how long to wait for the concurrent tests to run. This value should be set high enough to allow the other concurrent test to load up and execute without affecting the previous tests. However setting this value to a lower value would simulate a constant (albeit linear) stress factor of concurrent tests that load on top of concurrent tests before the previous tests have had a chance to complete.

Duration (sec) – the number of seconds to continue running the concurrent tests.

Number of tests – the number of iterations to run the complete duration of concurrent tests.

clip_image002

In the above test, I executed the Get Services REST endpoint for a SO-Aware instance I have installed. Below are the first five results from the Excel exported results workbook.

Making sense out of the first five results in the table:

Second

Count

Average

Errors

Passed

Thread Count

0

37

89.81081081

0

37

10

1

38

60.92105263

0

38

10

2

42

39.61904762

0

42

10

3

38

46.05263158

0

38

10

4

42

50.26190476

0

42

10

5

42

43.80952381

0

42

10

Basically what this boils down to is this, up to the first second, we had 37 tests run concurrently, and out of all these tests, it took an average of .0089 seconds (89.81 ms) for all 37 tests to complete. This yields an average of 1 test every 2.42 milliseconds. During the 89ms, there was a max count of 10 threads that occurred during the processing of these tests. Another way of looking at it is that every 3.7 tests a new thread may have been spawned. – Ok Great…

Now within the next second (up to the 2nd Second), we had 1 more test get crammed into the process to total 38 test being run concurrently. It took 61 seconds, on average. This is a speed increase of 30%.

On the next second (up to the 3rd second), we had 4 more test get added totaling 42 tests. These test ran the quickest at a blazing 40ms. That’s an average of 1 test every 0.94 ms which is .000094 seconds. Remember 2GHz is 2 billion ticks per second, and depending on the CPU architecture (# of Cores/RISC/CISC etc) different sets of instructions may execute on 1 tick (clock cycle).

On the fourth and fifth seconds the median appears to have been reached ranging from 43 – 53 milliseconds easily of 38 -42 tests, while taking the median of those ranges yields 50ms per grouping of 40 tests.

Simple Load tests help you determine how fast your service/operation will run with the same operation running at the same time. It will also allow you to see the fastest and slowest times on a given hardware design. This test will also allow you to see how many threads will yield performance degradation for a given operation. Hopefully this is enough to get you started on executing Unit Tests and Simple Load tests using the TWB. In the next posting I will talk about Linear Load Tests, so start testing your services today!!!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s