Microsoft Visual Studio Team Suite Load Test Part 1 – VSTS vs Mercury LoadRunner
I’ve been getting my hands dirty with Visual Studio Team Suite Load Test the past few months. In the coming few months, i will be posting several articles on my experiences with VSTS.
First i’ll start with a comparison between VSTS and Mercury LoadRunner. I’m including the version numbers as the newer versions of these 2 software promises much much more!
|Point of Comparison||Mercury Load Runner 6
||Visual Studio Load Test 2008
|Pricing||Unlimited number of agent computers, pricing is per Virtual User (VU)
Useful if you have a whole lot of old computers or if you want to spread the load around in your network to find connection bottlenecks
|Pricing is per physical CPU, Microsoft states that each logical CPU can take up to 1000VU
As this is on a per CPU basis, you may want to get a relatively powerful (read 4/6/8 Core CPU, >=8gb) machine to be the load test agent.
Also, DO NOT install this on a VM, be it ESX or Hypervisor, performance of the agent in a virtual environment is at least 50% worse as compared to having it installed on a physical machine.
|Monitoring Capabilities||By default, it does not monitor any server, client parameters, of course these are available if you pay
This means that during any test run, engineers need to be around to monitor the server loads
|Able to monitor perfmon statistics (i.e windows machines), you can also specify the statistics to gather and save in your load test report.
It also comes with SQL Tracing for SQL Server, so you can monitor the long running queries with respect to your load test script.
With real time display of the server/client statistics, you can easily tell how the load test is running
On the downside, you will still have to monitor non-windows machines manually.
|Virtual User Stepping||You can specify User Stepping or Constant User Load||You can specify User Stepping, Constant User Load or Goal based.
Goal based means VSTS will keep increasing the VU until a certain perfmon statistic is matched e.g web server 100% CPU
|Testing Flexibility||Able to test various systems, including web based, SAP etc
Databinding to text files is supported
Backend is something like C programming so you can easily change the parameters if required
|Able to test most systems, any additional functionality can be done via a custom app and using VSTS to run the app
Databinding to various data sources are supported
There are 2 formats to testing, coded and scripted tests.
Coded tests are simple C#/VB codes with events and stuff for ultimate flexibility at the expense of ease of editing.
Scripted tests is similar to what Mercury has to offer
One thing to note: If you have installed the VSTS SDK, you can actually use a program to run thru the scripts and change the parameters for a scripted test, e.g changing the think time for all web requests to 5 seconds.