Analyze Test Results

PushToTest TestMaker provides several methods for understanding and analyzing test results.

Live Results Charts for a load test scenario shows transactions per second (TPS) results as the test operates.
Transaction Distribution Charts for a load test scenario shows the range of response times for the duration of each testusecase.
Logged Results Files store the timing values for each transaction on each TestNode in a set of XML formatted files.
Resource Utilization shows CPU, network, and memory utilization for each TestNode for each testusecase.
Performance Comparison Utility compares the results of multiple test scenarios.

Read the Why Test? And How document to understand the concepts behind PushToTest TestMaker and how to analyze results sets.

Live Results Charts

As TestMaker operates a load test scenario the Controller panel displays a live view of the scalability index for the target service. The scalability index shows the average Transactions Per Second (TPS) throughput, measured by the TestNodes, at each concurrent requests level and message payload size. Figure 1 shows the live results chart for a load test scenario reporting 0.17 TPS at the first concurrent virtual users (CVUs) level and 0.08 TPS as the second level of CVUs operates:

Figure 1 - Operating Test Scenario Results

The live results chart provides several functions accessible using a right-click of the mouse on the chart. A pop-menu appears to let you save the chart to a PNG graphic file, zoom in/out, print, and set preferences for the chart, including fonts, axis settings, and titles.

Transaction Distribution Charts

For each testusecase, TestMaker creates a transaction distribution chart. This chart plots a vertical point indicating the transaction duration time (in milliseconds; 1000 milliseconds equals 1 second) measured at the TestNode for the duration of the testusecase. The transaction distribution chart is useful when auditing a test to validate the results by identifying unusually long or short response rates or trends. TestMaker creates one distribution chart for each testusecase. For instance, Figure 2 is the chart for a load test scenario operating the examples_usecase testusecase with a message payload size of 1 and 20 concurrent virtual users (CVUs).

Figure 2 - Transaction Distribution Chart

The transaction distribution chart provides several functions accessible with a right-click of the mouse on the chart. A popup menu appears allowing you to save the chart to a PNG graphic file, zoom in/out, print, and set preferences including fonts, axis settings, and titles.

Resource Utilization Charts

For each testusecase, TestMaker creates a resource utilization chart (also called a System Data Distribution chart). This chart plots a vertical point indicating the CPU, Memory, and Network resource utilization for the duration of the testusecase. The transaction distribution chart is useful when auditing a test to validate results by identifying resource contention, or savings from underutilized resources. TestMaker creates one resource utilization chart for each testusecase, each TestNode, and the target host. For instance, Figure 3 is a chart for a load test scenario operating the examples_usecase on the localhost TestNode with a message payload size of 1 and 20 concurrent virtual users (CVUs).

Note: You must install and operate the PushToTest TestMaker Monitor utility, because TestMaker cannot create resource utilization charts without a monitor.

Figure 3 - Resource Utilization Chart

The resource utilization chart provides several functions accessible with a right-click of the mouse on the chart. A popup menu appears allowing you to save the chart in a PNG graphic file, zoom in/out, print, and set preferences including fonts, axis settings, and titles.

Logged Results Files

TestScenarios log the performance statistics to results files. The default is to store results files in testmaker_home/results. The files are in XML format and are used by data analysis packages including Jasper Reports. The results file XML schema definition is found in testmaker_home/docs/results.xsd.

TestMaker creates a summary report when a load test scenario has completed. Table 1 is an example summary report.

Table 1 - Example Summary Report

Load Test Results												TestNode: localhost
Use Case	Target	Payload	CVUs	TPS	Min	Max	Avr	Errors	CPU	MEM	NET	localhost:TPS	Min	Max	Avr	Errors	CPU	MEM	NET
examples_usecase	notarget	1	10	0.20	16945	48723	32706	0	0.00	0.00	0.0000	0.20	16945	48723	32706	0	0.00	0.00	0.0000
examples_usecase	notarget	1	20	0.13	32829	49911	47112	0	0.00	0.00	0.0000	0.13	32829	49911	47112	0	0.00	0.00	0.0000

Each row of a summary report lists the results of one testusecase showing the data for the local TestNode, followed by all the other TestNodes. The data in each row includes Transactions Per Second (TPS), minimum, maximum, and average transaction response times, and resource utilization. Table 1 shows that test operated at 2 levels of Concurrent Virtual Users (CVUs), 10 and 20.

Performance Comparison Utility (PCU)

TestMaker automatically archives each load test-scenario test-result set into a directory in testmaker_home/results. TestMaker provides a Performance Comparison Utility (PCU) to compare multiple result sets. Access the PCU by selecting the Tools menu bar, Performance Comparison Utility menu command.

The PCU can analyze results from performance, regression, and stress testing and display the analysis data in a graphical environment. The PCU can perform a result set comparison, user result files comparison, and a check for results differences and similarities.

The PCU can processes these data sets:

Test-scenario files
Generated transaction Log
Generated CPU, Memory and Network usage log
Generated summary.
User generated information

Using The PCU

Here are the steps to use the PCU.

Select the result set directories to compare by clicking the Add Result Directory button and choosing the result set directories in the Open file selector dialog. Once selected, the system retrieves information from the directories and creates a list of use cases:

The PCU prepares result comparisons as an aggregate of a test or TestNode, and an operating parameter. To perform an aggregate test comparison, choose the desired comparison options: transaction (TPS) time or memory, CPU, and network utilization. For specific TestNode and operating parameters click the Summary Information tab.

Click on the first Directory to select it.
Choose one or more result sets from the Available Use Cases drop-down menu and click the Add To Comparison List button for each selection.
Click on the second Directory to select it.
Choose one or more result sets from the Available Use Cases drop-down menu and click the Add To Comparison List button for each selection.
Select the Transaction Time item from Comparison Options.
Click the Generate Graph button.

The comparison chart shows the two selected result sets.

The PCU displays comparison charts of two or more TestNodes for validating test results.

Click the Summary Information tab.

Additional documentation, product downloads and updates are at www.PushToTest.com. While the PushToTest TestMaker software is distributed under an open-source license, the documentation remains (c) 2007 PushToTest. All rights reserved. PushToTest is a trademark of the PushToTest company.