A TestScenario is a complete definition of a functional test, load test, and service monitor. TestScenarios identify the TestNodes that operate the test, the resources the TestNode needs to operate the test, and the dimensions of the test.
TestScenario documents follow an XML Schema Definition (XSD) found at http://downloads.pushtotest.com/tm5/testscenario.xsd. Examples of TestScenario documents used in functional, load, scalability, and performance tests plus service monitors are found in the Tutorials.
Defines the test scenario name and default directory.
<basics>
<name>GeospatialTest</>
<defaultdirectory>./geospatialtest</defaultdirectory/>
</basics>
The <name> element defines the test scenario name. TestMaker uses the name in the controller panel, in the status output panel, and in the results files. The <defaultdirectory> element defines the default path to find scripts, JAR files, and other resources.
Defines a TestNode name and URL for running the test scenario.
<testnodes>
<node
name="localhost" location="http://localhost:8080/TestNetwork/TestNode"/>
</testnodes>
Defines a set of name/value pairs passed to running tests; allows for parameterized test scenarios.
<arguments>
<argument
name="targethost" value="10.0.0.1/gis"/>
<argument
name="id" value="admin"/>
<argument
name="password" value="admin"/>
</arguments>
A running test has access to the name/value pairs through a dictionary (Jython) or HashMap (Java).
Identifies files to be dynamically loaded when the test is run on a TestNode.
<resources>
<jar path="./lib/test.jar"/>
<testgen4web path="./PTT_Examples_UnitTest.testgen4web"/>
<soapui path="example_agents/soapui_examples/PTT_Examples-soapui-project.xml"/>
<module name="myJythonModule" path="./example_agents/scriptTutorial/example1.py"/>
<module name="myRubyModule" path="./example_agents/rubyExample/calc.rb"/>
<data path="./example_agents/dplCSVExample/csv.txt"/>
</resources>Identifies Data Production Libraries providing dynamic data when the test is run.
<DataSources>
<dplDefType name="SimpleDPL"
library="./example_agents/dplExample/dplSimple.jar"
lang="java"
classname="com.dpl.SimpleDPL"/>
<dpl name="lingo" type="LingoDPL"/>
<dpl name="simple" type="SimpleDPL">
<argument name="argrun2" value="Data"/>
</dpl>
<dpl name="simple" type="SimpleDPL">
<argument name="argrun2" dpl="lingo" value="getNextData"/>
</dpl>
</DataSources>
See the
example_agents directory for examples of the DataSources function,
including dplCSVExample, dplExample, and counterDPLExample.
Defines one or more log handlers handling logged entries during a running test.
<logs>
<log
type="file" path="logs/mylog.txt" level="2"/>
</logs>
<test>
<run name="geo_insert" testclass="com.pushtotest.example.test"
method="insert" langtype="java"/>
<run name="geo_query" testclass="com.pushtotest.example.test"
method="query" langtype="java"/>
<run name="geo_update" testclass="com.pushtotest.example.test"
method="update" langtype="java"/>
<run name="geo_delete" testclass="com.pushtotest.example.test"
method="delete" langtype="java"/>
</test>
The enhancement specification for step logging is found here, here, and here.

<logs>
<log type="console" level="info"/>
<log type="file" path="mylog.txt" level="info"/>
<log type="database"
connector="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost/testmaker"
loggin="llara"
password="12345"/>
</logs>
<functionaltest repeat="1">
<loadtest>
<monitor timeBetweenTest="10000">
<until>
<exception type="java.lang.Exception"></exception>
</until>
</monitor>
Defines default and optional values for the test.
<options>
<sleeptime>0</sleeptime>
<delayBetweenStartingUseCases time="10"/>
<delayBetweenTestCases time="60"/>
<testTime time="60000"/>
</options>
With <sleeptime> set to 0 there is no delay before running the next use case; that is, once the <run> method operates, there is no delay before operating the <run> method again.
With <delayBetweenStartingUseCases> set to 1000 there is a 1000 millisecond (1 second) delay before starting each concurrently running thread. This prevents exhausting resources on the load generating (TestNode) and target host systems.
With <delayBetweenTestCases> set to 100, at the completion of the test period (set by <testTime>) there is a 100 millisecond (1/10 of 1 second) delay before the test creates the next set of threads.
With <testTime> set to 60000 there is a 60000 millisecond (60 seconds; 1 minute) time to operate the threads of each <crlevel>. Note the <testTime> value does not include the time setup the test - when the TestNodes instantiate the Threads - or the time it takes to teardown the threads.
Sets the notification parameters for the email notification triggered when a test changes state.
<notifications>
<email
on="start" email="fcohen@pushtotest.com"/>
<email
on="end" email="fcohen@pushtotest.com"/>
<email
on="interrupted" email="fcohen@pushtotest.com"/>
<email
on="testcasestart" email="fcohen@pushtotest.com"/>
<email
on="hourly" email="fcohen@pushtotest.com"/>
</notifications>
<mail>
<username value="yourUsename"/>
<password value="yourPassword"/>
<hostname value="YourMailHost"/>
</mail>
<notifications>
<email on="start" email="EmailDirection" subject="test scenario started"
message="The test started. - From Pushtotes"/>
<email on="testcasestart" email="EmailDirection"
subject="test case started"/>
<email on="end" email="EmailDirection"
message="The test scenario had ended. - From push to test"/>
</notifications>
Defines the functional unit test used during the test operation.
<testusecase> repeatedly operates a single unit test, a sequence
of unit tests, or a mix of sequences for the duration of the test.
These sections show how to configure these tests in a TestScenario.xml
file.
<testusecase name="simpletest">
<test>
<run name="geo_insert" testclass="com.pushtotest.example.test"
method="runTest" langtype="java"/>
</test>
</testusecase>
In this example TestMaker stages a single test to run for the default time of 1 minute. At the start of the test period TestMaker instantiates the Java class test in the com.pushtotest.example package. During the 1 minute test run, TestMaker calls the runTest method of the test object, sleeps for the default 1 second, then repeats the call to runTest until time expires.
This form is used when setup and teardown instructions are required by the service under test or the test itself:
<testusecase name="simpletest">
<setup name="geo_insert" testclass="com.pushtotest.example.test"
method="setUp" langtype="java"/>
<test>
<run name="geo_insert" testclass="com.pushtotest.example.test"
method="runTest" langtype="java"/>
</test>
<teardown name="geo_insert" testclass="com.pushtotest.example.test"
method="tearDown" langtype="java"/>
</testusecase>
In this example TestMaker stages a single test to run for the default time of 1 minute. At the start of the test period TestMaker instantiates the Java class test in the com.pushtotest.example package and runs the setup method of the test object once. During the 1 minute test run, TestMaker calls the runTest method of the test object, sleeps for the default 1 second, and repeats the call to runTest until time expires. At the end of the test period, TestMaker calls the teardown method once and the test ends. The langtype attribute specifies the language used, for example Java or Jython.
This form is used when you need to test a service by calling several functions in series:
<testusecase name="simpletest">
<setup name="geo_insert" testclass="com.pushtotest.example.test"
method="setUp" langtype="java"/>
<test>
<run name="geo_insert" testclass="com.pushtotest.example.test"
method="insert" langtype="java"/>
<run name="geo_insert" testclass="com.pushtotest.example.test"
method="query" langtype="java"/>
<run name="geo_insert" testclass="com.pushtotest.example.test"
method="update" langtype="java"/>
<run name="geo_insert" testclass="com.pushtotest.example.test"
method="delete" langtype="java"/>
</test>
<teardown name="geo_insert" testclass="com.pushtotest.example.test"
method="tearDown" />
</testusecase>
In this example TestMaker stages a single test to run for the default time of 1 minute. At the start of the test period TestMaker instantiates the Java class test in the com.pushtotest.example package and runs the setup method of the test object once. During the 1 minute test run, TestMaker calls the insert method of the test object, sleeps for the default 1 second, calls the query method, sleeps, runs the update method, sleeps, runs the delete method, sleeps, and repeats the call to insert until time expires. At the end of the test period, TestMaker calls the teardown method once and the test ends.
This example's use cases are functional tests to verify a service operates correctly. The recorded results show how long the service took to complete each operation.
This section shows how to turn a functional test into a load test to understand the scalability and performance of a target service.
A Scalability Index shows how a target service performs under various load levels. For instance, TestMaker runs a testusecase with the specified number of concurrent running threads, known as Concurrent Requests (CRs), and tracks the number of completed testusecases measured as Transactions Per Second (TPS). Contrasting TPS to CRs values yields the Scalability Index. This chart shows the TPS results for three CRs levels:
The chart shows the service does not scale in proportion to the number
of CRs.
With the performance of 8 TPS at 2 CRs, one would expect the service to
perform twice as many TPS at twice the CRs.
However, the chart shows at 4 CRs the service performs only 14 TPS,
and even worse, at 8 CRs the service performs only 17 TPS.
A perfectly scalable system has this Scalability Index:

The <testusecase> definition for load-tests builds on the prior examples with this form defining load tests at 2, 4, and 8 CRs:
<dimensions>
<crlevels>
<crlevel value="2" />
<crlevel value="4" />
<crlevel value="8" />
</crlevels>
<testusecases>
<testusecase name="simpletest">
<test>
<run
name="geo_insert" testclass="com.pushtotest.example.test"
method="runTest"
langtype="java"/>
</test>
</testusecase>
</testusecases>
</dimensions>
In this example TestMaker stages a load test to run for the default time of 1 minute. TestMaker instantiates two concurrently running threads at the start of the test period. In each thread TestMaker instantiates the Java class test in the com.pushtotest.example package. In each thread during the test, TestMaker calls the runTest method of the test object, sleeps (for the default 1 second), then repeatedly calls runTest until time expires. At the end of the test period, TestMaker ends the threads, starts the next test by repeating the test with 4 threads, then does the same operation again with 8 threads.
To make defining CRs easier, TestMaker supports a short-hand syntax. This is the CRs levels definition from the example:
<crlevels>
<crlevel value="2" />
<crlevel value="4" />
<crlevel value="8" />
</crlevels>
The equivalent definition using short-hand is:
<crlevels>
<crlevel start="2" multiple="2" steps="3"/>
<crlevels>
This defines the same 3 CRs levels (steps), starting with 2 (start), and increasing by a multiple of 2 (multiple) for subsequent steps, yielding the same 2, 4, and 8 CRs level values.
TestMaker uses the <crlevels> element to define a single dimension of a test, but TestMaker also supports multiple dimensions.
For instance, TestMaker can vary the message size of a call to a SOAP-based Web Service using the <messagesizes> element:
<messagesizes>
<messagesize value="1" />
<messagesize value="2" />
<messagesize value="3" />
</messagesizes>
From this definition, TestMaker operates a test with the message size of 1,
then repeats the test with message sizes of 2 and 3.
The test case object defines the meaning of the message size value.
Each dimension used geometrically increases the number of tests.
For instance, this test definition causes TestMaker to run 9 test
cases (3 CRs levels multiplied by 3 message size levels):
<dimensions>
<crlevels>
<crlevel value="2" />
<crlevel value="4" />
<crlevel value="8" />
</crlevels>
<messagesizes>
<messagesize value="1" />
<messagesize value="2" />
<messagesize value="3" />
</messagesizes>
<testusecases>
...
</dimensions>
This chart illustrates the resulting Scalability Index showing increasing CRs with increasing message sizes:

Consider the TPS results between the 2 and 8 CRs levels. At the 2 CRs level increasing message size slightly reduces TPS levels, but at the 8 CRs level increasing message size dramatically reduces TPS levels.
An earlier example showed testing a service by calling several functions in series; this series of functions is called a <sequence>. Real world use cases often need more than one sequence operating at a time and TestMaker supports operating multiple sequences for each <testusecase>. For instance, testing a database doing insert and update operations concurrently may uncover contention problems.
This is an example showing the definition of two sequences:
<crlevels>
<crlevel value="2" />
<crlevel value="4" />
<crlevel value="8" />
</crlevels>
...
<testusecase name="sequencetest">
<setup name="geo_insert" testclass="com.pushtotest.example.test"
method="setUp" langtype="java">
<argument name="geoval" value="1738"/>
</setup>
<sequence name="inserts" proportion="80">
<test name="test1">
<setup
name="geo_insert" testclass="com.pushtotest.example.test"
method="seqSetup"
langtype="java">
<argument
name="geoval" value="1738"/>
</setup>
<run
name="geo_insert" testclass="" method="" langtype="java">
<argument
name="geoval" value="1738"/>
</run>
<teardown
name="geo_insert" testclass="com.pushtotest.example.test"
method="geoteardown"
langtype="java"/>
</test>
</sequence>
<sequence name="queries" proportion="20">
<test>
<setup
name="geo_insert" testclass="com.pushtotest.example.test"
method="seqSetup"
langtype="java">
<argument
name="geoval" value="1738"/>
</setup>
<run
name="geo_insert" testclass="" method="" langtype="java">
<argument
name="geoval" value="1738"/>
</run>
<teardown
name="geo_insert" testclass="com.pushtotest.example.test"
method="geoteardown"
langtype="java"/>
</test>
</sequence>
<teardown name="geo_insert" testclass="com.pushtotest.example.test"
method="geoteardown" langtype="java"/>
</testusecase>
In the example, TestMaker operates the test at 2, 4, and 8 CRs levels. At each CRs level, TestMaker instantiates the corresponding number of concurrently running threads, and the threads instantiate one of the two <sequence>s. TestMaker sets 80% of the threads to operate the inserts sequence and the remaining 20% to operate the queries sequence.
A
<sequence> has a proportions attibute, which defines the number
of threads that will execute the actual sequence. This number of
threads is calculated from the crlevel.
In the case the number of threads ends in decimals (3.7 threads in a
sequence for instance) this number is rounded to the nearest integer
number (4 threads).
It can happen that the number of thread in some sequence falls below 0.5, and then the sequence won't be run (0 threads).
To allow the user have the capability of choosing which sequences to
run, a new attribute will be added to the sequence tag: <sequence
mustrun='true'>
The idea is that if the mustrun attribute is true, the sequence will be
run in AT LEAST one thread, even if the number of threads for this
sequence is zero.
<loadtest>
<testusecase>
<dimensions>
<crlevels>
<crlevel value="10"/>
<crlevel value="20"/>
</crlevels>
<usecases>
<usecase name="WHILE Test">
<sequence name="while test" proportion="100">
<setup
testclass="whiletest.While" method="setup" langtype="java">
<argument name="quantity" value="100"/>
</setup>
<test>
<while testclass="whiletest.While" method="mustContinue"
langtype="java"/>
<run testclass="whiletest.While" method="run"
langtype="java"/>
</test>
</sequence>
</usecase>
</usecases>
</dimensions>
</testusecase>
</loadtest>
<options>
<sleeptime>1</sleeptime>
<delayBetweenStartingUseCases time="1000"/>
<delayBetweenTestCases time="100"/>
<testTime time="20000"/>
</options>
A discussion of the <while> is found here. See PushToTest_home/example_agents/whiletest for an example.
PushToTest TestMaker provides a Monitor to correlate performance hotspots to testusecase activity. The Monitor installs on TestNodes and the target system. TestMaker TestScenarios use <monitoring> to identify the target host and turn monitoring on or off when presenting results:
<monitoring enablednodes="true" enabledtarget="false" targethost="http://localhost" />The Test Object Oriented Library (Tool) is a library of test objects that handle all the communication with HTTP, HTTPS, SOAP, .NET, XML-RPC, Telnet, eMail, and JDBC hosts. TOOL is designed to be called from a high level language, such as unit tests written in Java, Jython, Groovy, Ruby, PHP, and the other supported script languages. This guide covers the overall design of Tool and the objects available to build intelligent test agents.Goals
Modern software test automation tools need to meet these criteria:Tool is a free open-source design for a library of test objects. The Tool design enables developers to build and run intelligent test agents. The test agents drive Web applications and SOAP-based Web Services as an everyday user would. Multiple concurrent agents test an application for scalability and performance under near real production settings.
The design of Tool should be language and implementation independent. While we will try and use good Object Oriented design and architecture, it should not be assumed that someone would need to implement Tool in a language that explicitly supports OO structures. (e.g. a pure C implementation - using OO techniques in C should be possible.)
Tool is a library. This means that on its own, Tool will not accomplish anything. There is no user interface that would allow an end user to use Tool on its own. Tool is intended to be used in other projects that offer this UI to an end user. With that in mind, the audience for Tool is Software Developers - not end users.
Tool should offer software developers a flexible framework for building test suites to test Web systems. While we are targeting software developers with Tool, we ought to give them all of the help that we can with being able to create products that can be used by end users that are not developers.
(or namespace or modules depending on what language you use)
datasource:
These Objects are used get data for use with making requests or for other things that require access to data files, databases, etc.
protocolhandler:
This contains the Objects related to making requests via a specific protocol. This is the request portion of a transaction.
response:
This is the response portion of a transaction that is the information returned from a request made with a protocol handler.
util:
This is for generally useful utilities that do not directly relate to the core functionality, but could be used by them or in code that would use them. Examples: encoding, cryptographic function, properties handling
(Hopefully the pseudo-UML in the following section makes a sense to you.)
datasource:
- DataSource - An interface that needs to be implemented by DataSource
implementations to allow access to the data.
- Csv - Access data in a Comma Separated Values flat file.
- Tsv - Access data in a Tab Separated Values flat file.
- Db - Access information that is stored in a Database. If needed
by the language, make Database specific implementations of the
Db to access a specific database type.
Datasource <-- Csv
Tsv
Db <-- PostgreDb (if necessary)
OracleDb
protocolhandler:
- Protocol - Interface to implement to create a new Protocol type.
- Header - An interface to setting Header values for a protocol. Protocol
specific implementations will need to be provided.
- Body - An interface to setting the Body of a Protocol request. Protocol
specific implementations will need to be provided.
- ProtocolHandler - A facade in front of all of the Protocols that allows an
easy way to instantiate new objects (factory methods) and call methods on a
Protocol regardless of which specific type it is.
- *Protocol - The specific implementations of the different Protocol handlers.
Protocol <-- HTTPProtocol
SOAPProtocol
?
<>-- Header <-- HTTPHeader
IMAPHeader
?
Body <-- POP3Body
SOAPBody
?
ProtocolHandler <>-- Protocol
response:
- Response - Interface to implement to define a response that will be returned
by a specific protocol.
- ResponseLink - An interface that defines the access to Response items that
can be chained together with the ResponseChain to define how to deal with a
Response object. This is how searching responses and things of that sort
will be implemented.
- ResponseLinkConfig - An Object that is created to hold values used to
initialize a ResponseLink.
- *Response - The implementations of Protocol specific response types.
- ResponseChain - A Chain of Responsibility pattern class that allows a
response to be passed among many ResponseLinks.
Response <-- HTTPResponse
SOAPResponse
?
ResponseLink <-- HTTPSearchLink
?
<>-- ResponseConfig
ResponseChain <>-- ResponseLink
util:
- Codec - Interface to encoding/decoding methods.
- Crypt - Interface to cryptographic routines.
- Lingo - Utility that creates pseudo-text to use as subjects or bodies of
messages, form posts, etc.
- Queue - For languages where there is no Queue class, this provides a Queue
implementation.
- PropertyFinder - A class that hides the details of loading property files
that contain strings and other configuration directives and handles
dynamically changing the values of these.
Codec <-- Base64Codec
URLCodec
HexCodec
Crypt <-- ?
PropertyFinder
Lingo
Queue
TOOL is written in Java and comes with Javadoc code documentation. Javadoc is the standard way Java engineers annotate their code. The Javadoc code documentation for Tool is also found on-line at docs.pushtotest.com.
|