Tutorial: Testing and Monitoring Web Applications
This
set of tutorials will show you how to create a test of a
browser-based Web application. These tutorials use the FireFox browser
and TestMaker's included TestGen4Web add-on. An alternate method of
recording tests is provided using the Recorder.
Tutorial 1 - Build a Functional
Test of a Web application
In
this
tutorial we will build a functional test of a Web-application by using
the TestGen4Web plug-in to Firefox. (If FireFox is unavailable to you
then please consider using the Recorder
as an alternative.) To follow this step-by-step tutorial requires that
you first install FireFox, the TestGen4Web addon, and TestMaker 5 or
greater.
A functional test determines that a service operates correctly. In
TestMaker we create a unit test that speaks the native protocol of the
service. In this tutorial the service is a Web application using HTTP
and HTML protocols. TestMaker operates the unit test once and records
the results to a log to show how long each operation takes the service
to complete. We will create the unit test using the TestGen4Web plug-in
to FireFox. Then in the next tutorial we will see how to use the same
unit test as a load test to understand the scalability and performance
of a target service.
TestMaker comes with a TestNode - the test runtime
environment - installed on the LocalHost. The
TestNode software also includes an example Web application called the
Responder at http://localhost:8080/responder/htmlresponder.
The above screen shot shows FireFox displaying the Responder
application. Notice the TestGen4Web toolbar. Here you may start a
recording, pause and stop a recording, edit the recording, and open and
save recordings to files.
Use TestGen4Web to record a unit test of this application by
taking the following steps:
- Open FireFox and browse
http://localhost:8080/responder/htmlresponder
- Click the Record button in the TestGen4Web toolbar.
- Click the file1.html link
- Click the file2.html link
- Enter your name in the First Name and Last name fields of
the top-most Web form
- Enter 100 in the Account Number field
- Enter 5.05 in the Amount field
- Click the Transfer Funds button
- Click the file3.html link
While you were taking these steps TestGen4Web watched your use of the
browser and recorded a unit test of the Web application.
- Click the Stop button in the TestGen4Web toolbar.
- Click the Edit Steps button in the TestGen4Web toolbar.
The
Edit Steps
dialog appears in the FireFox browser to show you the steps it
recorded. Each step has a step number, a label you may provide, an
action, an XPath expression, and a value. The Edit Steps dialog enables
you to add New Steps, edit steps, and rearrange steps.
- Click the OK button. The Edit Steps dialog disappears.
- Click the Play button in the TestGen4Web toolbar to operate
the test.
TestGen4Web
plays-back recorded tests in FireFox as a means to qualify the unit
test. At this point you have created a unit test of this Web
application that verifies the function of the Responder application.
- Click the
Save button in the TestGen4Web toolsbar. A File dialog opens. Save the
recorded test to TestMaker_home/PTT_Examples_UnitTest.testgen4web
Next, we will operate the recorded test in a TestScenario as a
functional test.
- Open
TestMaker. From the desktop this normally means running the
TestMaker.sh (For Unix, Linux, and Mac OS X systems) or TestMaker.bat
(Windows platforms) start-up scripts.
When TestMaker starts it automatically starts a localhost TestNode. A
TestNode is a service that operates a test according to the
instructions in a TestScenario document. TestScenario is
an XML document that
defines the basics of the test, the location of the TestNode's, the
type
of test, resources needed for the test, and logging information.
- Click the New Functional Test icon in the PushToTest
toolbar in TestMaker.
- Save the new test in TestMaker_home/functionaltest.xml
TestMaker creates the TestScenario file named functionaltest.xml and
opens a new Controller panel.
The Controller panel is in the lower left portion of the TestMaker
window. A graphical representation of the TestScenario for the new
functional test appears within a new panel in the
Controller panel. The Controller panel shows controls to run, pause,
stop, edit, expand, and close for this functional test.
- Click the Edit icon in the Controller panel for the new
functional test.
Above the Controller panel and the Output panel
is the Editor area. The left side of the Editor area shows a list of
open Buffers and below it is a detail summary of the currently open
document. The rest of the Editor area is an emacs-style text editor
that is currently showing you the contents of the TestScenario for the
functional test.
Before
we
continue with the tutorial, please take a moment to look at the
following TestScenario definition. The following TestScenario operates
the unit test of the Responder Web
application as a functional test.
<?xml version="1.0" encoding="UTF-8"?>
<testscenario
xmlns="www.pushtotest.com/tm5"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="www.pushtotest.com/tm5" version="2.0">
<!-- PushToTest(tm) Test Scenario for a Functional
Test -->
<basics>
<name>Examples Functional
Test</name>
<defaultdirectory>.</defaultdirectory>
</basics>
<testnodes>
<node name="localhost"
location="http://localhost:8080/TestNetwork/ws/TestNode"/>
</testnodes>
<!-- Dynamically loaded jar files and other resources -->
<resources>
<testgen4web
path="./example_agents/testgen4web_examples/PTT_Examples_UnitTest.testgen4web"/>
</resources>
<functionaltest repeat="1">
<testusecase>
<dimensions>
<usecases>
<test>
<run name="MyTest" testclass="PTT_Examples_UnitTest.testgen4web"
method="testGen4Web" langtype="testgen4web"/>
</test>
</usecases>
</dimensions>
</testusecase>
</functionaltest>
<logs/>
<options>
<sleeptime>0</sleeptime>
<delayBetweenStartingUseCasses
time="1000"/>
<delayBetweenTestCases
time="100"/>
<testTime time="1000"/>
</options>
</testscenario>
TestScenario documents provide all the information needed for TestMaker
to operate a functional test. The
TestScenario begins with basic information about the test, defines the
test use cases, and identifies logging parameters. In the above
TestScenario we point out these elements:
- <basics> names the test and sets the default
directory
- <testnodes> identifies the location of the
TestNode(s) to
operate the test. In this tutorial we are using the default TestNode
that runs on the localhost (automatically installed when you installed
TestMaker.)
- <resources> identifies the files, JAR and WAR
files, and scripts needed to operate the test
- <functionaltest> identifies that this is a
functional test
and will operate the unit test one or more times with a single
concurrent virtual user (CVU)
- <dimensions>, <usecases>,
<tests> identify
the unit tests that make up this functional test. In this tutorial we
have only a single unit test recorded.
- <run> is a method that calls the recorded
unit test.
Notice the testclass, method, and langtype attributes identify the
TestGen4Web recorded unit test. This is the test the
<resources>
section makes available to TestMaker.
- <logs> identifies where and how to log the
results data
- <options> miscellaneous settings
Now
back to the tutorial...
- Copy the TestScenario document above and paste it into the
functional test.
You should now have a TestScenario document that appears as follows.
If you are uncertain that you correctly performed the previous steps we
have included a full written and tested script found in
TestMaker_home/example_agents/testgen4web_examples. Use the Open icon in the
PushToTest toolbar to navigate to the example_agents/testgen4web_examples
directory and open the functionaltest.xml file. TestMaker will display
this TestScenario in a new panel in the Controller, below the
TestScenario we previously created.
We are ready to run the functional test.
- Click the Run button in the Controller panel for the
TestMaker
operates the functional test and shows the results in the lower output
panel. You should see the following:
-
Validating test
-
The test is valid.
- The
functional test is running -
- Test
Resources -
XML Scenario:
/Users/frankcohen/dev/cvs/tm5/testmaker_distribution/example_agents/testgen4web_examples/FunctionalTest.xml
TestGen4Web File:
./example_agents/testgen4web_examples/PTT_Examples_UnitTest.testgen4web
- Loading
Scenario -
The node localhost is valid
The node localhost has loaded the scenario
Node: localhost Done test: 0
-
Functional Test results
-
The node localhost had finished the test successfully
Functional test succeed
In
this
tutorial we built a functional test of a Web-application by using
the TestGen4Web plug-in to Firefox. This
ends the tutorial. Ready for something more interesting? In the next
tutorial we show how to turn this same unit test into a load test
without any additional effort.
Tutorial 2 - Build A Load Test of a Web Application
In
this
tutorial we will build a load test of a Web-application by
using
TestScenario test definition system in TestMaker. To perform this
tutorial you must first follow the steps of Tutorial
1.
First we will give you a summary of the load testing method in
TestMaker called XSTest. Then we will show you step-by-step
instructions to use the TestGen4Web-recorded unit test of the Responder
Web application as a load, scalability, and performance test.
The
XSTest Method in TestMaker
TestMaker
provides load testing, scalability testing, and performance testing.
- Load
testing identifies the throughput of a service by making multiple
concurrent requests to a service and observing the upper range of the
service's ability to respond to concurrent requests. Load testing is
useful to identify the number of users a service can handle given a
certain number of resources (CPUs, network bandwidth, memory, threads,
database connections, servers, and more.)
- Scalability
testing identifies the scalability index of a service as the service
handles increasing levels of concurrent requests. Scalability testing
is useful to forecast resource needs (CPUs,
network bandwidth, memory, threads, database connections, servers, and
more.) Scalability testing enables a business to make tradeoffs at
development and configuration time.
- Performance
testing identifies changes in the scalability index of a service as the
service programming changes and the service resources change. For
instance, when installing a new patch to a database a performance test
will identify an improvement in performance as compared to a previous
test run.
TestMaker
provides the XSTest framework to operate unit tests as load tests,
scalability test, and performance tests. XSTest operates a unit test at
various levels of concurrent
request sizes to determine the scalability index of the target host.
The following chart explains the use case that XSTest implements.
In a TestMaker load test the TestScenario defines the parameters of the
test, the TestUseCase creates one thread for each concurrent virtual
user, and the threads operate concurrently for the duration of the
TestUseCase. Once the TestScenario creates all
of the threads then the TestScenario begins logging results to a
log file. When the test period ends the TestScenario ends the results
logging and ends all of the threads. The TestScenario then operates the
test as the next level of concurrent virtual users. Once all of the
TestUseCases operate the TestScenario tallies the results into a
Scalability Index.
A Scalability Index shows the performance of a target service at a
variety of load levels. For instance, the following chart shows the TPS
results as measured at 3 levels of concurrent requests (CRs.)
The
chart above shows that the service does not scale in proportion to
the number of CRs. For example, if a service performance 8 TPS at 2 CRs
one would expect the same service could perform twice as many TPS at
twice the CRs. The chart above shows that at 4 CRs the service performs
only 14 TPS. Even worse, at 8 CRs the service performs only 17 TPS. A
perfectly scalable system would produce a Scalability Index looking
like the following chart.
now that you have an understanding of the XSTest load testing method in
TestMaker we will show you step-by-step
instructions to use the TestGen4Web-recorded unit test of the Responder
Web application as a load, scalability, and performance test.
Turning
Unit Tests Into Load Tests
TestMaker TestScenario documents are XML documents containing all the
configuration and processing instructions needed for PushToTest
TestMaker to operate a load test. Here is a TestScenario that uses the
unit test we created using TestGen4Web in Tutorial
1.
<?xml version="1.0" encoding="UTF-8"?>
<testscenario
xmlns="www.pushtotest.com/tm5"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="www.pushtotest.com/tm5" version="2.0">
<!-- PushToTest(tm) Test Scenario for a Functional
Test -->
<basics>
<name>Examples Functional
Test</name>
<defaultdirectory>.</defaultdirectory>
</basics>
<testnodes>
<node name="localhost"
location="http://localhost:8080/TestNetwork/ws/TestNode"/>
</testnodes>
<resources>
<testgen4web
path="./example_agents/testgen4web_examples/PTT_Examples_UnitTest.testgen4web"/>
</resources>
<loadtest>
<testusecase>
<dimensions>
<!-- Concurrent request levels for each TestNode
-->
<crlevels>
<crlevel
value="10"/>
<crlevel
value="20"/>
</crlevels>
<usecases>
<usecase
name="Examples_usecase">
<sequence
name="Examples_sequence" proportion="100">
<test>
<run name="MyTest" testclass="PTT_Examples_UnitTest.testgen4web"
method="testGen4Web" langtype="testgen4web"/>
</test>
</sequence>
</usecase>
</usecases>
</dimensions>
</testusecase>
</loadtest>
<logs/>
<options>
<sleeptime>0</sleeptime>
<delayBetweenStartingUseCasses time="1000"/>
<delayBetweenTestCases time="100"/>
<testTime time="60000"/>
</options>
</testscenario>
In
the
above
TestScenario three elements stand out:
- The
<basics>, <testnodes>,
<resources>,
and <usecases> remain exactly the same from the
functional test
in Tutorial 1.
- <loadtest> identifies that this is a load
test.
- <crlevels> identifies concurrent request
levels. Load tests
use concurrently running threads to operate a unit test over-and-over
for the period of the test. The <crlevel> values tell
TestMaker
how many threads to operate. For this test, TestMaker instantiates 10
concurrently running threads, operates the threads for the 1 minute
period of the test, tears-down the running threads, and then operates
20 new threads for the period of the test. In a system with linear
scalability twice the number of concurrently running request agents
should generate twice the throughput, as measured in transactions per
second (TPS) at the client.
- <options> defines values used to time the
test.
- <sleeptime> is set to 0 meaning there will
be no delay
between running one usecase and the next. Once the <run>
method
operates there will be no delay before operating the
<run> method
again.
- <delayBetweenStartingUseCases> is set to
1000 meaning
there will be a 1000 millisecond (1 second) delay between starting each
of the concurrently running threads. This is done to not overwhelm the
resources of the load generating system (the TestNode) and the target
host system.
- <delayBetweenTestCases> is set to 100
meaning there will
be a 100 millisecond (1/10 of 1 second) delay once a test period (set
by the <testTime>) is completed before the test creates
the next
set of threads.
- <testTime> is set to 60000 meaning there
will be a 60000
millisecond (60 seconds, or 1 minute) time to operate the threads for
each <crlevel>. The <testTime> does NOT
include the time it
takes to instantiate or teardown the threads.
Next
we will operate the load test and observe the results.
- Click the Close button on any open TestScenarios in the
Controller panel.
- Click the New Load Test icon in the PushToTest
toolbar in TestMaker.
- Save the new test in TestMaker_home/loadtest.xml
TestMaker creates the TestScenario file named loadtest.xml and
opens a new Controller panel.
The Controller panel is in the lower left portion of the TestMaker
window. A graphical representation of the TestScenario for the new load test appears within a new panel in the
Controller panel. The Controller panel shows controls to run, pause,
stop, edit, expand, and close for this functional test.
- Click the Edit icon in the Controller panel for the new
load test.
- Copy the load testing TestScenario document above and paste
it into the load test.
If you
are uncertain that you correctly performed the previous steps we
have included a full written and tested script found in
TestMaker_home/example_agents/testgen4web. Click the close icon for the
TestScenario in the Controller panel. Use the Open icon in the
PushToTest toolbar to navigate to the example_agents/testgen4web
directory and open the loadtest.xml file. TestMaker will display
this TestScenario in a new panel in the Controller.
- Click the Run icon in the Controller panel for the load
test.
The
Output panel - to the right of the Controller panel - shows you
status as TestMaker stages the load test using the local TestNode. The
contents of the TestScenario panel update to show you the test progress
graphically.
Click the Expand icon in the Controller panel for the load test. The
Controller for the load test appears in a new window as shown below.
In the
Controller window for the running TestScenario starts at the top with
icons to run, pause, stop, and edit control.
Below that is a horizontal slider that, while the test is operating,
moves from left to right and allows you to change the order of the test
as the test operates.
Below that is an icon bar indicating the TestNode machines that
are actually running the tests in a distributed fashion, and this icon
shows you the status of the running TestNode.
Rolling the mouse over each one of these icons indicates special
information about how well that test machine is performing.
Below the TestNode icon bar is a live scalability index chart that
shows you the live scalability index results as the test runs.
This chart is followed by a transaction distribution chart showing you
whether there were any anomalies that happened while the test was
running.
Below the transaction distribution chart - when the Monitor is
installed - is several other charts to show you the CPU, network, and
memory utilization of both the TestNode machines that are generating
the test load, as well as the machine, the target machine that's
actually running the application.
This concludes Tutorial 2. In this tutorial we showed a summary of
the load testing method in
TestMaker called XSTest. Then we showed the step-by-step
instructions to use the TestGen4Web-recorded unit test of the Responder
Web application as a load, scalability, and performance test. We
repurposed the unit test from a Functional test in Tutorial 1 to a
load, scalability, and performance test with no extra effort.
Wouldn't it be great to turn the unit test from a load test into a service monitor? Get ready for Tutorial 3.
Tutorial 3 - Build A Service
Monitor of a Web application
Intm5mainft.pngtm5mainft.png
this
tutorial we will build a service monitor of a Web-application by
using a TestScenario test definition. To perform this
tutorial you must first follow the steps of Tutorial
1. A service monitor is a functional test that operates periodically.
A unit test uses the native protocols of a service to test the
operation of a function in a service. A service monitor in TestMaker
uses a unit test to periodically operate a service. The service monitor
records the success or failure of the operation, alerts an operator
when problems exist, and displays the service monitoring functions in a
dashboard.
TestScenario documents are XML documents containing all the
configuration and processing instructions needed for PushToTest
TestMaker to operate a service monitor. Here is a TestScenario that uses the
unit test we created using TestGen4Web in Tutorial
1.
<?xml version="1.0" encoding="UTF-8"?>
<testscenario xmlns="www.pushtotest.com/tm5"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="www.pushtotest.com/tm5
C:\DOCUME~1\llara\MYDOCU~1\testmaker\XSD\testscenario.xsd"
version="2.0">
<!--
PushToTest(tm) Test Scenario for a Service Monitor
PushToTest is the open-source SOA governance and test automation platform
For technical support, documentation, and help see http://www.pushtotest.com
-->
<basics>
<name>Examples Service Monitor Test</name>
<defaultdirectory>.</defaultdirectory>
</basics>
<!-- Location of TestNodes to run this test -->
<testnodes>
<node name="localhost" location="http://localhost:8080/TestNetwork/ws/TestNode"/>
</testnodes>
<!-- Dynamically loaded jar files and other resources -->
<resources>
<testgen4web
path="example_agents/testgen4web_examples/PTT_Examples_UnitTest.testgen4web"/>
</resources>
<monitor timeBetweenTest="10000">
<until>
<exception type="java.lang.Exception"></exception>
</until>
<testusecase>
<dimensions>
<usecases>
<usecase
name="Examples_usecase">
<sequence
name="Examples_sequence">
<test>
<run name="MyTest"
testclass="PTT_Examples_UnitTest.testgen4web"
method="testGen4Web"
langtype="testgen4web"/>
</test>
</sequence>
</usecase>
</usecases>
</dimensions>
</testusecase>
</monitor>
<logs/>
<options>
<sleeptime>0</sleeptime>
<delayBetweenStartingUseCasses time="1000"/>
<delayBetweenTestCases time="100"/>
<testTime time="1000"/>
</options>
</testscenario>
In
the
above
TestScenario three elements stand out:
- The
<basics>, <testnodes>,
<resources>,
and <usecases> remain exactly the same from the
functional test
in Tutorial 1 and the load test in Tutorial 2.
- <monitor> identifies that this is a service monitor. The
timeBetweenTest attribute tells TestMaker to operate this test and then
wait for 10 seconds (10,000 milliseconds) and then to repeat the test.
- <until> identifies the exceptions that will stop the
service monitor. Any other exceptions are logged but do not stop the
monitor.
Next
we will create and operate a service monitor and observe the results.
- Click the Close button on any open TestScenarios in the
Controller panel.
- Click the New Service Monitor icon in the PushToTest
toolbar in TestMaker.
- Save the new test in TestMaker_home/servicemonitor.xml
TestMaker creates the TestScenario file named servicemonitor.xml and
opens a new Controller panel.
The Controller panel is in the lower left portion of the TestMaker
window. A graphical representation of the TestScenario for the new service monitor appears within a new panel in the
Controller panel. The Controller panel shows controls to run, pause,
stop, edit, expand, and close for this functional test.
- Click the Edit icon in the Controller panel for the new
load test.
- Copy the service monitor TestScenario document above and paste
it into the service monitor.
If you
are uncertain that you correctly performed the previous steps we
have included a full written and tested script found in
TestMaker_home/example_agents/testgen4web_examples. Click the close icon for the
TestScenario in the Controller panel. Use the Open icon in the
PushToTest toolbar to navigate to the example_agents/testgen4web_examples
directory and open the servicemonitor.xml file. TestMaker will display
this TestScenario in a new panel in the Controller.
- Click the Run icon in the Controller panel for the load
test.
The
Output panel - to the right of the Controller panel - shows you
status as TestMaker operates the test. The
contents of the TestScenario panel update to show you the test progress
graphically. Many TestMaker users leave the controller panel open as a
dashboard to gain visibility into their service operations.
Click the Expand icon in the Controller panel for the service monitor. The
Controller for the service monitor appears in a new window as shown below.
The service monitor shows the most recent test results in the table.
The duration value is the amount of time taken to perform the use case
for the service monitor. The large image below the name of the service
monitor test shows an up-arrow during normal operation. The image turns
to a down-arrow when the service monitor operated a test that returned
an exception in the past 10 minutes.
This concludes Tutorial 3.
In this tutorial we showed a summary of the
load testing method in
TestMaker called XSTest. Then we showed the step-by-step
instructions to use the TestGen4Web-recorded unit test of the Responder
Web application as a load, scalability, and performance test. We
repurposed the unit test from a Functional test in Tutorial 1 to a
load, scalability, and performance test with no extra effort.
In
this
tutorial we showed you how to build a service monitor of a Web-application by
using a TestScenario test definition. We showed that a service monitor is a functional test that operates periodically.
We used the unit test we created with TestGen4Web to use the native protocols of a service to test the operation
of a function in a service. A service monitor in TestMaker uses a unit
test to periodically operate a service. The service monitor records the
success or failure of the operation, alerts an operator when problems
exist, and displays the service monitoring functions in a dashboard.
In these tutorials we demonstrated building a functional test of a
Web-browser-based application, building a scalability and performance
load test of a Web-browser-based application, and building a service
monitor. We accomplished all three with no programming experience
required and through the TestScenario system with a minimum of effort.
Where to go from here?
This tutorial recorded a Web-application using TestGen4Web and
FireFox. If you need to do anything more advanced than playback of the
recorded application then please see Tutorial 7, including writing tests in Java, Jython, JRuby, Groovy, Rhino and several other languages.
A list of
frequently asked questions (FAQ)
Known
problems and incomplete features in this release of PushToTest TestMaker
(online)
|
|
|
|
Additional
documentation, product downloads and updates are at www.PushToTest.com.
While the PushToTest TestMaker software is distributed under an
open-source
license, the documentation remains (c) 2007 PushToTest. All rights
reserved. PushToTest is a trademark of the PushToTest company.
|
|