Tutorial: Testing and Monitoring SOAP-based Web Services

This set of tutorials will show you how to create a test of a SOAP-based Web service. These tutorials use the Eviware soapUI utility that is included and integrated with TestMaker. Of course, you could write a test script from scratch that speaks native SOAP, Ajax, REST, and Ajax protocols using the TOOL protocol handler library. See the Advanced Topics tutorial for details. But for now, this tutorial uses the soapUI utility to require little or no programming experience to accomplish the following:

Build A Functional Test of a SOAP-based Web service
Build A Scalability and Performance Load Test of a SOAP-based Web service
Build A Service Monitor of a SOAP-based Web service

Tutorial 1 - Build a Functional Test of a SOAP-based Web Service

In this tutorial we will build a functional test of a Web service by using the soapUI utility that is integrated in TestMaker 5 or greater. To follow this step-by-step tutorial requires that you first install TestMaker 5 or greater. Also, this tutorial requires you to operate on a machine that has connectivity to the public Internet.

A functional test determines that a service operates correctly. In TestMaker we create a unit test that speaks the native protocol of the service. In this tutorial the service is a Web service using SOAP protocols. TestMaker operates the unit test once and records the results to a log to show how long each operation takes the service to complete. We will create the unit test using the soapUI utility. Then in the next tutorial we will see how to use the same unit test as a load test to understand the scalability and performance of a target service.

This tutorial will use an example Web service hosted by PushToTest at the examples.pushtotest.com service using SOAP Document-literal encoding (Message-style). The WSDL definition for this service is found at: http://localhost:8080/axis/services/MessageService?WSDL. The Web service offers a single function named echoMessage that returns the XML you send it.

In this tutorial we will start the soapUI utility, then identify the URL of the WSDL definition for the service, create a TestSuite, and then write a TestMaker TestScenario to turn the TestSuite into a functional test, a load test, and a service monitor.

Run TestMaker
Choose Start soapUI from the Tools drop-down menu. soapUI takes a few minutes to start-up.

The soapUI window divides into several areas. On the left is a navigation panel to work with one or more projects, TestSuites, and operations. The bottom of the navigation panel displays detailed information about the item selected in the navigation panel. To the right of the navigation panel is a desktop area where various content panels will appear. For instance, one content panel will show a skeleton of a Web service request and the service's response in XML format.

Click the Open WSDL Project from the File menu in soapUI.
Type 'Examples' as the Project Name
Type 'http://examples.pushtotest.com/axis/services/MessageService?wsdl' as the Initial WSDL
Click the OK button

soapUI will communicate with the examples.pushtotest.com service, retrieve the WSDL definition for the MessageService and create sample requests for the operations in this service. This takes a minute. soapUI will then ask for a file name to store this project.

Save the project by choosing a directory, file name, and clicking the Save button.

The Projects navigation panel will update to show an Examples project with a child element for MessageServiceSoapBinding.

Click the + expansion controls to view the echoElements and Request 1 operations.
Right-click on Request 1 to see the pop-up menu of options.

soapUI knows enough about this service to build a TestCase of this Web service but for now we will first work with the Request Editor to gain some familiarty with it.Internally soapUI uses a combination of XMLBeans and XFire to understand many of the commonly used WSDL and Web service protocols.

Click Open Request Editor from the pop-up menu of options for Request 1.

soapUI opens a new window in the desktop area titled echoElements - Request 1. This is a request/response editor. The request panel is populated with a skeleton of the SOAP request message the MessageService requires. The ? character indicates data that you may complete.

Replace the ? character with <hello>PushToTest Here</hello>
Click the Submit Request To Endpoint icon - the top, left-most icon - to send the request to the service.

After a few seconds soapUI displays the Web service response in a panel to the right of the request.

soapUI reports the time it took to send the request and receive the response at the bottom of the Request/Response editor. soapUI shows the SOAP response message in the panel to the right of the request panel. Next we will create a TestSuite that uses this request.

Right-click the mouse on the Request 1 entry in the navigation panel and choose the Add To TestCase command.

Name the new TestSuite simply 'TestSuite' and click the OK button.

TestSuites have one or more TestCases. For this tutorial, simply enter TestCase and click the OK button.

soapUI provides many features to make TestCase development easy. To accept the default enabled features click the 'OK' button.

soapUI creates the new TestSuite as a child to the Examples project listed in the navigation panel. A new TestCase window opens in the desktop area. Within the TestCase window you may create additional test steps, test the TestScenario, set optional parameters for security settings, URLs, attributes, add data that will be injected into the requests at run-time, and add response validation logic. Details are found in the soapUI documentation.

At this point you have created a unit test of this Web service that verifies the function of the MessageEcho service. soapUI saves the changes you made automatically to the file and location you chose in the previous steps.

Next, we will operate the recorded test in a TestScenario as a functional test.

Open TestMaker. From the desktop this normally means running the TestMaker.sh (For Unix, Linux, and Mac OS X systems) or TestMaker.bat (Windows platforms) start-up scripts.

When TestMaker starts it automatically starts a localhost TestNode. A TestNode is a service that operates a test according to the instructions in a TestScenario document. TestScenario is an XML document that defines the basics of the test, the location of the TestNode's, the type of test, resources needed for the test, and logging information.

Click the New Functional Test icon in the PushToTest toolbar in TestMaker.
Save the new test in TestMaker_home/functionaltest.xml

TestMaker creates the TestScenario file named functionaltest.xml and opens a new Controller panel.

The Controller panel is in the lower left portion of the TestMaker window. A graphical representation of the TestScenario for the new functional test appears within a new panel in the Controller panel. The Controller panel shows controls to run, pause, stop, edit, expand, and close for this functional test.

Click the Edit icon in the Controller panel for the new functional test.

Above the Controller panel and the Output panel is the Editor area. The left side of the Editor area shows a list of open Buffers and below it is a detail summary of the currently open document. The rest of the Editor area is an emacs-style text editor that is currently showing you the contents of the TestScenario for the functional test.

Before we continue with the tutorial, please take a moment to look at the following TestScenario definition. The following TestScenario operates the unit test of the Responder Web service as a functional test.

<?xml version="1.0" encoding="UTF-8"?>


<testscenario xmlns="www.pushtotest.com/tm5"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="www.pushtotest.com/tm5 


" version="2.0">


    <!--


  PushToTest(tm) Test Scenario for a Functional Test





  PushToTest is the open-source SOA governance and test automation platform


  For technical support, documentation, and help see http://www.pushtotest.com


  -->


    <basics>


        <name>SOAP Functional Test</name>


        <defaultdirectory>.</defaultdirectory>


    </basics>


    <testnodes>


        <node name="localhost" location="http://localhost:8080/TestNetwork/ws/TestNode"/>


    </testnodes>


    <!-- Dynamically loaded jar files and other resources -->


        <resources>


          <soapui
path="example_agents/soapui_examples/PTT_Examples-soapui-project.xml"/>


        </resources>


    <functionaltest repeat="1">


        <testusecase>


            <dimensions>


                <usecases>


           
        <usecase
name="SOAP_usecase">


           
            <sequence
name="SOAP_sequence">


           
           
    <test>

                           
  <run name="setuptest" testclass="testsuite" method="testcase"
langtype="soapui"/>


           
           
    </test>


           
           
</sequence>


                    </usecase>


                </usecases>


            </dimensions>


        </testusecase>


    </functionaltest>


    <logs/>


    <options>


        <sleeptime>0</sleeptime>


        <delayBetweenStartingUseCasses time="1000"/>


        <delayBetweenTestCases time="100"/>


        <testTime time="1000"/>


    </options>


</testscenario>

TestScenario documents provide all the information needed for TestMaker to operate a functional test. The TestScenario begins with basic information about the test, defines the test use cases, and identifies logging parameters. In the above TestScenario we point out these elements:

<basics> names the test and sets the default directory
<testnodes> identifies the location of the TestNode(s) to operate the test. In this tutorial we are using the default TestNode that runs on the localhost (automatically installed when you installed TestMaker.)
<resources> identifies the files, JAR and WAR files, and scripts needed to operate the test
<functionaltest> identifies that this is a functional test and will operate the unit test one or more times with a single concurrent virtual user (CVU)

<dimensions>, <usecases>, <tests> identify the unit tests that make up this functional test. In this tutorial we have only a single unit test recorded.

<run> is a method that calls the recorded unit test. Notice the testclass, method, and langtype attributes identify the TestGen4Web recorded unit test. This is the test the <resources> section makes available to TestMaker.

<logs> identifies where and how to log the results data
<options> miscellaneous settings

Now back to the tutorial...

Copy the TestScenario document above and paste it into the functional test.

You should now have a TestScenario document that appears as follows.

If you are uncertain that you correctly performed the previous steps we have included a full written and tested script found in TestMaker_home/example_agents/testgen4web. Use the Open icon in the PushToTest toolbar to navigate to the example_agents/testgen4web directory and open the functionaltest.xml file. TestMaker will display this TestScenario in a new panel in the Controller, below the TestScenario we previously created.

We are ready to run the functional test.

Click the Run button in the Controller panel for the

TestMaker operates the functional test and shows the results in the lower output panel. You should see the following:

    -    Validating test            -


Trying example_agents/soapui_examples/PTT_Examples-soapui-project.xml


Triying suite testsuite


Using com.eviware.soapui.impl.wsdl.testcase.WsdlTestCase@ea4467


The test is valid.


    -    The functional test is running    -


Running a functional test


Executing time 0


Trying example_agents/soapui_examples/PTT_Examples-soapui-project.xml


Triying suite testsuite


Using com.eviware.soapui.impl.wsdl.testcase.WsdlTestCase@49e11f


Functional test succesful

In this tutorial we built a functional test of a Web service by using the soapUI utility to create a TestScenario. This ends the tutorial. Ready for something more interesting? In the next tutorial we show how to turn this same unit test into a load test without any additional effort.

Tutorial 2 - Build A Load Test of a SOAP-based Web Service

In this tutorial we will build a load test of a Web-application by using TestScenario test definition system in TestMaker. To perform this tutorial you must first follow the steps of Tutorial 1.

First we will give you a summary of the load testing method in TestMaker called XSTest. Then we will show you step-by-step instructions to use the soapUI utility to create a TestSuite of the Web service as a load, scalability, and performance test.

The XSTest Method in TestMaker

TestMaker provides load testing, scalability testing, and performance testing.

Load testing identifies the throughput of a service by making multiple concurrent requests to a service and observing the upper range of the service's ability to respond to concurrent requests. Load testing is useful to identify the number of users a service can handle given a certain number of resources (CPUs, network bandwidth, memory, threads, database connections, servers, and more.)
Scalability testing identifies the scalability index of a service as the service handles increasing levels of concurrent requests. Scalability testing is useful to forecast resource needs (CPUs, network bandwidth, memory, threads, database connections, servers, and more.) Scalability testing enables a business to make tradeoffs at development and configuration time.
Performance testing identifies changes in the scalability index of a service as the service programming changes and the service resources change. For instance, when installing a new patch to a database a performance test will identify an improvement in performance as compared to a previous test run.

TestMaker provides the XSTest framework to operate unit tests as load tests, scalability test, and performance tests. XSTest operates a unit test at various levels of concurrent request sizes to determine the scalability index of the target host. The following chart explains the use case that XSTest implements.

In a TestMaker load test the TestScenario defines the parameters of the test, the TestUseCase creates one thread for each concurrent virtual user, and the threads operate concurrently for the duration of the TestUseCase. Once the TestScenario creates all of the threads then the TestScenario begins logging results to a log file. When the test period ends the TestScenario ends the results logging and ends all of the threads. The TestScenario then operates the test as the next level of concurrent virtual users. Once all of the TestUseCases operate the TestScenario tallies the results into a Scalability Index.

A Scalability Index shows the performance of a target service at a variety of load levels. For instance, the following chart shows the TPS results as measured at 3 levels of concurrent requests (CRs.)

The chart above shows that the service does not scale in proportion to the number of CRs. For example, if a service performance 8 TPS at 2 CRs one would expect the same service could perform twice as many TPS at twice the CRs. The chart above shows that at 4 CRs the service performs only 14 TPS. Even worse, at 8 CRs the service performs only 17 TPS. A perfectly scalable system would produce a Scalability Index looking like the following chart.

now that you have an understanding of the XSTest load testing method in TestMaker we will show you step-by-step instructions to use the soapUI-created TestSuite of the Responder Web application as a load, scalability, and performance test.

Turning Unit Tests Into Load Tests

TestMaker TestScenario documents are XML documents containing all the configuration and processing instructions needed for PushToTest TestMaker to operate a load test. Here is a TestScenario that uses the TestSuite we created using soapUI in Tutorial 1.

<?xml version="1.0" encoding="UTF-8"?>

<testscenario xmlns="www.pushtotest.com/tm5" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="www.pushtotest.com/XSD/testscenario.xsd" version="2.0">
    <basics>
        <name>SOAP Load Test</name>
        <defaultdirectory>.</defaultdirectory>
    </basics>
    
    <testnodes>
        <node name="localhost" location="http://localhost:8080/TestNetwork/ws/TestNode"/>
    </testnodes>
    
        <resources>
          <soapui path="example_agents/soapui_examples/PTT_Examples-soapui-project.xml"/>
        </resources>
    <loadtest>
        <testusecase>
            <dimensions>
                
                <crlevels>
                    <crlevel value="1"/>
                </crlevels>
                <usecases>
                    <usecase name="SOAP_usecase">
                        <sequence name="SOAP_sequence" proportion="100">
                            <test>
                                 <run name="setuptest" testclass="testsuite" method="testcase" langtype="soapui"/>
                            </test>
                        </sequence>
                    </usecase>
                </usecases>
            </dimensions>
        </testusecase>
    </loadtest>
    
    <logs/>
    
    <options>
        <sleeptime>0</sleeptime>
        <delayBetweenStartingUseCasses time="1000"/>
        <delayBetweenTestCases time="100"/>
        <testTime time="60000"/>
    </options>
</testscenario>

In the above TestScenario three elements stand out:

The <basics>, <testnodes>, <resources>, and <usecases> remain exactly the same from the functional test in Tutorial 1.
<loadtest> identifies that this is a load test.
<crlevels> identifies concurrent request levels. Load tests use concurrently running threads to operate a unit test over-and-over for the period of the test. The <crlevel> values tell TestMaker how many threads to operate. For this test, TestMaker instantiates 10 concurrently running threads, operates the threads for the 1 minute period of the test, tears-down the running threads, and then operates 20 new threads for the period of the test. In a system with linear scalability twice the number of concurrently running request agents should generate twice the throughput, as measured in transactions per second (TPS) at the client.
<options> defines values used to time the test.

<sleeptime> is set to 0 meaning there will be no delay between running one usecase and the next. Once the <run> method operates there will be no delay before operating the <run> method again.
<delayBetweenStartingUseCases> is set to 1000 meaning there will be a 1000 millisecond (1 second) delay between starting each of the concurrently running threads. This is done to not overwhelm the resources of the load generating system (the TestNode) and the target host system.
<delayBetweenTestCases> is set to 100 meaning there will be a 100 millisecond (1/10 of 1 second) delay once a test period (set by the <testTime>) is completed before the test creates the next set of threads.
<testTime> is set to 60000 meaning there will be a 60000 millisecond (60 seconds, or 1 minute) time to operate the threads for each <crlevel>. The <testTime> does NOT include the time it takes to instantiate or teardown the threads.

Next we will operate the load test and observe the results.

Click the Close button on any open TestScenarios in the Controller panel.
Click the New Load Test icon in the PushToTest toolbar in TestMaker.
Save the new test in TestMaker_home/loadtest.xml

TestMaker creates the TestScenario file named loadtest.xml and opens a new Controller panel.

The Controller panel is in the lower left portion of the TestMaker window. A graphical representation of the TestScenario for the new load test appears within a new panel in the Controller panel. The Controller panel shows controls to run, pause, stop, edit, expand, and close for this functional test.

Click the Edit icon in the Controller panel for the new load test.
Copy the load testing TestScenario document above and paste it into the load test.

If you are uncertain that you correctly performed the previous steps we have included a full written and tested script found in TestMaker_home/example_agents/soapui_examples. Click the close icon for the TestScenario in the Controller panel. Use the Open icon in the PushToTest toolbar to navigate to the example_agents/soapui_examples directory and open the loadtest.xml file. TestMaker will display this TestScenario in a new panel in the Controller.

Click the Run icon in the Controller panel for the load test.

The Output panel - to the right of the Controller panel - shows you status as TestMaker stages the load test using the local TestNode. The contents of the TestScenario panel update to show you the test progress graphically.

Click the Expand icon in the Controller panel for the load test. The Controller for the load test appears in a new window as shown below.

In the Controller window for the running TestScenario starts at the top with icons to run, pause, stop, and edit control. Below that is a horizontal slider that, while the test is operating, moves from left to right and allows you to change the order of the test as the test operates. Below that is an icon bar indicating the TestNode machines that are actually running the tests in a distributed fashion, and this icon shows you the status of the running TestNode. Rolling the mouse over each one of these icons indicates special information about how well that test machine is performing. Below the TestNode icon bar is a live scalability index chart that shows you the live scalability index results as the test runs. This chart is followed by a transaction distribution chart showing you whether there were any anomalies that happened while the test was running. Below the transaction distribution chart - when the Monitor is installed - is several other charts to show you the CPU, network, and memory utilization of both the TestNode machines that are generating the test load, as well as the machine, the target machine that's actually running the application.

This concludes Tutorial 2. In this tutorial we showed a summary of the load testing method in TestMaker called XSTest. Then we showed the step-by-step instructions to use the TestSuite we created using the soapUI utility as a load, scalability, and performance test. We repurposed the unit test from a Functional test in Tutorial 1 to a load, scalability, and performance test with no extra effort.

Wouldn't it be great to turn the unit test from a load test into a service monitor? Get ready for Tutorial 3.

Tutorial 3 - Build A Service Monitor of a SOAP-based Web Service

In this tutorial we will build a service monitor of a Web-application by using a TestScenario test definition. To perform this tutorial you must first follow the steps of Tutorial 1. A service monitor is a functional test that operates periodically.

A unit test uses the native protocols of a service to test the operation of a function in a service. A service monitor in TestMaker uses a unit test to periodically operate a service. The service monitor records the success or failure of the operation, alerts an operator when problems exist, and displays the service monitoring functions in a dashboard.

TestScenario documents are XML documents containing all the configuration and processing instructions needed for PushToTest TestMaker to operate a service monitor. Here is a TestScenario that uses the unit test we created using TestGen4Web in Tutorial 1.

<?xml version="1.0" encoding="UTF-8"?>


<testscenario xmlns="www.pushtotest.com/tm5"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="www.pushtotest.com/tm5
C:\DOCUME~1\llara\MYDOCU~1\testmaker\XSD\testscenario.xsd"
version="2.0">


    <!-- 


  PushToTest(tm) Test Scenario for a Service Monitor





  PushToTest is the open-source SOA governance and test automation platform


  For technical support, documentation, and help see http://www.pushtotest.com


  -->


    <basics>


        <name>SOAP Service Monitor Test</name>


        <defaultdirectory>.</defaultdirectory>


    </basics>


    <!-- Location of TestNodes to run this test -->


    <testnodes>


        <node name="localhost" location="http://localhost:8080/TestNetwork/ws/TestNode"/>


    </testnodes>


    <!-- Dynamically loaded jar files and other resources -->


        <resources>


          <soapui
path="example_agents/soapui_examples/PTT_Examples-soapui-project.xml"/>


        </resources>


    <monitor timeBetweenTest="10000">


        <until>


            <exception type="java.lang.Exception"></exception>


        </until>


        <testusecase>


            <dimensions>


                <usecases>


           
        <usecase
name="SOAP_usecase">


           
            <sequence
name="SOAP_sequence">


           
           
    <test>

                            
 <run name="setuptest" testclass="testsuite" method="testcase"
langtype="soapui"/>


           
           
    </test>


           
           
</sequence>


                    </usecase>


                </usecases>


            </dimensions>


        </testusecase>


    </monitor>


    <logs/>


    <options>


        <sleeptime>0</sleeptime>


        <delayBetweenStartingUseCasses time="1000"/>


        <delayBetweenTestCases time="100"/>


        <testTime time="1000"/>


    </options>


</testscenario>

In the above TestScenario three elements stand out:

The <basics>, <testnodes>, <resources>, and <usecases> remain exactly the same from the functional test in Tutorial 1 and the load test in Tutorial 2.
<monitor> identifies that this is a service monitor. The timeBetweenTest attribute tells TestMaker to operate this test and then wait for 10 seconds (10,000 milliseconds) and then to repeat the test.
<until> identifies the exceptions that will stop the service monitor. Any other exceptions are logged but do not stop the monitor.

Next we will create and operate a service monitor and observe the results.

Click the Close button on any open TestScenarios in the Controller panel.
Click the New Service Monitor icon in the PushToTest toolbar in TestMaker.
Save the new test in TestMaker_home/servicemonitor.xml

TestMaker creates the TestScenario file named servicemonitor.xml and opens a new Controller panel.

The Controller panel is in the lower left portion of the TestMaker window. A graphical representation of the TestScenario for the new service monitor appears within a new panel in the Controller panel. The Controller panel shows controls to run, pause, stop, edit, expand, and close for this functional test.

Click the Edit icon in the Controller panel for the new load test.
Copy the service monitor TestScenario document above and paste it into the service monitor.

If you are uncertain that you correctly performed the previous steps we have included a full written and tested script found in TestMaker_home/example_agents/soapui_examples. Click the close icon for the TestScenario in the Controller panel. Use the Open icon in the PushToTest toolbar to navigate to the example_agents/soapui_examples directory and open the servicemonitor.xml file. TestMaker will display this TestScenario in a new panel in the Controller.

Click the Run icon in the Controller panel for the load test.

The Output panel - to the right of the Controller panel - shows you status as TestMaker operates the test. The contents of the TestScenario panel update to show you the test progress graphically. Many TestMaker users leave the controller panel open as a dashboard to gain visibility into their service operations.

Click the Expand icon in the Controller panel for the service monitor. The Controller for the service monitor appears in a new window as shown below.

The service monitor shows the most recent test results in the table. The duration value is the amount of time taken to perform the use case for the service monitor. The large image below the name of the service monitor test shows an up-arrow during normal operation. The image turns to a down-arrow when the service monitor operated a test that returned an exception in the past 10 minutes.

This concludes Tutorial 3.

In this tutorial we showed a summary of the load testing method in TestMaker called XSTest. Then we showed the step-by-step instructions to use the soapUI-created TestSuite of the Web service as a load, scalability, and performance test. We repurposed the unit test from a Functional test in Tutorial 1 to a load, scalability, and performance test with no extra effort.

In this tutorial we showed you how to build a service monitor of a Web-application by using a TestScenario test definition. We showed that a service monitor is a functional test that operates periodically.

We used the TestSuite we created with soapUI to use the native protocols of a service to test the operation of a function in a service. A service monitor in TestMaker uses a unit test to periodically operate a service. The service monitor records the success or failure of the operation, alerts an operator when problems exist, and displays the service monitoring functions in a dashboard.

In these tutorials we demonstrated building a functional test of a Web-browser-based application, building a scalability and performance load test of a Web-browser-based application, and building a service monitor. We accomplished all three with no programming experience required and through the TestScenario system with a minimum of effort.

Where to go from here?

A list of frequently asked questions (FAQ)

Known problems and incomplete features in this release of PushToTest TestMaker (online)

Additional documentation, product downloads and updates are at www.PushToTest.com. While the PushToTest TestMaker software is distributed under an open-source license, the documentation remains (c) 2007 PushToTest. All rights reserved. PushToTest is a trademark of the PushToTest company.