The script development process in VUGen

1. Record the Script: Usually this is the first step of scripting where every user action is recorded into a script.

2. Replay and Verify: Once the script is recorded, reply the script to ensure its working right. Verify any impact through application frontend or database.

3. Enhance the Script: Once recording has been verified, enhance script by adding checkpoints, validating data, adding transactions and rendezvous points.

4. Replay and Verify: As earlier, re-play your script and verify that everything is working as intended.

5. Configure Runtime Settings: Configure and control pacing duration, think time variation, proxy settings and whether you wish to ignore any external resources.

6. Use for Load Scenarios: Formulate load scenarios based on test objectives. Use load distribution and geo-wide agents to make real like scenarios.

Compare recording between QTP and LoadRunner
Introduction to Web Tours Application
Understanding the VUGen Environment
Creating Your First VUGen Script
Using Transactions at Record Time
Insert Comments at Record Time:
Code Generation:
Deciding a Protocol and Protocol Advisor
Recording Options
Difference between HTML-based and URL-based Scripting

• Reduces need to capture dynamic values             

• Script is only as big as the business process–one step per page

•  Scripts are less scalable

•  Memory (cache) is searched during playback

• Flexibility

• Scalability

•  Scripts require more correlation (nothing is retrieved from the cache)

•  Context-sensitive checks won’t work (parser is disabled)*

•  Scripts are large (all images and frames are recorded as separate steps)

HTML Mode	URL Mode
Intuitive and easy to understand	Not as intuitive as the HTML scripts
Scripts are smaller, requests are encapsulated and easy to understand	Scripts are large, containing a call to each image, css, html, etc. thus making it difficult to understand.
Scalable	More scalable and effective for creating a load test

Use of Re-Generate Script
Playback a Script and understanding Log
Overview of Files Generated During Record & Playback

VUGen disregards UI Objects:

Unlike QTP scripts, LoadRunner scripts are independent of GUI. This is because the code generated does not work based on UI objects, rather, it works on the principal of client’s request to the server and expecting server response back to the client. This is why replaying LoadRunner scripts are always browser independent. The VUser can communicate directly with a server by executing calls to the server API-without relying on client software (SUL) this means there will no impact on scripts if the changes are made to the UI of the System Under Load. This tells; the performance testing is completely based on client/server communication (request and response) and not the GUI objects.

Please refer to below link for more details: http://youtu.be/FDORUeqpNJs?t=3m41s

We will see in more details how VUGen works below.

For all hands-on purpose, we will be using the Web Tours Application that comes bundled with LoadRunner.

The HP Web Tours requires a dependent “Web tours apache server” to be running before it can operate. To run the Web Tours Apache Server, go to Start Menu => HP Software => Samples => Web and then click on Start HP Web Tour Server.

Once the “server” has been activated, it will open a shell and output on the console.

The console should look like snapshot below, look for your IP address in the shell:

NOTE: if you close this window, the server will stop.

Now you can access the Web Tours Application –

Once launched, it will look like –

To run the VUGen, double click on the VUGen shortcut from the desktop, or go to Start Menu => HP Software => and then click on Virtual User Generator.

VUGen splash screen will appearas follows

The splash screen will subsequently open HP VUGen Home page, as shown below:

From the File menu, you can observe various command controls. The purpose of these controls are mentioned below:

New Script and Solution: is used to start creating a new script. Alternatively, you can create a new script from the toolbar button.

Open >> Script Solution: is used to open an existing, previously created script or solution from a directory.

Add >> New Script: is used to add a previously created script into current solution

We will get acquainted with other controls as we start creating our first script. Let’s continue learning.

Please be patient. The Video will load in some time. If you still face issue viewing video click here

Before you record, make sure the Web Tours App serveris running. Remember, you need to keep the server running so do not close it. If the window bothers you, you can minimize it.

Tip:In a real world problem, one needs to get acquainted with the subject application (SUL) especially if it involves complex business workflows and data stages. You can also interact with HP Web Tours to ensure you can repeat the exact steps when recording.

Before one can start with scripting of any application (SUL), one needs to get acquainted with the application flow. With this tutorial, let’s get acquainted with HP Web Tours which is shipped and automatically installed with HP LoadRunner.

Step 1)Click on File =>New Script and Solution as shown in the snapshot below:

You can use the shortcut (Ctrl + N) to do the same.

Step 2) A window to select the protocol is shown.  The list looks slightly different from previous versions

1. Select Single Protocol

2. Web

3. Enter Script Name

4. Click Create

Note: You can use Protocol Advisor to determine right protocol your application uses for communication. For HP Web Tours, we already know that it requires “Web – HTTP/HTML” protocol. We will learn at a later stage how to determine the right protocol for your SUL.

Tip: An application may require more than 1 Protocols sometimes to communicate with its server. For such a case, navigate to Multiple Protocols list on the left side of window.

HP Web Tours, however, requires only one protocol when scripting with VUGen.

Tip: You cannot rename your scripts easily, so be creative when giving your scripts a name. If you’ve to rename a script, use Save As feature and give a new name. A good nomenclature could be something like, WT01_Signup where WT is short form of the application name, 01 is a business process sequence, and Sign up refers to the business processes being scripted. Also note that you cannot use spaces in a script name.

Step 3) once you click Create, HP VUGen will open the IDE (Integrated Development Environment) or code editor. You can notice the script files are blank, except basic signature of Function Action. Below snapshot should help understand it more:

Step 4) Click the Recording  button which you can find in the toolbar. Refer to the snapshot below:

Step 5) a new window opens

1. If we study the above window, the name of the script is mentioned in the title bar of this popup. This will help you make changes to the right script when you are dealing with multiple scripts opened at a time.

2. If you do not see the rest of controls, click on the Fewer Options button to ensure you are not viewing fewer options.

3. Record into the action field determines where the code is generated after recording. VUGen facilitates switching to other actions or create new actions during and after recording.

4. Record: field tells VUGen that the subject application (SUL) is a web application or a desktop client. This option cannot be overridden once recording begins.

5. Application: field tells VUGen, which browser to use for recording. If you are not working with Web – HTTP/HTML protocol, you will select the desktop client with this field.

Tip: Unlike QTP scripts, LoadRunner scripts are independent of GUI. This is because the code generated does not work based on UI objects, rather, it works on the principle of client’s request to the server and expecting server response back to the client – usually, in the form of HTML (hence the name of protocol)

6. URL Address: The URL of the SUL is input here. VUGen will invoke this URL when “Start Recording” is clicked. The browser used will be the one mentioned under “Record:” field if it is different from your default browser.

7. Start Recording: Immediate vs. Delayed: If you’ve selected Immediate recording mode, you will notice that VUGen starts recording events as soon as you hit “Start Recording” button. However, if you select Delayed recording mode, the VUGen will invoke the system (SUL) but will not commence capturing events unless you click on the Recording button from the floating bar.

8. Working Directory: This is a temporary directory which VUGen will use while capturing events. A copy of your script will be available at this location as well, just in case if you lose your folder :)

9.Recording Options: These settings tell VUGen what type of URLs to capture and what recording mode to use.

10.Start Recording: Will start recording. Click on it

Step 6) Once recording starts, VUGen will invoke the HP Web Tours application using the browser selected.

Let’s Sign up a user at Web Tours application and record a basic script.

A floating recording bar will appear which will offer various controls over recording.  This tells the time elapsed and number of events automatically captured. Let’s look at the floating bar closely.

In the snapshot shown above, the SUL has fired 56 events and a total of 0 minutes, and 25 seconds of time duration have elapsed since recording begun.

You’ll also observe that no events are fired as you type. Ask yourself why. Can you answer?

This is because no client to server communication takes place as you type. Hence, no events are fired and no script is generated regarding typing. Code will be generated only once you submit the form (click Continue)

Recording can be pause  using the pause button. Please note, as long as the recording remains paused, all events being fired by the application will be disregarded. This also means that even if your application is sending requests and/or receiving response form SUL, the script will not be generated as long as you’ve paused the recording.

If you’ve paused recording, you will need to click recording button from the floating bar to resume recording.

While you are recording in vuser_init section, you’d notice that rendezvous button is not enabled. Switch to Action (or create a new Action) and continue recording. Refer to below snapshot to learn how to switch Action.

You’ll notice that after switching to an Action, the rendezvous  button will be enabled automatically. This is because VUGen does not allow insertion of rendezvous points in vuser_init.

Let’s assume you wish to gauge response time when X number of users simultaneously Sign up. To address this objective, let’s read further. 

By now you’ve recorded opening of the application in vuser_init action. After you’ve switched to Action, enter the user information required by SUL. Now before you hit “Continue” button, follow below steps:

Start a transaction by clicking  button in floating bar. Enter the name of the transaction, for example, 01_Signup.

Tip: It is a good practice to name your transaction with a prefix like 01_ to help keep them unique.

The floating bar will look like this:

Click OK to close the transaction label.

Click the Continue button of the application.

End transaction by clicking  button. Select the name of the transaction you wish to close, for example, 01_Signup in this case. Refer to below snapshot for illustration.

If you do not have multiple transactions opened, you’ll see only one name. If you’ve multiple, however, you’ll 0be able to select from the list.  

Now that you’ve successfully closed the transaction, you can put a comment at record time to give yourself a reminder when you study the code later. From the floating bar, click on the button to insert a comment. A text box will appear adjacent to the floating bar where you can enter any text.

This comment will be inserted into the code after the steps (code) you’ve just finished before clicking OK button.

From the application, you can click continue. You’ll observe a welcome note by application.

Now select vuser_end action and click Sign off from the application. Close the application and click stop button from the floating bar.

Once recording has been stopped, you’ll notice VUGen post generation activity. This includes generating code, transactions and comments based on your recording. Refer to snapshot below.

Immediately after VUGen finished “Performing post generation operations” it will start “Performing scan for correlation post generation operation” We will see in more detail what correlation means.  

Once post generation operation is finished, you will see Design Studio window. If any candidates for correlation are found, you’ll be able to review them in this window.  

You can safely click on the Close button, and VUGen will take you to the code that has been generated.

The window will look like this:

As highlighted in the above figure, the editor contains easy to understand, readable code. You can have a look on the parameters used while recording.

The left side of VUGen is called Step Navigator that lets you understand the “script” without looking at the granularities of the code. For example, you can read steps as, opening a URL, spend Think Time and submit the form. This encapsulates all the parameters associated with each request.

Your script has been generated, click on the File menu and then click on Save Script As to browser the location where you wish to save your script. VUGen will automatically suggest the last directory used if you’ve already saved a script earlier. The name suggested will be the name you mentioned when you started recording.

All done.

Now, congratulate yourself for having your first script successfully generated and saved.

You might have questioned yourself why did we use Web – HTTP/HTML protocol. How we decided which protocol should we use? The answer is not that simple.

There is an architectural foundation set of skills you need to have in place as a prerequisite to answering this question. If you are a beginner, you can pair up with someone who has solid client-side architectural and development skills for your SUL. You can speak with the developers of your SUL and find out which interfaces your application leverages. This should lead you on a natural path to the interfaces that you will be using for your virtual user script development and protocol selection.

To address the needs of beginners less knowledgeable in architectural skills, LoadRunner introduced a feature called Protocol Advisor in LoadRunner 9.1. While this has made life easier for many, one should rely more on the architectural and development skills instead of protocol advisor and receiving information from the development team about underlying development technologies. Protocol may not suggest correct protocol in all cases.

To use Protocol Advisor, go to Record => Protocol Advisor => Analyze Application

Refer to snapshot below:

This will open the main window of Protocol Advisor. If you notice, this resembles a bit with the window appearing for recording. Let’s have a look at the window below:

Select the Web Browser since we are using a web based application.

Specify the URL of the application that will subsequently be invoked. Working directory can be left as such since this is merely a temporary directory for VUGen to use. Ensure you’ve read and write access on this directory.

Click the Start Analyzing button.

A floating bar, somewhat similar to the record time floating bar will appear. Have a look at the snapshot:

The process will tell the time elapsed and number of events fired. However, this information is not necessary. The only benefit of this events counter is, you know your client, SUL, is communicating with the server.

It is a good practice to analyze only one business process at a time since various business process in a large enterprise application may involve various protocols. For example, a dashboard in some application may have Ajax or Flex, etc. but this will not be present on the login page.

Once you’ve finished executing a particular business process, you can hit the Stop button. The VUGen protocol advisor will come up with a summary report on the protocol suggestion. Have a look how it looks like:

You can see the suggestions from Protocol Advisor. These may or may not be the best choices to pick.

You’ve learned to use Protocol Advisor by now. However, this could be help for beginners or for situation where you need “another opinion” – rely on your architectural sense, programming knowledge, development skills and information received from development team to decide on the protocol.

Whenever VUGen generates a script, the code generated is based on various configurations that can be found under the “Recording Options” – or you can press Ctrl + F7 to view the Recording Options.

Let’s have a look at recording options window before we discuss all configurations:

There are various categories of configurations like General, Correlations, Network and Data Format Extension. Let’s understand most significant among these, one by one.  

General => Recording:

This topic requires detail understanding. Hence this is discussed separately.

General => Script: 

Have a look at the snapshot for a glimpse:

 

You’ll notice that Language dropdown is disabled. A common myth is that the LoadRunner does not generate code in any other language. Another myth is that it requires a license to work in other languages.

Both are false. LoadRunner decides for itself which language to use when generating the script. In almost all cases, you’ll find yourself working with C Language.

For certain java applications (like Java applets) the code being generated will be in Java Script Language.

VUGen will generate a script in VBScript Language only for applications developed in Visual Basic classic (MS Visual Studio 2002)

Scripting Options:

You can opt to “Generate fixed think time after end transaction”. This means, no matter how much a user wait, the think time generated (the delay) will be equal to value specified. The value is in seconds.

Maximum number of lines in the action file refers to the maximum number of lines VUGen will generate in an action. If the script is larger, VUGen will automatically create a new action. The default is set to 60,000. The maximum value which can be specified is 65,000

You may find this configuration helpful when dealing with a desktop application with Oracle on the backend.

General => Protocol gives you an option to select and deselect any protocols you’ve selected at the start of recordingp

Essentially, this will be used only when you wish to Re-Generate Script.

Have a look at the screen:

This is helpful when you’ve used multi protocols at the time of recording a script. You can regenerate the script and deselect the protocols you don’t wish and get a new script without having to re-record it.

General => Code Generation:

Have a look at the snapshot below:

This configuration tells VUGen to find candidates for correlation at record time. If you do not wish for Automatic Correlation, then you might wish to turn off this feature.

Correlation => Configuration:  

Have a look at the screenshot below and familiarize yourself with the screen.

Although automatic correlation is helpful from 5% to 10% only, yet you can select “Rules Scan” and “Automatically correlate values found”. However, if your script doesn’t play, you can consider restoring to defaults by clicking on  button.

Correlation => Rules:

Go to Rules, and here you can see various rules VUGen is using to find correlation candidates. You can add custom rules if you know what your application (SUL) is using as parameters. However, this is an advanced use of record time settings. If you’re a beginner, you can safely skip this topic. 

HTTP Properties => Advanced:

This frame offers various settings related to HTTP binding.

Reset context for each action, enabling this option instructs VUGen to reset all HTP contexts between actions to their initial state before recording, providing a clean beginning for the recording session. The option is enabled by default.

You can leave the rest of configurations intact, unless required.

Network => Port Mapping:

This frame should be left intact. If you’re recording a desktop application, then you may have to choose WinINet level data.

You can go to Options (as long as you’re using Socket level data) and select various options like SSL version or other types of Secure Socket Layer. If you’re a beginner level or do not require these options, you can skip. Have a look to get yourself acquainted with the screen.

Now you’re done with most of the Record Time options, let’s move to the next topic and understand the difference between HTML and URL based scripting.

You may have noticed an options to pick either HTML-based script or URL-based script. Have a look at the snapshot for a flashback.

So what is this option and which one to pick?

The HTML-based script is based on user actions, and the scripts contain functions that correspond directly to the action taken. Let’s understand example of a small piece of code:

Example:

...

web_link(“Enterprise Systems Performance",

"Text=Enterprise Systems Performance,"

"Snapshot=t4.inf",

LAST);

The URL-based script is based on HTTP requests sent to the server as a result of user actions.

Here is an example of code for URL mode for the same actions performed as above (in HTML mode)

Example:

...

web_url(“Enterprise Systems Performance",

"URL=http://www.guru99.com/esp.html",

"TargetFrame=",

"Resource=0",

"RecContentType=text/html",

"Referer=http://www.guru99.com/atc?. . . ,

"Snapshot=t4.inf",

"Mode=URL",

LAST);

Tip: It’s best to experiment yourself before you move forward. Change the record time settings and record same script twice i.e. once with HTML mode and once with URL mode – then compare both. Keep the script short so you can understand the difference.

How do we decide on which mode to use?

Let’s understand the pros and cons of both modes so understand which mode is more suitable under certain situations:

Benefits of HTML Recording

                     - Action tag values and hidden data are NOT hardcoded

                     - They are retrieved from memory during playback

                     - If they are dynamic, the VUser still run

Disadvantages of HTML Recording

                      - requires more memory

                      - requires more CPU power

Benefits of URL Recording

                      - Support for Java Applets and ActiveX objects on the page

                      - Ability to replay on UNIX

                      - Scripts are more scalable than HTML scripts because they require fewer resources

Disadvantages of URL recording

Here is a quick illustration:

Let’s suppose you want to record the same script you just recorded, but with different record time settings. In such a case, you can use the regenerate script feature.

You can access it under Record => Regenerate Script or with hot key Ctrl+Shift+R

Once you click on the menu, VUGen will give you a warning that your existing script and all changed you’ve made to your existing script will be lost. The warning message looks like this:

You can also click on Options to open Record Time Options from here.

Click OK to proceed with Re-Generation of script.

Once you’ve saved the script, you can click on the  in the toolbar to replay or hit F5.

You can find this button in the toolbar:

You need to ensure the server is running (which is required for application to work properly)

When you replay the script, you’ll notice that unlike QuickTest Professional, it doesn’t open any browser to replay. Remember, this execution will simulate only 1 (single) user load on the SUL. The purpose of this execution is to ensure your script is working.

Tip: You’ll need to verify the impact from application itself. For example, if you’re creating a record, go to the application and verify manually that your script actually created a record. Your scripts, most likely, will not be tested by yet another Testing or QA team so you need to be very careful with your script and ensure these are thoroughly tested.

You can leave the replay log active since this will be a great help in identifying candidates for correlation and any errors and warning you might come across. Since generating log takes ample resources, it is best-turned off when you’re done with debugging of scripts and using them for scenarios.

Let’s close the VUGen and have a look at the files it has created in the script folder.

VUGen creates a series of configuration files, data files and source code files which contain VUser run-time and setup information. The results of each iteration of the script are stored separately. If you’ve executed your script at least once, you will notice a directory by the name result1. This directory is for system use and should be ignored by the tester.

Important files which you need to understand:

VUGen will create one .c (C Language Code file) for each action. Thus, at the last you’ll have vuser_init.c and vuser_end.c and Action.c – if you’ve more actions created, you will see corresponding files too. For example, myAction.c

The replay log is saved in a file called output.txt. If you’ve replaced it multiple times, output.txt will contain the last execution log whereas, output.bak will contain previous to the last run.

<script_name>.usr file will contain all the run time configurations you’ve customized. Even if you’ve left all the configurations to default, this usr file will contain the information. This file also contains the version of LoadRunner used for creating script. This information is helpful if you’re reading old scripts for which you can’t recall the version number.

You will see a folder named “data”. This folder keeps an image of the events as well as copy of your code. VUGen makes use of these files when you “ReGenerate” your code.

5 tips to solve the most common problems seen in Web recording

Recording a LoadRunner test script in Virtual User Generator (VuGen) is usually very simple - click the record button, do whatever needs to be done in the browser, and click the ‘Stop Recording’ button.  But occasionally, the results might not be what you expected.

Here are the five most common problems seen when recording a test using the Web (HTTP/HTML) protocol, and how to resolve them. 

Problem 1: No events are being recorded

Here are the most common reasons why no events are showing in the script:

The events counter keeps increasing, while the generated script is empty

This may be because VuGen’s recording mechanism fails to identify HTTP data. To fix it, ensure that your application indeed uses HTTP Web traffic. If the application uses SSL connections, make sure you choose the correct SSL version (SSL 2, SSL 3, TLS) through the Port Mapping dialog, available by clicking the ‘Options…’ button on the Record > Recording Options… dialog:

The Advanced Port Mapping Settings dialog opens when you click ‘Options…’:

The events counter shows less than five events, while the application keeps getting data from the server

In this case, VuGen’s recording mechanism fails to capture any network activity at all. Ensure that your application really does provide some network traffic, ie. it sends and receives data through the IP network. If an antivirus program is running, turn it off during recording. Check the recording log for any clues about the recording failure. Messages such as “connection failure” or “connection not trapped” can be a sign of the wrongly configured Port Mapping settings.

In addition, if you are recording on a Chrome or Firefox browser, make sure that all the instances of the browser are closed prior to the recording.

Problem 2: Specific events are not recorded

This problem could be caused by the event being dropped as “uninteresting”. By default, VuGen’s Web (HTTP/HTML)protocol records client requests that return an HTTP response status of 2xx or 302, and discards all other requests. If a missing request returns a response that was discarded, such as 301, you can make a modification to the registry to instruct VuGen to generate a command for it:

• Locate the following registry key:
  • [HKEY_CURRENT_USER\Software\Mercury Interactive\Networking\Multi Settings\QTWeb\Recording]
• Add the following string value to it:
  • "GenerateApiFuncForCustomHttpStatus"="301"

Problem 3: The recorded application becomes unresponsive during the recording

This could be caused by VuGen’s recording mechanism not being able to connect to the application’s server. Network connection errors can be seen in the Recording Log:

[Net An. Warning  (1068:197c)] Request Connection: Remote Server @ 123.123.123.123 - 5222 (Service=) Failed attempt #1. Unable to connect to remote server: rc = -1 , le = 10060)
[Net An. Warning  (1068:197c)] Request Connection: Remote Server @ 123.123.123.123 - 5222 (Service=) Failed attempt #2. Unable to connect to remote server: rc = -1 , le = 10060)

Note: the Recording log is in the ‘Output’ pane.  Make sure ‘Recording’ is selected in the combo-box on the left:

In order to fix this problem, the Port mapping for the specific IP and port should be added to the Port Mapping dialog (under the Record > Recording Options… menu), and the entry should be unchecked. That will ensure that the above IP and port are not recorded – that application simply connects to them without any LoadRunner involvement. For the messages above the correct setting would be:

This workaround can be done in case the communication to 123.123.123.123:5222 is not important for the business process and can be omitted. If that’s not the case, the same entry should still be added, but left as checked. That will ensure correct traffic capture to this address.

Problem 4: During the recording the recorded application shows an error message about the wrong server certificate

The problem is caused by the inability of the client side to verify the validity of the server certificate. In order to fix this problem, the LoadRunner Certificate Authority (CA) file should be added to the machine’s “Trusted Root Certificate Authorities” certificate store (in case of a Java client application, LoadRunner’s CA should be added to Java’s trusted CA list using the keytool). This file is supplied with LoadRunner, and is called wplusCAOnly_Expiration_2022.crt. It’s located in the <LR_folder\bin\certs> folder.

To add it to the store, double-click on the file to open the certificate:

Then click ‘Install Certificate…’ to open the Certificate Import Wizard’:

Make sure you check ‘Place all certificates in the following store’ and select ‘Trusted Root Certification Authorities’.  When the wizard is completed, you should be able to record the application.

Problem 5: The browser crashes during recording when using the Ajax Click and Script protocol

This can happen when some of the Ajax controls inside the application are not recognized by VuGen. In order to fix it:

• Go to the  <LR_folder\dat\protocols> folder and open the WebAjax.lrp file
• Comment out the following: DllGetClassObject:jscript.dll=DllGetClassObjectHook:ajax_hooks.dll (simply put a semi-colon (‘;’) in front of the line)

You should now be able to record the application.

Core Activities of Performance Testing

1. Performance testing is typically done to help identify bottlenecks in a system, establish a baseline for future testing, support a performance tuning effort, determine compliance with performance goals and requirements, and/or collect other performance-related data to help stakeholders make informed decisions related to the overall quality of the application being tested. In addition, the results from performance testing and analysis can help you to estimate the hardware configuration required to support the application(s) when you “go live” to production operation.
2. Activity 1. Identify the Test Environment.  Identify the physical test environment and the production environment as well as the tools and resources available to the test team. The physical environment includes hardware, software, and network configurations. Having a thorough understanding of the entire test environment at the outset enables more efficient test design and planning and helps you identify testing challenges early in the project. In some situations, this process must be revisited periodically throughout the project’s life cycle.
3. Activity 2. Identify Performance Acceptance Criteria.  Identify the response time, throughput, and resource utilization goals and constraints. In general, response time is a user concern, throughput is a business concern, and resource utilization is a system concern. Additionally, identify project success criteria that may not be captured by those goals and constraints; for example, using performance tests to evaluate what combination of configuration settings will result in the most desirable performance characteristics.
4. Activity 3. Plan and Design Tests.  Identify key scenarios, determine variability among representative users and how to simulate that variability, define test data, and establish metrics to be collected. Consolidate this information into one or more models of system usage to be implemented, executed, and analyzed.   
5. Activity 4. Configure the Test Environment.  Prepare the test environment, tools, and resources necessary to execute each strategy as features and components become available for test. Ensure that the test environment is instrumented for resource monitoring as necessary.
6. Activity 5. Implement the Test Design.  Develop the performance tests in accordance with the test design.
7. Activity 6. Execute the Test.  Run and monitor your tests. Validate the tests, test data, and results collection. Execute validated tests for analysis while monitoring the test and the test environment.
8. Activity 7. Analyze Results, Report, and Retest.  Consolidate and share results data. Analyze the data both individually and as a cross-functional team. Reprioritize the remaining tests and re-execute them as needed. When all of the metric values are within accepted limits, none of the set thresholds have been violated, and all of the desired information has been collected, you have finished testing that particular scenario on that particular configuration.
9. Why Do Performance Testing?
10. Assessing release readiness by:
11. Enabling you to predict or estimate the performance characteristics of an application in production and evaluate whether or not to address performance concerns based on those predictions. These predictions are also valuable to the stakeholders who make decisions about whether an application is ready for release or capable of handling future growth, or whether it requires a performance improvement/hardware upgrade prior to release.
12. Providing data indicating the likelihood of user dissatisfaction with the performance characteristics of the system.
13. Providing data to aid in the prediction of revenue losses or damaged brand credibility due to scalability or stability issues, or due to users being dissatisfied with application response time.
14. Assessing infrastructure adequacy by:
15. Evaluating the adequacy of current capacity.
16. Determining the acceptability of stability.
17. Determining the capacity of the application’s infrastructure, as well as determining the future resources required to deliver acceptable application performance.
18. Comparing different system configurations to determine which works best for both the application and the business.
19. Verifying that the application exhibits the desired performance characteristics, within budgeted resource utilization constraints.
20. Assessing adequacy of developed software performance by:
21. Determining the application’s desired performance characteristics before and after changes to the software.
22. Providing comparisons between the application’s current and desired performance characteristics.
23. Improving the efficiency of performance tuning by:
24. Analyzing the behavior of the application at various load levels.
25. Identifying bottlenecks in the application.
26. Providing information related to the speed, scalability, and stability of a product prior to production release, thus enabling you to make informed decisions about whether and when to tune the system.
27. Project Context
28. The overall vision or intent of the project
29. Performance testing objectives
30. Performance success criteria
31. The development life cycle
32. The project schedule
33. The project budget
34. Available tools and environments
35. The skill set of the performance tester and the team
36. The priority of detected performance concerns
37. The business impact of deploying an application that performs poorly
38. Project vision.  Before beginning performance testing, ensure that you understand the current project vision. The project vision is the foundation for determining what performance testing is necessary and valuable. Revisit the vision regularly, as it has the potential to change as well.
39. Purpose of the system.  Understand the purpose of the application or system you are testing. This will help you identify the highest-priority performance characteristics on which you should focus your testing. You will need to know the system’s intent, the actual hardware and software architecture deployed, and the characteristics of the typical end user.
40. Customer or user expectations.  Keep customer or user expectations in mind when planning performance testing. Remember that customer or user satisfaction is based on expectations, not simply compliance with explicitly stated requirements.
41. Business drivers.  Understand the business drivers – such as business needs or opportunities – that are constrained to some degree by budget, schedule, and/or resources. It is important to meet your business requirements on time and within the available budget.
42. Reasons for testing performance.  Understand the reasons for conducting performance testing very early in the project. Failing to do so might lead to ineffective performance testing. These reasons often go beyond a list of performance acceptance criteria and are bound to change or shift priority as the project progresses, so revisit them regularly as you and your team learn more about the application, its performance, and the customer or user.
43. Value that performance testing brings to the project.  Understand the value that performance testing is expected to bring to the project by translating the project- and business-level objectives into specific, identifiable, and manageable performance testing activities. Coordinate and prioritize these activities to determine which performance testing activities are likely to add value.
44. Project management and staffing.  Understand the team’s organization, operation, and communication techniques in order to conduct performance testing effectively.
45. Process.  Understand your team’s process and interpret how that process applies to performance testing. If the team’s process documentation does not address performance testing directly, extrapolate the document to include performance testing to the best of your ability, and then get the revised document approved by the project manager and/or process engineer.
46. Compliance criteria.  Understand the regulatory requirements related to your project. Obtain compliance documents to ensure that you have the specific language and context of any statement related to testing, as this information is critical to determining compliance tests and ensuring a compliant product. Also understand that the nature of performance testing makes it virtually impossible to follow the same processes that have been developed for functional testing.
47. Project schedule.  Be aware of the project start and end dates, the hardware and environment availability dates, the flow of builds and releases, and any checkpoints and milestones in the project schedule.
48. The Relationship Between Performance Testing and Tuning
49. Cooperative Effort
50. Product vendors
51. Architects
52. Developers
53. Testers
54. Database administrators
55. System administrators
56. Network administrators
57. Tuning Process Overview
58. Tests are conducted with the system or application deployed in a well-defined, controlled test environment in order to ensure that the configuration and test results at the start of the testing process are known and reproducible.
59. When the tests reveal performance characteristics deemed to be unacceptable, the performance testing and tuning team enters a diagnosis and remediation stage (tuning) that will require changes to be applied to the test environment and/or the application. It is not uncommon to make temporary changes that are deliberately designed to magnify an issue for diagnostic purposes, or to change the test environment to see if such changes lead to better performance.
60. The cooperative testing and tuning team is generally given full and exclusive control over the test environment in order to maximize the effectiveness of the tuning phase.
61. Performance tests are executed, or re-executed after each change to the test environment, in order to measure the impact of a remedial change.
62. The tuning process typically involves a rapid sequence of changes and tests. This process can take exponentially more time if a cooperative testing and tuning team is not fully available and dedicated to this effort while in a tuning phase.
63. When a tuning phase is complete, the test environment is generally reset to its initial state, the successful remedial changes are applied again, and any unsuccessful remedial changes (together with temporary instrumentation and diagnostic changes) are discarded. The performance test should then be repeated to prove that the correct changes have been identified. It might also be the case that the test environment itself is changed to reflect new expectations as to the minimal required production environment. This is unusual, but a potential outcome of the tuning effort.
64. Performance, Load, and Stress Testing
65. Performance testing.  This type of testing determines or validates the speed, scalability, and/or stability characteristics of the system or application under test. Performance is concerned with achieving response times, throughput, and resource-utilization levels that meet the performance objectives for the project or product. In this guide, performance testing represents the superset of all of the other subcategories of performance-related testing.
66. Load testing.  This subcategory of performance testing is focused on determining or validating performance characteristics of the system or application under test when subjected to workloads and load volumes anticipated during production operations.  
67. Stress testing.  This subcategory of performance testing is focused on determining or validating performance characteristics of the system or application under test when subjected to conditions beyond those anticipated during production operations. Stress tests may also include tests focused on determining or validating performance characteristics of the system or application under test when subjected to other stressful conditions, such as limited memory, insufficient disk space, or server failure. These tests are designed to determine under what conditions an application will fail, how it will fail, and what indicators can be monitored to warn of an impending failure.
68. Baselines
69. A baseline can be created for a system, component, or application.  A baseline can also be created for different layers of the application, including a database, Web services, and so on.
70. A baseline can set the standard for comparison, to track future optimizations or regressions.  It is important to validate that the baseline results are repeatable, because considerable fluctuations may occur across test results due to environment and workload characteristics.
71. Baselines can help identify changes in performance.  Baselines can help product teams identify changes in performance that reflect degradation or optimization over the course of the development life cycle. Identifying these changes in comparison to a well-known state or configuration often makes resolving performance issues simpler.
72. Baselines assets should be reusable.  Baselines are most valuable if they are created by using a set of reusable test assets. It is important that such tests accurately simulate repeatable and actionable workload characteristics.
73. Baselines are metrics.  Baseline results can be articulated by using a broad set of key performance indicators, including response time, processor capacity, memory usage, disk capacity, and network bandwidth.
74. Baselines act as a shared frame of reference.  Sharing baseline results allows your team to build a common store of acquired knowledge about the performance characteristics of an application or component.
75. Avoid over-generalizing your baselines.  If your project entails a major reengineering of the application, you need to reestablish the baseline for testing that application. A baseline is application-specific and is most useful for comparing performance across different versions. Sometimes, subsequent versions of an application are so different that previous baselines are no longer valid for comparisons.
76. Know your application’s behavior.  It is a good idea to ensure that you completely understand the behavior of the application at the time a baseline is created.  Failure to do so before making changes to the system with a focus on optimization objectives is frequently counterproductive.
77. Baselines evolve.  At times you will have to redefine your baseline because of changes that have been made to the system since the time the baseline was initially captured.
78. Benchmarking
79. You need to play by the rules.  A benchmark is achieved by working with industry specifications or by porting an existing implementation to meet such standards. Benchmarking entails identifying all of the necessary components that will run together, the market where the product exists, and the specific metrics to be measured.
80. Because you play by the rules, you can be transparent.  Benchmarking results can be published to the outside world. Since comparisons may be produced by your competitors, you will want to employ a strict set of standard approaches for testing and data to ensure reliable results.
81. You divulge results across various metrics. Performance metrics may involve load time, number of transactions processed per unit of time, Web pages accessed per unit of time, processor usage, memory usage, search times, and so on.
82. Terminology

Term / Concept	Description
Capacity	The capacity of a system is the total workload it can handle without violating predetermined key performance acceptance criteria.
Capacity test	A capacity test complements load testing by determining your server’s ultimate failure point, whereas load testing monitors results at various levels of load and traffic patterns. You perform capacity testing in conjunction with capacity planning, which you use to plan for future growth, such as an increased user base or increased volume of data. For example, to accommodate future loads, you need to know how many additional resources (such as processor capacity, memory usage, disk capacity, or network bandwidth) are necessary to support future usage levels. Capacity testing helps you to identify a scaling strategy in order to determine whether you should scale up or scale out.
Component test	A component test is any performance test that targets an architectural component of the application. Commonly tested components include servers, databases, networks, firewalls, and storage devices.
Endurance test	An endurance test is a type of performance test focused on determining or validating performance characteristics of the product under test when subjected to workload models and load volumes anticipated during production operations over an extended period of time. Endurance testing is a subset of load testing.
Investigation	Investigation is an activity based on collecting information related to the speed, scalability, and/or stability characteristics of the product under test that may have value in determining or improving product quality. Investigation is frequently employed to prove or disprove hypotheses regarding the root cause of one or more observed performance issues.
Latency	Latency is a measure of responsiveness that represents the time it takes to complete the execution of a request. Latency may also represent the sum of several latencies or subtasks.
Metrics	Metrics are measurements obtained by running performance tests as expressed on a commonly understood scale. Some metrics commonly obtained through performance tests include processor utilization over time and memory usage by load.
Performance	Performance refers to information regarding your application’s response times, throughput, and resource utilization levels.
Performance test	A performance test is a technical investigation done to determine or validate the speed, scalability, and/or stability characteristics of the product under test. Performance testing is the superset containing all other subcategories of performance testing described in this chapter.
Performance budgets or allocations	Performance budgets (or allocations) are constraints placed on developers regarding allowable resource consumption for their component.
Performance goals	Performance goals are the criteria that your team wants to meet before product release, although these criteria may be negotiable under certain circumstances. For example, if a response time goal of three seconds is set for a particular transaction but the actual response time is 3.3 seconds, it is likely that the stakeholders will choose to release the application and defer performance tuning of that transaction for a future release.
Performance objectives	Performance objectives are usually specified in terms of response times, throughput (transactions per second), and resource-utilization levels and typically focus on metrics that can be directly related to user satisfaction.
Performance requirements	Performance requirements are those criteria that are absolutely non-negotiable due to contractual obligations, service level agreements (SLAs), or fixed business needs. Any performance criterion that will not unquestionably lead to a decision to delay a release until the criterion passes is not absolutely required ― and therefore, not a requirement.
Performance targets	Performance targets are the desired values for the metrics identified for your project under a particular set of conditions, usually specified in terms of response time, throughput, and resource-utilization levels. Resource-utilization levels include the amount of processor capacity, memory, disk I/O, and network I/O that your application consumes. Performance targets typically equate to project goals.
Performance testing objectives	Performance testing objectives refer to data collected through the performance-testing process that is anticipated to have value in determining or improving product quality. However, these objectives are not necessarily quantitative or directly related to a performance requirement, goal, or stated quality of service (QoS) specification.
Performance thresholds	Performance thresholds are the maximum acceptable values for the metrics identified for your project, usually specified in terms of response time, throughput (transactions per second), and resource-utilization levels. Resource-utilization levels include the amount of processor capacity, memory, disk I/O, and network I/O that your application consumes. Performance thresholds typically equate to requirements.
Resource utilization	Resource utilization is the cost of the project in terms of system resources. The primary resources are processor, memory, disk I/O, and network I/O.
Response time	Response time is a measure of how responsive an application or subsystem is to a client request.
Saturation	Saturation refers to the point at which a resource has reached full utilization.
Scalability	Scalability refers to an application’s ability to handle additional workload, without adversely affecting performance, by adding resources such as processor, memory, and storage capacity.
Scenarios	In the context of performance testing, a scenario is a sequence of steps in your application. A scenario can represent a use case or a business function such as searching a product catalog, adding an item to a shopping cart, or placing an order.
Smoke test	A smoke test is the initial run of a performance test to see if your application can perform its operations under a normal load.
Spike test	A spike test is a type of performance test focused on determining or validating performance characteristics of the product under test when subjected to workload models and load volumes that repeatedly increase beyond anticipated production operations for short periods of time. Spike testing is a subset of stress testing.
Stability	In the context of performance testing, stability refers to the overall reliability, robustness, functional and data integrity, availability, and/or consistency of responsiveness for your system under a variety conditions.
Stress test	A stress test is a type of performance test designed to evaluate an application’s behavior when it is pushed beyond normal or peak load conditions. The goal of stress testing is to reveal application bugs that surface only under high load conditions. These bugs can include such things as synchronization issues, race conditions, and memory leaks. Stress testing enables you to identify your application’s weak points, and shows how the application behaves under extreme load conditions.
Throughput	Throughput is the number of units of work that can be handled per unit of time; for instance, requests per second, calls per day, hits per second, reports per year, etc.
Unit test	In the context of performance testing, a unit test is any test that targets a module of code where that module is any logical subset of the entire existing code base of the application, with a focus on performance characteristics. Commonly tested modules include functions, procedures, routines, objects, methods, and classes. Performance unit tests are frequently created and conducted by the developer who wrote the module of code being tested.
Utilization	In the context of performance testing, utilization is the percentage of time that a resource is busy servicing user requests. The remaining percentage of time is considered idle time.
Validation test	A validation test compares the speed, scalability, and/or stability characteristics of the product under test against the expectations that have been set or presumed for that product.
Workload	Workload is the stimulus applied to a system, application, or component to simulate a usage pattern, in regard to concurrency and/or data inputs. The workload includes the total number of users, concurrent active users, data volumes, and transaction volumes, along with the transaction mix. For performance modeling, you associate a workload with an individual scenario.

2. Summary
3. 
4. Figure 1.1 Core Performance Testing Activities
5. The performance testing approach used in this guide consists of the following activities:
6. At the highest level, performance testing is almost always conducted to address one or more risks related to expense, opportunity costs, continuity, and/or corporate reputation. Some more specific reasons for conducting performance testing include:
7. For a performance testing project to be successful, both the approach to testing performance and the testing itself must be relevant to the context of the project. Without an understanding of the project context, performance testing is bound to focus on only those items that the performance tester or test team assumes to be important, as opposed to those that truly are important, frequently leading to wasted time, frustration, and conflicts.
8. The project context is nothing more than those things that are, or may become, relevant to achieving project success. This may include, but is not limited to:
9. Some examples of items that may be relevant to the performance-testing effort in your project context include:
10. When end-to-end performance testing reveals system or application characteristics that are deemed unacceptable, many teams shift their focus from performance testing to performance tuning, to discover what is necessary to make the application perform acceptably. A team may also shift its focus to tuning when performance criteria have been met but the team wants to reduce the amount of resources being used in order to increase platform headroom, decrease the volume of hardware needed, and/or further improve system performance.
11. Although tuning is not the direct responsibility of most performance testers, the tuning process is most effective when it is a cooperative effort between all of those concerned with the application or system under test, including:
12. Without the cooperation of a cross-functional team, it is almost impossible to gain the system-wide perspective necessary to resolve performance issues effectively or efficiently.
13. The performance tester, or performance testing team, is a critical component of this cooperative team as tuning typically requires additional monitoring of components, resources, and response times under a variety of load conditions and configurations. Generally speaking, it is the performance tester who has the tools and expertise to provide this information in an efficient manner, making the performance tester the enabler for tuning.
14. Tuning follows an iterative process that is usually separate from, but not independent of, the performance testing approach a project is following. The following is a brief overview of a typical tuning process:
15. Performance tests are usually described as belonging to one of the following three categories:
16. Creating a baseline is the process of running a set of tests to capture performance metric data for the purpose of evaluating the effectiveness of subsequent performance-improving changes to the system or application. A critical aspect of a baseline is that all characteristics and configuration options except those specifically being varied for comparison must remain invariant. Once a part of the system that is not intentionally being varied for comparison to the baseline is changed, the baseline measurement is no longer a valid basis for comparison.
17. With respect to Web applications, you can use a baseline to determine whether performance is improving or declining and to find deviations across different builds and versions. For example, you could measure load time, the number of transactions processed per unit of time, the number of Web pages served per unit of time, and resource utilization such as memory usage and processor usage. Some considerations about using baselines include:
18. Benchmarking is the process of comparing your system’s performance against a baseline that you have created internally or against an industry standard endorsed by some other organization. 
19. In the case of a Web application, you would run a set of tests that comply with the specifications of an industry benchmark in order to capture the performance metrics necessary to determine your application’s benchmark score. You can then compare your application against other systems or applications that also calculated their score for the same benchmark. You may choose to tune your application performance to achieve or surpass a certain benchmark score. Some considerations about benchmarking include:
20. The following definitions are used throughout this guide. Every effort has been made to ensure that these terms and definitions are consistent with formal use and industry standards; however, some of these terms are known to have certain valid alternate definitions and implications in specific industries and organizations. Keep in mind that these definitions are intended to aid communication and are not an attempt to create a universal standard.
21. Performance testing helps to identify bottlenecks in a system, establish a baseline for future testing, support a performance tuning effort, and determine compliance with performance goals and requirements. Including performance testing very early in your development life cycle tends to add significant value to the project.
22. For a performance testing project to be successful, the testing must be relevant to the context of the project, which helps you to focus on the items that that are truly important.
23. If the performance characteristics are unacceptable, you will typically want to shift the focus from performance testing to performance tuning in order to make the application perform acceptably. You will likely also focus on tuning if you want to reduce the amount of resources being used and/or further improve system performance.
24. Performance, load, and stress tests are subcategories of performance testing, each intended for a different purpose.
25. Creating a baseline against which to evaluate the effectiveness of subsequent performance-improving changes to the system or application will generally increase project efficiency.