Obtaining the source code

IMS Bench SIPp is released under the GNU GPL license. All the terms of the license apply.

The complete source tree containing all the components of IMS Bench SIPp can be obtained from the Subversion repository at sipp.svn.sourceforge.net/svnroot/sipp/sipp/branches/ims_bench. For example, the following command creates the ims_bench directory and populates it with the latest version of the sources:

 svn co https://sipp.svn.sourceforge.net/svnroot/sipp/sipp/branches/ims_bench ims_bench
         

Pre-requisites

• In order to achieve around millisecond precision in scenario attempt scheduling and in timing measurements, the underlying operating system must provide sufficiently fined grained scheduling. On most Linux distributions, this requires that the kernel be rebuilt with the kernel "Timer frequency" changed to 1000 HZ. For example, on FC6:

   rpm -i kernel-2.6.18-1.2798.fc6.src.rpm
   cd /usr/src/redhat/SPECS
   rpmbuild -bp --target=i686 kernel-2.6.spec
   cd /usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18.i686
   make menuconfig
    Change:
       Processor type and features  --->
         Timer frequency (1000 HZ)  --->
       General setup  --->
         () Local version - append to kernel release  <- set your own kernel prefix
  
   make dep bzImage modules modules_install install
  
   (modify /etc/grub.conf to point to this new kernel)

• In order for the timing precision to remain when measuring a time difference between two different physical systems, all systems that constitute the Test System should be synchronized with a better precision than what the standard NTP protocol achieves. A simple way of doing this is to use the Precision Time Protocol (IEEE 1588) deamon , ptpd (ptpd.sourceforge.net)

   svn co https://ptpd.svn.sourceforge.net/svnroot/ptpd ptpd
   cd ptpd/trunk/src
   make
   ./ptpd -g (client side)

• Random number generation for the statistical distributions (scenario arrival, pauses in scenarios) require the GSL library. It can be obtained from http://www.gnu.org/software/gsl and compiled from sources:

   tar xvfz gsl-1.9.tar.gz
   cd gsl-1.9/
   ./configure
   make 
   make install

  You may need to add the path to the library (/usr/local/lib by default) to the LD_LIBRARY_PATH environment variable or to the /etc/ld.so.conf file:

   echo /usr/local/lib/ >>/etc/ld.so.conf
   ldconfig

• In order to be able to use the menu-driven benchmark configuration tool and the report generation tool, the following components must be installed.

  • Perl XML::Simple module - http://search.cpan.org/dist/XML-Simple/

     perl -MCPAN -e shell
     {reply with default answers... just select the local ftp server}
     cpan> install XML::Simple
     cpan> quit

  • Gnuplot 4.2 - http://gnuplot.sourceforge.net/

     tar xvfz gnuplot-4.2.0.tar.gz
     cd gnuplot-4.2.0
     ./configure --without-x
     make
     make install

• Configure Virtual IPs

  In case you want your test systems to support large numbers of users, you'll probably want to configure multiple virtual IP addresses on your network adapters. The actual number of IP addresses to configure will depend on the transport option you select: a single IP address per SIPp instance, in which case you need many IP addresses as you'll run SIPp instances on a same physical system, or multiple IP addresses per SIPp instance in which case you will want plenty of IP addresses.

  There are at least two ways to configure virtual IP addresses:

  1. Through ifconfig command execution (probably from within a script)

      ifconfig eth0:0 192.168.1.76/24 up

  2. or through the network adapter configuration files (/etc/sysconfig/network-scripts/ifcfg-eth0:x), and applying the changes with "service network restart"

      DEVICE=eth0:0
      BOOTPROTO=static
      TYPE=Ethernet
      IPV6INIT=no
      HWADDR=00:15:17:01:E2:E2
      IPADDR=192.168.1.76
      NETMASK=255.255.255.0
      NETWORK=192.168.1.0
      ONBOOT=yes

• Modify System Limits (/etc/security/limits.conf) to allow SIPp process to open a large number of sockets, and add:

   * soft nofile  102400
   * hard nofile  409600

Building IMS Bench SIPp components

To build SIPp and the manager in the way appropriate for the benchmark, perform the following make invocations:

 cd ims_bench
 make rmtl
 make ossl
 make mgr

Alternatively, the default make invocation (no argument) will build all these components as well as cpum, the system resource monitoring component. This might however not be what you need as the latter must be built on the system where it needs to run, i.e. the SUT, which might not be compatible with binaries built on your test systems.

To build cpum on the SUT, you will need the GNU development toolchain on your SUT or on compatible development environment. You can then copy the IMS Bench SIPp source tree and simply build cpum and its required dependencies:

 make rmtl           (on the SUT)
 make cpumem         (")

Configuration

When configuring the test system for a benchmark run, there are two possible approaches:

1. Use the ims_bench.pl perl script to enter the benchmark parameters using a menu driven user interface and automatically generate all the necessary configuration files and execution scripts
2. Configure all elements manually

Obviously the first approach is the easiest but is somewhat limited to configuring benchmark runs in close accordance to the TS 186 008 specification. As IMS Bench SIPp is based on SIPp which was already very flexible and as the new features and new components (Manager, CpuMem...) were designed in the same spirit, one may configure quite a large variety of benchmark runs. When configuring manually however, the benchmark parameters as specified in TS 186 008 do not appear as clearly anymore since many of them are implemented using SIPp features that were not directly implemented based on that specification. For this reason, the names don't match and some parameters of the specification may be linked to multiple configuration bits in the IMS Bench SIPp configuration.

Manager Configuration

The configuration file for the manager, manager.xml, is an XML file with one global configuration section and one or more "run" sections.

If you used the ims_bench tool to configure your benchmark run, it will have generated this file for you in a target directory. Otherwise, you can start from the example manager.xml file provided in the source tree.

• <configuration> section

  This section of the manager configuration contains the global configuration (independent of the individual "runs" defined in subsequent sections).

  • global parameters

    Each parameter entry is specified using the following syntax: <param name="name" value="value"/>

    Possible parameters are described in the table below:

    number_test_systems	Number of Test Systems that the manager will wait for before starting the load generation. A value of 0 indicates that the manager should start immediately with the number of SIPp instances connected at the time the user presses the 'e' key. When the manager is used in full automatic mode (-e command line flag), a non-0 value must be specified.
    prep_offset	Time (in milliseconds) allocated for the preparation portion (usually dedicated to the user reservation procedure) of a client-side scenario before its real SIP scenario portion is expected to start (according to the scenario initiation distribution). This is the scenario portion between the start and the <sync> command of the scenario. Once the scenario reaches its <sync> command, SIPp will put it to sleep until the time the SIP scenario was scheduled to start. This parameter should be set to a value high enough to guarantee that all scenarios can reach their <sync> command in advance of their actual scheduled start time, but not too large in order not to have too many users "consumed" by scenarios in their preparation phases (risk of running out of users).
    rand_seed	Initial seed value that will be distributed to all random number generators in the complete test system to compute their own individual seed value (this seed value is derived from the global rand_seed value, the SIPp instance ID and the specific random generator it is used for). This scheme guarantees that each SIPp instance starts at a different place in the pseudo-random sequences it uses and still allows re-generating almost exactly the same load as for a previous run by assigning the same rand_seed value. A value of 0 tells the manager to generate the seed at random. The actual rand_seed value used is always logged into the report.xml file generated by the manager so that one can then later force the rand_seed to this value to re-generate almost the exact same load.
    report	Report generation (1 = generate report; 0 = don't generate). Must be set to 1 in order to be able to use the report generation tool.
    log	Manager logging (0 = disabled; 1 = enabled). The manager can log details of its activities in a manager.log file. This includes the same data as the manager screen output with the highest verbosity.
    transient_time	Transient time (in seconds) at the begining of each step (change in the load applied to the SUT) during which scenario outcomes are ignored when computing the IHS percentage (percentage of Inadequately Handled Scenarios).
    scenario_path	Path prefix where the scenario xml files are located.
    max_time_offset	Maximum offest (in microseconds) allowed between each TS and the manager (0 = not checked)

    
    
  • Scenario Parameters

    This provides an extension of SIPp's -key command line mechanism. It allows the manager to define the value of global generic parameters that scenarios can refer to.

    Example: <scen_param name="RingTime" value="5000"/>

    Scenarios can then use this value, for example in a pause command: <pause poisson="true" mean="%RingTime"/>

    The name can be any name that does not conflict with pre-defined scenario keywords.

  • Scenario List

    Each scenario that must be loaded is specified using the following syntax: <scenario name="name" max_ihs="value"/>

    The name must represent an existing xml scenario file. max_ihs is an optional attribute defining the maximum percentage of inadequately handled scenario attempts allowed for this scenario (to be specified only for client side - "UAC-like" - scenarios).

  Example of (partial) <configuration> section:

   <configuration>
     <param name="rand_seed" value="0"/>
     <scen_param name="RingTime" value="5000"/>
     <scenario name="ims_uac" max_ihs="0.1"/>
   </configuration>

• <run> sections

  A run section defines a series of steps in the traffic time profile (evolution of the traffic over time) as well as which traffic set to use (percentage of each scenario). All steps of a run use the same traffic parameters except the scenario attempt rate which increases from step to step within a same run.

  A configuration can have as many runs as desired to make the scenario mix change over time, or to change other parameters that are specific to a run. Runs are executed in sequence in the same order as they appear in the manager config file.

  The following arguments can be specified for any run:

  cps	Initial rate of scenario attempts (CPS) for this step (Use 0 to define a "pause" step)
  max_calls	Number of calls to generate for this step. Generaly used only for a pre-registration phase where only SIP registration scenarios are executed.
  distribution	Distribution used for scenario initiation over time. Possible values: "constant", "poisson"
  duration	Duration (in seconds) of a single step of the run.
  step_increase	Load (scenario attempt rate) increase when moving to a next step within the run.
  num_steps	Number of steps in the runs. 0 = no increase (a single flat load) n = load is increased n times during the run.
  report	This attribute has no impact on the run-time behavior of the test system but is passed on to the report generation tool. If set to "no", the steps of the run will not appear as distinct steps in the generated report and will not be listed in summary tables, etc. This is typically the case for a run used to warm up the SUT before the actual benchamrking phase starts.
  sync_mode	If set to "off", the SIPp instances will bypasses the "sync" state of the scenarios. In this case, SIPp makes no attempt to avoid that preparation part of the scenarios delays the actual start of its SIP portion. This is generaly used for a pre-registration phase only. See also the prep_offset global parameter.
  use_scen_max_ihs	If set to "yes" or not set (default), the max_ihs value defined for each scenario is used as threshold value against which the IHS percentage of the step is compared (on a per scenario basis) to decide whether to stop the benchmark or to move on to a next step or run. If set to "no", the value specified under max_global_ihs run parameter (see below) is used as threshold value for all scenarios and the per scenario max_ihs threshold is ignored for this run. This can be useful for example in warm-up runs done as part of the overall benchmark execution (so it is always done the same way) and where a higher number of failures can be accepted. These warm-up runs would then use a higher IHS threshold than the subsequent "real" runs.
  max_global_ihs	Maximum allowed percentage of IHS for the run, evaluated over all scenarios executed regardless of their specific max_ihs value. See also the use_scen_max_ohs parameter.
  stats	Interval (in milliseconds) at which the manager requests counters from the SIPp instances to evaluate the progress of the test and display it.

  The scenario mix is also part of the configuration of a run. The first run must specify the ratio of each scenario. Subsequent run sections only need to specify this if the scenario mix must be changed. Otherwise, the same ratios as in the previous run are used.

  Scenario ratios are specified using the following syntax: <scenario name="name" ratio="value"/>

  Only the client scenarios (intiating side) should be specified. The sum of the ratios of all scenarios must be equal to 100, including the ratios that have been configured in earlier runs. This means that if a specific scenario was executed in the previous run and should not be executed anymore, one must explicitly set its ratio to 0 in the new run.

  Example run section:

  <!-- Stir phase to warm up the SUT -->
  <run cps="40" duration="75" step_increase="20" num_steps="3" distribution="poisson" \
       use_scen_max_ihs="no" max_global_ihs="1" stats="2000" report="no">
    <scenario name="ims_reg"   ratio="2.5"/>
    <scenario name="ims_uac"   ratio="50"/>
    <scenario name="ims_dereg" ratio="2.5"/>
    <scenario name="ims_msgc"  ratio="30"/>
    <scenario name="ims_rereg" ratio="15"/>
  </run>
  
  

Benchmark Execution

Running

In case you configured your test system using the ims_bench Perl script, you will have received detailed instructions at time of exiting the configuration tool. Please follow those instructions which will guide you through the deployment of the components (using the prepare.sh script) on the systems used for the test and the starting of all components (manager and SIPp instances).

In all cases, the first step is to start the manager on the system that you intend to use as central controller for the benchmark.

./manager [-f manager.xml]

It will then read its configuration file and wait for SIPp agents and resource monitoring agents (cpum) to connect.

The next step consists in starting the SIPp instances. Each instance must be started with the necessary options to make it use the IP address(es) you configured for it, to connect back to the manager as remote control and to load the user information for the users it will represent. The file containing the user data must be present on the system where the SIPp instance runs. The scenario files however are not needed locally since they will be sent over the network by the manager.
Each instance should also have its test system ID specified by means of the -i option (although this is optional, it simplifies things a bit because SIPp will then be able to use, in csv file names for example, this TS ID which is guaranteed to be unique even across multiple systems, instead of the local process id). The -trace_scen and -trace_retrans options are also required if you want to generate a report for the run (usually the case).

Here is an example of the command to issue on one of the test systems to start one of the SIPp instances, assuming that the manager is at 192.168.1.1, that the instance will use 192.168.1.20 and that the SUT is at 192.168.1.100 and listens for SIP traffic on port 5060:

./sipp -id 1 -i 192.168.1.20 -user_inf ./ims_users_1.inf
       -rmctrl 192.168.1.1:5000 192.168.1.100:5060
       -trace_err -trace_cpumem -trace_scen -trace_retrans

If you used the ims_bench tool to prepare the benchmark configuration, it will have created the necessary scripts for you (run_x.sh) and you can simply start those on the test systems as you'll have been instructed by the tool.

If you have built the resource monitoring tool - cpum - for your system under test, you should start it on the SUT now (unless it's already running from a previous run). It will connect to the manager and report the SUT CPU and memory utilization data.

./cpum 192.168.1.1:5000  (on the SUT)

You can watch on the console of the manager, the various systems connecting to it. Once all components are started, you can start the actual execution by pressing the 'e' key in the manager console.

While the test system manager executes the runs according to its configuration, it also monitors the percentage of inadequately handled scenarios (IHS) during each step, and decides, based on the configured maximum value allowed for the IHS percentage, whether to perform the next step, increasing the load on the SUT, or not.

As the manager moves from step to step within a configured run and from one run to the next, it writes these transitions to a report file it generates. This report file, report.xml, also contains information about the test systems used, the overall benchmark configuration, etc.

Once the IHS threshold has been exceeded, the manager instructs the SIPp instances to stop applying load to the SUT and reports that the test is finished. You can then press the 'q' key in the manager console. This will stop all connected SIPp instances and the manager itself.

Gathering Results

As each SIPp instance dumps most of its statistics on the local system it runs on (that's because sending it in real time to the manager could make the manager a bottleneck in the system), if you used multiple physical systems to execute the benchmark, you will need to gather together the csv files from each SIPp instance. In addition, prior to running the report generation tool, it is required to merge together the data from all the SIPp instances.
A simple script is provided that reads the manager report file to learn the IP addresses of the test systems and the PID or TS ID of their SIPp instances, then grabs the corresponding files using scp and merges them together (assuming you are in a subdirectory as created by ims_bench script or that you created yourself for the execution):

../scripts/getResults.pl

In case you copied the files manually from the test systems, you can use the same script to only perform the merging operation:

../scripts/getResults.pl -merge

This merging operation can take some time if the amount of data collected was very large. It produces the following files:

• sipp.csv resulting from the merge of all sipp_TS<ts_id>_scen.csv files
• sipp_retrans.csv resulting from the merge of all sipp_TS<ts_id>_retrans.csv files

Note: After the merge completes, you can delete the partial files by running

../scripts/getResults.pl -clean

Screens and Keys

Manager

• Main keyboard keys:

  Key	Description
  #	Change the display verbosity level
  e	Execute the benchmark
  0-9,<,>,D,q	Keys are directly sent to all SIPp clients
  T	Measure the time difference in micro seconds (Manual/Debug)
  t	Request the Date Time Stamp in text format (Manual/Debug)
  g	Request Counters (Manual/Debug)
  r	Reset Clients (Manual/Debug)
  l	Load Scenarios (Manual/Debug)
  W/w	Request CPU (Manual/Debug)

• When starting, the manager displays a summary of the configuration and the requested runs.

  

• After launching the benchmark execution (pressing 'e' key), the manager starts executing the runs. The first run could for example consist in a pre-registration phase where a certain percentage of the user population is registered with the SUT before the actual benchmark run really starts. In this example, only the ims_reg scenario is active.

  

• At run time, the manager displays global summary and the current percentage of Inadequately Handled Scenarios for the step. If cpumem is connected, the cpu utilization of the SUT(s) will be reported too.
  

  

CpuMem

• The following screen represents the CpuMem utility output.

  

  There is no runtime key. Press ctrl-c to exit the utility.

SIPp

• Main keyboard keys:

  Key	Description
  <,>	Select a particular scenario. Most data displayed on the screen is related to the currently selected scenario.
  1	Switch to the 'Scenario' screen
  2	Switch to the 'Statistics' screen
  3	Switch to the 'Repartition' screen
  4	Switch to the 'Variables' screen
  5	Switch to the 'TDM map' screen
  6,7,8,9,0	Switch to the corresponding 'Secondary repartition' screen ('6' for RTD 1, '7' for RTD 2, etc.)
  D	Debug screen (dump internal variables)

• Key '1': Scenario screen. It displays a call flow of the scenario as well as some important informations.
  • Screen Layout

    
    <TS_id>- <scenario_name>-<scen_slot>-   Scenario Screen    - [1-9]: Change Screen - <PID>
    
    

    Client:

      Call-rate(length)     Port   Total-time  Total-calls  Remote-host
       [Desactivated](0 ms)/1.000s   5060      10.00 s            0  <sut_address>(UDP)
      0 new calls during 1.000 s period      1 ms scheduler resolution
      0 calls (limit 0)                      Peak was 0 calls, after 0 s
      0 Running, 0 Paused, 0 Woken up, 0 Sync
      0 out-of-call msg (discarded)
      0 open sockets
    

    Server:

      Port   Total-time  Total-calls  Transport
      5060     695.00 s            0  UDP
    
      0 new calls during 1.000 s period      1 ms scheduler resolution
      0 calls                                Peak was 0 calls, after 0 s
      0 Running, 0 Paused, 0 Woken up, 0 Sync
      0 open sockets
    

    The following screenshots give examples with some of the scenarios included with IMS Bench SIPp.

  • IMS Registration Scenario

    

  • IMS UAC Scenario

    

  • IMS Messaging (Client) Scenario

    

  • IMS UAS Scenario

    

• Key '2': Statistics screen. It displays the main statistics counters. The "Cumulative" column gathers all statistics, since SIPp has been launched. The "Periodic" column gives the statistic value for the period considered (specified by -f frequency command line parameter).

  

• Key '3': Repartition screen. It displays the distribution of response time and call length, as specified in the scenario.

  

• Key '4': Variables screen. It displays information on actions in scenario as well as scenario variable information.
  
  • IMS Registration scenario
    

    

  • IMS UAC scenario
    

    

Generating Reports

A perl script, doReport.pl, can be used to generate a report in MHT format (Multipurpose Internet Mail Extension HTML - RFC 2557), containing graphs and statistics about the test.

Note: As of this writing, the Mozilla Firefox browser did not support this format out of the box. Microsoft Internet Explorer 6 supports it natively. Although the MHT format is very convenient to group the HTML and picture files, one can also view the HTML directly as long as the picture files remain at the same relative paths.

This tool takes as input data from the following sources:

• SIPp metrics data contained in the merged csv files (resulting from the usage of the getResults.pl script (scenario attempts, outcome, timings).
• CPU and memory utilization data gathered from the cpum resource monitoring tool running on the System Under Test and on the test systems
• general information about the run (step start times, scenario attempt rate,...) from the file generated by the manager during the benchmark run (report.xml)
• metric related information (name, mapping with csv file...) from the XML scenario files.

The report is made up of multiple sections. The first section, the summary, is only configurable as far as the static text included is concerned and is otherwise built automatically by the tool. The subsequent sections contain graphs and statistics tables representing measurements like Scenario Attempts per Second (SAPS), response times, CPU utilization, etc. Those are configurable. See below for details on the report configuration.

Configuring the Report Content

The configuration for the report generation tool is located in the reportConfig.xml file (default - can be overwritten on the command line). It tells the tool which graphs must be plotted, gives descriptions and titles for these graphs and also contains some general parameters.

The reportConfig.xml file included in the source tree allows you to generate a report matching the requirements of the ETSI TS 186 008 specification (within the existing limitations of the current IMS Bench SIPp implementation). You only need to modify this file if you intend to change what data is reported (for example you added new scenarios for a new use case) or the way the data is presented (type of graph, legend, description...).

General parameters

Graphs

All measurements can be represented as time based graphs and/or as histograms.

Time based graphs usually have the time as X-coordinate, the measure as Y1-coordinate and SAPS as Y2-coordinate. The SAPS (scenario attempts per second) in such graphs can be calculated for the whole system (default), per use case (use case attempts per seconds, for example grouping together REGISTER, DE-REGISTER and RE-REGISTER scenarios) or per scenario (only REGISTER scenarios per second for example).

Each graph is configured within a <measure> section in the report configuration. As already mentioned, a measure can be evaluated for the whole system (default), in which case the <measure> section should appear at the top-level (within <config> section). But the measure can also be done per use case, in which case the <measure> section should appear within the corresponding <use_case> section.
Finally, the measure can also be done per scenario, in which case the <measure> section must appear within a <scenario> section.
Use case names and scenario names are the names referenced in the scenario XML files. Check out the provided default reportConfig.xml file for examples.

The following parameters describing the way to present the measurement may appear within a <measure> section: (parameters in italic are optional):

Measurements can be plotted in different forms. All forms can be used for all measurements, but some forms are more appropriate than others for some measurements. The type of graph for measurement is specified by including one of the following parameters within the <measure> section.

Each of the four graph types can have sub-parameters:

Executing doReport.pl

The help screen of doReport.pl shows the command line options it supports:

 Syntax: doReport.pl [-r <report_file>] [-c <report_config_file>]
                     [-i <ims_bench_file>] [-F<0|1>]

 -r specifies the raw benchmark report file (default: report.xml) resulting
    from the run you want to generate a graphical report for.
 -c specifies the configuration file for this script (default: reportConfig.xml)
 -b specifies an optional benchmark info HTML fragment file to include as
    introduction in the report (a default generic sentence is otherwise provided)
 -i specifies the ims_bench config file (in case the benchmark run was
    configured using the ims_bench script), containing IMS benchmark parameters.
 -F specifies whether gnuplot should be forked (to benefit from multiple CPU
    cores). -F0 disables the forking (default: enabled)
 -? to get this help.
        

doReport.pl expects to find the csv data files, the report.xml file and the scenario files in the current directory. You can however execute it from anywhere where you have these files present as it will look for its own files (reportConfig.xml unless specified through -c command line option, some small picture files, etc.) at the path present in the command line.
It is therefore common to execute it from the same location where you ran the manager during the benchmark run (the ims_bench_xyz directory in case the benchmark run was prepared by the ims_bench tool).
For example:

../scripts/doReport.pl -i ims_bench.xml

You can however overwrite the reportConfig.xml file as well as the logo file (logo.png - displayed in the top left corner of reports) by putting your own versions of these files in the current directory. doReport.pl first looks for them in the current directory and then at the same location where the script itself resides.

Multi-scenario mode

NEW!A key feature in IMS Bench SIPp is its support for multiple scenarios within a single SIPp instance. Multiple scenarios are uploaded by the manager to the SIPp instance(s) and each call is executing one of the scenarios.

Scenarios can be classified as either client-side or server-side. A client-side scenario is a scenario that starts by initiating a SIP transaction or a non-SIP message exchange with a partner SIPp instance (usually in a preparation phase of the scenario where scenario and user reservation is performed).
A server-side scenario is one that starts by the reception of the first message of a SIP transaction or the reception of a message from a partner SIPp instance.

Client-side scenarios are initiated by the SIPp instances according to the scenario initiation scheduling (e.g. Poisson distribution of the delays between two consecutive scenarios) and start executing their sequence immediately. The exact scenario to execute from the list of client-side scenarios loaded is selected at random according to the configured relative occurrences of the scenarios in the scenario mix.
In the benchmark configuration, client-side scenarios are identified by the fact that they have a ratio attribute that specifies their relative occurrence in a specific run section of the benchmark configuration (See also Manager Configuration - <run> sections).

In IMS Bench SIPp, server-side scenarios are only instantiated when receiving, from a partner SIPp instance executing a client-side scenario, a request for preparing execution of a specific server-side scenario. The client-side scenario is therefore the controlling side and a server-side scenario always has at least one associated client-side scenario that will trigger its invocation. If a server-side scenario has no client-side counterpart in the benchmark configuration, it will never be executed.

A client-side scenario Si running on SIPp instance X requests a partner SIPp instance Y (association made at random for the duration of the call) to instantiate the server-side scenario Si' by sending a non-SIP req_user message to Y, telling it the ID of the server-side scenario to instantiate as well as the SIP URI of the emulated user at X from whom the first SIP message of the actual SIP scenario portion will come (SIP From header). SIPp instance Y will then be ready to receive this first SIP message and will match it, based on the received From header, with the call instantiated for the server-side scenario Si'. SIPp instance Y will then update its internal SIP CallId map so that it can, from then on, directly dispatch subsequent messages for the same call to the corresponding call running the Si' scenarion.

Note: This fairly complex mechanism was designed to allow multiple SIPp instances to place calls between them through a System Under Test that could potentially modify the SIP CallId between both call legs. This is typically the case with SUTs behaving as a B2BUAs. IMS Bench SIPp should work in the situation just the same way as it works with SUTs that simply proxy the calls leaving the CallId unmodified. It also provides for a very realistic test system where a specific test system agent (SIPp instance) from the setup not only places calls towards itself but also to all other test system agents. Otherwise, users represented by a SIPp instance X would only call (or interact with) other users also represented by SIPp instance X.

User oriented mode

Also quite central to IMS Bench SIPp for its implementation of the IMS Performance Benchmark is its user oriented mode. It is triggered by the usage of the -user_inf command line parameter which specifies a file containing data for the SIP users that the SIPp instance will represent in its interactions with the SUT.

The way this is implemented is very simple and relies on the following basic elements:

1. SIPp maintains user entities that contain static data fields, and variables
2. SIPp also maintains a set of user pools into which users are placed. The actual meaning of these pools is really defined by the way the scenarios use them but they are meant to loosely represent user state.
3. New actions allow XML scenarios to assign a user from a specific pool to a call (scenario instance) and to move the currently assigned user to a different pool.
   This effectively gives a meaning to each pool. For example, a registration scenario will always pick users from a pool that represents the not-registered users, and upon successfull registration will move them to the pool of registered users. A successful calling scenario would then pick users from the pool of registered users, etc.
4. Similarly to call variables, values resulting for example from regular expression matching can be assigned to user variables of the user currently assigned to the call. The interest of user variables vs call variables is that they preserve their value between multiple scenario invocations. For example, the Service-Route header returned during a registration can be stored in a user variable in then later injected as Route header in the INVITE a calling scenario creates.

In addition, IMS Bench SIPp will assign a different combination of IP address and UDP port number to each user that it represents. This makes the traffic more realistic. Depending on the transport mode used, it will distribute the users on the configured IP addresses and then on the available ports on that address (see also SIPp Transport Modes).

Time Metrics

SIPp supports starting timers and stopping timers. It also supports specifying timeouts on <recv> commands. However, the original SIPp did not provide a way to verify that a measured time (called Response Time Duration, RTD) is within an allowed range for the scenario to be considered as correctly handled unless it exactly matched a receive timeout. IMS Bench SIPp provides such a mechanism by which a call can be marked as inadequately handled if one of the measured RTD exceeds a predefined maximum value, even though the scenario executed correctly from the sequence and SIP protocol point of view.

This is then reflected in statistics as well as in the percentage of inadequately handled scenarios that the IMS Bench manager determines at run-time when deciding whether to move to a next step in the load profile or not.

In IMS Bench SIPp, the timing measurements that must be collected in the scenario CSV result file (when using the -trace_scen option) and that can be checked against a specified maximum value are called "metrics".

These metrics are defined within the scenario file in a new <info> section, as in the following example:

  <info>
   <metric ref="PX_TRT-REG1" rtd="1" max="2000"/>
   <metric ref="PX_TRT-REG2" rtd="2" max="4000"/>
  </info>

The above example defines two time metrics to be checked against corresponding maximum values. For each metric, the RTD (SIPp timer) into which it will be computed by the scenario is specified as well as the maximum accepted value.

The checks are done at the end of the scenario execution (if successful from a message sequence and protocol timeouts point of view), and in case a maximum value is exceeded, the call is marked as failed.

The metric name specified in the ref attribute is not used by SIPp itself but by the report generation tool. It makes the link between the time metric name (for example as defined in a benchmark specification) and the RTD used to measure it within the SIPp scenario.

In the example above, the first metric is declared to be computed in RTD 1 and is not allowed to exceed 2000 milliseconds. The second one is computed in RTD 2 and may not exceed 4000 ms. The RTD values result from the usage of the start_rtd and rtd attributes on <send> or <recv> commands, or from computations performed on RTD values by RTD related Actions.

The <metric> elements also tell SIPp which rtd values to dump into the scenario CSV result file when the -trace_scen command line option is used. Also note that the max attribute is actually optional so that it's possible to dump an RTD to the scenario CSV file even when it does not need to be checked against a maximum value.

Traffic control

In IMS Bench SIPp, the traffic is controlled by the benchmark manager according to its configuration. The SIPp instances generate SIP traffic (scenario mix, average number of new scenario attempts per second) according to the instructions they receive from the manager. The traditional keys used in the original SIPp to control the number of calls started per second are disabled in IMS Bench SIPp mode.

You can still pause the traffic by pressing the 'p' key and resume it by pressing 'p' again, but this will of course disturb your benchmark run. SIPp will stop placing new calls and will continue executing the scenario of already running calls.

In IMS Bench mode, SIPp normally quits when you press 'q' in the console of the manager or when the manager exits.

The 'q' key is however still handled in the SIPp instance as well. If you press it, SIPp will stop placing new calls and will wait until all current calls go to their end. During this phase, (NEW!) SIPp will regularly look at all calls that are executing a pause command and will shorten the duration of this pause so as to speed up the exit while still trying to complete all calls in their normal flow. SIPp will then exit.

You can also force SIPp to quit immediatly by pressing the 'Q' key, or by pressing the 'q' key again several times. Current calls will be terminated by sending a BYE or CANCEL message (depending if the calls have been established or not).

List of commands with their attributes

  <send next="12">
    <![CDATA[

      ACK sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: ...
      From: ...
      To: ...
      Call-ID: ...
      Cseq: ...
      Contact: ...
      Max-Forwards: ...
      Subject: ...
      Content-Length: 0

    ]]>
  </send>

  <send next="6" test="4">
    <![CDATA[

      ACK sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: ...
      From: ...
      To: ...
      Call-ID: ...
      Cseq: ...
      Contact: ...
      Max-Forwards: ...
      Subject: ...
      Content-Length: 0

    ]]>
  </send>

<recv response="200">
 <action>
  <ereg regexp="([0-9]{1,3}\.){3}[0-9]{1,3}:[0-9]*"
    search_in="msg"
    check_it="true"
    assign_to="1,2"/>
  </action>
 </recv>

  <recv response="100"
        optional="true">
  </recv>
  <recv response="180" optional="true">
  </recv>
  <recv response="403" optional="true" next="5">
  </recv>
  <recv response="200">
  </recv>

  <recv response="100"
        optional="true">
  </recv>
  <recv response="180" optional="true">
  </recv>
  <recv response="403" optional="true" next="5" test="3">
  </recv>
  <recv response="200">
  </recv>

  <recv response="403" optional="true" next="5" test="3" chance="0.90">
  </recv>

<recv request="MESSAGE|PUBLISH|SUBSCRIBE" crlf="true" regexp_match="true">
</recv>

<pause milliseconds="4000" next="7"/>

<nop>
  <action>
    <exec play_pcap_audio="pcap/g711a.pcap"/>
  </action>
</nop>

<sendCmd>
  <![CDATA[
    Call-ID: [call_id]
    [$1]

   ]]>
</sendCmd>

<recvCmd>
  <action>
     <ereg regexp="Content-Type:.*"
           search_in="msg"
           assign_to="2"/>
  </action>
</recvCmd>

<label id="13"/>

<sync crlf="true">
 <action>
  <exec int_cmd="set_start_time"/>
 </action>
</sync>

<sendRmt type="req_user">
 <param name="scenario" value="ims_uas"/>
 <param name="from_uri" value="[field0]@[field1]"/>
 <param name="call_id" value="[call_id]"/>
</sendRmt>

<recvRmt type="req_user">
 <action>
  <assign_user pool="2"/>
  <move_user pool="3"/>
</recvRmt>

<recvRmt type="res_user" timeout="8000">
 <action>
   <store_param param="user_name" assign_to="1"/>
 </action>
</recvRmt>

Structure of client (UAC like) XML scenarios

A client scenario is a scenario that starts with a "send" command. So let's start:

<scenario name="Basic Sipstone UAC">
  <send>
    <![CDATA[
    
      INVITE sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[call_number]
      To: sut <sip:[service]@[remote_ip]:[remote_port]>
      Call-ID: [call_id]
      Cseq: 1 INVITE
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Type: application/sdp
      Content-Length: [len]

      v=0
      o=user1 53655765 2353687637 IN IP[local_ip_type] [local_ip]
      s=-
      t=0 0
      c=IN IP[media_ip_type] [media_ip]
      m=audio [media_port] RTP/AVP 0
      a=rtpmap:0 PCMU/8000


    ]]>
  </send>

Inside the "send" command, you have to enclose your SIP message between the "<![CDATA" and the "]]>" tags. Everything between those tags is going to be sent toward the remote system. You may have noticed that there are strange keywords in the SIP message, like [service], [remote_ip], .... Those keywords are used to indicate to SIPp that it has to do something with it.

Here is the list:

Now that the INVITE message is sent, SIPp can wait for an answer by using the "recv" command.

  <recv response="100"> optional="true"
  </recv>

  <recv response="180"> optional="true"
  </recv>

  <recv response="200">
  </recv>

100 and 180 messages are optional, and 200 is mandatory. In a "recv" sequence, there must be one mandatory message.

Now, let's send the ACK:

  <send>
    <![CDATA[

      ACK sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[call_number]
      To: sut <sip:[service]@[remote_ip]:[remote_port]>[peer_tag_param]
      Call-ID: [call_id]
      Cseq: 1 ACK
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Length: 0

    ]]>
  </send>

We can also insert a pause. The scenario will wait for 5 seconds at this point.

  <pause milliseconds="5000"/>

And finish the call by sending a BYE and expecting the 200 OK:

    <send retrans="500">
     <![CDATA[

      BYE sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port]
      From: sipp  <sip:sipp@[local_ip]:[local_port]>;tag=[call_number]
      To: sut  <sip:[service]@[remote_ip]:[remote_port]>[peer_tag_param]
      Call-ID: [call_id]
      Cseq: 2 BYE
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Length: 0

    ]]>
   </send>

   <recv response="200">
   </recv>

And this is the end of the scenario:

</scenario>

Creating your own SIPp scenarios is not a big deal. If you want to see other examples, use the -sd parameter on the command line to display embedded scenarios.

Structure of server (UAS like) XML scenarios

A server scenario is a scenario that starts with a "recv" command. The syntax and the list of available commands is the same as for "client" scenarios.

But you are more likely to use [last_*] keywords in those server side scenarios. For example, a UAS example will look like:

  <recv request="INVITE">
  </recv>

  <send>
    <![CDATA[

      SIP/2.0 180 Ringing
      [last_Via:]
      [last_From:]
      [last_To:];tag=[call_number]
      [last_Call-ID:]
      [last_CSeq:]
      Contact: <sip:[local_ip]:[local_port];transport=[transport]>
      Content-Length: 0

    ]]>
  </send>

The answering message, 180 Ringing in this case, is built with the content of headers received in the INVITE message.

Actions

In a "recv" or "recvCmd" command, you have the possibility to execute an action. Several actions are available:

• Regular expressions (ereg)
• Log something in aa log file (log)
• Execute an external (system), internal (int_cmd) or pcap_play_audio/pcap_play_video command (exec)
• User-related Actions (assign_user, move_user)
• RTD-related Actions (rtd_eval, rtd_store, rtd_op)

Regular expressions

Using regular expressions in SIPp allows to

• Extract content of a SIP message or a SIP header and store it for future usage (called re-injection)
• Check that a part of a SIP message or of a header is matching an expected expression