|
HTTPD::Bench::ApacheBench - Perl API for Apache benchmarking and regression testing. |
HTTPD::Bench::ApacheBench - Perl API for Apache benchmarking and regression testing.
use HTTPD::Bench::ApacheBench;
my $b = HTTPD::Bench::ApacheBench->new;
# global configuration
$b->concurrency(5);
$b->priority("run_priority");
# add HTTP request sequences (aka: runs)
my $run1 = HTTPD::Bench::ApacheBench::Run->new
({ urls => ["http://localhost/one";, "http://localhost/two";] });
$b->add_run($run1);
my $run2 = HTTPD::Bench::ApacheBench::Run->new
({ urls => ["http://localhost/three";, "http://localhost/four";],
cookies => ["Login_Cookie=b3dcc9bac34b7e60;"],
order => "depth_first",
repeat => 10,
memory => 2 });
$b->add_run($run2);
# send HTTP request sequences to server and time responses my $ro = $b->execute;
# calculate hits/sec print (1000*$b->total_requests/$b->total_time)." req/sec\n";
# show request times (in ms) for $run1, 1st repetition
print join(', ', @{$run1->request_times}) . "\n";
# show response times (in ms) for $run2, 7th repetition
print join(', ', @{$run2->iteration(6)->response_times}) . "\n";
# dump the entire regression object (WARNING, this could be a LOT OF DATA) use Data::Dumper; my $d = Data::Dumper->new([$ro]); print $d->Dumpxs;
This project is meant to be the foundation of a complete benchmarking and regression testing suite for an advanced, transaction-based mod_perl site. We need to be able to stress our server to its limit while also having a way to verify the HTTP responses for correctness. Since our site is transaction-based (as opposed to content-based), we needed to extend the single-URL ab model to a multiple-URL sequence model.
ApacheBench is based on the Apache 1.3.12 ab code (src/support/ab.c).
Note: although this tool was designed to be used on an Apache mod_perl site, it is generally applicable to any HTTP-compliant server. Beware, however, that it sends a high volume of HTTP requests in a very short period of time, which may overwhelm some weaker HTTP server implementations like NT/IIS.
ApacheBench sends sequences of HTTP requests to an HTTP server and keeps track of the time taken to receive a response, the data that was returned, the size of the data that was returned, and various other bits of information.
Since it is implemented in C, it sends HTTP requests in a tight loop which can stress your server to 100% capacity, especially if invoked in multiple concurrent instances. It gives accurate time measurements down to the millisecond for each HTTP request-response interval.
Included is a simplified re-implementation of ab using the ApacheBench Perl API. This should help get you started with ApacheBench.
You need to tell ApacheBench what the requests will be, in what order to send them, and how to prioritize sending them. ApacheBench was designed to simulate many users logged in simultaneously, each of whom may be doing many different transactions at once.
Global configuration methods apply to all benchmarking runs associated with this ApacheBench object.
new()If set to ``equal_opportunity'', all benchmark runs that are configured (see below) under this ApacheBench object are given equal access to the concurrency level. This means requests are taken from each run and sent in parallel (the level of parallelism defined by concurrency()).
If set to ``run_priority'', the benchmark runs that are configured first
get the highest priority. This means all requests in $b->run(0) will
be sent before any requests in $b->run(1) are sent, provided that
$b->run(0)->order eq ``breadth_first'' (see below).
See EXAMPLES near the bottom of this manual for a tutorial on the effects of ``equal_opportunity'' vs. ``run_priority''.
(default: ``equal_opportunity'')
(default: 1)
Warning: If you configure runs which contain requests to more than one
hostname/port, be aware that setting $b->keepalive() may not improve
performance. See the discussion in the $run->keepalive() section for more
details.
(default: 0)
Per-url/per-run time limits can also be specified using the $run->timelimits()
method, but it does not override this global setting. If $b->timelimit()
is set, ApacheBench will exit after $sec_to_max_wait (or slightly over, due to
final connection reads and building regression data), regardless of any other
settings.
(default: 0, which means wait an unlimited amount of time)
memory() < 3.
This can be overridden on a per-run basis (see below).
(default: 16384)
(default: 2048)
(default: 1)
Key: $memlvl => Description
0 => Remember nothing. (actually still keeps global
regression data: total_time, bytes_received,
and warnings)
1 => Remember connect/response times and minimal
summary information about size of requests and
responses.
2 => Remember connect/response times and all
information about size of requests and responses.
Also keeps an array of all HTTP response
headers returned by the server for each request.
3 => Remember connect/response times, all information
about request/response sizes, HTTP response
headers, and also all content of every HTTP
response returned by the server. B<Warning>:
this can quickly eat up all available main
memory if used with large runs.
You need to configure one or more benchmark runs. A run is defined as an ordered sequence of HTTP requests which can be repeated multiple times, and scheduled to be sent in different ways.
(default: 1, or whatever is specified in global configuration)
If you need to set different cookies within a single URL sequence, use
the request_headers() method.
Example usage: You could simulate $n users all doing the same transaction simultaneously by giving $n different login cookies here. Say you have login cookies in an array called @login of length $n. Set $run->repeat($n), $run->order(``breadth_first''), $run->cookies([map {$login[$_]} 0..$n]), and ApacheBench will perform the transaction sequence set by $run->urls $n times, each with a separate login cookie.
head_requests() below). If this option is omitted, all requests
for this run will be GET (or HEAD) requests. Length of @postdata should
equal the length of @url_list. The @postdata array should consist of
strings corresponding exactly to what you want sent in the HTTP request
body as a POST request. For example,
@postdata = (undef, undef, "cgikey1=val1&cgikey2=val2");
will send two GET requests, then a POST request with the CGI parameter 'cgikey1' set to 'val1' and the CGI parameter 'cgikey2' set to 'val2'.
postdata() value is defined, in which case a POST will be sent).
You can configure a run composed of any combination of the three HTTP
request types (GET, HEAD, and POST), but note that an individual URL with a
defined postdata() value will cause a POST request regardless of whether
head_requests() is set for that URL. The following precedence table
illustrates which type of request will be sent for URL $url_no in the sequence.
defined $run->postdata->[$url_no] ? ==> POST request $run->head_requests->[$url_no] ? ==> HEAD request else ==> GET request
The following example for a @url_list of length 4 produces two requests with no extra headers, one with 1 extra header, and one with 2 extra headers.
$run->request_headers([ undef, undef, "Extra-Header: bread",
"Extra-Header1: butter\r\nExtra-Header2: toaster" ])
To achieve full performance benefits from this feature, you need to be sure your Keep-Alive requests are consecutive. If a non-Keep-Alive request or a request for a different hostname or port immediately follows a Keep-Alive request in the connection slot, the connection will be closed and a new connection will be opened.
Further, keep in mind that for $b->concurrency() > 1, there are many
connection slots open and even though requests in @url_list will be sent
in order, there is no guarantee they will all use the same connection slot.
The HTTP Keep-Alive feature only yields performance benefits when consecutive
Keep-Alive requests use the same connection slot. Otherwise ApacheBench has
to close and re-open connections, resulting in the same performance as not
using keepalive() at all.
To guarantee consecutive Keep-Alive requests with $b->concurrency() > 1,
I recommend you either declare all URLs in all runs as keepalive()
(this can be done by setting $b->keepalive( 1 ) and not overriding it by
calling keepalive() for any runs), or set $run->order( ``depth_first'' ) and
$b->priority( ``run_priority'' ). This is the only combination of configuration
options that guarantees consecutive, same-slot Keep-Alive requests
regardless of your concurrency setting.
For $b->concurrency() == 1, things are simpler. At any given time, there
is only one connection slot open, so just make sure your keepalive URLs are
consecutive within each run (if in ``run_priority'' mode), or that
equal-numbered repetitions of URLs in all runs are keepalive (if in
``equal_opportunity'' mode), and be sure that all requests are to the
same hostname/port.
Warning: once the specified number of seconds have elapsed on the specified URL, ApacheBench will close the connection immediately. This can cause strange results in the regression data for this request. It could also result in partially received responses, which will cause broken sockets for the server.
``breadth_first'' mode sends $n of the first request in the @url_list, then $n of the second request in the @urls_list, then $n of the third... and so on. (e.g. If $n == 3 and @url_list contains two requests, then ApacheBench would send the first request 3 times, and then the second request 3 times.)
``depth_first'' mode ensures that HTTP requests in the sequence are
sent in order, completing a full sequence before starting again for the
next repeat() iteration. (e.g. If $n == 3 and @url_list contains two
requests, then ApacheBench would send the @url_list sequence in order,
then send it again, and then again. A total of six requests would be sent.)
See EXAMPLES near the bottom of this manual for a tutorial on the effects of ``breadth_first'' vs. ``depth_first''.
(default: ``breadth_first'')
Note: if $run->repeat() == 1, or the length of $run->urls() is 1, then the
order option has no effect
memory() < 3.
(default: 16384, or whatever is specified in global configuration)
(default: 1, or whatever is specified in global configuration)
execute() should complete safely. Returns 0 to
indicate some more parameters need setting before $b->execute() should be
called. Some of these parameters are automatically set by
prepare_for_execute() (below).
ready_to_execute() to return 1.
Returns 0 otherwise.
Note: this method is automatically called from execute() before entering
any XS code.
All of the following methods will return undef unless the underlying ApacheBench object has been execute()'d at least once.
If $url_no is not given, returns a reference to an array the same length as
$run->urls() which contains the number of successful HTTP requests sent
for each url in this run.
If $url_no is not given, returns a reference to an array the same length as
$run->urls() which contains the number of successful HTTP responses
received for each HTTP request in this run.
If $url_no is not given, returns a reference to an array the same length as
$run->urls() which contains the number of unsuccessful HTTP responses
received for each HTTP request in this run.
If $url_no is not given, returns a reference to an array the same length as
$run->urls() which contains connection times, in milliseconds, for each HTTP
request in this iteration of the sequence.
If $url_no is not given, returns a reference to an array the same length as
$run->urls() which contains the times taken to send a request to the server,
in milliseconds, for each HTTP request in the sequence.
If $url_no is not given, returns a reference to an array the same length as
$run->urls() which contains the times taken to receive a response from the
server, in milliseconds, for each HTTP request in the sequence.
If $url_no is not given, returns a reference to an array the same length as
$run->urls() which contains the number of bytes posted to the server for each
HTTP request in the sequence.
(return value will be undefined if memory < 2)
If $url_no is not given, returns a reference to an array the same length as
$run->urls() which contains the total number of bytes read from the server in
each HTTP response in the sequence.
(return value will be undefined if memory < 2)
If $url_no is not given, returns a reference to an array the same length as
$run->urls() which contains the full HTTP request (including HTTP request
headers) sent to the server for each URL in the sequence.
(return value will be undefined if memory < 3)
If $url_no is not given, returns a reference to an array the same length as
$run->urls() which contains the HTTP response headers returned by the server
for each URL in the sequence.
(return value will be undefined if memory < 2)
If $url_no is not given, returns a reference to an array the same length as
$run->urls() which contains the full HTTP responses returned by the server for
each URL in the sequence.
(return value will be undefined if memory < 3)
If $url_no is not given, returns a reference to an array the same length as
$run->urls() which contains the length of the document (in bytes) returned by
the server for each HTTP response in the sequence. This should be equivalent
to the Content-Length: line in the response headers, if the server set them
correctly. The following should also be true,
for 1 <= $j <= size($run->urls()):
( $i->response_body_lengths()->[$j] == $i->bytes_read()->[$j] - length($i->response_headers()->[$j]) )
(return value will be undefined if memory < 2)
The following examples of ApacheBench usage are paired with the resulting
output from an Apache access_log. This should give you a feel for how the
global $b->priority() method and the per-run $run->order() method affect how
HTTP requests are sent.
First, let's set $b->priority(``equal_opportunity'') (its default).
my $b = HTTPD::Bench::ApacheBench->new;
$b->concurrency(1);
$b->priority("equal_opportunity");
Add a single run and execute, then look at what gets sent to Apache.
my $run = HTTPD::Bench::ApacheBench::Run->new
({
repeat => 3,
urls => [ "http://localhost/";,
"http://localhost/server-status"; ],
order => "breadth_first",
});
$b->add_run($run);
$b->execute;
Apache access_log output:
127.0.0.1 - - [20/Sep/2000:18:43:32 -0400] "GET / HTTP/1.0" 200 5565 127.0.0.1 - - [20/Sep/2000:18:43:32 -0400] "GET / HTTP/1.0" 200 5565 127.0.0.1 - - [20/Sep/2000:18:43:32 -0400] "GET / HTTP/1.0" 200 5565 127.0.0.1 - - [20/Sep/2000:18:43:32 -0400] "GET /server-status HTTP/1.0" 200 5294 127.0.0.1 - - [20/Sep/2000:18:43:32 -0400] "GET /server-status HTTP/1.0" 200 5294 127.0.0.1 - - [20/Sep/2000:18:43:32 -0400] "GET /server-status HTTP/1.0" 200 5294
Let's add another run and execute, and see what Apache sees.
my $run2 = HTTPD::Bench::ApacheBench::Run->new
({
repeat => 3,
urls => [ "http://localhost/perl-status";,
"http://localhost/proxy-status"; ],
order => "breadth_first",
});
$b->add_run($run2);
$b->execute;
Notice that both the first and second runs get equal opportunity. Apache access_log output:
127.0.0.1 - - [20/Sep/2000:18:49:10 -0400] "GET / HTTP/1.0" 200 5565 127.0.0.1 - - [20/Sep/2000:18:49:11 -0400] "GET / HTTP/1.0" 200 5565 127.0.0.1 - - [20/Sep/2000:18:49:11 -0400] "GET / HTTP/1.0" 200 5565 127.0.0.1 - - [20/Sep/2000:18:49:11 -0400] "GET /perl-status HTTP/1.0" 200 848 127.0.0.1 - - [20/Sep/2000:18:49:11 -0400] "GET /perl-status HTTP/1.0" 200 848 127.0.0.1 - - [20/Sep/2000:18:49:11 -0400] "GET /perl-status HTTP/1.0" 200 848 127.0.0.1 - - [20/Sep/2000:18:49:11 -0400] "GET /server-status HTTP/1.0" 200 5294 127.0.0.1 - - [20/Sep/2000:18:49:11 -0400] "GET /server-status HTTP/1.0" 200 5294 127.0.0.1 - - [20/Sep/2000:18:49:11 -0400] "GET /server-status HTTP/1.0" 200 5294 127.0.0.1 - - [20/Sep/2000:18:49:11 -0400] "GET /proxy-status HTTP/1.0" 200 5886 127.0.0.1 - - [20/Sep/2000:18:49:11 -0400] "GET /proxy-status HTTP/1.0" 200 5888 127.0.0.1 - - [20/Sep/2000:18:49:11 -0400] "GET /proxy-status HTTP/1.0" 200 5889
Now let's set $b->priority(``run_priority'').
$b->priority("run_priority");
$b->execute;
Notice that now ApacheBench completes the entire first run before it starts the second. Here's the Apache access_log output:
127.0.0.1 - - [20/Sep/2000:18:52:47 -0400] "GET / HTTP/1.0" 200 5565 127.0.0.1 - - [20/Sep/2000:18:52:47 -0400] "GET / HTTP/1.0" 200 5565 127.0.0.1 - - [20/Sep/2000:18:52:47 -0400] "GET / HTTP/1.0" 200 5565 127.0.0.1 - - [20/Sep/2000:18:52:47 -0400] "GET /server-status HTTP/1.0" 200 5294 127.0.0.1 - - [20/Sep/2000:18:52:47 -0400] "GET /server-status HTTP/1.0" 200 5294 127.0.0.1 - - [20/Sep/2000:18:52:47 -0400] "GET /server-status HTTP/1.0" 200 5294 127.0.0.1 - - [20/Sep/2000:18:52:47 -0400] "GET /perl-status HTTP/1.0" 200 848 127.0.0.1 - - [20/Sep/2000:18:52:47 -0400] "GET /perl-status HTTP/1.0" 200 848 127.0.0.1 - - [20/Sep/2000:18:52:47 -0400] "GET /perl-status HTTP/1.0" 200 848 127.0.0.1 - - [20/Sep/2000:18:52:47 -0400] "GET /proxy-status HTTP/1.0" 200 5858 127.0.0.1 - - [20/Sep/2000:18:52:47 -0400] "GET /proxy-status HTTP/1.0" 200 5861 127.0.0.1 - - [20/Sep/2000:18:52:47 -0400] "GET /proxy-status HTTP/1.0" 200 5864
Let's now set our runs to $run->order(``depth_first'') instead of
``breadth_first''. With ``depth_first'', $b->priority() has no effect,
since each run can only use a maximum of one concurrent server
(by definition of ``depth_first'', it can only be sending one request
at a time).
$b->run(0)->order("depth_first");
$b->run(1)->order("depth_first");
$b->execute;
Notice each sequence gets sent in full before it repeats. Apache access_log output:
127.0.0.1 - - [20/Sep/2000:19:02:01 -0400] "GET / HTTP/1.0" 200 5565 127.0.0.1 - - [20/Sep/2000:19:02:01 -0400] "GET /server-status HTTP/1.0" 200 5294 127.0.0.1 - - [20/Sep/2000:19:02:01 -0400] "GET / HTTP/1.0" 200 5565 127.0.0.1 - - [20/Sep/2000:19:02:01 -0400] "GET /server-status HTTP/1.0" 200 5294 127.0.0.1 - - [20/Sep/2000:19:02:01 -0400] "GET / HTTP/1.0" 200 5565 127.0.0.1 - - [20/Sep/2000:19:02:01 -0400] "GET /server-status HTTP/1.0" 200 5294 127.0.0.1 - - [20/Sep/2000:19:02:01 -0400] "GET /perl-status HTTP/1.0" 200 848 127.0.0.1 - - [20/Sep/2000:19:02:01 -0400] "GET /proxy-status HTTP/1.0" 200 5858 127.0.0.1 - - [20/Sep/2000:19:02:01 -0400] "GET /perl-status HTTP/1.0" 200 848 127.0.0.1 - - [20/Sep/2000:19:02:01 -0400] "GET /proxy-status HTTP/1.0" 200 5860 127.0.0.1 - - [20/Sep/2000:19:02:01 -0400] "GET /perl-status HTTP/1.0" 200 848 127.0.0.1 - - [20/Sep/2000:19:02:01 -0400] "GET /proxy-status HTTP/1.0" 200 5860
Now let's send the same runs, but with a higher concurrency level.
$b->concurrency(2); $b->execute;
Notice that ApacheBench sends requests from all runs in order to fill up the specified level of concurrent requests. Apache access_log output:
127.0.0.1 - - [20/Sep/2000:19:04:38 -0400] "GET / HTTP/1.0" 200 5565 127.0.0.1 - - [20/Sep/2000:19:04:38 -0400] "GET /perl-status HTTP/1.0" 200 848 127.0.0.1 - - [20/Sep/2000:19:04:38 -0400] "GET /server-status HTTP/1.0" 200 5294 127.0.0.1 - - [20/Sep/2000:19:04:38 -0400] "GET /proxy-status HTTP/1.0" 200 5891 127.0.0.1 - - [20/Sep/2000:19:04:38 -0400] "GET / HTTP/1.0" 200 5565 127.0.0.1 - - [20/Sep/2000:19:04:38 -0400] "GET /perl-status HTTP/1.0" 200 848 127.0.0.1 - - [20/Sep/2000:19:04:38 -0400] "GET /server-status HTTP/1.0" 200 5294 127.0.0.1 - - [20/Sep/2000:19:04:38 -0400] "GET /proxy-status HTTP/1.0" 200 5878 127.0.0.1 - - [20/Sep/2000:19:04:38 -0400] "GET / HTTP/1.0" 200 5565 127.0.0.1 - - [20/Sep/2000:19:04:38 -0400] "GET /perl-status HTTP/1.0" 200 848 127.0.0.1 - - [20/Sep/2000:19:04:38 -0400] "GET /server-status HTTP/1.0" 200 5294 127.0.0.1 - - [20/Sep/2000:19:04:38 -0400] "GET /proxy-status HTTP/1.0" 200 5878
Let's take a look at the regression data from that last execute() call.
foreach my $run_no (0..1) {
foreach my $repeat (0..2) {
print "response times (in ms) for run $run_no, iteration ".($repeat+1).":\n ";
print join("\n ", @{$b->run($run_no)->iteration($repeat)->response_times});
print "\n";
}
}
Perl output:
response times (in ms) for run 0, iteration 1:
69
39
response times (in ms) for run 0, iteration 2:
67
39
response times (in ms) for run 0, iteration 3:
65
41
response times (in ms) for run 1, iteration 1:
67
40
response times (in ms) for run 1, iteration 2:
66
39
response times (in ms) for run 1, iteration 3:
65
39
ApacheBench/Perl has been tested and verified to work on the following platforms:
kernel Perl version compiler architecture distribution ------ ------------ -------- ------------ ------------ FreeBSD 4.4 Perl 5.005_03 gcc 2.95.3 x86 Linux 2.2.x Perl 5.005_03 egcs-2.91.66 x86 RedHat 6.2 Linux 2.4.x Perl 5.6.0 gcc 2.96 x86 RedHat 7.1 SunOS 5.8 Perl 5.6.1 gcc 2.95.2 x86
Please send patches for ports to other platforms to me (Adi Fairbank). I have received numerous reports of segmentation faults on Solaris, but I tested it on a SunOS 5.8 machine running on i386 architecure and it worked fine. It may have problems running on Sun hardware, or newer versions of Solaris.
Error checking is getting better but is still not great. If you do not configure correctly, you may experience a segmentation fault on execute().
The response body of any response which is larger than the buffersize
applicable to it will be truncated to zero length. This is contrary to
what the documentation says above. This should be fixed soon. For now,
just set your $run->buffersize() big enough for the largest page you anticipate
receiving.
If you are running in perl's taint-checking mode, and you pass tainted data to ApacheBench (e.g. a tainted URL), it will barf. Don't ask me why.
The ApacheBench Perl API is based on code from Apache 1.3.12 ab (src/support/ab.c), by the Apache group.
The ApacheBench Perl API was originally written by Ling Wu <ling@certsite.com>
Rewritten and currently maintained by Adi Fairbank <adi@adiraj.org>
Recent efforts have been made to incorporate the newest ab code from the Apache Group. As of version 0.62, most features of Apache 1.3.22 ab are supported. The main exception being SSL.
Please e-mail either Adi or Ling with bug reports, or preferably patches.
This package is free software and is provided AS IS without express or implied warranty. It may be used, redistributed and/or modified under the terms of the Perl Artistic License (http://www.perl.com/perl/misc/Artistic.html)
Dec 02, 2001
|
HTTPD::Bench::ApacheBench - Perl API for Apache benchmarking and regression testing. |