Benchmarking Tips
This guide will enable you to use a selection of benchmarking tools to make the fairest comparisons possible between different web servers. Testing such as this is essential to determining website capacity and capabilities for web applications.
These are the basic steps that will be covered:
- Prepare a Client Server and a Test Server
- Install web servers to be benchmarked on Test Server
- Test the web servers' general configurations
- Run tests from the Client Server with your choice of tools:
Prepare a Client Server and a Test Server
When preparing your servers, keep these principles in mind.
Low Latency
Low latency is best achieved by launching your Test and Client servers in the same zone or country, or on a local network. You can test latency by sending a ping command from the Client Server:
ping -c5 Test_Server
--- 142.93.185.51 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4099ms rtt min/avg/max/mdev = 0.504/0.591/0.677/0.062 ms
High Bandwidth Network
Make sure the bandwidth between the two servers is enough. You don't want the network become your benchmark's bottleneck. You can easily verify sufficient bandwidth by using the iperf tool.
Install the tool on Ubuntu:
apt-get install iperf
yum install epel-release yum update yum install iperf
Run the command from the Client Server:
iperf -c TEST_SERVER_IP -i1
Run it from the Test Server:
iperf -s -i1
The default iperf port is 5001.
Your firewall needs to be open for the default iperf testing port of 5001. Or, you can specify a different port with the -p parameter.
If you have bandwidth results of 1 GB or more, that should be sufficient.
A Powerful Client Server
To fully demonstrate the power of the web servers you are testing, the Client Server needs to be more powerful then the Test Server.
No Proxy or CDN
To avoid any potential bottlenecks on the Client Server, avoid using Proxies, Firewalls, Load Balancers or CDNs in front of the Test Server.
Example of Server Specs
We've found that servers with the following specs are appropriate for benchmarking:
| Server | CPU | Memory | NetWork | latency | Zone |
|---|---|---|---|---|---|
| Test Client | 2 | 2GB | 2 Gbits/sec | 0.59 ms | NYC1 |
| Test Server | 1 | 1GB | 2 Gbits/sec | 0.59 ms | NYC1 |
Install web servers to be benchmarked on Test Server
When setting up your Test Server, it is helpful to keep the following concepts in mind:
Multiple Web Servers on the Same Test Server
Install all of the web servers you wish to test on the same Test Server. You will not be running all of the servers at the same time, but bringing up each one as needed.
Same Modules
If you want to test benchmarks between different types of web servers, you will want to have all modules as identical as possible.
Switching Web Servers
With a Control Panel: It may be easier to test LSWS/Apache/Nginx on a control panel such as cPanel, which has a web server switch function built in, and where all PHP modules are quite similar to each other.
Without a Control Panel: You will probably need to manually stop web server A, and then bring up web server B. You can build PHP modules with remi repo.
Similar Nginx and OpenLiteSpeed PHP Modules
If you're looking to compare OpenLiteSpeed and nginx, you'll want to refer to this chart to make sure that you have similar modules installed between the two servers.
| Server | Nginx | OpenLiteSpeed |
|---|---|---|
| PHP version | 7.2 | 7.2 |
| Unix Socket | On | On |
| Loaded Config File | /etc/php/7.2/fpm/php.ini | /usr/local/lsws/lsphp72/etc/php/7.2/litespeed/php.ini |
| Module | module_bcmath | module_bcmath |
| Module | module_calendar | module_calendar |
| Module | module_cgi-fcgi | module_core |
| Module | module_core | module_ctype |
| Module | module_ctype | module_curl |
| Module | module_curl | module_date |
| Module | module_date | module_dom |
| Module | module_dom | module_enchant |
| Module | module_enchant | module_exif |
| Module | module_exif | module_fileinfo |
| Module | module_fileinfo | module_filter |
| Module | module_filter | module_ftp |
| Module | module_ftp | module_gd |
| Module | module_gd | module_gettext |
| Module | module_gettext | module_gmp |
| Module | module_gmp | module_hash |
| Module | module_hash | module_iconv |
| Module | module_iconv | module_json |
| Module | module_json | module_libxml |
| Module | module_libxml | module_mbstring |
| Module | module_mbstring | module_mysqli |
| Module | module_mysqli | module_mysqlnd |
| Module | module_mysqlnd | module_openssl |
| Module | module_openssl | module_pcntl |
| Module | module_pcre | module_pcre |
| Module | module_pdo | module_pdo |
| Module | module_pdo_mysql | module_pdo_mysql |
| Module | module_phar | module_phar |
| Module | module_posix | module_posix |
| Module | module_pspell | module_pspell |
| Module | module_readline | module_readline |
| Module | module_recode | module_recode |
| Module | module_reflection | module_reflection |
| Module | module_session | module_session |
| Module | module_simplexml | module_simplexml |
| Module | module_soap | module_soap |
| Module | module_sockets | module_sockets |
| Module | module_sodium | |
| Module | module_spl | module_spl |
| Module | module_standard | module_standard |
| Module | module_sysvmsg | module_sysvmsg |
| Module | module_sysvsem | module_sysvsem |
| Module | module_sysvshm | module_sysvshm |
| Module | module_tidy | module_tidy |
| Module | module_tokenizer | module_tokenizer |
| Module | module_xml | module_xml |
| Module | module_xmlreader | module_xmlreader |
| Module | module_xmlwriter | module_xmlwriter |
| Module | module_xsl | module_xsl |
| Module | module_zend+opcache | module_zend+opcache |
| Module | module_zip | module_zip |
| Module | module_zlib | module_zlib |
php.ini
We copy the php.ini file from one server to another, so the contents are identical.
Nginx and OpenLiteSpeed Cache Types
When comparing cache solutions between OpenLiteSpeed and nginx, use these types.
| Server | Nginx | OpenLiteSpeed |
|---|---|---|
| Cache type | fastcgi_cache | LSCache Plugin |
Test the Web Servers' General Configurations
For best results, keep these configuration suggestions in mind:
Have a Large HTTP/HTTPS Connection Number
Keep your connections number high.
PHP-Related Settings Should Have High Numbers
Maximize suEXEC Max Conn, PHP Child number, and PHP Max number when you are testing PHP-based apps such as WordPress.
Tip
If you are testing other language, e.g. Python, please do increase the Max Connections as well)
Configure all Web Servers the Same
- Same SSL Ciphers
- OCSP ON
- Same Document Root
- Same level of compression
- Same number of workers
- Keep-alive enabled
- Debug log off
Example Nginx and OpenLiteSpeed Setup
| Server | Nginx | OpenLiteSpeed |
|---|---|---|
| Version | 1.15.8 | 1.4.44 |
| Config Port | 80/443 | 80/443 |
| Certificate | /etc/letsencrypt/live/wp-benchmark.tk | /etc/letsencrypt/live/wp-benchmark.tk |
| doc root | /var/www/html | /var/www/html |
| worker_processes | 1 | 1 |
| user | www-data | www-data |
| ssl_ciphers | ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384 | ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384 |
| Gzip | On | On |
| Compression Level | 5 | 5 |
| gzip_min_length | 300 | 300 |
| gzip_types | text/plain text/css text/xml text/javascript application/x-javascript application/xml image/svg+xml | text/*, application/x-javascript, application/javascript, application/xml, image/svg+xml |
| keepalive_timeout | 15 | 5 |
| ssl_session_timeout | 10m | On |
| ssl_session_cache | shared:SSL:10m | On |
| OCSP | On | On |
See our Benchmarks Shootout spreadsheet for more example configuration information.
Run tests from the Client Server with your choice of tools
These are some recommended benchmarking tools. The parameters we've chosen to use from the Client Server, are intended to simulate real users as closely as possible.
ApacheBench
Installation
Ubuntu:
apt-get install apache2-utils -y
yum install httpd-tools -y
Command
ab -n 1000 -c 100 -k -H "Accept-Encoding: gzip,deflate" http://Test_Server_Domain/
-n: Number of requests to perform for the benchmarking session-c: Number of multiple requests to perform at a time-k: Enable the HTTP KeepAlive feature-H: custom-header
Documentation: Apache.org
Output:
Server Software: LiteSpeed Server Hostname: Test_Server_Domain Server Port: 80 Document Path: / Document Length: 3749 bytes Concurrency Level: 100 Time taken for tests: 0.210 seconds Complete requests: 1000 Failed requests: 0 Keep-Alive requests: 1000 Total transferred: 4048000 bytes HTML transferred: 3749000 bytes Requests per second: 4756.31 [#/sec] (mean) Time per request: 21.025 [ms] (mean) Time per request: 0.210 [ms] (mean, across all concurrent requests) Transfer rate: 18802.29 [Kbytes/sec] received
Siege
Installation
Ubuntu:
apt-get install siege -y
yum install epel-release yum install siege -y
Command
siege -c 10 -r 100 -b http://Test_Server_Domain/
Parameters :
-b: runs the test with NO DELAY for throughput benchmarking-c: Set the number of concurrent users-r: Allows you to run the siege for NUM repetitions
Output:
Transactions: 6000 hits Availability: 100.00 % Elapsed time: 4.39 secs Data transferred: 247.85 MB Response time: 0.00 secs Transaction rate: 1366.74 trans/sec Throughput: 56.46 MB/sec Concurrency: 3.66 Successful transactions: 6000 Failed transactions: 0 Longest transaction: 0.03 Shortest transaction: 0.00
h2load
If you are specifically testing HTTP2, you may want to give h2load a try.
Installation
Ubuntu:
apt-get update apt-get install nghttp2-client -y
yum install epel-release yum install nghttp2 -y
For old OS version
You may need to build the h2load if Ubuntu version under 18 or CentOS under 7. Download source from https://github.com/nghttp2/nghttp2.git
Command
h2load -n 1000 -c 10 -t 1 -m 10 https://Test_Server_Domain/
Parameters :
-c: Number of concurrent clients-n: Number of requests across all clients-t: Number of native threads-m: The max concurrent streams to issue per client
Output:
starting benchmark... spawning thread #0: 100 total client(s). 10000 total requests TLS Protocol: TLSv1.2 Cipher: ECDHE-RSA-AES256-GCM-SHA384 Server Temp Key: X25519 253 bits Application protocol: h2 progress: 10% done progress: 20% done progress: 30% done progress: 40% done progress: 50% done progress: 60% done progress: 70% done progress: 80% done progress: 90% done progress: 100% done finished in 10.58s, 945.15 req/s, 47.52MB/s requests: 10000 total, 10000 started, 10000 done, 10000 succeeded, 0 failed, 0 errored, 0 timeout status codes: 10000 2xx, 0 3xx, 0 4xx, 0 5xx traffic: 502.82MB (527242343) total, 140.02KB (143384) headers (space savings 95.72%), 501.64MB (526010000) data
Jmeter
Installation
Ubuntu:
apt install openjdk-11-jre-headless -y
yum install java-11-openjdk-devel -y
You can download it here. This example shows the installation of version 5.1.1, the install process is the same for other versions.
wget http://apache.osuosl.org//jmeter/binaries/apache-jmeter-5.1.1.tgz
tar xf apache-jmeter-*.tgz
cd apache-jmeter*/bin/
Command
Start JMeter in command line mode
./jmeter.sh -n -t examples/TESTPLAN.jmx
Example of the Test Plan file
bash <?xml version="1.0" encoding="UTF-8"?> <jmeterTestPlan version="1.2" properties="5.0" jmeter="5.0 r1840935"> <hashTree> <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Test Plan" enabled="true"> <stringProp name="TestPlan.comments"></stringProp> <boolProp name="TestPlan.functional_mode">false</boolProp> <boolProp name="TestPlan.tearDown_on_shutdown">true</boolProp> <boolProp name="TestPlan.serialize_threadgroups">false</boolProp> <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true"> <collectionProp name="Arguments.arguments"/> </elementProp> <stringProp name="TestPlan.user_define_classpath"></stringProp> </TestPlan> <hashTree> <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true"> <stringProp name="ThreadGroup.on_sample_error">continue</stringProp> <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true"> <boolProp name="LoopController.continue_forever">false</boolProp> <intProp name="LoopController.loops">-1</intProp> </elementProp> <stringProp name="ThreadGroup.num_threads">10</stringProp> <stringProp name="ThreadGroup.ramp_time">60</stringProp> <boolProp name="ThreadGroup.scheduler">true</boolProp> <stringProp name="ThreadGroup.duration">300</stringProp> <stringProp name="ThreadGroup.delay">0</stringProp> </ThreadGroup> <hashTree> <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="HTTP Request" enabled="true"> <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true"> <collectionProp name="Arguments.arguments"/> </elementProp> <stringProp name="HTTPSampler.domain">hostname</stringProp> <stringProp name="HTTPSampler.port">80</stringProp> <stringProp name="HTTPSampler.protocol">http</stringProp> <stringProp name="HTTPSampler.contentEncoding"></stringProp> <stringProp name="HTTPSampler.path">/</stringProp> <stringProp name="HTTPSampler.method">GET</stringProp> <boolProp name="HTTPSampler.follow_redirects">true</boolProp> <boolProp name="HTTPSampler.auto_redirects">false</boolProp> <boolProp name="HTTPSampler.use_keepalive">true</boolProp> <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp> <boolProp name="HTTPSampler.BROWSER_COMPATIBLE_MULTIPART">true</boolProp> <stringProp name="HTTPSampler.embedded_url_re"></stringProp> <stringProp name="HTTPSampler.connect_timeout"></stringProp> <stringProp name="HTTPSampler.response_timeout"></stringProp> </HTTPSamplerProxy> <hashTree/> <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="HTTP Header Manager" enabled="true"> <collectionProp name="HeaderManager.headers"> <elementProp name="" elementType="Header"> <stringProp name="Header.name">Accept-Encoding</stringProp> <stringProp name="Header.value">gzip, deflate, sdch</stringProp> </elementProp> </collectionProp> </HeaderManager> <hashTree/> </hashTree> </hashTree> </hashTree> </jmeterTestPlan>
Parameters :
-n: Specified HNeter is to run in command line mode-t: Name of the file that contains the Test Plan
Output:
summary + 1 in 00:00:00 = 2.5/s Avg: 58 Min: 58 Max: 58 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0 summary + 45568 in 00:00:25 = 1851.0/s Avg: 1 Min: 0 Max: 17 Err: 0 (0.00%) Active: 5 Started: 5 Finished: 0 summary = 45569 in 00:00:25 = 1820.8/s Avg: 1 Min: 0 Max: 58 Err: 0 (0.00%) summary + 107678 in 00:00:30 = 3589.4/s Avg: 1 Min: 0 Max: 18 Err: 0 (0.00%) Active: 10 Started: 10 Finished: 0 summary = 153247 in 00:00:55 = 2785.0/s Avg: 1 Min: 0 Max: 58 Err: 0 (0.00%)
wrk
wrk is a modern HTTP benchmarking tool capable of generating significant load when run on a single multi-core CPU.
Installation
Ubuntu:
apt-get install build-essential libssl-dev git -y
yum groupinstall 'Development Tools'
yum install -y openssl-devel git
git clone https://github.com/wg/wrk.git wrk && cd wrk make
Command
wrk -c100 -t1 -d10s http://Test_Server_Domain/
-c: Total number of HTTP connections to keep open with each thread-t: Total number of threads-d: Duration of the test
Output:
Running 10s test @ http://Test_Server_Domain/
1 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 19.62ms 0.98ms 34.28ms 93.53%
Req/Sec 5.11k 179.79 5.31k 91.00%
50870 requests in 10.00s, 542.09MB read
Requests/sec: 5084.80
Transfer/sec: 54.19MB