Determine how sites react under increased requests with curl

There are times when you test out your site using curl, and it loads fine. But yet when multiple users or a search bot is going through your site, you notice the page request times skyrocket. There could be many causes, though usually it revolves around a lack of caching since repeated requests for the same page need to first check in with the database, then get passed over to PHP to finish processing the request.

Okay great, cache is king. We know this. But how can we actually confirm this is a problem? Well, this is where curl comes to the rescue.

First, establish a baseline by sending a single request to each website:

[user@workstation01 ~]# for i in www.example1.com www.example2.com www.example3.com; do echo -n "$i "; (time curl -IL $i -XGET) 2>&1 | grep -E "real|HTTP"; echo; done

www.example1.com HTTP/1.1 200 OK
real	0m0.642s

www.example2.com HTTP/1.1 200 OK
real	0m2.234s

www.example3.com HTTP/1.1 200 OK
real	0m0.421s

So based off those results, we established that 2 of the sites take about half a second to load, and the other one (www.example2.com) takes about 2 seconds to load when hit with a single request.

As www.example2.com takes 2 seconds to load with a single request, lets see what happens during increased traffic. This could be from valid users, search bots, or something else. How much will the load times increase for the site? Here is an example sending over 25 requests:

[user@workstation01 ~]# for i in {1..25}; do (time curl -sIL http://www.example2.com -X GET &) 2>&1 | grep -E "real|HTTP" & done

HTTP/1.1 200 OK
real	0m11.297s
HTTP/1.1 200 OK
real	0m11.395s
HTTP/1.1 200 OK
real	0m11.906s
HTTP/1.1 200 OK
real	0m12.079s
...
HTTP/1.1 200 OK
real	0m11.297s

So with the increased requests, the page load times increase from 2 seconds all the way up to 11-12 seconds!

Determining why this happens will involve investigation outside the scope of this article. However at least now we know which site doesn’t perform well under increased requests.

While every site is different, lets say www.example2.com was a WordPress site. Try installing the WordPress W3 Total Cache (W3TC) plugin. Basic items to enable/disable within W3TC would be:

- Enable Page Cache
- Disable Minify options
- Enable Browser Cache

Next, sign up with a free CloudFlare account, then add the following PageRules so CloudFlare actually starts caching your WordPress site:

1. *example2.com/wp-admin* : Cache Level (Bypass)
2. *example2.com/wp-login.php* : Cache Level (Bypass)
3. *example2.com/* : Cache Level (Cache Everything)
* The order is extremely critical here.  Make sure the pages you do not wish to cache are above the last line!
** Every site is different.  These rules may or may not work for you.

Then retest to see if your changes helped improve the times.

[user@workstation01 ~]# for i in {1..25}; do (time curl -sIL http://www.example2.com -X GET &) 2>&1 | grep -E "real|HTTP" & done

HTTP/1.1 200 OK
real	0m1.347s
HTTP/1.1 200 OK
real	0m1.342s
HTTP/1.1 200 OK
real	0m1.237s
HTTP/1.1 200 OK
real	0m1.021s
...
HTTP/1.1 200 OK
real	0m1.532s