Prior to the existence of Mod Proxy AJP being built into Apache HTTP we used Mod JK for any Tomcat based instance we hosted here. All of our Resin containers would use Mod Caucho, and all the Tomcat containers used Mod JK, and all was well in the world. We never had any significant complaints with Mod JK (Caucho’s another story for another day), other than it being a bit of a pain to setup, and it needing to be built from source. This created a situation where maintaining Mod JK was problematic, because it wouldn’t automatically be updated in our routine RHEL updates forcing us to manually rebuild for critical security updates. Then you added in the fact that the configuration files totalled in at three, with a mod_jk.conf, a worker.properties file, and the actually JkMount itself in the virtual host configuration, and you had yourself a maintenance headache. Not too mention the slightest of typos and a simple Apache reload turns into Apache going belly up like a poisoned whale.

Needless to say I suppose, when Proxy AJP came into our lives we as engineers were smitten. We were all quite familiar with ProxyPass, and simply swapping “http://” with “ajp://” to proxy to Tomcat containers seemed like the best solution we had ever seen. We were practically partying over the idea that we could finally ditch JK like a bad habit, and ride off into the sunset with our new ProxyPassing beast of a friend. The only downside is it required Apache 2.2, so our RHEL 5 boxes were all set, but the RHEL 4 boxes left us in the hands of old faithful, Mod JK. The ultimate question continued to linger though: Which would hold it’s own under high stress more effectively? In terms of average browsing, I don’t think you’d ever notice a difference, even after the tests I’ve done. The difference, to the human eye at least, is negligible. Nevertheless, I constructed a few test to put both through their paces.

Test Environment:

• Apache 2.2
• Tomcat 5.5.20
• Java 1.6 64 bit
• PostgreSQL 8.3.5 (tweaked for performance)
• Confluence 2.10.1

Test Tools:

• Pylot
• BrowserMob

To get the ball rolling I started with Pylot. This was a quick and free way to start some preliminary testing. My initial tests fired up a constant flow of 200 (virtual) users at the instance for 10 minutes to see what would happen. To my initial surprise, Proxy AJP was a pretty clear winner to start things off. It was able to serve 1-2 more requests per second, while serving an average response time of roughly 1 second better than JK. Surely that couldn’t be right, it couldn’t possibly be that easy right? Right, not that easy. With some adjustments to Mod JK, like giving it a connection pool, I was able to get the results from my tests with Pylot to average out to be roughly the same. Under constant pounding from 200 agents, both were able to average an average response time of approximately 8.3 seconds, while serving roughly 23 requests per second. I was pretty pleased with my test environment at this point, and ready to kick it up a notch.

We got in touch with Patrick from BrowserMob, which is a site that offers real load testing with real browsers. I was hoping that with his service, I’d be able to find distinguishing differences between the two AJP connectors for Apache. After crafting myself a script using Selenium, and testing it ad nausem locally, I was prepared for my first test against Proxy AJP. I ran a 15 minutes ramp up to 100 users, and then put a constant load of 100 users across 45 minutes on the new kid in town. Surprisingly, the throughput looked pretty incredible:

If you click the image you can get a better view of the results, and see that we topped out around 140MB per minute being served. You’ll also see that the ebb and flow of data being pushed out was incredibly spiky. There are numerous jumps up and down the graph as more and less intense pages are hit throughout the tests. At first this seemed a bit scary, because I somewhat expected a smoother line across the board. I left all configurations the same though, and proceeded to test Mod JK, where I received the following throughput graph:

In a side by side comparison (or for better results lay the images over the top), Mod JK suffers from similar, but less severe spikes. It never quite reaches the highs that Proxy was able to reach, and never dips to the lows the Proxy had succumbed to. However, there’s something else at play here that caused this to happen. Mod JK caused Apache to reach MaxClients, and hold fairly steady there. There were only 100 users, and there were 256 MaxClients available in Apache, so numerically speaking, Mod JK should have been able to handle this without cracking under the pressure.

However, since it was at max connections, it didn’t have the downward spikes that Proxy did, because it was still trying to serve past requests. At the end of the day, we saw roughly 630 timeouts from JK in the tests, with the tests timing out at 30 seconds. Yes, I could have cranked up the timeouts to allow every step to completely finish, but this is real world testing. When was the last time you allowed a page more than 30 seconds to load? Proxy AJP on the other hand only sufferred from 450 such timeouts, showing me that it was able to pump out requests, and break off connections faster than Mod JK. The way I currently see it, JK failed to serve 200 people. That’s 200 people that think my site’s down, or heaven forbid, a Google bot that is prepping to denote me as a downed site. That’s 200 requests I’m not able to serve advertisements on, and thus not able to make money off of. Might not seem like much, but it is a big deal to some degree.

Now it would be interesting to crank up the MaxClients, and see what happens in that scenario. However, I needed to keep parity between the tests, so they were both performed within the exact same scenarios. We can always do more tweaking, and make more adjustments though. However, at the end of the day, Proxy has a lot going for it. It’s a built in module in 2.2, configuration is insanely easy, balancing is a cinch, it works with the balancer manager of mod_proxy_balancer, it receives automatic security fixes via RHEL updates, and so far, it out performs JK. We plan on doing more tests with altered configuration in the future, so look forward to future rounds. Perhaps JK can make up some ground in Round 2? Perhaps you’ll also get some insight on methods to tweak your Confluence installation to squeeze a bit more performance out of it as well. Many updates to come…