Server performance measurement

Like everything in life, webservers sometimes have issues and even break down. One major way to address this, is by monitoring or measuring the performance of webservers.

This article is only about LAMP servers running WooCommerce shops
This article is not only about monitoring server performance, but also about notifications related to this
There is quite an overlap with performance aspects of individual webshops, like measuring site performance to draw conclusions about a server's state - And that's quite handy
Including solutions for solving common performance issues.

Why?

Some overlapping reasons for server performance monitoring:

Business case optimization

I like to think in business cases. So, how to find the optimum in this business case:

When stuff breaks down or doesn't work as intended, it costs money
When stuff works really well, it makes money
Monitoring costs some money.

Continuity

From the business case to a more practical approach: Probably the first primary goal of server performance monitoring, is to assure continuity of services, like a webserver with a high uptime.

Optimization

Probably the second practical primary goal of server performance monitoring: Optimize webserver performance.

Here, that probably means the same as webshop performance optimization as so far this performance is related to the server or platform on which a webshop is running.

Prevention

Prevention: Get alerted of issues before they develop into problems. E.g.:

Performance degredation due to site popularity
Performance degredation due to crawlers
Running out of storage space.

Notification

Amongst other things, this article is about automation of performance measurements and reporting, as this is something that computers can do much better than humans.

When business expands, it can easily happen that something goes wrong, and nobody is aware of it for a long time. Just getting notified that there is a problem, is already quite an improvement.

Early alert

Get alerted as soon as there is a problem. E.g., immediately when a site is down, rather than hours or maybe days later.

Diagnosis

When the shit does hit the fan, be able to do a proper diagnosis of what happened and how to improve on that.

Logging and reporting (e.g., by email) might play quite a role here.

How?

How to do webserver performance monitoring? Some ideas:

Measure input

Track the input; the stuff 'under the hood', the parts that together make a webserver. Here, static aspects like the number of threads, aren't relevant. It's about dynamic or changing parts. Some ideas:

Memory usage
Bandwidth usage
Storage space usage
Storage space bandwidth
CPU usage
CPU usage by Apache

Measure output

Measure the output; the end result of what a webserver is supposed to do.

E.g.:

That sites are still online
Web site speed
Checkout speed?

Note that these example measure website metrics and not any webserver metrics: A webserver is a means, not a goal - The goal is (mostly) fast webshops.

You need both

In the end, output is the only relevant measure. However, you do need the input measurements as well:

Signaling: To be able to solve issues before they affect server performance
Diagnosis: To figure out the cause of a problem, once it happened.

Criteria

Case: The opposite of WordFence

The following paragraphs, discuss some of the criteria that emerged to me.

I can't stress enough how important understanding and inspection is for me: I found most tools that I come across (especially WordFence and iThemes Security) terribly opaque: I have no real idea what they are doing, how they are doing it, and why there are doing it. To make this even more specific: I can't figure out if I actually need WordFence at all:

Over the years, I think I have had two hacked WordPress sites, and in both cases, the culprit were page builders, especially WPBakery Page Builder, if I remember correctly
In 2023 and 2024, we have had issues with crawlers overwhelming sites. Appearantly, WordFence didn't fix that
Over the years, I've asked some collegues if you actually need a security plugin like WordFence, and I usually got an answer like "of course", but when I pressed for why this is, I don't think I ever got a real answer.

And this would actually be a nice use case concerning server performance monitoring and deploying related tools: Make sure that I need a plugin like WordFence, or actually not needing it at all.

Understandable reporting

Seems obvious that the reporting of monitoring tools should be understandable, but it isn't. In general, I find security and monitoring tools hard to understand, to use and to rely upon:

E.g.: I found the WordPress plugins WordFence and iThemes Security not understandable or intuitive at all and I suspect this is done on purpose: To create a sense of urgency to buy the payed version of these plugins and as a means to develop their market
This seem ssimilar to virusscanners and spyware removers.

This is probably one of the reasons why I like DIY solutions: To have a thorough understanding of what is actually happening and learn about service performance and monitoring along the way while building this.

Understandable working

See the previous paragraph concerning the problem.

Examples of stuff that is understandable to me:

sudo tail -f /var/log/apache2/access.log to see real-time visits
Using iptables for blocking and filtering traffic.

Verifiable

Quite related to understandable: Having stuff be verifiable. E.g., if I change firewall settings, that I instantly can see the change in traffic.

Manageability

I want to be in control of the monitoring solutions:

What does it exactly do?
What are in the updates?
What recources does it take?

Again, this is quite the opposite of opaque tools like WordFence or iThemes Security and maybe why I like DIY solutions.

Paper trail

I find it important that it is documented why settings are as they are. E.g.:

iptables doesn't allow for comments to be included with rules
The possibility to include comments in Bash scripts, including scripts to import rules into iptables.

Low level

I prefer low-level, 'elimentary' tools. E.g.: Rather using iptables than some fancy WordPress plugin.

DIY

I quite like Do-it-yourself (DIY) solutions (e.g., in Bash), so I understand exactly what is happening, why, and how.

Server monitoring tools

New Relic

Features: Provides detailed performance metrics, including response times, error rates, and server resource usage
Setup: Install the New Relic PHP agent on your server and configure it to monitor your applications.

Datadog

Features: Offers comprehensive monitoring, including APM (Application Performance Monitoring) and infrastructure metrics
Setup: Install Datadog’s Agent on your server and integrate it with your web applications.

Prometheus with Grafana

Features: Prometheus collects metrics, and Grafana visualizes them. Useful for custom dashboards
Setup: Install Prometheus to scrape metrics from your web server and set up Grafana for visualization.

DIY

This is what I have been doing so far (Bash), and I quite like it

Site performance monitoring tools

Pingdom

Features: Provides response time monitoring and uptime checks from multiple locations
Setup: Configure Pingdom to monitor your website and get alerts for performance issues.

GTmetrix

Features: Analyzes page load times and provides recommendations for performance improvements
Setup: Use GTmetrix for periodic checks and performance analysis.

Might be interesting to use GTmetrix, as we already use this for performance optimization, aka site tuning.

WebPageTest

Features: Offers detailed performance tests and analysis from different locations and devices
Setup: Run tests from WebPageTest to understand how your site performs under various conditions.

DIY - curl

I like DIY, but would it be possible to measure this from the server itself? Or from another server from the same hosting company? Maybe with curl? → curl

DIY - wget

See wget for details.

Server-sided monitoring

Apache Status Module

Features: Provides a real-time overview of Apache server performance
Setup: Enable the mod_status module in Apache and configure it in your httpd.conf or apache2.conf file.

MySQL Slow Query Log

Features: Identifies slow-running queries that might be impacting performance
Setup: Enable slow query logging in MySQL to track and optimize problematic queries.

Server resource usage monitoring tools

Munin

Features: Provides graphs for various server metrics like CPU usage, memory usage, and network traffic
Setup: Install Munin on your server and configure it to collect and display data.

Zabbix

Features: A comprehensive monitoring solution that tracks server health, network performance, and application performance
Setup: Install Zabbix Server and Agents, configure monitoring for your LAMP stack.

ELK Stack

Use tools like ELK Stack (Elasticsearch, Logstash, Kibana) for log analysis.

Load testing

Simulate traffic to test how a server is doing under high load. E.g.:

Apache JMeter
Locust.

tail -f access.log

This really helps to get a real-time impression of requests made on the server - E.g., to see how much of the traffic is from bots:

sudo tail -f /var/log/apache2/access.log

free -h

Use free -h to check for available memory. E.g.:

$ free -h

Memory usage:
              total        used        free      shared  buff/cache   available
Mem:           47Gi        24Gi       591Mi       118Mi        21Gi        21Gi
Swap:         8.0Gi       399Mi       7.6Gi

Only 591Mi out of 47Gi is free, or 1.26%. That sounds problematic, probably leading to MySQL connection errors.