In this blog post, we will provide an overview of which of the seemingly endless number of metrics in a database should be monitored in order to quickly identify problem hotspots. Examples will illustrate possible effects. Although this blog post was written on a Postgres basis, the findings can be applied to any database.
Proactive and reactive monitoring
As with all monitoring, databases can be monitored proactively and reactively. The aim of proactive monitoring is to detect problems before they become too big. Certain metrics are examined and people are alerted in the event of anomalies. This is also the preferred type of monitoring. Reactive monitoring takes place when an incident has occurred (usually in the event of security incidents or performance problems).
Overview of the metrics to be monitored
A interesting podcast on the topic as well as the resulting Community Document form the basis for the recommended metric overview:
System monitoring
Database monitoring
- TPS (= Transactions per Second) and QPS (= Queries per Second)
- Average query duration
- Longest transactions (max transaction age or top-n transactions by age)
- Commits & rollbacks - how many transactions are reversed
- Transaction Wraparound - how many transactions until the limit
- Replication Lags
- Bytes in replication slot
- Unused replication slots
- Number of WAL files waiting to be archived
- WAL generation rates
- Locks and deadlocks
- Query graph (Top n by total_time)
System monitoring
Before we look at database monitoring, let's take a look at the system on which the database is running. The following metrics should be emphasized:
A high CPU load can mean that the database cannot respond quickly enough to queries. This can lead to delays in the provision of results and slow down the response time of the database.
Load Average
A high load average can also lead to slow database response times due to delayed request response. Data loss is also possible if not all incoming requests can be processed. It should be noted that a load average not only takes into account the CPU, but also the number of processes waiting for CPU access. This means that a high load average does not necessarily mean that the CPU load is also high.
Swap I/O
As access to the swap memory is slower than to the RAM, this also has a negative impact on the response times of the database if it has to access data in the swap memory. If the swap is full, this can have a negative impact on system stability.
Read and write latency
In addition to potentially slow response times, high read / write latency can lead to delays in transactions. If transactions are not executed quickly enough in the event of latency problems, this can result in transactions taking too long and/or failing.
Disk space
Whether writing to the database or backing up data. Without sufficient disk space, data loss can occur.
Page cache hit/miss
A high miss rate means that the database has to read more data from the hard disk than is available in the page cache. This leads to performance problems such as slow response times or delayed request processing. A high hit rate means that the page cache needs more storage space. If the hit rate is too high and the page cache becomes too large, this has a negative impact on other applications, which can also lead to performance problems.
Database monitoring
Now that we know our system is monitored and healthy, let's take a look at some specific database metrics:
TPS (= Transactions per Second) and QPS (= Queries per Second)
TPS is the number of transactions, QPS the number of queries that can be processed by the database per second. These can be determined by dividing the number of processed transactions/queries in a certain period of time by the time.
If the TPS or QPS drops, queries and transactions may take longer to complete. Furthermore, the database may not be able to scale accordingly as the load increases.
Average query duration
If the average time for queries increases, this may indicate that queries are working inefficiently or that the database is overloaded due to an infrastructure bottleneck.
Longest transactions (max transaction age or top-n transactions by age)
Long transactions can cause some problems. As transactions take up and block database resources, it can happen that bottlenecks occur and the database stops responding sooner or later.
Commits & rollbacks - how many transactions are reversed
In the database, commits and rollbacks are used to successfully complete transactions and to commit or cancel changes and undo changes.
If there are many rollbacks in the database, it may be an indication that there are problems with data consistency. It can also be an indicator that the workload is high or that queries are being executed inefficiently.
Transaction Wraparound - how many transactions until the limit
Most databases use a 32-bit signed integer transaction ID to track transactions. This means that there can be approximately 2.1 billion transactions before the highest transaction ID is reached. When this happens, a transaction wraparound can occur.
In general, this means that a transaction ID is no longer unique and can collide with other transactions. As a result, the database is no longer able to process transactions, so there is a risk of data loss.
Replication Lags
Replication lags usually occur when replication does not take place in real time and there are delays between writing the data to the primary node and transferring the data to the replicas. Small lags can normally be tolerated.
Regular or prolonged high replication lags can lead to problems, e.g. database responsiveness can be impaired so that replicas are not up-to-date, which can lead to inconsistencies or data loss.
Bytes in replication slot
The number of bytes in a replication slot depends on how many changes have been made on the primary node and how often replication takes place. However, bytes in the replication slot are not yet a problem.
If the replication slot is full, replication may be stopped and there is a risk of data loss. A high value can affect the working memory of the database, which in turn has a negative impact on database performance.
Unused replication slots
A replication slot reserves memory space for replication on the primary node. If a slot is created, part of the working memory is reserved on the primary node.
If replication slots remain unused, it can happen that WAL files pile up on the primary node. This storage space problem can have a negative impact on database performance.
Number of WAL files waiting to be archived
This metric indicates how many WAL files (= Write-Ahead Log) are still waiting to be archived. WAL files are log files that record database changes so that the database can be consistently restored after a system crash.
If the number of WAL files waiting to be archived is too high, the available storage space for WAL files may be exhausted. If this happens, there is a risk of data loss as no more WAL files can be written. Furthermore, a high archiving lag can indicate that there is a problem with the archiving process, e.g. because the archiving storage is too slow or the network has problems.
As all open transactions and the changes saved in the WAL must be restored before normal operation when the database is started, the start process may take longer if there are many WAL files waiting to be archived.
WAL generation rates
This metric indicates the speed at which the database generates WAL files.
If the value is too high, this means that many database changes are being made. This can have a negative impact on performance, as transaction processing may be slowed down. Furthermore, a high generation rate can cause the WAL memory to overflow so that the database can no longer record and there is a risk of system failure.
Locks and deadlocks
A lock is a mechanism with which the database prevents simultaneous data access and modification by multiple users. A deadlock occurs when several transactions lock resources that are required by the other transaction.
If locks and deadlocks block the database, performance can suffer as transactions and queries are executed more slowly. If the locks and deadlocks increase, this can also have a negative impact on the scalability of the database. Negative side effects can also occur if transactions are not completed and have to be aborted due to deadlocks.
Query graph (Top n by total_time)
Long queries can lead to bottlenecks in the database, which have a negative impact on performance and lead to delayed data output, for example.
