The Wondrous Guide To Understanding mysqlreport

This wondrous guide to understanding mysqlreport is comprehensive again. The current version of mysqlreport (v3.0a) can generate a large, comprehensive report that covers practically every applicable MySQL status value, or a short, quick report that covers only the most important status values. The comprehensive report includes all 14 report sections for a wondrous total of 121 lines. The quick report includes 6 report sections for a total of 29 lines of no less importance.

This guide to understanding mysqlreport explains everything mysqlreport can report in a comprehensive report. It also teaches you how to interpret and understand all the reports in context so that after reading a mysqlreport report ("a report") you can answer the fundamental question that mysqlreport was created to help answer: "How well is the MySQL server running?"

To facilitate better understanding and insight, this guide is written as a walk-through of a comprehensive report. We'll start with the very first line and examine and consider every report section and line to the end. When we're finished, you should be completely equipped with the know-how to mysqlreport any server and, from the report, say how well or not the server is running.

To begin, here is the report for our example server. This comprehensive report was generated with the command line option --all. Line numbers are added here to make examining the report easier. Most line numbers are links that will take you directly to the explanation of that line.
   1 MySQL 5.0.3              uptime 0 0:34:26       Fri Sep  1 19:46:02 2006
   2
   3 __ Key _________________________________________________________________
   4 Buffer used   380.00k of 512.00M  %Used:   0.07
   5   Current      59.32M            %Usage:  11.59
   6 Write ratio      0.93
   7 Read ratio       0.00
   8 
   9 __ Questions ___________________________________________________________
  10 Total          98.06k   47.46/s
  11   DMS          81.23k   39.32/s  %Total:  82.84
  12   QC Hits      16.58k    8.02/s           16.91
  13   COM_QUIT        200    0.10/s            0.20
  14   Com_            131    0.06/s            0.13
  15   -Unknown         82    0.04/s            0.08
  16 Slow                0    0.00/s            0.00  %DMS:   0.00
  17 DMS            81.23k   39.32/s           82.84
  18   SELECT       64.44k   31.19/s           65.72         79.33
  19   INSERT       16.75k    8.11/s           17.08         20.61
  20   UPDATE           41    0.02/s            0.04          0.05
  21   REPLACE           0    0.00/s            0.00          0.00
  22   DELETE            0    0.00/s            0.00          0.00
  23 Com_              131    0.06/s            0.13
  24   change_db       119    0.06/s            0.12
  25   show_fields       9    0.00/s            0.01
  26   show_status       2    0.00/s            0.00
  27 
  28 __ SELECT and Sort _____________________________________________________
  29 Scan               38    0.02/s %SELECT:   0.06
  30 Range              14    0.01/s            0.02
  31 Full join           3    0.00/s            0.00
  32 Range check         0    0.00/s            0.00
  33 Full rng join       0    0.00/s            0.00
  34 Sort scan          14    0.01/s
  35 Sort range         26    0.01/s
  36 Sort mrg pass       0    0.00/s
  37
  38 __ Query Cache _________________________________________________________
  39 Memory usage   17.81M of  32.00M  %Used:  55.66
  40 Block Fragmnt  13.05%
  41 Hits           16.58k    8.02/s
  42 Inserts        48.50k   23.48/s
  43 Prunes         33.46k   16.20/s
  44 Insrt:Prune    1.45:1    7.28/s
  45 Hit:Insert     0.34:1
  46
  47 __ Table Locks _________________________________________________________
  48 Waited          1.01k    0.49/s  %Total:   1.24
  49 Immediate      80.04k   38.74/s
  50
  51 __ Tables ______________________________________________________________
  52 Open              107 of 1024    %Cache:  10.45
  53 Opened            118    0.06/s
  54
  55 __ Connections _________________________________________________________
  56 Max used           77 of  600      %Max:  12.83
  57 Total             202    0.10/s
  58
  59 __ Created Temp ________________________________________________________
  60 Disk table         10    0.00/s
  61 Table              26    0.01/s
  62 File                3    0.00/s
  63
  64 __ Threads _____________________________________________________________
  65 Running            55 of   77
  66 Cache               0              %Hit:    0.5
  67 Created           201    0.10/s
  68 Slow                0    0.00/s
  69
  70 __ Aborted _____________________________________________________________
  71 Clients             0    0.00/s
  72 Connects            8    0.00/s
  73
  74 __ Bytes _______________________________________________________________
  75 Sent           38.46M  18.62k/s
  76 Received        7.98M   3.86k/s
  77
  78 __ InnoDB Buffer Pool __________________________________________________
  79 Usage           3.95M of   7.00M  %Used:  56.47
  80 Read ratio      0.000
  81 Pages
  82   Free            195            %Total:  43.53
  83   Data            249                     55.58 %Drty:   0.00
  84   Misc              4                      0.89
  85   Latched           0                      0.00
  86 Reads         574.56k     0.6/s
  87   From file       176     0.0/s            0.03
  88   Ahead Rnd         4     0.0/s
  89   Ahead Sql         2     0.0/s
  90 Writes        160.82k     0.2/s
  91 Flushes         1.04k     0.0/s
  92 Wait Free           0       0/s
  93
  94 __ InnoDB Lock _________________________________________________________
  95 Waits               0       0/s
  96 Current             0
  97 Time acquiring
  98   Total             0 ms
  99   Average           0 ms
  100  Max               0 ms
  101
  102 __ InnoDB Data, Pages, Rows ____________________________________________
  103 Data
  104   Reads           225     0.0/s
  105   Writes          799     0.0/s
  106   fsync           541     0.0/s
  107   Pending
  108     Reads           0
  109     Writes          0
  110     fsync           0
  111
  112 Pages
  113   Created          23     0.0/s
  114   Read            226     0.0/s
  115   Written       1.04k     0.0/s
  116
  117 Rows
  118   Deleted      25.04k     0.0/s
  119   Inserted     25.04k     0.0/s
  120   Read         81.91k     0.1/s
  121   Updated           0       0/s

Report Header: Line 1

The first line of a report has three bits of information you may already know: the version of the MySQL server, its uptime, and the server's current date and time. Some people run mysqlreport at regular intervals (with cron), so the report header is a way to distinguish one report from another. For people who use mysqlreport on servers that are not theirs (consultants, datacenter administrators, etc.), the report header tells them what they're up against. The MySQL server version indicates what features the server does or doesn't have, and its uptime indicates how representative the values in the report will be. This later point (uptime) is important for assessing the report because the values tend to be skewed and misleading if the server has not be running for at least an hour. Even an hour may not be enough if, for example, the server has only been running for six hours in the middle of the night with almost no usage. Ideally, you want the server to have been up for at least one whole day. That way, you know the values in the report reflect the slow and busy times for the server, and not just one or the other. In this example, the server has only been up for 34 minutes so the report is not as ideally representative as we would like, but it's good enough for now.

Key Report: Lines 3 - 7

The first major report section of any mysqlreport report is the Key report because keys (indexes) are the most important things. Although the report cannot tell you if the server is well indexed or not, it can tell you generally how the shared key buffer is being utilized. Note that this report only applies to the default, shared key buffer for MyISAM tables. InnoDB tables use a different key buffer that mysqlreport currently doesn't look at. Nor does it currently look at other key buffers (created by the administrator, like hot and cold key buffers).

Buffer used: Line 4

The first question one usually asks of a MySQL server is: "How much of the key buffer is being used?" If it's not being used much, that's okay because MySQL only allocate system RAM for the key buffer when needed. That is, if your key buffer size is set to 512M, MySQL doesn't not automatically allocate 512M of system RAM when it starts. MySQL allocates up to 512M of system RAM when it needs to. The fourth line, Buffer used, is supposed to indicate the maximum amount of key buffer MySQL has ever used at once. Hence, the past tense usage of the word "used" because MySQL may currently being using less or, strangely, more. MySQL calls this a "high water mark," but we'll see in the next line that this is not always so. Regardless, this line is usually indicative of whether or not the key_buffer_size system variable is sufficiently large. If this line indicates that MySQL has used upwards of 80% to 90% of the key buffer, you may want to increase key_buffer_size. Note, however, that this line will probably never indicate above 95% used because, as the MySQL documentation states, some of the shared key buffer is used for internal data structures, which mysqlreport cannot account for. Therefore, 95% used is practically 100% used. In this example, the server has used 380k of 512M, or 0.07%, so the key buffer is plenty large. However, regard the next line...

Current: Line 5

This line is new in mysqlreport v2.5 thanks to a user who suggested mysqlreport use the Key_blocks_unused status value to determine the current, actual key buffer usage of MySQL, not the "high water mark." This line only appears in MySQL servers version 4.1.2+ because Key_blocks_unused was added in version 4.1.2. This line indicates how much key buffer MySQL is using right now. If the previous line is actually a high water mark, then this line should always be less than or equal to it, but as we can see in this example, that's not always the case. Currently, this server is using about 60M of the key buffer (12%), which is good because it's nowhere near full capacity. It does make one wonder, though, how MySQL can currently be using more key buffer than it supposedly has ever used before. This is a discrepancy that will hopefully be figured out later. For now, this line in combination with the preceding line gives a good indication of whether or not key_buffer_size is set sufficiently large.

Write ratio: Line 6

Indexes (keys) are inherently RAM-based. Their usefulness is due in part to the fact that they exist in RAM which is very quick to access instead of existing only on a hard disk which is very slow to access. However, MySQL inevitably must write and read indexes to and from a hard disk at times. This line, Write ratio, indicates the ratio of key writes to hard disk to key writes to RAM. It's not uncommon to have a key write ratio near 1.0. As the MySQL manual says, if your database usage is primarily updates, inserts, etc. the write ratio will be near 1.0. A key write ratio over 1.0 indicates that MySQL is more often writing keys to hard disk than RAM, which can be a bad, slow thing. The highest key write ratio I can remember seeing is 1.15; most I see are below 1.0 even on really busy servers.

Read ratio: Line 7

More important than the key write ratio is the key read ratio. This line indicates the ratio of key reads from hard disk to key reads from RAM. Key read ratio should be 0.00 or 0.01. Any more indicates a problem, usually that the key buffer is too small which keeps MySQL from loading more indexes into RAM and so it has to revert to reading them from the hard disk instead, which is terribly slow and completely negates the point of an index. It is common, however, for this value to be over 0.01 within the first hour or so of starting MySQL. After an hour, it should definitely be down to 0.01 or zero.

Questions Report: Lines 9 - 26

The second major report section, Questions, is the second most important because it tells you a lot about what MySQL is busy doing, and how busy it's doing all that it is. Questions includes SQL queries and MySQL protocol communications. Most people are concerned about how many queries per second the server is doing, but this is actually a very arbitrary number when considered out of a larger context. The larger context is all the other questions MySQL is handling. This report provides the whole context.

Total: Line 10

The first line is simple the total number of all questions MySQL has answered (first column) and the rate of those answers over its update (second column). This rate is what most people mean when they say things like, "My server averages one hundred queries a second." You should ask them: "And of those one hundred, how many are really getting something done?" mysqlreport can answer this in the following lines...

Distribution of Total Queries (DTQ): Lines 11 - 15

All questions can largely be divided into five categories: Data Manipulation Statements (DMS), query cache hits (QC Hits), COM_QUIT, all other Com_ commands, and the lovely Unknown. These five categories are indicated in the five lines 11 through 15 but their order is dynamic: mysqlreport sorts them in descending order based on total number (first column). Therefore, by looking at this sub-report (which is invoked by the --dtq command line option) you can quickly tell what MySQL is most busy doing. Ideally, you want MySQL to be most busy with DMS or QC Hits because these are the categories of questions that are really getting things done. COM_QUIT, Com_, and Unknown are necessary but should play only a minor role. Before explaining each category further, I'll mention that the third column for this and other sub-reports in the Questions report shows the percentage of that line's total value to all questions (Total, line 10). In this example, DMS questions account for 82.84% of all questions that the MySQL server handles, which is a really good percentage.

Data manipulation statements include: SELECT, INSERT, REPLACE, UPDATE, and DELETE. (Technically, there are others but mysqlreport uses only these.) Basically, DMS is what you think of when you think of MySQL doing something useful. Hence, you want DMS to be what MySQL is most busy doing. This category is expanded in more detail in the DMS sub-report later, lines 17 through 22.

QC Hits is self-evident: it is the number of queries MySQL answered by pulling the answer from the query cache instead of having to actually execute the query. Having a high percentage of QC Hits is coveted because serving answers from the QC is very fast. However, my experience has been that most servers don't have a very effective QC cache for various reasons. In this example, QC Hits account for 16.91% of all questions, which is pretty good. However, don't be mislead by this: the Query Cache report (lines 38 through 45) can tell a very different story. This is an example of the kind of deep, cross-comparative analysis mysqlreport can help you do. Whereas QC Hits here seems pretty good, this server's query cache isn't all that spectacular, as we'll see later.

COM_QUIT is a category which I have written another article about: COM_QUIT and Questions. It's not a very important category; therefore, I'll leave it to you to read that article if you wish to understand this line. Otherwise, don't be concerned with it. It's mostly there to geek out about.

The category Com_ represents all the various commands MySQL handles, usually protocol related. Ideally, you want this category to be low because when it's high it's like MySQL is spinning it wheels really fast but getting nowhere. A high value for this category can indicate some weird problems, as we'll discuss later when we discuss the Com_ sub-report that expands on this in more detail (lines 23 through 26, usually).

Unknown is an inferred category. Ideally, the preceding four categories should add up to equal total questions, but they usually do not. This is because there are a few questions that MySQL handles and increments the total questions counter for but does not otherwise maintain a separate status value for. This line is dynamic in that it can read "+Unknown" or "-Unknown." +Unknown means there's more total questions than mysqlreport can account for; -Unknown means mysqlreport counted questions than total questions. This category varies greatly: on some servers it's near the top, but on most it's at the very bottom. We want it to be at the very bottom. If it's at the very top, I would say something is wrong, but since these are "unknown" questions, I can't say what might be wrong. Eventually (hopefully), I'll go through the MySQL source code again and discover the nature of these unknowns.

Slow: Line 16

Line 16 is very important: it indicates the number of slow queries MySQL has executed. What constitutes "slow" is set by the system variable long_query_time, which by default is 10 (seconds). I usually set this to 5. Ideally, we want zero slow queries, but usually we have a few. Generally, Slow as a percentage of total questions (third column) should be 0.05 or less. There can be a lot of slow queries (first column), but it's the percentage of them all compared to total that indicates a problem. This line also adds a fourth column: percentage of DMS questions. For Slow, zero is best, but this column is more useful later in the DMS sub-report.

DMS: Lines 17 - 22

The DMS sub-report, like the DTQ sub-report, is sorted in descending order of value (first column). This sub-report is invoked by the --dms command line option. Its 6 lines, 17 through 22, represent the data manipulation statements mentioned earlier (SELECT, INSERT, etc.), and the first line (17) is the total of all these again (identical to line 11 in the DTQ sub-report). This sub-report tells us, in a sense, what "kind" of MySQL server this is: is it SELECT heavy, or INSERT heavy, etc. Most MySQL servers, it seems, are SELECT heavy. Know what kind of MySQL server a server is help orient our thoughts and understanding about other the other values. For example, an INSERT heavy server should have a write ratio very near 1.0, and it will probably have high values regarding table locks. It would also be a candidate for InnoDB tables. A SELECT heavy server had better have a read ratio of zero and very low table lock values. It also stands in relation to what the Query Cache report can tell us. In this example, the server is SELECT heavy: 65.72% of all questions are SELECTs (third column), and 79.33% of all DMS questions are SELECTs (fourth column). Clearly, this server is oriented towards SELECT statements above everything else. Knowing that shapes how one approaches all aspects of optimization.

Com_: Lines 23 - (26)

The Com_ sub-report like other sub-report so-far is sorted and invoked by a command line option, --com. The contents of this sub-report vary from server to server because each line (default 3; more can be specified like --com 10) represents some Com_ status value which in turn most often represent some COM_ command in the MySQL protocol. Most of the names are intuitive like, Com_change_db. This sub-report matters when Com_ in the DTQ sub-report is near the top because it indicates MySQL is busy doing "program things" instead of answering SQL queries. In one case I saw a server that had very high numbers of Com_rollback. If you use transactions you know that a rollback occurs when a transaction fails and this is usually not a good thing. That server was failing nearly every transaction so clearly something was very wrong. Without mysqlreport, the DTQ sub-report, and this sub-report it was practically impossible to otherwise tell that that server had any problem. For most servers, the Com_ sub-report indicates nothing weird, but it's good to check it from time to time.

SELECT and Sort Report: Lines 28 - 36

The SELECT and Sort Report is invoked by the command line option --sas. It details the various Select_ status value which are described in another article: MySQL Select and Sort Status Variables. I will leave it to that article to explain the details of all these values. What concerns us most here are lines 29 and 31: Scan and Full join. Scan indicates how many SELECT statements resulted in a full table scan, which is a slow process. Full join is like Scan except that it applies to tables being joined in a multi-table query. Such tables are joined by process of a full table scan, but in the context of a join, a table scan is even slower. Therefore, ideally you want these two values to be as low as possible, but there is no real standard for "low" here. Some servers which are running really well have a relatively high percentage of Scan to all SELECT statements (third column).

Query Cache Report: Lines 38 - 45

The Query Cache report is invoked with the --qcache command line option, but it only appears if, one, the MySQL version supports query cache and, two, query cache is enabled.

Memory usage: Line 39

This first line of this report is self-evident: it indicates how much of the query cache memory is being used. If it's at max capacity, this will probably also be reflected in the Prunes value below since queries in the QC are pruned when memory is low.

Block Fragmnt: Line 40

Line 40, Block Fragmnt (Fragmentation), indicates a condition particular to the way the query cache functions. I quote from the MySQL manual section 5.14.3. Query Cache Configuration:
The default value of query_cache_min_res_unit is 4KB. This should be adequate for most cases. ... If you have a lot of queries with small results, the default block size may lead to memory fragmentation, as indicated by a large number of free blocks. Fragmentation can force the query cache to prune (delete) queries from the cache due to lack of memory. In this case, you should decrease the value of query_cache_min_res_unit. The number of free blocks and queries removed due to pruning are given by the values of the Qcache_free_blocks and Qcache_lowmem_prunes status variables.
This value is a percentage of free QC blocks to total blocks. The higher the percentage, the more the QC memory is fragmented. 10% to 20% seems to be what I usually see on servers. In this example, block fragmentation is at 13.05%. I would call that acceptable, but I would also play around with query_cache_min_res_unit to see if I could get it lower.

Hits, Inserts, Prunes: Lines 41 - 43

Query cache Hits, Inserts, and Prunes are indicated on lines 41, 42, and 43. Hits is the most important because it indicates how many SELECT statements were served from the cache, so the more the better. Inserts and Prunes are better understood in terms of the ratio on line 44. Although, as mentioned earlier, a high rate of Prunes can be indicative of the QC size being too small, but not always. In this example, only 55% of the QC is in use, with relatively low fragmentation, yet prunes are pretty high; prunes are occurring at the rate of 16/s, double the rate of QC hits. In a sense, this server's QC is like an apple tree where the limbs are being cut off faster than the apples are being picked.

Insrt:Prune and Hit:Insert Ratios: Lines 44 - 45

The QC Insert:Prune ratio on line 44 is an indicator of QC volatility. In a highly stable QC, more queries will be inserted than are pruned. In a volatile QC, this ratio will be one-to-one or heavy on the prune side, indicating a kind of evacuation of queries from the QC. We want a stable QC because a stable QC implies that the cached queries are being used often. A volatile QC can indicate two things: either one, the QC size is too small so MySQL has to keep pruning and inserting queries, or two, MySQL is trying to cache everything to a self-defeating end. In the first case, simply increase the QC size may help. This type of volatility may be further indicated by high block fragmentation and QC memory usage. The second type of volatility, I think, is more common because MySQL does try to cache nearly everything it can when the QC is enabled with the default type 1. Type 1 means (quoting the manual): "Cache all query results except for those that begin with SELECT SQL_NO_CACHE." I rarely if ever see people use SQL_NO_CACHE. A better way to enable the QC is with type 2 "DEMAND": "Cache results only for queries that begin with SELECT SQL_CACHE." Demand caching requires more work on your part, because you have to explicitly add SQL_CACHE to the queries you want MySQL to cache, but the advantage is that you probably know what queries are good, stable cache candidates. The other ratio is Hit:Insert. This ratio indicates QC effectiveness. Ideally, we want to insert a bunch of stable queries into the QC, then get a lot more hits on them. Therefore, this ratio should be heavy on the hit side if the QC is effective. If it is heavy on the insert side, then the QC isn't really helping much and it's probably too volatile. Consider a Hit:Insert ratio of 1:1. This practically means that a cached result is only used once before it's replaced. This completely defeats the idea of a query cache though. A worse ratio, like in this example of 0.34:1, indicates that some results aren't even hit before they're pruned or replaced. In other words, we're putting more into the QC than we're getting out of it. We are wasting effort with the QC. So, as mentioned much earlier in the DTQ sub-report, even though QC Hits account for a good percentage of total questions, our actually QC effectiveness is really low as indicated by the Hit:Insert ratio being terribly Insert heavy. I think this server would benefit from demand caching since QC memory usage and fragmentation is not bad. Chances are, MySQL is just defeating itself trying to cache everything.

Table Locks Report: Lines 47 - 49

The Table Locks report consists of two lines: the first, Waited, shows the number of table locks that MySQL had to wait to obtain, and the second, Immediate, shows the number of table locks MySQL obtained immediately. Waiting is almost always a bad thing in database terms, therefore, table locks waited should be as low as possible, relative to your server. What's most indicative of table locking problems is third column of table locks waited: %Total of all table locks. The percentage of table locks that had to wait should be 10% or less. Higher percentage can indicate poor table/query indexing or slow queries.

Tables Report: Lines 51 - 53

The Tables report is also two lines: the first, Open, shows how many tables are open right now (first column), of how many total possible (table cache; second column), and the percentage of table cache usage (third column). The second line, Opened, shows the total number of tables MySQL has ever opened and this value over its uptime (second column). Two things concern us here: first is the table cache usage. It's not a bad thing to have 100% table cache usage, but if you're getting close to 100% you may benefit from increasing the table_cache system variable. Second, the rate of opening tables can also help determine if table_cache is too low. Generally, it's nice to have this value less than 1/s. However, I have seen a busy and well running server that was opening 7 tables/s and running 100% table cache.

Connections Report: Lines 55 - 57

Another two line report, the Connection report is practically identical to the Tables report and so I won't explain it again. Obviously, if the max number of connections used is approaching 100% (first line, third column), you might want to increase the max_connections system variable. However, this is often misleading. I constantly see servers with very high max_connection for no good reason. The default value is 100 and this works for even extremely busy, well-optimized servers. A connection to MySQL should last a fraction of a second, so even 100 connections goes a long way. If max connections on your server is very high or slow crawls up over time, the problem might be elsewhere, like slow queries, poor indexing, or even slow DNS resolution. Before setting max_connections above 100, I would discover the fundamental reason why 100 connections at once is not enough to serve your MySQL needs and verify that it's a legitimate need for an insanely busy server and not another problem that manifests itself as too few connections. Regarding the number of connections per second, this value can be rather high. In fact, if it is high and everything else is working well, it's usually a good indication. One server I worked on was doing 7 connections/s and 154 DMS/s. However, most server's connections/s are well under 5/s.

Created Temp Report: Lines 59 - 62

MySQL can create temporary tables on hard disk, in RAM, and temporary files. Each of these three corresponds to the three lines of the Created Temp report. These value are mostly relative; there is no standard for them. Since temporary tables on hard disk are the slowest (indicated by the first line, Disk table), it's best if this value is the lowest of the three. A temporary table is created on hard disk only if it can't fit into a temporary table in RAM which is limited by the system variable tmp_table_size. Temporary tables in RAM (indicated by the second line, Table) and temporary files are so common that these value are relative to your server.

Threads, Aborted, Bytes Reports: Lines 64 - 76

The final three reports, Threads, Aborted, and Bytes, are altogether invoked by the --tab or --all command line options. These reports are the least important. Therefore, I will not bore you with all their details as they are mostly self-evident. There is, however, one line of particular interest: line 66 of the Threads report, Cache and specifically %Hit. This was added in mysqlreport v2.3 at the request of a user, and it was a good request because the cache hit rate is something to regard. If you were unaware, every connection to MySQL is handled by a separate thread. At startup, MySQL creates a few threads and keeps a few in a thread cache so that it doesn't have to constantly keep killing and creating threads. Although threads aren't expensive to make, it's nonetheless uncivilized to "thread thrash." When the number of connections/s to MySQL exceeds the thread cache (set by the system variable thread_cache_size) MySQL starts to thread thrash: it goes crazy creating threads to keep up with the demand for new connections. When this happens, the thread cache hit rate drops. In this example, it is a very poor 0.05% which means nearly every new connection causes MySQL to create a new thread. It's easy to see why: the first column of the same line (66) says there are zero threads left in the cache. Therefore, thread_cache_size should be increased. Also notice the correlation between line 67, threads created, and the earlier line 57, total connections: 201 threads created, 202 total connections. Hence, the near-zero thread cache hit rate. Does this matter? Jeremy Zawondy once blogged:
So the moral of the story is this: If you have a busy server that's getting a lot of quick connections, set your thread cache high enough that the Threads_created value in SHOW STATUS stops increasing. Your CPU will thank you. ... Thread caching really wasn't the worst of our problems. But it became the worst after we had fixed all the bigger ones.
So there you have it. Don't thread thrash.

InnoDB Buffer Pool Report: Lines 78 - 92

The InnoDB reports that follow were added in mysqlreport v3.0. The Innodb_ status values (in SHOW STATUS;) are only available in MySQL v5.0.2 and later. Therefore, mysqlreport may not show these InnoDB reports even if the MySQL is running the InnoDB storage engine. In short: mysqlreport's InnoDB reports only work with MySQL v5.0.2 and later.

A central feature of the InnoDB storage engine is the buffer pool in which InnoDB caches table data and indexes. Internally, the buffer pool is composed of 16Kb pages which contain different types of data. The InnoDB Buffer Pool Report contains values pertaining to the pages in the buffer pool.

NOTE: I have not seen enough mysqlreports from MySQL servers that are both version 5.0.2 or newer and relying heavily on InnoDB. Therefore, the guide covering the InnoDB reports may seem less thorough than the guide covering the previous reports. Although MySQL v5 has been the GA release for awhile now, it is amazing how common v4.0 and v4.1 still are.

Usage: Line 79

Line 79, InnoDB Buffer Pool Usage is similar to line 4, Key Buffer used. However, the MyISAM engine stores only indexes in its key buffer (hence the name), but the InnoDB engine stores indexes and other data in the buffer pool. Therefore, this line shows how much of the buffer pool is being used but it does not show what accounts for the usage. To get an idea of what accounts for the buffer pool usage, one must look at lines 81 through 85.

Obviously, you want to avoid running out of buffer pool space. With the MyISAM engine, running out of key buffer space may only cause performance problems (because of the adverse effect on table indexes). With the InnoDB engine, running out of buffer pool space can cause many more problems because nearly everything relies on the buffer pool. It is possible to configure an auto-extending buffer pool.

Read ratio: Line 80

The InnoDB Buffer Pool Read ratio is very similar to Key Read ratio on line 7. However, given again that InnoDB stores more than just indexes in the buffer pool, this value is more general then Key Read ratio.

InnoDB Buffer Pool Read ratio is the ratio of buffer pool page reads from hard-disk to buffer pool reads from RAM. Therefore, this ratio should be very small. In the limited number of cases that I have seen, this ratio was zero. It is difficult to say what a large Read ratio might be indicative of because this Read ratio applies to different types of pages. I have seen cases where the Read ratio was 0.1 but the buffer pool usage was very low. Perhaps such a high Read ratio but low buffer pool usage reflects the initial page reads at startup to load the buffer pool, but then the engine was not used.

Pages: Lines 81 - 85

These lines are a very broad look into the substance of the buffer pool. Each line (82 - 85) corresponds to a different kind of buffer pool page: free pages (line 82), data pages (83), miscellaneous pages (84), and "latched" pages (85).

Free pages are self-describing. The far-right column, "%Total:", says what percentage of all buffer pool pages are free (or data, misc, and latched correspondingly). This line is the opposite of line 79: instead of saying how much of the buffer pool is used (line 79), this line says how much of the buffer pool is free.

Data pages are also self-describing. Currently there is no way to know the kinds of data that these pages comprise. This line has one extra column: %Drty (%Dirty). This column says what percentage of data pages have been modified (are dirty) but have not been flushed/saved back to hard-disk.

Not much can be said about the remaining two kinds of pages: misc and latched. Regarding miscellaneous pages, the MySQL manual simply says: "The number of pages that are busy because they have been allocated for administrative overhead such as row locks or the adaptive hash index." And regarding latched pages: "These are pages currently being read or written or that cannot be flushed or removed for some other reason."

Reads: Lines 86 - 89

The following four lines give us some idea of InnoDB's buffer pool reading activity. The first line, line 86, is simply a metric of the number of buffer pool reads from RAM. On busy servers using InnoDB, this value should be high because InnoDB should be reading most of its pages from the buffer pool. This metric can been considered a measure of InnoDB buffer pool throughput. Since almost everything InnoDB needs is kept in and retrieved from its buffer pool, you want buffer pool reads to be as fast as possible. For example, an InnoDB Buffer Pool Reads rate (second column) of over 200k/s is not impossible.

Line 87, however, you want to be much smaller in value. Line 87 lists "The number of logical reads that InnoDB could not satisfy from the buffer pool and had to do a single-page read." In other words, how many hard-disk reads.

Line 88, Ahead Rnd (Random), lists "The number of random read-aheads initiated by InnoDB. This happens when a query scans a large portion of a table but in random order."

Line 89, Ahead Sql (Sequential), lists "The number of sequential read-aheads initiated by InnoDB. This happens when InnoDB does a sequential full table scan." As always, full table scans are usually a bad thing and should be minimized.

Writes: Line 90

Like line 86, InnoDB Buffer Pool Writes can been considered a measure of InnoDB buffer pool throughput. This line lists the number and rate of writes to the buffer pool. This value will probably be high if the server does a lot of UPDATEs or INSERTs.

Flushes: Line 91

This line is simply the number of buffer pool page-flush requests.

Wait Free: Line 92

Quoting from the MySQL manual regarding this status value:
Normally, writes to the InnoDB buffer pool happen in the background. However, if it is necessary to read or create a page and no clean pages are available, it is also necessary to wait for pages to be flushed first. This counter counts instances of these waits. If the buffer pool size has been set properly, this value should be small.

InnoDB Lock Report: Lines 94 - 100

InnoDB row lock status values were added in MySQL 5.0.3. MyISAM is a table-level locking storage engine, but InnoDB is a row-level locking store engine. Therefore, these values are important to consider when using the InnoDB engine.

Waits: Line 95

This line shows "The number of times a row lock had to be waited for." Zero is best.

Current: Line 96

This line shows "The number of row locks currently being waited for." Zero is best.

Time acquiring: Lines 97 - 100

Lines 98, 99, and 100 show millisecond times corresponding to the Total number of milliseconds that row locks had to wait (line 98), the Average wait time (99), and the Maximum wait time (100). For all three metrics, zero is best.

InnoDB Data, Pages, Rows Report: Lines 102 - 121

The last InnoDB report, Data, Pages, Rows, is optionally invoked by either the --dpr or --all command line options. The InnoDB Data, Pages, Rows Report is optional because its values are very general and, as such, are probably only useful as general indicators of InnoDB engine throughput. The three sections, Data, Pages, and Rows, give you a very broad look into InnoDB's activity.

Data: Lines 103 - 110

The first section, Data, lists four categories corresponding to InnoDB data: Reads, Writes, fsync, Pending. The first category, Reads, refers to the total number of data reads done by the InnoDB engine. This does not mean the total number of data bytes read. It only means how many times InnoDB has read data; it does not indicate what kind of data or how much data was read.

The second category, Writes, is just like Reads: it refers to the total number of data writes done by InnoDB, but it does not indicate what kind of data or how much data was written.

The third category, fsync, refers to the total number of file system syncs. In other words: how many times InnoDB has saved data from RAM back to hard-disk. This value will tend to be lower than Reads or Writes.

Pending, the final category, is further divided into 3 lines (lines 108, 109, 110): Reads, Writes, fsync. Correspondingly, these lines refer to the current number of Data Reads, Writes, and fsyncs that are pending (waiting). Zero is best.

Pages: Lines 112 - 115

The Pages section has three categories: Created, Read, Written. Each category is self-describing and all three refer to pages in the InnoDB buffer pool. These values allow you to see the number and rate at which pages in the buffer pool have been created, read, and written. However, none of the values indicate what kind of pages. Therefore, these values are also only useful as general indicators of InnoDB engine throughput.

Rows: Lines 117 - 121

The final section, Rows, is last because it is the most general. Each of the four categories in this section (Deleted, Inserted, Read, Updated) refer to rows in InnoDB tables. Therefore, these values tend to be very large and, while their counts (first column) may indicate little, their rates (second column) are another indicator of InnoDB engine throughput.

Conclusion

Now that we've read the entire mysqlreport report and considered it all, we can make a general assessment of this example server.

In general, the server is running very well according to a number of big indicators: key buffer usage is only at 12%, key ratios are good, DMS and QC Hits account for over 99% of all questions, no weird Com_ problems, table locks are good, table cache is only at 10% usage, and relatively low and slow number of connections.

Concerning the InnoDB engine, it appears to be in use, but not heavily. As best as the InnoDB status values can inform us, nothing is out of the ordinary in this case.

Things we could work on include, first and foremost, the query cache because it's too volatile, and secondly we must set thread_cache_size higher until the thread cache hit rate comes back up.

That's all there is too it. If you have further questions you can ask me via the feedback form. And if you didn't notice, there are a number of other example mysqlreport reports on the mysqlreport web page. Although the example reports are from varying older versions of mysqlreport, the format is still similar.

(Doc rev: Apr 25 2007)