« Contact Form in PHP | Main | VMSTAT part 2 »

Monitoring CPU usage with VMSTAT

I thought I'd do a little CPU monitoring to determine if our VPS is still meeting our needs. As with most things it was more complicated than I thought.

I noticed that MT 3.34 is supposed to support FastCGI which l think would leave MT memory resident to improve performance. But I was wondering if I needed this, just how busy is my server?

Plesk reports some simple load numbers in it statistics page. These are the load average numbers from uptime. But these numbers aren't easy to understand. From what I understand load average is actually an average queue length of running or runable processes. At load average of: 0.21, 0.11, 0.05 I'm thinking my VPS is not loaded at all.

I want to know if my CPU allocation is maxed out. Do I have an idle CPU or is it pretty much busy. I also want to build an extensive CPU usage history and generate some reports for average CPU usage plus peak periods and checking for runaway programs.

I ran some vmstat commands and noticed the CPU numbers generally indicate 95% or more idle, with that dropping a lot when I did something like rebuilding a MT blog. This seems to confirm the load average numbers that my VPS is lightly loaded.

I suspect my low CPU usage is due to the fact that our web pages are mostly static. I do notice that doing something like a blog rebuild takes time and does seem to max out CPU. But for now that seems to be ok. If we ever need to host a forum or other type of dynamic site or rebuild times get excessive then we'll have to upgrade our server.

But I still want to moniter what we have to confirm these suspicions. I built a couple of Perl programs. The first writes the output of the vmstat command to a MySQL table.

#!/usr/bin/perl
use DBI;
use strict;
my $user = 'some_user';
my $password = 'some_pw';
my $dsn = 'DBI:mysql:cpu_stat:localhost';
my $dbh = DBI->connect($dsn, $user, $password,
                      { RaiseError => 1, AutoCommit => 0 })
   or die "Couldn't connect to database: " . DBI->errstr;
my $insert_cpu =
            $dbh->prepare_cached("INSERT INTO vm_stat VALUES (CURRENT_TIMESTAMP,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)");

my $vm_stat = `vmstat`;

$vm_stat =~ m/(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)$/;

$insert_cpu->execute($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16)
                        or die "Couldn't insert row into cpu_stat table: " . DBI->errstr;

$insert_cpu->finish();
$dbh->disconnect();

The second creates a little report of the busiest times of the day. I'll also do some monitoring mid-day to see what the server load is like during normal periods.

#!/usr/bin/perl
use DBI;
use strict;
my $user = 'some_user';
my $password = 'some_ps';
my $dsn = 'DBI:mysql:cpu_stat:localhost';
my $dbh = DBI->connect($dsn, $user, $password,
                      { RaiseError => 1, AutoCommit => 0 })
   or die "Couldn't connect to database: " . DBI->errstr;
my $select_cpu =
 $dbh->prepare_cached('select dt, cpu_us, cpu_sy, cpu_id, cpu_wa from vm_stat where (dt + INTERVAL 1 day) > now() order by cpu_id');

$select_cpu->execute()
    or die "Couldn't select from cpu_stat table: " . DBI->errstr;

my $i = 1;
print "Date/time, cpu_us, cpu_sy, cpu_id, cpu_wa\n";
while ((my $row_ref = $select_cpu->fetchrow_hashref()) and ($i < 15))
{
    print "$row_ref->{dt}, $row_ref->{cpu_us}, $row_ref->{cpu_sy}, $row_ref->{cpu_id}, $row_ref->{cpu_wa}\n";
   $i = $i + 1;
}
$select_cpu->finish();
$dbh->disconnect();

I quickly realized I have a problem with vmstat. Although it seemed to work fine at first. After running it for a day I noticed that the CPU stats were not changing, even though I ran MT rebuilds to force some extra activity. The other stats like memory used were changing, just not the CPU stats.

Another issue is a top 15 report isn't too useful. I'll probably convert the daily stats into a daily graph. Much better at showing average usuage over time as well as any peaks.

I decided to reboot and see what happened. Sure enough after the reboot vmstat started reporting increased CPU usuage, especially as the reboot was still going on. I did this just before midnight and by 4am the CPU numbers once again were frozen.

I'm going to leave the CPU moniter program running just to see if it gets unstuck. If not I'll have to do some digging to figure out what's going on.

After another day, it was still stuck. But I think I figured something out. If I run vmstat from the command line I almost always get the following:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0 344528  35192 108012 365760    0    0     0     0    0   150  2  1 96  0

With the CPU stats almost always 2 1 96 0. But if I run vmstat 1 60 to have vmstat report at 1 second intervals for 60 repetitions I get:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0 344528  31336 107924 365440    0    0     0     0    0   151  3  6 91  0
 0  0 344528  31720 107924 365440    0    0     0     0    0   210  1  0 99  0
 0  0 344528  31720 107924 365440    0    0     0     0    0   139  1  0 99  0
 0  0 344528  32296 107924 365440    0    0     0     0    0   119 10  9 81  0
 0  1 344528  39632 107924 365440    0    0     0     0    0   176  4  6 84  6
 0  0 344528  39568 107932 365432    0    0     0     0    0   254  4  7 70 19
 0  0 344528  39568 107936 365428    0    0     0     0    0   272  4  6 90  0
 0  0 344528  39312 107936 365496    0    0     0     0    0   317  2  0 98  0

Clearly vmtstat when run one time is just reporting CPU usage for itself. But if left running it will show useful CPU stats. I'll do a Part 2 post showing the revised code.


© 2016 Mike Silversides

About

This page contains a single entry from the blog posted on March 13, 2007 3:58 PM.

The previous post in this blog was Contact Form in PHP.

The next post in this blog is VMSTAT part 2.

Many more can be found on the main index page or by looking through the archives.