I wrote a few munin plugins to monitor memory, CPU and IO usage per process. After collecting the stats for a few days I noticed that MySQL CPU usage had a constant value of 9.1, however all other processes collected CPU usage correctly..

CPU usage

proc=mysqld
value=$(ps u -C $proc | awk 'BEGIN { sum = 0 } NR > 1 { sum += $3 }; END { print sum }')

Memory usage – VSZ – Virtual Memory

proc=mysqld
value=$(ps u -C $proc | awk 'BEGIN { sum = 0 } NR > 1 { sum += $6 }; END { print sum * 1024 }')

The strange part was that top displayed varying CPU usage for the process and did not match what “ps aux” states.

db1vivo:/etc/munin/plugins# top -n1 | grep mysql | awk '{print $10}'
97
db1vivo:/etc/munin/plugins# top -n1 | grep mysql | awk '{print $10}'
83
db1vivo:/etc/munin/plugins# top -n1 | grep mysql | awk '{print $10}'
33

db1vivo:/etc/munin/plugins# top -b -n1  | grep mysql
 2909 mysql     20   0 8601m 7.9g 6860 S   28 50.2  32843:58 mysqld
 2870 root      20   0 17316 1456 1160 S    0  0.0   0:00.02 mysqld_safe
30442 prodadmi  20   0 31120  376  228 R    0  0.0   0:00.00 mysql      

db1vivo:/etc/munin/plugins# ps fuax | grep mysql
root      2870  0.0  0.0  17316  1456 ?        S     2009   0:00 /bin/sh /usr/bin/mysqld_safe
mysql     2909  9.1 50.2 8807588 8275404 ?     Sl    2009 32842:49  _ /usr/sbin/mysqld --basedir=/usr --datadir=/mnt/data/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
root      2910  0.0  0.0   3780   596 ?        S     2009   0:00  _ logger -p daemon.err -t mysqld_safe -i -t mysqld

I have never actually read the man pages for “ps” and my assumption had always been that “ps aux” gave you current CPU utilization. However, reading the man pages for both “ps” and “top” it’s pretty clear what the CPU values are based on and the reason other processes varied were due to child processes.

PS

cpu utilization of the process in “##.#” format. Currently, it is the CPU time used divided by the time the process has been running (cputime/realtime ratio), expressed as a percentage. It will not add up to 100% unless you are lucky.

TOP

The task’s share of the elapsed CPU time since the last screen update, expressed as a percentage of total CPU time. In a true SMP environment, if ‘Irix mode’ is Off, top will operate in ‘Solaris mode’ where a task’s cpu usage will be divided by the total number of CPUs.

The munin plugin resulted in the following proc_cpu

#!/bin/sh
#
# (c) 2010, Andrew Johnstone andrew @ajohnstone.com
# Based on the 'proc_mem' plugin, written by Rodrigo Sieiro rsieiro @gmail.com
#
# Configure it by using the processes env var, i.e.:
#
# [proc_mem]
# env.processes        php mysqld apache2
#

. $MUNIN_LIBDIR/plugins/plugin.sh

if [ "$1" = "autoconf" ]; then
        echo yes
        exit 0
fi

processes=${processes:="php mysqld apache2"}

if [ "$1" = "config" ]; then

        NCPU=$(egrep '^cpu[0-9]+ ' /proc/stat | wc -l)
        PERCENT=$(($NCPU * 100))
        if [ "$scaleto100" = "yes" ]; then
                graphlimit=100
        else
                graphlimit=$PERCENT
        fi
        SYSWARNING=`expr $PERCENT '*' 30 / 100`
        SYSCRITICAL=`expr $PERCENT '*' 50 / 100`
        USRWARNING=`expr $PERCENT '*' 80 / 100`

        echo 'graph_title CPU usage by process'
        echo "graph_args --base 1000 -r --lower-limit 0 --upper-limit $graphlimit"
        echo 'graph_vlabel %'
        echo 'graph_category processes'
        echo 'graph_info This graph shows the cpu usage of several processes'

        for proc in $processes; do
                  echo "$proc.label $proc"
        done
        exit 0
fi

TMPFILE=`mktemp -t top.XXXXXXXXXX` && {

  top -b -n1 > $TMPFILE

  for proc in $processes; do
    value=$(cat $TMPFILE | grep $proc | awk 'BEGIN { SUM = 0 } { SUM += $9} END { print SUM }')
    echo "$proc.value $value"
  done
  rm -f $TMPFILE
}