How to limit web services so they don’t kill CPU

Featured Image © by groenmen/sxc

Hi! As any developer, I’ve encountered a small problem when working with PHP under Apache. It’s all good until you have to test the “real life” situations. For example – if you use imagemagick library in your application. It can get quite heavy on server load. So if you want to read a solution for it, read on.

First of all, what do I mean by “quite heavy”. Well, imagemagick as my example is great image manipulation tool, but it takes quite a lot of CPU power compared to gd library that PHP uses. And if you have a site that has something around 50 users at the same moment who use imagemagick in the same time, with poor server they’ll mots probably create a mini DDOS attack on you, non intentionally. A solution for Linux based servers is “CPU limit”.

About CPU limit

CPU limit is open source Linux based solution that gives you option to limit each process’s (by PID) CPU power consumption in percentage. So for example, you can always limit your apache to 2% of CPU, and MySQL to let’s say 10%. Its pretty much straightforward, so you can fit your needs.

Installation process on Ubuntu

There are few steps to complete this tasks, so here they are in right order:

First, 2 packages are needed, and you can install those with following commands:

sudo apt-get install cpulimit
sudo apt-get install gawk

Next step is to create Daemon for CPU limit:

#!/bin/bash
# ==============================================================
# CPU limit daemon - set PID's max. percentage CPU consumptions
# ==============================================================

# Variables
CPU_LIMIT=20       	# Maximum percentage CPU consumption by each PID
DAEMON_INTERVAL=3  	# Daemon check interval in seconds
BLACK_PROCESSES_LIST=   # Limit only processes defined in this variable. If variable is empty (default) all violating processes are limited.
WHITE_PROCESSES_LIST=   # Limit all processes except processes defined in this variable. If variable is empty (default) all violating processes are limited.

# Check if one of the variables BLACK_PROCESSES_LIST or WHITE_PROCESSES_LIST is defined.
if [[ -n "$BLACK_PROCESSES_LIST" &&  -n "$WHITE_PROCESSES_LIST" ]] ; then    # If both variables are defined then error is produced.
   echo "At least one or both of the variables BLACK_PROCESSES_LIST or WHITE_PROCESSES_LIST must be empty."
   exit 1
elif [[ -n "$BLACK_PROCESSES_LIST" ]] ; then                                 # If this variable is non-empty then set NEW_PIDS_COMMAND variable to bellow command
   NEW_PIDS_COMMAND="top -b -n1 -c | grep -E '$BLACK_PROCESSES_LIST' | gawk '\$9>CPU_LIMIT {print \$1}' CPU_LIMIT=$CPU_LIMIT"
elif [[ -n "$WHITE_PROCESSES_LIST" ]] ; then                                 # If this variable is non-empty then set NEW_PIDS_COMMAND variable to bellow command
   NEW_PIDS_COMMAND="top -b -n1 -c | gawk 'NR>6' | grep -E -v '$WHITE_PROCESSES_LIST' | gawk '\$9>CPU_LIMIT {print \$1}' CPU_LIMIT=$CPU_LIMIT"
else
   NEW_PIDS_COMMAND="top -b -n1 -c | gawk 'NR>6 && \$9>CPU_LIMIT {print \$1}' CPU_LIMIT=$CPU_LIMIT"
fi
 
# Search and limit violating PIDs
while sleep $DAEMON_INTERVAL
do
   NEW_PIDS=$(eval "$NEW_PIDS_COMMAND")                                                                    # Violating PIDs
   LIMITED_PIDS=$(ps -eo args | gawk '$1=="cpulimit" {print $3}')                                          # Already limited PIDs
   QUEUE_PIDS=$(comm -23 < (echo "$NEW_PIDS" | sort -u) <(echo "$LIMITED_PIDS" | sort -u) | grep -v '^$')   # PIDs in queue

   for i in $QUEUE_PIDS
   do
       cpulimit -p $i -l $CPU_LIMIT -z &   # Limit new violating processes
   done
done
[/sourcecode] 
 
If you wish to auto-start the Daemon on PC start, you have to do the following 3 steps: 
Step 1: 
[sourcecode="php"]
sudo chmod 700 /usr/bin/cpulimit_daemon.sh 
[/sourcecode] 
 
Step 2: Now save this to /etc/init.d/cpulimit: 
[sourcecode="php"] 
#!/bin/sh
#
# Script to start CPU limit daemon
#
set -e
 
case "$1" in
start)
if [ $(ps -eo pid,args | gawk '$3=="/usr/bin/cpulimit_daemon.sh"  {print $1}' | wc -l) -eq 0 ]; then
    nohup /usr/bin/cpulimit_daemon.sh >/dev/null 2>&1 &
    ps -eo pid,args | gawk '$3=="/usr/bin/cpulimit_daemon.sh"  {print}' | wc -l | gawk '{ if ($1 == 1) print " * cpulimit daemon started successfully"; else print " * cpulimit daemon can not be started" }'
else
    echo " * cpulimit daemon can't be started, because it is already running"
fi
;;
stop)
CPULIMIT_DAEMON=$(ps -eo pid,args | gawk '$3=="/usr/bin/cpulimit_daemon.sh"  {print $1}' | wc -l)
CPULIMIT_INSTANCE=$(ps -eo pid,args | gawk '$2=="cpulimit" {print $1}' | wc -l)
CPULIMIT_ALL=$((CPULIMIT_DAEMON + CPULIMIT_INSTANCE))
if [ $CPULIMIT_ALL -gt 0 ]; then
    if [ $CPULIMIT_DAEMON -gt 0 ]; then
        ps -eo pid,args | gawk '$3=="/usr/bin/cpulimit_daemon.sh"  {print $1}' | xargs kill -9   # kill cpulimit daemon
    fi
 
    if [ $CPULIMIT_INSTANCE -gt 0 ]; then
        ps -eo pid,args | gawk '$2=="cpulimit" {print $1}' | xargs kill -9                    # release cpulimited process to normal priority
    fi
    ps -eo pid,args | gawk '$3=="/usr/bin/cpulimit_daemon.sh"  {print}' | wc -l | gawk '{ if ($1 == 1) print " * cpulimit daemon can not be stopped"; else print " * cpulimit daemon stopped successfully" }'
else
    echo " * cpulimit daemon can't be stopped, because it is not running"
fi
;;
restart)
$0 stop
sleep 3
$0 start
;;
status)
ps -eo pid,args | gawk '$3=="/usr/bin/cpulimit_daemon.sh"  {print}' | wc -l | gawk '{ if ($1 == 1) print " * cpulimit daemon is running"; else print " * cpulimit daemon is not running" }'
;;
esac
exit 0

Step 3:

sudo chown root:root /etc/init.d/cpulimit
//Set the ownership
sudo chmod 755 /etc/init.d/cpulimit
//Set the privileges
sudo update-rc.d cpulimit defaults
//Add script to boot

And only thing left to do is test it out. You can do that with following commands (which are self-explanatory):

sudo service cpulimit status
sudo service cpulimit start
sudo service cpulimit stop
sudo service cpulimit restart

Conclusion

Well, I not the author of those scripts, but I think it is quite useful, and I wanted to share it with all of you. So have fun with limiting your processes! 😀


4 comments

  1. Hi,

    it’s first time I hear about “Silverline”, and thanks for the info! I’ll check it out.

    Cheers!

  2. You can try Librato “Silverline” to manage resource consumption of specific processes or applications. You can give an application specific resource quota, in which case it is guaranteed to receive resources up to its quota, but will get more if other workloads do not need it. Setting the quota to 0 will cause the application to only receive resources not required by other workloads.Resources not used by workloads within their quota are fair shared. 30 days free and free forever on up to 8 CPU cores.

    Fred van den Bosch
    Librato

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <blockquote cite=""> <code> <del datetime=""> <em> <s> <strike> <strong>. You may use following syntax for source code: <pre><code>$current = "Inchoo";</code></pre>.