Previously, I explained why you need to set limits for container resource usage and how to limit a container’s memory resources. Today, I’ll show you how to configure cpu resource limits.
The first thing to point out is that CPU resources are often a bit more fluid than memory. When a program needs to do work, it executes instructions, the cpu processes these as quickly as it can, and then both the program and cpu return to idle unless and until there is more work to do. The program may use one percent of a cpu core in one second, 100% of a core the next, then back down to idle.
Here’s a cpu utilization chart depicting cpu usage of an application processing web requests. Even with usage aggregated to the minute level, we see the program operating in a range of 1% to 24% cpu utilization.
A program’s CPU usage is highly quantized and consumed in many very short bursts “to 100%” each second. This means the tactics for limiting usage can be different than memory usage. Server applications may allocate memory statically on startup, build in-memory caches, and consume memory in other ways that aren’t directly related to processing the workload.
Docker exposes two main Linux cpu usage controls to you via the container’s cgroup:
- cpu shares
- cpu quota
CPU shares
The cpu-shares option allows you to specify the relative share of cpu a container will receive when there is contention for cpu.
When there is no contention for cpu, a container will get to use however much it wants, no matter what the ‘limit’ is.
When there is contention, a container configured for 2048 shares of a cpu will get twice as much cpu time as a container that requested 1024 cpu shares.
The cpu-shares resource constraint feature has been available since early versions of Docker.
Let’s work through an example. I am using a Docker on a machine with 4 cores.
Start by launching docker stats in one terminal, showing just the container name, cpu usage, and memory usage:
docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}"In a second terminal, launch the Docker in Action ‘stresser’ image used in Chapter 6:
docker container run --name stresser-1024 \
--cpu-shares 1024 \
dockerinaction/ch6_stresserThe Docker stats command shows that stresser-1024 container uses 398% cpu — all four cores:
NAME CPU % MEM USAGE / LIMIT MEM %
stresser-1024 398.29% 1.102MiB / 1.945GiB 0.06%Now let’s see what happens when running a second container with twice as many cpu shares:
docker container run -d --name stresser-2048 \
--cpu-shares 2048 \
dockerinaction/ch6_stresserand restart the stresser-1024 container:
docker container start stresser-1024Now that there are two processes that both want to take 100% of CPU resources, Linux is dividing those resources between them proportionally:
NAME CPU % MEM USAGE / LIMIT MEM %
stresser-2048 263.26% 1.078MiB / 1.945GiB 0.05%
stresser-1024 131.42% 1.035MiB / 1.945GiB 0.05% And indeed, stresser-2048 gets twice (263%) as much cpu as stresser-1024 (131%).
Notice that while we specified the cpu shares with what looks like an implied number of desired processors multiplied by 1024, this didn’t limit the processes to one or two cores. All four cores on the machine are used.
If you want to enforce an absolute limit, you need to specify a cpu quota.
CPU quota
Docker permits you to configure absolute cpu quotas easily through the --cpus option introduced in Docker 1.13. This cpu quota specifies the fixed share of cpu that the container is entitled may use before it is throttled. The quota is defined at the container’s cgroup and enforced by Linux’s Completely Fair Scheduler.
Let’s see it in action by starting stresser containers with quotas of 1 and 2 cpus each:
docker container run -d --name stresser-1-cpus \
--cpus 1 \
dockerinaction/ch6_stresser
docker container run -d --name stresser-2-cpus \
--cpus 2 \
dockerinaction/ch6_stresserNow the stats show a much different story:
NAME CPU % MEM USAGE / LIMIT MEM %
stresser-1-cpus 100.17% 1.098MiB / 1.945GiB 0.06%
stresser-2-cpus 201.14% 1.07MiB / 1.945GiB 0.05%The stresser programs are limited to precisely the number of cpus we specified. When a program exhausts its quota, the Linux kernel will delay running the program until the quota is replenished. The quota is allocated and enforced every 100 milliseconds.
Applying CPU constraints
Both of the cpu share and quota resource constraint mechanisms can be useful.
CPU shares can help you establish a relative priority between containerized processes sharing a host. Also, because CPU shares was available before the absolute CPU constraint option, most container orchestrators know how to work with cpu shares expressed in ‘millicores’. These orchestrators will take a request or limit expressed in millicores, look for a container host with sufficient resources under the assumption that each core is worth 1024 millicores, and then configure the cpu-shares for the container accordingly. This approximates the ability to specify a certain number of cpus well for many use cases.
One of the places CPU shares breaks down is when you don’t want a container to be able to burst above its share when there is no contention on the host. There are a number of valid use cases such as:
- preventing a load testing of an isolated service from using a whole machine in a test environment when the service will only have part of a machine in production
- establishing a hard budget that limits the resources available to a service for scheduling, security, billing, or accounting reasons
There are several other cpu constraints and ways to apply them. Check out Chapter 6 of Docker in Action, 2ed for more or hit reply with a question.
Stephen
#NoDrama
p.s. you can delete all the containers used in this post with:
docker container rm -f stresser-1024 stresser-2048 stresser-1-cpus stresser-2-cpus
