Ok, so maybe you’re convinced why you need to set limits for container resource usage, especially when running on shared container application platforms like ECS, Kubernetes, and Swarm. In this post, I’ll show you how to set memory limits for a basic Java web app built with Spring Boot. Most container clusters are memory bound, so this is the best place to start.

First let’s run the application in the background with no limits enabled:

docker container run -d --name no-limits -p 9000:8080 \
    qualimente/container-limits:java

We can see the application is working by sending a few requests:

for i in `seq 1 3`; do curl http://localhost:9000; echo ""; done;
Hello Docker World
Hello Docker World
Hello Docker World

Docker’s stats command is a convenient way to check what resources a container is using along with the limit for that resource. If you run docker stats on that host, you should see something like:

NAME                       CPU %               MEM USAGE / LIMIT     MEM %
no-limits                  0.50%               224.5MiB / 1.945GiB   12.53%

This output shows the no-limits container is using 224.2MiB of memory against a limit of 1.945GiB. The ‘limit’ in this case is basically the entirety host’s 2GiB of RAM. This means the web application’s Java Virtual Machine (JVM) may consume all of the host’s memory if it decides to.

This lack of deterministic resource usage could be a big problem for other applications on the host or container orchestrators trying to pick an appropriate host for the container to run on.

There’s a second problem here too. In its current configuration, the JVM will play a guessing game as to what the maximum amount of memory it should use for the Java heap. Don’t rely on this guessing game for a real deployment. Over the past 18 year’s I’ve tuned JVMs with heaps from 256MiB to 40GiB. I can assure you that you can easily do better than the defaults, especially when it comes to memory limits and high throughput workloads.

We will approach this problem from two directions, first we’ll configure the JVM to use a deterministic amount of memory and then size a container reasonably for that limit.

Configure JVM memory limits

The Java Virtual Machine supports many command line options, including those that configure the size of the ‘heap’ and the garbage collector. The JVM heap is where objects that live past a temporary calculation (on the stack) are stored. Suppose I’ve done some load testing and used garbage collection information to decide that this application works best with a heap size of 256MiB. This image’s default program to run on startup (entrypoint) can be overwritten to with heap configuration options like so:

docker container run -d --name limit-heap -p 9001:8080 \
    --entrypoint "" \
    qualimente/container-limits:java \
    java -XX:+UseG1GC -Xms256m -Xmx256m -jar /app.jar

The -Xmx option configures the maximum size of the heap and the -Xmx option configures the minimum size. Configuring the maximum and minimum heap sizes to be identical is a best practice for optimizing garbage collector performance and also helps make the JVM’s footprint in RAM more deterministic.

After issuing a few requests to port 9001, I can see in docker stats that the JVM is oscillating between 275MiB and 285MiB of RAM.

Now, this isn’t meant to be a tutorial on JVM tuning, but I’ll let you in on a ‘secret.’ The JVM’s footprint in RAM is significantly more than the heap size, especially under heavy workloads and non-transient state. For the G1 collector, based on experience, I start with the assumption that the RAM footprint will be 35% more than the maximum heap size.

Configure container memory limits

So how should we size the container in this case?

The JVM will probably use about 135% of the heap size, but about if it uses more? I’m not a fan of denying malloc to hungry, but well-behaving applications. I recommend starting with a container memory limit that is 150% of the maximum JVM heap size to start with. In this case, that’s 384MiB. Remember:

1. this is a starting point that you can adjust using real world data
2. the JVM will use this extra memory servicing the twin goals of throughput and responsiveness, so it’s generally better to size the container too big than too small

The Docker container create and run commands support a --memory option that limits the amount of memory a container may use. Let’s run the app again with the memory limited to 384MiB:

docker container run -d --name limit-heap-and-container -p 9002:8080 \
    --entrypoint "" \
    --memory 384m \
    qualimente/container-limits:java \
    java -XX:+UseG1GC -Xms256m -Xmx256m -jar /app.jar

after running a few thousand requests through each of these services, docker stats shows:

NAME                       CPU %               MEM USAGE / LIMIT     MEM %
limit-heap-and-container   0.50%               308.6MiB / 384MiB     80.36%
limit-heap                 0.38%               300.8MiB / 1.945GiB   15.10%
no-limits                  0.50%               249.5MiB / 1.945GiB   12.53%

The limit-heap-and-container container is using 308MiB of its 384MiB limit (80%). The limit-heap container is close behind at 301MiB used and the no-limits container is now up to 249MiB.

These memory measurements come via the Linux cgroup facility, which is the kernel’s built-in resource accounting and enforcement mechanism. The measurements include memory usage by all the processes in the container. In this case, each container only runs a java process. However, if there were other processes running in the container or the Java program used “off heap” in-memory caches, they would be accounted for in the container’s memory usage.

The limit-heap-and-container container is a bit over-provisioned on memory. However, this exercise demonstrated that the JVM needed significantly more memory (20%) that what is implied by its heap settings, even for nearly the simplest possible Java webapp. “Real” applications are usually closer to the suggested 1.5x JVM heap guideline.

What we can be sure of at this point is that the JVM or any other process running inside the limit-heap-and-container will only be able to use up to a maximum of 384MiB of RAM.

Application engineers and operators can monitor the memory usage of their containers to identify changes in the application’s runtime resource consumption, particularly release to release or changing workloads. I suggest playing around with an insufficient memory limit like 200MiB to see what that looks like.

Memory limits are very useful for planning application deployments and capacity planning. Operators and container orchestrators can deploy applications to hosts with precise knowledge of what the application may use. This greatly simplifies the task of finding hosts with sufficient resources and running those hosts at high utilization, all while achieving other operational goals such as always running applications fully in RAM, avoiding swap memory.

If you have any questions, feel free to hit reply or take a look at Chapter 6 of Docker in Action, 2ed!

#NoDrama

KEEP LEARNING

 

 

 

Receive #NoDrama articles in your inbox whenever they are published. Reply to Stephen and the QualiMente team when you want to dig deeper into a topic.

SUBSCRIBE!

You have Successfully Subscribed!