Linux cgroups Explained: How Containers Use It
Cgroups (control groups) is a Linux kernel feature that allows limiting how much resources (for example cpu and memory) processes can consume. There are other traditional systems for doing so, such as POSIX rlimit interface, but the most flexible option on Linux systems is now the cgroup kernel feature.
The idea is that we can place several processes into a cgroup, which allows us to manage the resources that they consume on a group-wide basis. After creating a cgroup, we can add processes to it and then use a controller to change how those processes behave.
cgroup versions
There are two versions of cgroups, 1 and 2, and both are currently in use and can be configured simultaneously on a system. We can sum up the structural differences between the versions as follows:
- In cgroups v1 each type of controller (cpu, memory, and so on) has its own set of cgroups. A process can belong to multiple cgroups. For example, in v1, a process can belong to a cpu cgroup and a memory cgroup.
- In cgroups v2 a process can belong to only one cgroup and we can set up different types of controllers for each group.
For example, if we have 2 processes A and B, and we want to use the cpu and memory controllers on each of them, we need 4 cgroups total, because each cgroup is limited to a single controller.
Compared to the v2 we need only 2 cgroups in total, as for each cgroup we can create multiple controllers.
It seems like v1 has more flexibility over v2 because we can assign different combinations of cgroups to processes. However, it turns out that this approach was more complicated to set up and implement than simply having one cgroup per process. If a controller is used in cgroups v1, it cannot be used in v2 simultaneously due to potential conflicts.
Working with cgroups
Unlike traditional Unix system call interface for interacting with kernel, cgroups are accessed through the filesystem, which is usually mounted as a cgroup2 filesystem under /sys/fs/cgroup .
Let’s run the following command and see the different types of cgroups on my system.
root@ubuntu-linux-2404:~# ls /sys/fs/cgroup/
cgroup.controllers cpu.pressure init.scope memory.reclaim sys-kernel-debug.mount
cgroup.max.depth cpuset.cpus.effective io.cost.model memory.stat sys-kernel-tracing.mount
cgroup.max.descendants cpuset.cpus.isolated io.cost.qos memory.zswap.writeback system.slice
cgroup.pressure cpuset.mems.effective io.pressure misc.capacity user.slice
cgroup.procs cpu.stat io.prio.class misc.current
cgroup.stat cpu.stat.local io.stat proc-sys-fs-binfmt_misc.mount
cgroup.subtree_control dev-hugepages.mount memory.numa_stat sys-fs-fuse-connections.mount
cgroup.threads dev-mqueue.mount memory.pressure sys-kernel-config.mountManaging cgroups involves reading and writing to the files and directories within these hierarchies. Creating a subdirectory creates a cgroup, and the kernel automatically populates the directory with the various files that represent parameters and statistics about the cgroup.
root@ubuntu-linux-2404:~# mkdir /sys/fs/cgroup/test
root@ubuntu-linux-2404:~# ls /sys/fs/cgroup/test
cgroup.controllers cgroup.stat cpu.stat memory.events memory.peak memory.swap.peak
cgroup.events cgroup.subtree_control cpu.stat.local memory.events.local memory.pressure memory.zswap.current
cgroup.freeze cgroup.threads cpu.uclamp.max memory.high memory.reclaim memory.zswap.max
cgroup.kill cgroup.type cpu.uclamp.min memory.low memory.stat memory.zswap.writeback
cgroup.max.depth cpu.idle cpu.weight memory.max memory.swap.current pids.current
cgroup.max.descendants cpu.max cpu.weight.nice memory.min memory.swap.events pids.events
cgroup.pressure cpu.max.burst io.pressure memory.numa_stat memory.swap.high pids.max
cgroup.procs cpu.pressure memory.current memory.oom.group memory.swap.max pids.peakBut how can we check cgroups for a specific process? One option is to use the /proc filesystem to view the cgroups that are being used for a specific process. In this case, the bash shell of the running user. First let’s get the PID of our shell.
root@ubuntu-linux-2404:~# ps -fC bash
UID PID PPID C STIME TTY TIME CMD
root 3373 3372 0 11:30 pts/1 00:00:00 -bashWe can now use the PID to reveal the cgroup “slice” and “scope” to which the process belongs (slices and scopes are utilized to organize cgroups and processes).
root@ubuntu-linux-2404:~# cat /proc/3373/cgroup
0::/user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-3fb230df-3687-4ba6-95e7-d476955fef64.scopeTo see the available resources that can be modified for that process, you can look in the /sys filesystem, which corresponds to the information we obtained from the previous command:
root@ubuntu-linux-2404:~# ls /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-3fb230df-3687-4ba6-95e7-d476955fef64.scope/
cgroup.controllers cgroup.procs cpu.stat.local memory.max memory.stat memory.zswap.max
cgroup.events cgroup.stat io.pressure memory.min memory.swap.current memory.zswap.writeback
cgroup.freeze cgroup.subtree_control memory.current memory.numa_stat memory.swap.events pids.current
cgroup.kill cgroup.threads memory.events memory.oom.group memory.swap.high pids.events
cgroup.max.depth cgroup.type memory.events.local memory.peak memory.swap.max pids.max
cgroup.max.descendants cpu.pressure memory.high memory.pressure memory.swap.peak pids.peak
cgroup.pressure cpu.stat memory.low memory.reclaim memory.zswap.currentNow that we can see the cgroup files, it’s important to understand that the cgroup filesystem is a “virtual interface”. Meaning when we run ls or cat commands, we are NOT looking at data on a disk; we are looking at the live kernel variables. When we create/modify cgroup files in the filesystem, we are sending a message to a specific kernel subsystem (like cpu controller).
Get David Mosyan’s stories in your inbox
Join Medium for free to get updates from this writer.
And because the kernel sits between the processes and the hardware (CPU/RAM), it uses specific mechanisms to enforce the numbers we put in those files. Now let’s talk about some of the files we saw in the output of the last command.
cgroup.procs — lists every PID belonging to this group. The kernel treats all these PIDs as one single entity for accounting.
memory.max — If we echo 500M > memory.max, the kernel will kill processes if the group exceeds 500MB.
cpu.max — Tells the scheduler exactly how much CPU time to allow per millisecond.
memory.current — When we cat this, the kernel sums up all the RAM pages currently mapped to the PIDs in cgroup.procs .
By using a hierarchy of folders, the kernel allows for inheritance. If you set a limit on user.slice (one of the folders where we run ls command) every sub-folder automatically shares or stays within that limit without the kernel needing to track thousands of individual settings manually.
Containers
When we start a container, the runtime creates new cgroups for it. We can use a utility called lscgroup to help see these cgroups from the host. First, let’s install it and save the current state of the cgroups in a file.
root@ubuntu-linux-2404:~# apt install cgroup-tools
root@ubuntu-linux-2404:~# lscgroup > before_container.txtOnce we have a “snapshot” of the cgroup file system before running the container, we can proceed to start a container.
root@ubuntu-linux-2404:~# docker run -d --name test-container nginx
3323b0e3ca585834fa312bba4633211224337500324a4e1bc8ab436cf9fcb583
root@ubuntu-linux-2404:~# lscgroup > after_container.txtNow let’s see the diff of the text files and check the newly created cgroup folder.
root@ubuntu-linux-2404:~# diff before_container.txt after_container.txt
120a121
> cpuset,cpu,io,memory,hugetlb,pids,rdma,misc:/system.slice/docker-3323b0e3ca585834fa312bba4633211224337500324a4e1bc8ab436cf9fcb583.scope
root@ubuntu-linux-2404:~# ls /sys/fs/cgroup/system.slice/docker-3323b0e3ca585834fa312bba4633211224337500324a4e1bc8ab436cf9fcb583.scope/
cgroup.controllers cpuset.cpus.exclusive.effective hugetlb.2MB.events.local io.max memory.swap.events
cgroup.events cpuset.cpus.partition hugetlb.2MB.max io.pressure memory.swap.high
cgroup.freeze cpuset.mems hugetlb.2MB.numa_stat io.prio.class memory.swap.max
cgroup.kill cpuset.mems.effective hugetlb.2MB.rsvd.current io.stat memory.swap.peak
cgroup.max.depth cpu.stat hugetlb.2MB.rsvd.max io.weight memory.zswap.current
cgroup.max.descendants cpu.stat.local hugetlb.32MB.current memory.current memory.zswap.max
We can clearly see that the file hierarchy of the new cgroup contains the container ID returned by the docker run command. Docker automatically creates its own cgroups of each type. When we start a container, it automatically creates another set of cgroups within the docker cgroups.
Summary
Cgroups limit the resources available to different Linux processes. We don’t have to use containers to take advantage of cgroups, but Docker and other container runtimes provide a convenient interface for using them: it’s very easy to set resource limits at the point where you run a container, and those limits are being set in the cgroup file hirarchy which is then policed by cgroups.
This helps to protect the system from a class of attacks that attempt to disrupt the system by consuming excessive resources. It’s always recommended to set memory and CPU limits when running container applications.
👏 and subscribe for more articles! 🖥️
Sources
How Linux Works by Brian Ward
Container Security by Liz Rice
Datadog security labs blog post







