Monitoring and inspecting Docker containers & images with Osquery
Osquery provides a huge amount of flexibility for querying various different aspects of your system which we’ve covered in many posts. What’s really awesome is Osquery also lets you query Docker containers, images, networks and their respective volumes. Which gives you a great insight into sub-systems running on your host you may not ordinarily be able to inspect with conventional tooling.
As of 4.5.1 there are currently 17 tables within Osquery dedicated to Docker. In this post we’re going to explore each one and see how they work.
Prerequisites
This goes without saying that’ll you’ll obviously need Docker installed and running on either Mac OSX or Linux. Windows is currently unsupported.
Docker will also need to be running with a few containers deployed. Otherwise, you’ll get nothing returned when you run your queries.
Docker system
The first two tables we’re going to look are docker_info and docker_version . These two tables provide information on the underlying version, system and configuration of Docker.
docker_info
There’s quite a lot of information packed into the docker_info table. I’ve grouped together a few of the available options. Hopefully to give you a better breakdown of whats available within this table.
The first query I have shows the current number of deployed containers. Including a breakdown of those running, paused and stopped. This query also includes the number of images currently available on the host.
osquery> SELECT containers, containers_running AS running, containers_paused AS paused, containers_stopped AS stopped, images FROM docker_info;+------------+---------+--------+---------+--------+
| containers | running | paused | stopped | images |
+------------+---------+--------+---------+--------+
| 68 | 5 | 0 | 63 | 460 |
+------------+---------+--------+---------+--------+
This next query will give you a feel for both the system and the resources allocated on the system toward Docker. The cpus represents the number of cores allocated and the memory is the available memory to Docker represented in bytes.
osquery> SELECT os, os_type, architecture, cpus, memory FROM docker_info;+----------------+---------+--------------+------+------------+
| os | os_type | architecture | cpus | memory |
+----------------+---------+--------------+------+------------+
| Docker Desktop | linux | x86_64 | 2 | 2084458496 |
+----------------+---------+--------------+------+------------+
We can also get an idea of Dockers configuration:.
osquery> .mode line
osquery> SELECT CASE WHEN memory_limit = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS memory_limit,
CASE WHEN swap_limit = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS swap_limit,
CASE WHEN kernel_memory = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS kernel_memory,
CASE WHEN cpu_cfs_period = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS cpu_cfs_period,
CASE WHEN cpu_cfs_quota = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS cpu_cfs_quota,
CASE WHEN cpu_shares = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS cpu_shares,
CASE WHEN cpu_set = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS cpu_set,
CASE WHEN ipv4_forwarding = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS ipv4_forwarding,
CASE WHEN bridge_nf_iptables = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS bridge_nf_iptables,
CASE WHEN bridge_nf_ip6tables = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS bridge_nf_ip6tables,
CASE WHEN oom_kill_disable = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS oom_kill_disable,
logging_driver,
cgroup_driver
FROM docker_info; memory_limit = ENABLED # memory limit support
swap_limit = ENABLED # swap limit support
kernel_memory = ENABLED # kernel memory limit
cpu_cfs_period = ENABLED # Completely Fair Scheduler period
cpu_cfs_quota = ENABLED # Completely Fair Scheduler quota
cpu_shares = ENABLED # CPU share weighting
cpu_set = ENABLED # CPU set selection support
ipv4_forwarding = ENABLED # IPv4 forwarding
bridge_nf_iptables = ENABLED # if bridge netfilter iptable
bridge_nf_ip6tables = ENABLED # if bridge netfilter ip6tables
oom_kill_disable = ENABLED # Out-of-memory kill
logging_driver = json-file
cgroup_driver = cgroupfs
There are a few other columns around version information. Which are also covered in docker_version . The only other column worth a mention is the id field. Which is the docker system ID. Which provides a useful identifier if you’ve got numerous systems deployed.
osquery> .mode pretty
osquery> SELECT id FROM docker_info;
+-------------------------------------------------------------+
| id |
+-------------------------------------------------------------+
| QJMQ:CCZO:7MBG:AFWL:JE6C:XTWY:2E3M:DFAO:GCME:LWZ3:CXJG:CMRC |
+-------------------------------------------------------------+docker_version
This table is pretty straight forward. It just shows you the version and release information for the version of Docker you have installed.
osquery> SELECT version, os, arch, git_commit FROM docker_version;+----------+-------+-------+------------+
| version | os | arch | git_commit |
+----------+-------+-------+------------+
| 19.03.13 | linux | amd64 | 4484c46d9d |
+----------+-------+-------+------------+
Docker images
Now all the configuration and version information is out of the way we can get onto the interesting stuff. First up are our Docker images.
Docker images are a read-only template that contains a set of instructions for creating a container. These containers are a convenient way of packaging up and distributing applications.
docker_images
When running a container i.e. docker run hello-world . The associated image will be downloaded and stored on your local machine. The docker_images table is a convenient way of showing all these images and their versions (tags).
osquery> SELECT SUBSTR(id, 0, 8) AS id, strftime(
'%d-%m-%Y ', datetime(created, 'unixepoch')
) AS created, size_bytes, tags
FROM docker_images
WHERE LENGTH(tags) < 20
AND tags <> ''
AND tags != '<none>:<none>' LIMIT 5;+---------+-------------+------------+------------------+
| id | created | size_bytes | tags |
+---------+-------------+------------+------------------+
| 37f4608 | 10-08-2020 | 384577927 | dev_nginx:latest |
| 4b52913 | 22-07-2020 | 313001393 | postgres:12.3 |
| 30df784 | 17-07-2020 | 370479878 | golang:alpine |
| 9fc56f7 | 09-06-2020 | 132097078 | nginx:stable |
| 5738956 | 09-06-2020 | 100637041 | debian:stretch |
+---------+-------------+------------+------------------+
docker_image_labels
Labels can be attached to images to help you find an image. Whilst you can SELECT every label on across all images stored on your machine. The majority of the time you’ll want to query labels contained within Docker files (using the LABEL tag) to find their image id. Or use the image id to find information on that image or potentially links to documentation or support.
osquery> SELECT key, value FROM docker_image_labels WHERE id = 'dcdb7fb8';+-------------+-----------------------+
| key | value |
+-------------+-----------------------+
| description | Zercurity Nginx proxy |
| maintainer | tim@zercurity.com |
| version | 1.0 |
+-------------+-----------------------+
docker_image_layers
Docker images are made up of layers. A layer is created when an instruction has been run to modify the parent layer. These layers come with a checksum (sha256) — which is the layer id.
Layers can be shared between other images as a base to build from. These layer ids can be extremely useful for identifying known or unknown layers within your docker infrastructure. This is particularly useful for vulnerability and patch management of images.
osquery> SELECT SUBSTR(layer_id, 0, 8) AS layer, layer_order
FROM docker_image_layers WHERE id = 'dcdb7fb8' ORDER BY layer_order ASC LIMIT 5;+---------+-------------+
| layer | layer_order |
+---------+-------------+
| 831c562 | 1 |
| ce342cf | 2 |
| e993d2b | 3 |
| 09f4a2d | 4 |
| f1b5753 | 5 |
+---------+-------------+
Docker containers
Containers are segregated processes that are allocated a share of resources form the host machine. These containers run from an image template that provides everything the processes needs to run within its environment.
docker_containers
You can quickly get a list of all the running containers and their states from the docker_containers table.
osquery> SELECT SUBSTR(id, 0, 8) AS container, SUBSTR(image_id, 0, 8) AS image_id, name, state, status FROM docker_containers;+-----------+----------+-----------------+---------+------------+
| container | image_id | name | state | status |
+-----------+----------+-----------------+---------+------------+
| 419ff6c | 37f4608 | /dev_nginx_1 | running | Up 2 weeks |
| fdaf4bc | fb6a0e0 | /dev_backend_1 | running | Up 2 weeks |
| aa5b373 | 86a53f9 | /dev_postgres_1 | running | Up 2 weeks |
| c2677b7 | b4d11e7 | /dev_frontend_1 | running | Up 2 weeks |
+-----------+----------+-----------------+---------+------------+
docker_container_labels
Just like with the image labels. Docker containers can also have labels applied to them to make identification easier.
osquery> SELECT SUBSTR(id, 0, 8) AS container, key, value FROM docker_container_labels WHERE key = 'maintainer';+-----------+------------+-----------------------+
| container | key | value |
+-----------+------------+-----------------------+
| 419ff6c | maintainer | support@zercurity.com |
+-----------+------------+-----------------------+
docker_container_mounts
Containers are designed to be stateless. Sometimes you’ll want to provide persistent storage to your container like a database. So that you can swap out container images for updates and retain the underlying data.
The docker_container_mounts table lets you see all these mounts including their attributes.
osquery> SELECT type, name, source, destination, driver, mode, rw FROM docker_container_mounts WHERE id = 'aa5b3731';+------+------+---------+-------------+--------+------+----+
| type | name | source | destination | driver | mode | rw |
+------+------+---------+-------------+--------+------+----+
| bind | | /tmp/db | /tmp/data | | rw | 1 |
+------+------+---------+-------------+--------+------+----+
docker_container_networks
Docker networks are a way of creating private networks in order for containers to talk to one another. The query below shows several containers all connected to a single network called dev_zercurity.
osquery> SELECT SUBSTR(id, 0, 8) AS container, name, SUBSTR(network_id, 0, 8) AS network, gateway, ip_address FROM docker_container_networks WHERE name = 'dev_zercurity';+-----------+---------------+---------+------------+------------+
| container | name | network | gateway | ip_address |
+-----------+---------------+---------+------------+------------+
| 419ff6c | dev_zercurity | bfa5d96 | 172.22.0.1 | 172.22.0.3 |
| fdaf4bc | dev_zercurity | bfa5d96 | | |
| aa5b373 | dev_zercurity | bfa5d96 | 172.22.0.1 | 172.22.0.4 |
| c2677b7 | dev_zercurity | bfa5d96 | 172.22.0.1 | 172.22.0.5 |
+-----------+---------------+---------+------------+------------+
docker_container_ports
You’ll often want to expose ports from your container to the host system. Osquery can show you all the exposed ports without a WHERE clause.
Or you can limit the scope down to individual containers. The example below shows the exposed ports on our NGINX server. Which we’d expect to be port 80 (http) and 443 (https).
osquery> SELECT type, port, host_ip, host_port FROM docker_container_ports WHERE id = '419ff6ca';+------+------+---------+-----------+
| type | port | host_ip | host_port |
+------+------+---------+-----------+
| tcp | 443 | 0.0.0.0 | 443 |
| tcp | 80 | 0.0.0.0 | 80 |
+------+------+---------+-----------+
This particular table is useful for security and DevOps teams to check ports that haven’t been exposed accidentally. You can quickly check for open development ports, administration services and insecure services like so:
osquery> SELECT SUBSTR(id, 0, 8) AS container, host_ip, host_port FROM docker_container_ports WHERE port IN (21, 22, 23, 5000);Depending on your internal policies you may also want to ensure that container processes exposing ports, aren’t running as root . All processes are using ports above 1024.
osquery> SELECT SUBSTR(id, 0, 8) AS container, host_ip, host_port FROM docker_container_ports WHERE port <= 1024;docker_container_processes
This is another really cool table, that lets you query the running processes within a container. From a security perspective if you know what processes ought to be running. You can effectively define an allow-list of known processes and alert. Should the event arise of an unexpected processes starting.
A container id must be provided as part of the WHERE clause.
osquery> SELECT pid, name, cmdline FROM docker_container_processes WHERE id = '419ff6ca';+------+------+--------------------------------------------+
| pid | name | cmdline |
+------+------+--------------------------------------------+
| 2361 | | /bin/sh -c ./start.sh |
| 2553 | | {start.sh} /bin/bash ./start.sh |
| 2596 | | nginx: master process nginx -g daemon off; |
| 2621 | | nginx: worker process |
| 2622 | | nginx: worker process |
+------+------+--------------------------------------------+
docker_container_stats
This is a really useful table for monitoring the performance of your containers. There are quite a few columns available so I’ve broken them down into CPU, memory, disk and networking.
When dealing with calculating CPU stats on systems you’ll only get the amount of time that has been spent on a task. To actually calculate the CPU usage as a percentage you need to keep track on the time between the two polling intervals.
Helpfully this is provided to you via the read and preread columns. Which can be used to help in the CPU calculations under the next section.
A container id must be provided as part of the WHERE clause.
osquery> .mode line
osquery> SELECT name, pids, read, preread, interval FROM docker_container_stats WHERE id = '419ff6ca';name = /dev_nginx_1
pids = 4 # Number of processes
read = 1607679236 # UNIX time when stats were read
preread = 1607679235 # UNIX time when stats were last read
interval = 1008060650 # Diff of read and preread in nano-seconds
CPU
There are quite a number of avaliable CPU options:
osquery> SELECT name, num_procs, cpu_total_usage, cpu_kernelmode_usage, cpu_usermode_usage, system_cpu_usage, online_cpus, pre_cpu_total_usage, pre_cpu_kernelmode_usage, pre_cpu_usermode_usage, pre_system_cpu_usage, pre_online_cpus FROM docker_container_stats WHERE id = '419ff6ca'; name = /dev_nginx_1
num_procs = 0
cpu_total_usage = 157615787
cpu_kernelmode_usage = 90000000
cpu_usermode_usage = 60000000
system_cpu_usage = 1543430000000
online_cpus = 2
pre_cpu_total_usage = 157615787
pre_cpu_kernelmode_usage = 90000000
pre_cpu_usermode_usage = 60000000
pre_system_cpu_usage = 1541780000000
pre_online_cpus = 2
However, to calculate the actual CPU usage as a percentage. The pre and present stats are included automatically for you.
osquery> SELECT ((
(cpu_total_usage-pre_cpu_total_usage)*1.0 /
(system_cpu_usage-pre_system_cpu_usage)*1.0
) * online_cpus) * 100 AS usage
FROM docker_container_stats WHERE id = '419ff6ca';Mem
Simply shows the memory allocated to the container versus whats avaliable.
osquery> SELECT name, memory_usage, memory_max_usage, memory_limit FROM docker_container_stats WHERE id = '419ff6ca'; name = /dev_nginx_1
memory_usage = 13541376 # Current container usage
memory_max_usage = 40849408 # The max limit for the container
memory_limit = 2084458496 # The system limit for docker
Disk
Simply shows the number of bytes writen to and from the host disk.
osquery> SELECT name, disk_read, disk_write FROM docker_container_stats WHERE id = '419ff6ca';name = /dev_nginx_1
disk_read = 27443200
disk_write = 4096
Network
Simply shows the number of bytes sent and recived from the container.
osquery> SELECT name, network_rx_bytes, network_tx_bytes FROM docker_container_stats WHERE id = '419ff6ca'; name = /dev_nginx_1
network_rx_bytes = 13376 # Recived bytes
network_tx_bytes = 3455 # Sent bytes
docker_container_fs_changes
These are changes that have been make to the container’s filesystem since the container was started. As containers use an image as a base for their filesystem. We can derive a list of changes since the container started.
A container id must be provided as part of the WHERE clause.
osquery> SELECT path, (CASE
WHEN change_type = 'C' THEN 'CREATED'
WHEN change_type = 'A' THEN 'ADDED'
WHEN change_type = 'R' THEN 'REMOVED'
ELSE 'UNKNOWN' END) AS type
FROM docker_container_fs_changes
WHERE id = '10d666fb9752' LIMIT 5;+----------------+---------+
| path | type |
+----------------+---------+
| /run | CREATED |
| /run/nginx.pid | ADDED |
| /var | CREATED |
| /var/lib | CREATED |
| /var/lib/nginx | CREATED |
+----------------+---------+
Docker networks
As mentioned earlier with docker_container_networks table. Containers can be attached to their own private networks (network_id). To allow inter-container communication.
docker_network_labels
Just like with the container and image labels. Networks can also have labels attached to make discovery easier.
osquery> SELECT SUBSTR(id, 0, 8) AS id, key, value FROM docker_network_labels LIMIT 5;+---------+----------------------------+-----------+
| id | key | value |
+---------+----------------------------+-----------+
| bfa5d96 | com.docker.compose.network | zercurity |
| bfa5d96 | com.docker.compose.project | dev |
| bfa5d96 | com.docker.compose.version | 1.26.2 |
+---------+----------------------------+-----------+
docker_networks
Most of the information within this table is already avaliable in docker_container_networks. However , you can fetch all the configured networks like so:
osquery> SELECT SUBSTR(id, 0, 8) AS id, name, driver, subnet, gateway FROM docker_networks;
+---------+------------+--------+------------------+---------------+
| id | name | driver | subnet | gateway |
+---------+------------+--------+------------------+---------------+
| ebd527e | dev_defa.. | bridge | 192.168.160.0/20 | 192.168.160.1 |
| 7ecbb21 | hive_def.. | bridge | 192.168.0.0/20 | 192.168.0.1 |
| 809b959 | host | host | | |
| bfa5d96 | dev_zerc.. | bridge | 172.22.0.0/16 | 172.22.0.1 |
| b32881c | zercurit.. | bridge | 192.168.224.0/20 | 192.168.224.1 |
| be8e04e | none | null | | |
| 657ee7e | bridge | bridge | 172.17.0.0/16 | 172.17.0.1 |
+---------+------------+--------+------------------+---------------+Docker volumes
As mentioned earlier with docker_container_mounts table. Containers can have attached persistent volumes.
docker_volume_labels
To make the identification of volumes easier you can query their labels.
osquery> SELECT * FROM docker_volume_labels LIMIT 5;
+-------------------+----------------------------+-----------+
| name | key | value |
+-------------------+----------------------------+-----------+
| wordpress_db_data | com.docker.compose.project | wordpress |
| wordpress_db_data | com.docker.compose.version | 1.22.0 |
| wordpress_db_data | com.docker.compose.volume | db_data |
+-------------------+----------------------------+-----------+docker_volumes
Lastly, just like with the docker_networks table. We can retrive a compelte system inventory of all the create volumnes and their mount points.
osquery> SELECT SUBSTR(name, 0, 8) AS id, driver, mount_point AS mount FROM docker_volumes LIMIT 5;+---------+--------+-----------------------------------------------+
| id | driver | mount |
+---------+--------+-----------------------------------------------+
| 11bd27d | local | /var/lib/docker/volumes/11bd2..0e7d1820/_data |
| f562970 | local | /var/lib/docker/volumes/f5629..670bbd70/_data |
| fa4bf78 | local | /var/lib/docker/volumes/fa4bf..abe163de/_data |
| 0c17c32 | local | /var/lib/docker/volumes/0c17c..c6fda879/_data |
| 7e5ab04 | local | /var/lib/docker/volumes/7e5ab..2d084acb/_data |
+---------+--------+-----------------------------------------------+
Troubleshoorting
Whilst the docker tables are straight forward there are a few gotachs.
Nothing returned in query
If you run a query against one of the docker based tables such as SELECT * FROM docker_containers; and you get nothing back. It maybe that your Docker service isn’t running. Or your user does not have the correct permissions to read from the Docker socket.
WHERE clause
This error simply means you need to specify a container id in your SQL statement. As the query itself is used to fetch specific information about a container. You don’t even have to provide the full container id. A partial id can also be provided.
virtual_table.cpp:966] Table <table_name> was queried without a required column in the WHERE clauseSELECT * FROM docker_container_ports WHERE id = '419ff6ca';
Its all over!
Hopefully that’s given you a quick dive into how to use Docker with Osquery. We’ll be following up on this topic in the near future. As well as how Zercurity uses Docker and Kubernetes at scale in production. However, that’s all for now. Please feel free to get in touch if you have any questions.
