Monitoring and inspecting Docker containers & images with Osquery

Osquery provides a huge amount of flexibility for querying various different aspects of your system which we’ve covered in many posts. What’s really awesome is Osquery also lets you query Docker containers, images, networks and their respective volumes. Which gives you a great insight into sub-systems running on your host you may not ordinarily be able to inspect with conventional tooling.

Docker’s logo

As of 4.5.1 there are currently 17 tables within Osquery dedicated to Docker. In this post we’re going to explore each one and see how they work.

Prerequisites

Docker will also need to be running with a few containers deployed. Otherwise, you’ll get nothing returned when you run your queries.

Docker system

docker_info

The first query I have shows the current number of deployed containers. Including a breakdown of those running, paused and stopped. This query also includes the number of images currently available on the host.

osquery> SELECT containers, containers_running AS running, containers_paused AS paused, containers_stopped AS stopped, images FROM docker_info;+------------+---------+--------+---------+--------+
| containers | running | paused | stopped | images |
+------------+---------+--------+---------+--------+
| 68 | 5 | 0 | 63 | 460 |
+------------+---------+--------+---------+--------+

This next query will give you a feel for both the system and the resources allocated on the system toward Docker. The cpus represents the number of cores allocated and the memory is the available memory to Docker represented in bytes.

osquery> SELECT os, os_type, architecture, cpus, memory FROM docker_info;+----------------+---------+--------------+------+------------+
| os | os_type | architecture | cpus | memory |
+----------------+---------+--------------+------+------------+
| Docker Desktop | linux | x86_64 | 2 | 2084458496 |
+----------------+---------+--------------+------+------------+

We can also get an idea of Dockers configuration:.

osquery> .mode line
osquery>
SELECT CASE WHEN memory_limit = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS memory_limit,
CASE WHEN swap_limit = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS swap_limit,
CASE WHEN kernel_memory = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS kernel_memory,
CASE WHEN cpu_cfs_period = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS cpu_cfs_period,
CASE WHEN cpu_cfs_quota = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS cpu_cfs_quota,
CASE WHEN cpu_shares = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS cpu_shares,
CASE WHEN cpu_set = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS cpu_set,
CASE WHEN ipv4_forwarding = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS ipv4_forwarding,
CASE WHEN bridge_nf_iptables = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS bridge_nf_iptables,
CASE WHEN bridge_nf_ip6tables = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS bridge_nf_ip6tables,
CASE WHEN oom_kill_disable = 1 THEN
'ENABLED' ELSE 'DISABLED' END AS oom_kill_disable,
logging_driver,
cgroup_driver
FROM docker_info;
memory_limit = ENABLED # memory limit support
swap_limit = ENABLED # swap limit support
kernel_memory = ENABLED # kernel memory limit
cpu_cfs_period = ENABLED # Completely Fair Scheduler period
cpu_cfs_quota = ENABLED # Completely Fair Scheduler quota
cpu_shares = ENABLED # CPU share weighting
cpu_set = ENABLED # CPU set selection support
ipv4_forwarding = ENABLED # IPv4 forwarding
bridge_nf_iptables = ENABLED # if bridge netfilter iptable
bridge_nf_ip6tables = ENABLED # if bridge netfilter ip6tables
oom_kill_disable = ENABLED # Out-of-memory kill
logging_driver = json-file
cgroup_driver = cgroupfs

There are a few other columns around version information. Which are also covered in docker_version . The only other column worth a mention is the id field. Which is the docker system ID. Which provides a useful identifier if you’ve got numerous systems deployed.

osquery> .mode pretty
osquery>
SELECT id FROM docker_info;
+-------------------------------------------------------------+
| id |
+-------------------------------------------------------------+
| QJMQ:CCZO:7MBG:AFWL:JE6C:XTWY:2E3M:DFAO:GCME:LWZ3:CXJG:CMRC |
+-------------------------------------------------------------+

docker_version

osquery> SELECT version, os, arch, git_commit FROM docker_version;+----------+-------+-------+------------+
| version | os | arch | git_commit |
+----------+-------+-------+------------+
| 19.03.13 | linux | amd64 | 4484c46d9d |
+----------+-------+-------+------------+

Docker images

Docker images are a read-only template that contains a set of instructions for creating a container. These containers are a convenient way of packaging up and distributing applications.

docker_images

osquery> SELECT SUBSTR(id, 0, 8) AS id, strftime(
'%d-%m-%Y ', datetime(created, 'unixepoch')
) AS created, size_bytes, tags
FROM docker_images
WHERE LENGTH(tags) < 20
AND tags <> ''
AND tags != '<none>:<none>' LIMIT 5;
+---------+-------------+------------+------------------+
| id | created | size_bytes | tags |
+---------+-------------+------------+------------------+
| 37f4608 | 10-08-2020 | 384577927 | dev_nginx:latest |
| 4b52913 | 22-07-2020 | 313001393 | postgres:12.3 |
| 30df784 | 17-07-2020 | 370479878 | golang:alpine |
| 9fc56f7 | 09-06-2020 | 132097078 | nginx:stable |
| 5738956 | 09-06-2020 | 100637041 | debian:stretch |
+---------+-------------+------------+------------------+

docker_image_labels

osquery> SELECT key, value FROM docker_image_labels WHERE id = 'dcdb7fb8';+-------------+-----------------------+
| key | value |
+-------------+-----------------------+
| description | Zercurity Nginx proxy |
| maintainer | tim@zercurity.com |
| version | 1.0 |
+-------------+-----------------------+

docker_image_layers

Layers can be shared between other images as a base to build from. These layer ids can be extremely useful for identifying known or unknown layers within your docker infrastructure. This is particularly useful for vulnerability and patch management of images.

osquery> SELECT SUBSTR(layer_id, 0, 8) AS layer, layer_order 
FROM docker_image_layers WHERE id = 'dcdb7fb8' ORDER BY layer_order ASC LIMIT 5;
+---------+-------------+
| layer | layer_order |
+---------+-------------+
| 831c562 | 1 |
| ce342cf | 2 |
| e993d2b | 3 |
| 09f4a2d | 4 |
| f1b5753 | 5 |
+---------+-------------+

Docker containers

docker_containers

osquery> SELECT SUBSTR(id, 0, 8) AS container, SUBSTR(image_id, 0, 8) AS image_id, name, state, status FROM docker_containers;+-----------+----------+-----------------+---------+------------+
| container | image_id | name | state | status |
+-----------+----------+-----------------+---------+------------+
| 419ff6c | 37f4608 | /dev_nginx_1 | running | Up 2 weeks |
| fdaf4bc | fb6a0e0 | /dev_backend_1 | running | Up 2 weeks |
| aa5b373 | 86a53f9 | /dev_postgres_1 | running | Up 2 weeks |
| c2677b7 | b4d11e7 | /dev_frontend_1 | running | Up 2 weeks |
+-----------+----------+-----------------+---------+------------+

docker_container_labels

osquery> SELECT SUBSTR(id, 0, 8) AS container, key, value FROM docker_container_labels WHERE key = 'maintainer';+-----------+------------+-----------------------+
| container | key | value |
+-----------+------------+-----------------------+
| 419ff6c | maintainer | support@zercurity.com |
+-----------+------------+-----------------------+

docker_container_mounts

The docker_container_mounts table lets you see all these mounts including their attributes.

osquery> SELECT type, name, source, destination, driver, mode, rw FROM docker_container_mounts WHERE id = 'aa5b3731';+------+------+---------+-------------+--------+------+----+
| type | name | source | destination | driver | mode | rw |
+------+------+---------+-------------+--------+------+----+
| bind | | /tmp/db | /tmp/data | | rw | 1 |
+------+------+---------+-------------+--------+------+----+

docker_container_networks

osquery> SELECT SUBSTR(id, 0, 8) AS container, name, SUBSTR(network_id, 0, 8) AS network, gateway, ip_address FROM docker_container_networks WHERE name = 'dev_zercurity';+-----------+---------------+---------+------------+------------+
| container | name | network | gateway | ip_address |
+-----------+---------------+---------+------------+------------+
| 419ff6c | dev_zercurity | bfa5d96 | 172.22.0.1 | 172.22.0.3 |
| fdaf4bc | dev_zercurity | bfa5d96 | | |
| aa5b373 | dev_zercurity | bfa5d96 | 172.22.0.1 | 172.22.0.4 |
| c2677b7 | dev_zercurity | bfa5d96 | 172.22.0.1 | 172.22.0.5 |
+-----------+---------------+---------+------------+------------+

docker_container_ports

Or you can limit the scope down to individual containers. The example below shows the exposed ports on our NGINX server. Which we’d expect to be port 80 (http) and 443 (https).

osquery> SELECT type, port, host_ip, host_port FROM docker_container_ports WHERE id = '419ff6ca';+------+------+---------+-----------+
| type | port | host_ip | host_port |
+------+------+---------+-----------+
| tcp | 443 | 0.0.0.0 | 443 |
| tcp | 80 | 0.0.0.0 | 80 |
+------+------+---------+-----------+

This particular table is useful for security and DevOps teams to check ports that haven’t been exposed accidentally. You can quickly check for open development ports, administration services and insecure services like so:

osquery> SELECT SUBSTR(id, 0, 8) AS container, host_ip, host_port FROM docker_container_ports WHERE port IN (21, 22, 23, 5000);

Depending on your internal policies you may also want to ensure that container processes exposing ports, aren’t running as root . All processes are using ports above 1024.

osquery> SELECT SUBSTR(id, 0, 8) AS container, host_ip, host_port FROM docker_container_ports WHERE port <= 1024;

docker_container_processes

A container id must be provided as part of the WHERE clause.

osquery> SELECT pid, name, cmdline FROM docker_container_processes WHERE id = '419ff6ca';+------+------+--------------------------------------------+
| pid | name | cmdline |
+------+------+--------------------------------------------+
| 2361 | | /bin/sh -c ./start.sh |
| 2553 | | {start.sh} /bin/bash ./start.sh |
| 2596 | | nginx: master process nginx -g daemon off; |
| 2621 | | nginx: worker process |
| 2622 | | nginx: worker process |
+------+------+--------------------------------------------+

docker_container_stats

When dealing with calculating CPU stats on systems you’ll only get the amount of time that has been spent on a task. To actually calculate the CPU usage as a percentage you need to keep track on the time between the two polling intervals.

Helpfully this is provided to you via the read and preread columns. Which can be used to help in the CPU calculations under the next section.

A container id must be provided as part of the WHERE clause.

osquery> .mode line
osquery>
SELECT name, pids, read, preread, interval FROM docker_container_stats WHERE id = '419ff6ca';
name = /dev_nginx_1
pids = 4 # Number of processes
read = 1607679236 # UNIX time when stats were read
preread = 1607679235 # UNIX time when stats were last read
interval = 1008060650 # Diff of read and preread in nano-seconds

CPU
There are quite a number of avaliable CPU options:

osquery> SELECT name, num_procs, cpu_total_usage, cpu_kernelmode_usage, cpu_usermode_usage, system_cpu_usage, online_cpus, pre_cpu_total_usage, pre_cpu_kernelmode_usage, pre_cpu_usermode_usage, pre_system_cpu_usage, pre_online_cpus FROM docker_container_stats WHERE id = '419ff6ca';                    name = /dev_nginx_1
num_procs = 0
cpu_total_usage = 157615787
cpu_kernelmode_usage = 90000000
cpu_usermode_usage = 60000000
system_cpu_usage = 1543430000000
online_cpus = 2
pre_cpu_total_usage = 157615787
pre_cpu_kernelmode_usage = 90000000
pre_cpu_usermode_usage = 60000000
pre_system_cpu_usage = 1541780000000
pre_online_cpus = 2

However, to calculate the actual CPU usage as a percentage. The pre and present stats are included automatically for you.

osquery> SELECT ((
(cpu_total_usage-pre_cpu_total_usage)*1.0 /
(system_cpu_usage-pre_system_cpu_usage)*1.0
) * online_cpus) * 100 AS usage
FROM docker_container_stats WHERE id = '419ff6ca';

Mem
Simply shows the memory allocated to the container versus whats avaliable.

osquery> SELECT name, memory_usage, memory_max_usage, memory_limit FROM docker_container_stats WHERE id = '419ff6ca';            name = /dev_nginx_1
memory_usage = 13541376 # Current container usage
memory_max_usage = 40849408 # The max limit for the container
memory_limit = 2084458496 # The system limit for docker

Disk
Simply shows the number of bytes writen to and from the host disk.

osquery> SELECT name, disk_read, disk_write FROM docker_container_stats WHERE id = '419ff6ca';name = /dev_nginx_1
disk_read = 27443200
disk_write = 4096

Network
Simply shows the number of bytes sent and recived from the container.

osquery> SELECT name,  network_rx_bytes, network_tx_bytes FROM docker_container_stats WHERE id = '419ff6ca';            name = /dev_nginx_1
network_rx_bytes = 13376 # Recived bytes
network_tx_bytes = 3455 # Sent bytes

docker_container_fs_changes

A container id must be provided as part of the WHERE clause.

osquery> SELECT path, (CASE 
WHEN change_type = 'C' THEN 'CREATED'
WHEN change_type = 'A' THEN 'ADDED'
WHEN change_type = 'R' THEN 'REMOVED'
ELSE 'UNKNOWN' END) AS type
FROM docker_container_fs_changes
WHERE id = '10d666fb9752' LIMIT 5;
+----------------+---------+
| path | type |
+----------------+---------+
| /run | CREATED |
| /run/nginx.pid | ADDED |
| /var | CREATED |
| /var/lib | CREATED |
| /var/lib/nginx | CREATED |
+----------------+---------+

Docker networks

docker_network_labels

osquery> SELECT SUBSTR(id, 0, 8) AS id, key, value FROM docker_network_labels LIMIT 5;+---------+----------------------------+-----------+
| id | key | value |
+---------+----------------------------+-----------+
| bfa5d96 | com.docker.compose.network | zercurity |
| bfa5d96 | com.docker.compose.project | dev |
| bfa5d96 | com.docker.compose.version | 1.26.2 |
+---------+----------------------------+-----------+

docker_networks

osquery> SELECT SUBSTR(id, 0, 8) AS id, name, driver, subnet, gateway FROM docker_networks;
+---------+------------+--------+------------------+---------------+
| id | name | driver | subnet | gateway |
+---------+------------+--------+------------------+---------------+
| ebd527e | dev_defa.. | bridge | 192.168.160.0/20 | 192.168.160.1 |
| 7ecbb21 | hive_def.. | bridge | 192.168.0.0/20 | 192.168.0.1 |
| 809b959 | host | host | | |
| bfa5d96 | dev_zerc.. | bridge | 172.22.0.0/16 | 172.22.0.1 |
| b32881c | zercurit.. | bridge | 192.168.224.0/20 | 192.168.224.1 |
| be8e04e | none | null | | |
| 657ee7e | bridge | bridge | 172.17.0.0/16 | 172.17.0.1 |
+---------+------------+--------+------------------+---------------+

Docker volumes

docker_volume_labels

osquery> SELECT * FROM docker_volume_labels LIMIT 5;
+-------------------+----------------------------+-----------+
| name | key | value |
+-------------------+----------------------------+-----------+
| wordpress_db_data | com.docker.compose.project | wordpress |
| wordpress_db_data | com.docker.compose.version | 1.22.0 |
| wordpress_db_data | com.docker.compose.volume | db_data |
+-------------------+----------------------------+-----------+

docker_volumes

osquery> SELECT SUBSTR(name, 0, 8) AS id, driver, mount_point AS mount FROM docker_volumes LIMIT 5;+---------+--------+-----------------------------------------------+
| id | driver | mount |
+---------+--------+-----------------------------------------------+
| 11bd27d | local | /var/lib/docker/volumes/11bd2..0e7d1820/_data |
| f562970 | local | /var/lib/docker/volumes/f5629..670bbd70/_data |
| fa4bf78 | local | /var/lib/docker/volumes/fa4bf..abe163de/_data |
| 0c17c32 | local | /var/lib/docker/volumes/0c17c..c6fda879/_data |
| 7e5ab04 | local | /var/lib/docker/volumes/7e5ab..2d084acb/_data |
+---------+--------+-----------------------------------------------+

Troubleshoorting

Nothing returned in query

WHERE clause

virtual_table.cpp:966] Table <table_name> was queried without a required column in the WHERE clauseSELECT * FROM docker_container_ports WHERE id = '419ff6ca';

Its all over!

Real-time security and compliance delivered.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store