Connection (network socket) monitoring with Osquery

Osquery offers a number of ways to monitor outbound network connections from your Windows, Linux or Mac OS hosts. This post was written using Osquery 4.5.1. Support for new tables and platforms may have been added since this article was posted. Please check the Osquery website for the latest query schema.

Socket events captured in Zercurity with Osquery

A little bit about network protocols

Firstly, if you’re not that familiar with networking protocols there are a few things of note. As all the tables we’ll be exploring in this post will use numerical representations for address families and protocols. For example:

osquery> SELECT DISTINCT family, protocol FROM process_open_sockets;
+--------+----------+
| family | protocol |
+--------+----------+
| 1      | 17       |
| 1      | 6        |
| 2      | 6        |
| 2      | 17       |
+--------+----------+

Whilst it looks fairly daunting, if we take each column. Firstly, the address protocol family — which for all intensive purposes the two most common values you’ll see are for IPv4 (1) and IPv6 (2). A full list is available here:

Address Family Numbers

Similarly, for the protocol column we can do the same. Again, the two most common numbers you’ll see are TCP (6) and UDP (17) you may also encounter ICMP (1), HOPOPT (0) and maybe GRE (47) which is mainly used for VPN traffic. A full breakdown of all the protocol numbers is available here:

Protocol Numbers

Armed with these two references we can actually convert these values on the fly with SQL. However, I’d recommend converting the numbers in code post query. As storing them numerically will give you better performance for searching and sorting.

osquery> SELECT (
  CASE family 
  WHEN 2 THEN 'IP4' 
  WHEN 10 THEN 'IP6' 
  ELSE family END
) AS family, (
  CASE protocol 
  WHEN 6 THEN 'TCP' 
  WHEN 17 THEN 'UDP' 
  ELSE protocol END
) AS protocol, local_address, local_port, 
  remote_address, remote_port 
FROM process_open_sockets 
WHERE family IN (2, 10) 
AND protocol IN (6, 17) 
LIMIT 4;
+--------+----------+------------+--------+-------------+--------+
| family | protocol | local_addr | l_port | remote_addr | r_port |
+--------+----------+------------+--------+-------------+--------+
| IP4    | TCP      | 192.168..  | 56493  | 140.123..   | 443    |
| IP4    | TCP      | 192.168..  | 55620  | 35.123..    | 443    |
| IP4    | TCP      | 192.168..  | 55210  | 192.116..   | 443    |
| IP4    | TCP      | 192.168..  | 55527  | 34.111..    | 443    |
+--------+----------+------------+--------+-------------+--------+

Osquery socket tables

There are a few key tables used to give you access to the socket events being created on your computer. First and foremost is the socket_events table which will give you a complete audit of connections for both Linux and Mac OS as of Osquery 4.5.1. However, the socket_events table does require some additional configuration which we’ll dive into. If however, you’re looking for a quicker insight into what connections processes are marking then listening_ports and process_open_ports require no additional configuration and work across all platforms.

socket_events

This is the most relevant table for tracking system network connections to and from the system. Unfortunately, as of 4.5.1 the socket_events table only works with Linux and Mac OS. There are however, a few additional command line arguments you need to add in order for the socket_events table to work. Otherwise, you’ll get this error:

W1115 virtual_table.cpp:967 Table socket_events is event-based but events are disabled
W1115 virtual_table.cpp:974] Please see the table documentation: https://osquery.io/schema/#socket_events

Note (Mac OS): If you’re running Mac OS you’ll need to ensure that openbsm is correctly configured. If you check your sudo nano /etc/security/audit_control it should resemble this:

#
# $P4: //depot/projects/trustedbsd/openbsm/etc/audit_control#8 $
#
dir:/var/audit
flags:ex,pc,ap,aa,lo,ad,nt
minfree:5
naflags:no
policy:cnt,argv,arge
filesz:2M
expire-after:10M
superuser-set-sflags-mask:has_authenticated,has_console_access
superuser-clear-sflags-mask:has_authenticated,has_console_access
member-set-sflags-mask:
member-clear-sflags-mask:has_authenticated

You’ll want to ensure that the flags configuration option has the nt option set. A full list of of flags can be found here. The nt flag enables the auditing of socket events. Once you’ve updated the file you can either run audit -s or restart the system.

183:AUE_SOCKET:socket(2):nt
184:AUE_SENDTO:sendto(2):nt

Note (Linux): If you’re running Linux however, you firstly need to ensure that auditd is installed.

sudo apt install auditd

Lastly, you’ll also need to use the osqueryd or osqueryi command with the following arguments in order to see the audited socket events:

Mac OSX:
sudo osqueryd -S --disable_events=false --audit_allow_sockets --disable_audit=false
Linux:
sudo osqueryi --audit_allow_config=true --audit_allow_sockets=true --audit_persist=true --disable_events=false

Note (Troubleshooting): Firstly the --verbose flag can be added to both the Mac and Osquery command line to enable additional output. You can also double check that the socket_events table is active like so:

osquery> SELECT publisher, type, subscriptions, events, refreshes FROM osquery_events WHERE name = 'socket_events' AND active = 1;
+-----------+------------+---------------+--------+-----------+
| publisher | type       | subscriptions | events | refreshes |
+-----------+------------+---------------+--------+-----------+
| openbsm   | subscriber | 2             | 0      | 0         |
+-----------+------------+---------------+--------+-----------+

Note (Troubleshooting Mac OS): If you’re using Mac OS you can also check that openbsm is enabled and working correctly:

sudo ls -l /var/audit/
total 17520
2097381 15 Nov 20201115142839.20201115142901
2098197 15 Nov 20201115142901.20201115142925
2100045 15 Nov 20201115142925.20201115142958
2097392 15 Nov 20201115142958.20201115143027
 446409 15 Nov 20201115143027.not_terminated
     40 15 Nov current -> /var/audit/20201115143027.not_terminated

These openbsm audit files can be parsed using the following command to check that socket events are being correctly traced:

sudo praudit /var/audit/current | grep socket
socket-inet,2,22443,192.168.1.110
socket-inet,2,22443,192.168.1.110
socket-inet,2,22443,192.168.1.110

Note (Troubleshooting Linux): If you get the netlink error on Linux its because auditd is installed and enabled.

I1115 auditdnetlink.cpp:647] Failed to set the netlink owner

You’ll need to use this command instead to let Osquery know to use auditd instead of the netlink socket.

sudo osqueryi --audit_allow_config=true --audit_allow_sockets=true --audit_persist=false  --disable_audit=false --disable_events=false

--disable_audit=false By default this is set to true and prevents Osquery from opening the kernel audit's netlink socket. By setting it to false, we are telling Osquery that we want to enable auditing functionality.

--audit_persist=true By default this is true and instructs Osquery to “regain” the audit netlink socket if another process also accesses it. However, you should do your best to ensure there will be no other program running which is attempting to access the audit netlink socket.

Querying the socket_events table

Phew! — and with all that you should now be able to query the socket_events table.

osquery> SELECT pid, remote_address AS address, 
  remote_port AS port, family, path, time AS timestamp
FROM socket_events
WHERE remote_address <> ""
  AND remote_port != 0
  AND pid > 0;
+------+----------+-------+--------+------------------+------------+
| pid  | address  | port  | family | path             | timestamp  |
+------+----------+-------+--------+------------------+------------+
| 433  | 18.203.. | 443   | 2      | /Systtem.parsecd | 1605449986 |
| 443  | 2a02:2.. | 47873 | 10     | /Syste.commerce  | 1605449986 |
| 443  | 2a00:2.. | 47873 | 10     | /System.commerce | 1605449986 |
| 405  | 2a01:b.. | 47873 | 10     | /Syste.Spotlight | 1605449989 |
+------+----------+-------+--------+------------------+------------+

process_open_sockets

This table will show you all the open socket connections in use by processes on the system. This table will show all the inbound and outbound connections to and from running processes. Please be aware this will only give you a snapshot in time and not a complete audit of all the socket events.

osquery> SELECT pos.pid, local_address, local_port, 
  remote_address, remote_port, family, protocol, 
  COALESCE(NULLIF(pos.path,''), p.path) AS path
FROM process_open_sockets AS pos
INNER JOIN processes AS p ON p.pid = pos.pid
WHERE remote_address <> ""
  AND remote_port != 0
  AND pos.pid > 0
LIMIT 5;
+------+-------+-------+-------+-------+--------+-------+----------+
| pid  | laddr | lport | raddr | rport | family | proto | path     |
+------+-------+-------+-------+-------+--------+-------+----------+
| 1135 | 192.. | 56493 | 140.. | 443   | 2      | 6     | ..firefox |
| 1135 | 192.  | 55620 | 35..  | 443   | 2      | 6     | ..firefox |
| 1135 | 192.  | 56536 | 104.. | 443   | 2      | 6     | ..firefox |
| 1135 | 192.  | 55527 | 34..  | 443   | 2      | 6     | ..firefox |
| 1135 | 192.  | 56531 | 216.. | 443   | 2      | 6     | ..firefox |
+------+-------+-------+-------+-------+--------+-------+----------+

listening_ports

Lastly, the listening ports table will show you all the current running processes with listening (bound) network sockets and ports. Similar to the lsof command on Linux. Whilst this table won’t give you any network events. It can still be useful to at least show what, if any what ports are open on the system. Which can be tied back to netflow events on the network if you have monitoring there. Which can be a better strategy to monitoring network events rather than taxing each host.

The listening_ports is currently supported across all platforms.

osquery> SELECT p.name, address, port, family, protocol, 
  COALESCE(NULLIF(pos.path,''), p.path) AS path
FROM listening_ports AS pos
INNER JOIN processes AS p ON p.pid = pos.pid
WHERE address <> ""
  AND port != 0
  AND pos.pid > 0
LIMIT 5;
+----------------+----------+-------+--------+-------+-------------+
| name           | addr     | port  | family | proto | path        |
+-----------------+---------+-------+--------+-------+-------------+
| SystemUIServer | 0.0.0.0  | 57645 | 2      | 17    | /System..   |                                                                                                
| postgres       | ::1      | 5432  | 10     | 6     | /Applic..   |
| postgres       | 127.0..  | 5432  | 2      | 6     | /Applic..   |
| trezord        | 127.0..  | 21325 | 2      | 6     | /Applic..   |
| Chrome Helper  | 0.0.0.0  | 5353  | 2      | 17    | /Applic..   |
+----------------+----------+-------+--------+-------+-------------+

Its all over

I hope that’s helped get you up and running with connection monitoring on Osquery. If you need some help do drop a comment or get in touch.