File retrieval with Osquery using carves on Zercurity
With remote server support in Osquery you can remotely grab files from systems using the
carves table within Osquery as simply as:
path LIKE '/Users/tim/Downloads/%'
AND carve = 1
The query above will fetch all the files residing within the users home directory. The wildcard
% can be used anywhere within the directory path. Carving is supported across Windows, Mac OSX and Linux.
Osquery will round up all the files that match the path and bundle them up into an archive as either a
.zst and uploaded back to the remote server.
Extracting the archives
To open your carved archive you’ll need to use either
tar to extract the archive.
untar bundle.tar # extract TAR archive
tar -I zstd -xvf bundle.zst # extract ZST archive
If you get the following error:
tar (child): zstd: Cannot exec: No such file or directory . You’ll need to install Facebook’s zstd package.
sudo apt install zstd
Osquery carver remote server settings
In order to get carving working you will require a remote Osquery server. You can download an example server here. Or you can spin up Zercurity using docker-compose here (which is configured with carving enabled by default).
Alongside these flags for letting the Osquery agent know about the remote server. The flags highlighted in bold are required. However, for ad-hoc carves you’ll need to configure a distributed TLS endpoint.
Osquery carver server
There are two important resources that are needed in order for the client to upload data back to server.
The /start resource
The start resource is initially hit by the client to retrieve a session token and let the server know how many blocks will be uploaded the server. As the archive is chunked up into blocks as specified by the
carver_block_size. The requesting payload is as follows:
block_count is the number of blocks the archive has been divided up into. During the
/upload resource being called the client will provide a
block_id to indicate the current block being uploaded. Once all 17 blocks have been uploaded they can be stitched together to form the final file.
block_size is the clients
carver_block_size as configured by the remote server of local config file.
carve_size is the total payload size which can be checked against the final size of stitched together archive.
carve_id is the
UUID given to the current carve job. This is not supplied in the
request_id is name or identifier of the query that kicked off the carve job. Zercurity uses
UUIDs for each query. However, this may just the the query name depending on how your remote server is configured.
Note: There is currently a bug where the
request_id may be incorrect and the
id of another job currently in-flight is used instead. Please see the GitHub issue for more information.
node_key is the shared secret used to authenticate the client with the remote server. This is also not provided during the subsequent
The client expects a
session_id . This should be a unique token to allow the server to both authenticate the client and identify the current carve job for the subsequent
/upload requests. Using a HMAC token with a payload is a useful way of passing data to the client that can be relayed back to the server to help identify the current carve.
The /upload resource
/start request has completed and a
session_id returned. The client will start to upload the necessary blocks. The payload for each request will be as follows:
Given the cave task
block_id is the current segment being uploaded to the server. This will form part of the total blocks as per the
session_id is the token provided to the client in the response during the
As before the
request_id is the identifier of the requesting cave.
data field houses a base64 encoded value of the current segment. This needs to be decoded and then either stored to disk or in memory
The Osquery client will not post a final request to let the server know it has finished uploading all the blocks. Instead the server needs to keep track of all the blocks uploaded. Once all the blocks have been received. Matching the final
block_count . All the uploaded blocks can be stitched back together to form the final archive.
The last step is to work out the archive type. Which will be either a
.zst archive. By checking the first 4 bytes of the file for the following magic bytes:
Below is some example python code for pulling all the blocks together and working out the archive file format:
file_ext = 'tar'
file_size = 0
fd, path = tempfile.mkstemp()
f = open(fd, 'wb')
for block_id in range(0, carve.block_count):
raw_data = get_block(carve.uuid, block_id)
file_size += len(raw_data)
if block_id == 0 and raw_data[0:4] == b'\x28\xB5\x2F\xFD':
file_ext = 'zst'
except Exception as e:
return path, file_ext, file_size
Feel free to reach out directly if you have any questions.