Open in app

Sign in

Write

Sign in

Building packages on AWS Lambda

Zercurity
8 min readMar 30, 2018

Zercurity uses Serverless containers to respond to API requests. This is useful as we’re only billed for the API requests we respond to. We have no running infrastructure that sits idle, ready to process requests. Your application is split up into “functions” and these functions are only run once they’re called. They’re also kept in a “warm” state to reduce the time cost, having to start up a container each time.

Zercurity builds custom installers for; Windows, Linux and Mac atop this serverless infrastructure. We use these installers to allow our customers to quickly download and install Zercurity without having to configure anything, enter in secrets or license keys etc. It’s all in a self-contained installer.

Without serverless, it’d be pretty straightforward to create these installers as the server you’re deploying too, is yours. It isn’t destroyed whenever the container expires. Everything persists. Including all your packages and dependencies. This means that any dependencies we require for building; MSIs, RPMs, DEBs and PKGs must be provided within our Lambda function.

The AWS Lambda image is based on RHEL / Fedora. Which comes with a predefined set of binaries and dependencies installed. Which means if you want to add anything you’ll have to compile it and deploy it alongside your application.

How do we do this?

Fortunately, there are a few open source projects which help make building binaries for AWS Lambda pretty straightforward. There is a GitHub project that mirrors the AWS Lambda image as a Docker image. Which you can shell onto and compile your build dependencies. Copy them off and then package them up, alongside your AWS Lambda function.

Assuming you’ve already got Docker installed. You can fire up the Docker image by running:

docker run -v /tmp/build:/build -it lambci/lambda:build bash

This will create a new Docker instance with the AWS Lambda image. It’ll also create a shared volume between the host and the container. You can then copy whatever you’ve compiled into the /build. Which will then be accessible from /tmp/build on your host machine.

Building Debian DEBs on AWS Lambda

Debian packages are pretty straightforward. We opted not to compile dpkg-deb in order to build our deb package. You only need ar and tar to create a deb package, as it only consists of:

  • debian-binary contains version information for the deb. “2.0”
  • control.tar.gz a gzip archive containing all the; install scripts, revision history, rules etc
  • data.tar.gz a gzip archive containing the files to be installed. The archive directory structure will need to mirror the Debian filesystem.

You can find more information here on the structure of a Debian package.

ar -r zercurity.deb debian-binary control.tar.gz data.tar.gz

With all that said we still need to compile ar within our AWS Lambda Docker image in order to create our package.

docker run -v /tmp/deb:/deb -it lambci/lambda:build bashyum -y install wget
wget http://ftp.gnu.org/gnu/binutils/binutils-2.29.tar.gz
tar -zxvf binutils-2.29.tar.gz
cd binutils-2.29/
./configure
make
make installcp `which ar` /deb

The compiled ar binary will now be available in /tmp/deb on your host.

Building Mac OSX PKGs on AWS Lambda

Creating Mac OSX packages are a little trickier to build and sign. There are only two binaries we need to compile on our AWS Lambda Docker container.

docker run -v /tmp/osx:/osx -it lambci/lambda:build bashyum -y install libxml2-devel wget bzip2-devel
wget https://github.com/downloads/mackyle/xar/xar-1.6.1.tar.gz
tar -zxvf xar-1.6.1.tar.gz
cd xar-1.6.1
./configure
make
make installcd ..wget https://github.com/hogliux/bomutils/archive/debian/0.2-1.tar.gz
tar -zxvf 0.2-1.tar.gz
cd bomutils-debian-0.2-1/
make
make installcp `which xar` /osx/.
cp `which mkbom` /osx/.

The compiled xar and mkbom binaries will now be available in /tmp/osx on your host.

Creating an OSX package bundle

All the major walkthroughs regarding creating OSX packages involve using the X-Code distributed binaries to create your OSX package.

This will hopefully provide you with a quick walkthrough on how to create an OSX package bundle. A bundle is a collection of packages that get installed as a simultaneously. In the example below, we’ll only be creating one package as part of our bundle but you can create a bundle together as many as you like.

Let's get started by creating our first package.

OSX Package

zercurity.pkg/Scripts
Create a new temporary directory that contains two files called “preinstall.sh” and “postinstall.sh”. You can pop whatever scripts you like in here. As the name suggests these two scripts will be executed pre and post install.

(cd '/path/to/_scripts' && find . | cpio -o --quiet --format odc --owner 0:0 | gzip -c) > 'Scripts'

zercurity.pkg/Payload
Create a new temporary directory that contains the file structure that you’re after including the files that you want copied across during the install. Your pre and post install scripts will be wrapped around this event.

(cd '/path/to/_payload' && find . | cpio -o --quiet --format odc --owner 0:0 | gzip -c) > 'Payload'

zercurity.pkg/PackageInfo
Create a new file with the following content. Ensure you update the XML attributes that reflect your Scripts and Payload archives.

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<pkg-info postinstall-action="none" format-version="2" identifier="com.zercurity" version="0.1" generator-version="0.1" install-location="/" auth="root">
    <payload numberOfFiles="10" installKBytes="1337"/>
    <bundle-version/>
    <upgrade-bundle/>
    <update-bundle/>
    <atomic-update-bundle/>
    <strict-identifier/>
    <relocate/>
    <scripts>
        <preinstall file="preinstall.sh"/>
        <postinstall file="postinstall.sh"/>
    </scripts>
</pkg-info>

zercurity.pkg/Bom
Create the OSX Bom file. This creates a “bill of materials” for the installer.

mkbom -u 0 -g 0 /path/to/_payload Bom

This gives the basic structure to our package which will be included within our bundle.

OSX Bundle

Our bundle will encapsulate our “zercurity.pkg” package.

Distribution
Copy and paste the following into your “Distribution” file. Make a note here of the items highlighted in bold as they are explained under the Distribution example file.

<?xml version="1.0" encoding="UTF-8"?>
<installer-gui-script minSpecVersion="1"><title>Zercurity Installer</title>
  <options allow-external-scripts="no" customize="allow" />
  <domains enable_anywhere="true" />
  <background file="background.png" alignment="topleft" scaling="none" />
  <license file="license.txt" />
  <welcome file="welcome.txt" />
  <readme file="readme.txt" />
  <conclusion file="conclusion.txt" /><choices-outline>
    <line choice="zercurity" />
  </choices-outline>><choice description="zercurity" id="zercurity" title="Zercurity pre/post install scripts">
    <pkg-ref id="com.zercurity" />
  </choice>
  <pkg-ref auth="root" id="com.zercurity" installKBytes="28" version="0.1">#zercurity.pkg</pkg-ref>
  <pkg-ref id="com.zercurity">
    <bundle-version />
  </pkg-ref>
</installer-gui-script>

background.png An image that, in this example will sit in the top left of the OSX installer window.

The text files below will be shown in the order they are listed. All text will appear as a scrollable text window. All these text files are optional.

welcome.txt Welcome message.

readme.txt Quick readme or overview on what to expect from the installer.

license.txt License agreement for the user to accept or decline.

conclusion.txt Will be shown following a successful installation. This can contain a message with a link back to your website for example or manual post-installation steps.

The final item highlighted in bold #zercurity.pkg (including the #) is the directory containing your Scripts, Payload, Bom and Distribution file.

In all, you should end up with the following directory structure:

/zercurity.pkg/Scripts
/zercurity.pkg/Payload
/zercurity.pkg/PackageInfo
/zercurity.pkg/Bom
/Resources/en.lproj/background.png
/Resources/en.lproj/license.txt
/Resources/en.lproj/welcome.txt
/Resources/en.lproj/readme.txt
/Resources/en.lproj/conclusion.txt
/Distribution

Llooking good? Let us create our package. The -C flag is the path to the directory structure above.

xar --compression none -c -C /path/to/sources -f 'zercurity.pkg' .

If all is well. You’ll end up with an unsigned Mac OSX package. To sign the package you can use OpenSSL.

Building RHEL RPMs on AWS Lambda

In order to build your .spec file as an rpm you’ll require the rpmbuild binary which thankfully does come with the AWS image. As the AWS Lambda image is a derivative of RHEL Fedora. However, you’ll still need to compile the rpmbuild binary as the older version of the software has a bug where you can’t build packages as a non-root user. You’ll end up with this error.

error: Bad owner/group rpmbuild

Which is annoying. This will let you build the most recent (at the time of writing) version of rpmbuild.

docker run -v /tmp/rpm:/rpm -it lambci/lambda:build bashyum -y install wget file-devel nss-devel popt-devel libarchive-devel db4-develwget https://github.com/rpm-software-management/rpm/archive/rpm-4.14.0-release.tar.gz
tar -zxvf rpm-4.14.0-release.tar.gz
cd rpm-rpm-4.14.0-release/
sh autogen.sh -prefix=/usr -without-lua
make
make installcp `which rpmbuild` /rpm/.

The compiled rpmbuild binary will now be available in /tmp/rpm on your host. For more information on using rpmbuild to turn your .spec file into an rpm check out this walkthrough.

Building Windows MSI on AWS Lambda

This might surprise you but you can actually build Windows (MSI) installers on Linux. There are a few ways to do this but we chose to use msitools which mirrors a subset of the Wix functionality.

I’ve left this one until last as it’s slightly trickier to get this all working as there are a few dependencies that need to be compiled from source (please let me know if any of the links stop working).

Here we go (this will take about 30 minutes to compile) …

docker run -v /tmp/msitools:/msitools -it lambci/lambda:build bashyum -y install wget glib2-devel gtk-doc libxml2-devel gobject-introspection libuuid-devel libmount-devel libtool-ltdl-devel curl-devel libxml2-devel libxslt-devel glib2-devel libIDL-devel dbus-devel dbus-glib-devel polkit-devel flex popt-devel bison bzip2-devel libgcrypt-devel libcanberra-devel gtk2-devel libart_lgpl-devel libglade2-devel libtasn1-tools libxklavier-devel libsoup-devel icon-naming-utils unique-devel libcanberra-gtk2 libcanberra-devel libwnck-devel librsvg2-devel libSM-devel libXdamage-devel gobject-introspection-devel upower-devel intltool  libtasn1-devel libtool gamin-devel rarian-devel dconf-devel libsecret-devel libgnome-keyring-develwget https://github.com/GNOME/glib/archive/2.55.1.tar.gz
tar -zxvf 2.55.1.tar.gz
cd glib-2.55.1/
sh autogen.sh -with-pcre=internal
make
make installcd ..wget https://graphviz.gitlab.io/pub/graphviz/stable/SOURCES/graphviz.tar.gz
tar -zxvf graphviz.tar.gz
cd graphviz-2.40.1/
sh autogen.sh
make
make installcd ..export PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig/:/usr/local/lib/pkgconfig/wget https://download.gnome.org/sources/vala/0.39/vala-0.39.7.tar.xz
tar xf vala-0.39.7.tar.xz
cd vala-0.39.7/
./configure
make
make installcd ..wget http://www.python.org/ftp/python/3.6.0/Python-3.6.0.tar.xz
tar xf Python-3.6.0.tar.xz
cd Python-3.6.0/
./configure
make
make installcd ..wget https://bootstrap.pypa.io/get-pip.py
python3.6 get-pip.py
pip3.6 install pathlib
pip3.6 install mesonwget https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip
unzip ninja-linux.zip
cp ninja /usr/bin/
chmod +x /usr/bin/ninjawget https://download.gnome.org/sources/gcab/1.0/gcab-1.0.tar.xz
tar xf gcab-1.0.tar.xz
mkdir build/
cd gcab-1.0/
meson . ../build/
cd ../build/
ninja all
ninja installcd ..wget http://ftp.gnu.org/pub/gnu/gettext/gettext-0.19.8.tar.gz
tar -zxvf gettext-0.19.8.tar.gz
cd gettext-0.19.8/
sh autogen.sh --skip-gnulib
./configure
make
make installcd ..wget https://github.com/GNOME/libgsf/archive/LIBGSF_1_14_42.tar.gz
tar -zxvf LIBGSF_1_14_42.tar.gz
cd libgsf-LIBGSF_1_14_42/
sh autogen.sh
make
make installcd ..wget http://ftp.gnome.org/pub/GNOME/sources/msitools/0.97/msitools-0.97.tar.xz
tar xf msitools-0.97.tar.xz
cd msitools-0.97/
sh autogen.sh
make
make installcd ..cp `which msibuild` /msitools/.
cp `which wixl` /msitools/.

For code signing, you can also compile osslsigncode. Which will allow you to buy a code signing certificate and sign your complied MSI.

wget https://10gbps-io.dl.sourceforge.net/project/osslsigncode/osslsigncode/osslsigncode-1.7.1.tar.gz
tar -zxvf osslsigncode-1.7.1.tar.gz
cd osslsigncode-1.7.1
./configure
make
make installcp `which osslsigncode` /msitools/.

With wixl and msibuild you can follow these docs on how to build an MSI installer from a wix xml spec file.

Done and dusted

Phew, well done if you got through all of that. Feel free to get in touch if you need any help working through any of that.

Docker
AWS Lambda
Aws Api Gateway
Linux
Wix

--

--

Zercurity

Written by Zercurity

169 Followers

Real-time security and compliance delivered.

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams