Serverless architectures are getting a lot of attention lately – and for good reason. I won’t rehash the definition of the architecture because Mike Roberts did a fine (and exhaustive) job over at MartinFowler.com.

However, practical illustrations of patterns and implementations are exceptionally hard to find. This series of posts will attempt to close that gap by providing both the purpose, design and implementation of a complete serverless application on Amazon Web Services.

Part 1 – The setup…

Every application needs a reason to exist – so before we dive into the patterns and implementation we should first discuss the purpose of the application.

Nest wants how much for cloud storage and playback?

I have 14 security cameras deployed, each captures video and still images when motion is detected. These videos and images are stored on premises – but getting them to “the cloud” is a must have – after all if someone breaks in and takes the drive they are stored on all the evidence is gone.

If I were to swap all of the cameras out for Nest cameras cloud storage and playback would cost $2250/year – clearly this can be done cheaper… so…

For several years now I’ve been sending those images – in near real time – to s3. This however, doesn’t allow anyone to review those images and movies simply.

The basic requirements are as follows:

• Keep a catalog of the images and videos that are uploaded to s3.
• Provide basic analysis/analytics of the catalog data.
• Optionally – transcode the videos to a lower bit rate for mobile playback.
• Provide a responsive web user interface which:
  • Allows specific users to log in
  • See the last x motion detection videos chronologically
  • See the last x videos for a given camera chronologically
  • Allows the end user to select either high definition or reduced resolution playback
• Does all of this securely with no servers – in this case EC2 instances.

In essence, this is IoT data collection and serverless application delivery for the IoT system.

Installed System

While it isn’t particularly important to the serverless discussion, it is worth discussing the physical systems I have deployed.

Cameras

All of the cameras are made by Foscam – I predominantly use the C1. Some are connected via wifi and some physically connect to the network. Each is set up to FTP both still images and video when motion is detected.

This post is not an endorsement of Foscam. As a matter of fact, the camera is the least important and interesting part of this setup.

FTP Server

While this series of posts will focus on serverless application architecture, patterns and implementation, there is also a significant IoT data collection use case which requires some local server resources to ochestrate the overall system. In this case we need a local FTP server to serve as the initial landing point for the videos and images sent from the cameras.

Obviously, I’d prefer the cameras sent the videos and images directly to s3 – but I’m also not sure I’d be okay with the security implications of putting an s3 keypair in the camera configuration.

For my purposes I’ve deployed a single RaspberryPi 3 Model B to serve as the FTP Server.

Moving the videos and images to s3

In order to move the videos and images to s3 (from the previously discussed FTP server) I’ve created a python process. This process can be found in this github repository (which will will discuss more in the next post in the series).

The python file is ftpfiletos3.py which leverages the configuration config.json which you should create from example-config.json.

Setup

The raspberrypi is running VSFTPD and the python daemon reads (tail -f equivalent) the VSFTPD log file looking for uploaded files. It then parses the logged information and discerns the information to put the file in s3.

Note: You’ll notice that the python includes transcoding the Foscam .mkv file to a .mp4 file – this is done to simplify HTML5 video playback.

I won’t go in to great detail on the setup of VSFTPD or the details of how the files are stored in the s3 bucket – but suffice it to say, this is the most complex piece of code you will find in the entire code base.

I’d also note that one raspberrypi handles both VSFTPD, this python daemon and a variety of other tasks easily.

In the next post in this series I’ll describe the target s3 bucket, how that bucket is managed and using AWS Lambda to process the security camera images and videos as they arrive in s3.