Series – Part 2: Serverless Architecture – a practical implementation: IoT Device data collection, processing and user interface.

In part one of this series I briefly discussed the purpose of the application to be built and reviewed the IoT local controller & gateway pattern I’ve deployed. To recap, I have a series of IP cameras deployed and configured to send (via FTP) images and videos to a central controller (RaspberryPI 3 Model B). The controller processes those files as they arrive and pushes them to Amazon S3. The code for the controller process can be found on GitHub.

In this post we will move on to the serverless processing of the videos when they arrive in S3.

Continue reading “Series – Part 2: Serverless Architecture – a practical implementation: IoT Device data collection, processing and user interface.”

The future of data is still Polyglot…

… but vendors will fight you every step of the way.

Remember – every vendor wants to get as much of your data in their database as possible. And every data approach (Relational, Document, Key/Value Store and Graph) can be forced to do just about anything; and given the opportunity the vendor will tell you their solution is the right choice. 

The challenge for the modern Enterprise Data Architect is maintaining a cogent point of view about assembling a polyglot solution to make each use case (or micro service) less complex, more scalable and easier improve/enrich over time.

Series – Part 1: Serverless Architecture – a practical implementation: IoT Device data collection, processing and user interface.


AWS Lambda

Serverless architectures are getting a lot of attention lately – and for good reason. I won’t rehash the definition of the architecture because Mike Roberts did a fine (and exhaustive) job over at

However, practical illustrations of patterns and implementations are exceptionally hard to find. This series of posts will attempt to close that gap by providing both the purpose, design and implementation of a complete serverless application on Amazon Web Services.

Part 1 – The setup…

Every application needs a reason to exist – so before we dive into the patterns and implementation we should first discuss the purpose of the application.

Nest wants how much for cloud storage and playback?

I have 14 security cameras deployed, each captures video and still images when motion is detected. These videos and images are stored on premises – but getting them to “the cloud” is a must have – after all if someone breaks in and takes the drive they are stored on all the evidence is gone.

If I were to swap all of the cameras out for Nest cameras cloud storage and playback would cost $2250/year – clearly this can be done cheaper… so…

For several years now I’ve been sending those images – in near real time – to s3. This however, doesn’t allow anyone to review those images and movies simply.

The basic requirements are as follows:

  • Keep a catalog of the images and videos that are uploaded to s3.
  • Provide basic analysis/analytics of the catalog data.
  • Optionally – transcode the videos to a lower bit rate for mobile playback.
  • Provide a responsive web user interface which:
    • Allows specific users to log in
    • See the last x motion detection videos chronologically
    • See the last x videos for a given camera chronologically
    • Allows the end user to select either high definition or reduced resolution playback
  • Does all of this securely with no servers – in this case EC2 instances.

In essence, this is IoT data collection and serverless application delivery for the IoT system.

Installed System

While it isn’t particularly important to the serverless discussion, it is worth discussing the physical systems I have deployed.


All of the cameras are made by Foscam – I predominantly use the C1. Some are connected via wifi and some physically connect to the network. Each is set up to FTP both still images and video when motion is detected.

This post is not an endorsement of Foscam. As a matter of fact, the camera is the least important and interesting part of this setup.

FTP Server

While this series of posts will focus on serverless application architecture, patterns and implementation, there is also a significant IoT data collection use case which requires some local server resources to ochestrate the overall system. In this case we need a local FTP server to serve as the initial landing point for the videos and images sent from the cameras.

Obviously, I’d prefer the cameras sent the videos and images directly to s3 – but I’m also not sure I’d be okay with the security implications of putting an s3 keypair in the camera configuration.

For my purposes I’ve deployed a single RaspberryPi 3 Model B to serve as the FTP Server.

Moving the videos and images to s3

In order to move the videos and images to s3 (from the previously discussed FTP server) I’ve created a python process. This process can be found in this github repository (which will will discuss more in the next post in the series).

The python file is which leverages the configuration config.json which you should create from example-config.json.


The raspberrypi is running VSFTPD and the python daemon reads (tail -f equivalent) the VSFTPD log file looking for uploaded files. It then parses the logged information and discerns the information to put the file in s3.

Note: You’ll notice that the python includes transcoding the Foscam .mkv file to a .mp4 file – this is done to simplify HTML5 video playback.

I won’t go in to great detail on the setup of VSFTPD or the details of how the files are stored in the s3 bucket – but suffice it to say, this is the most complex piece of code you will find in the entire code base.

I’d also note that one raspberrypi handles both VSFTPD, this python daemon and a variety of other tasks easily.

In the next post in this series I’ll describe the target s3 bucket, how that bucket is managed and using AWS Lambda to process the security camera images and videos as they arrive in s3.



The time for consolidation in the DB market has come; RethinkDB to shut down.

Too many databases with too little differentiation. The time has come for consolidation – including more news like RethinkDB shutting down:

Via HackerNews

Document oriented databases will be first – but expect additional consolidation in other segments.

To be clear – we are not heading back to the bad old days of one database (vendor) to rule them all – we will just see definitive winners in the key segments.