Series – Part 3: Serverless Architecture – a practical implementation: Serverless REST API

In part two of this series I discussed creating a serverless data collection and processing fabric for an IoT deployment. To recap, we’ve now reviewed the local devices and controller/gateway pattern for the security cameras deployed. We’ve also discussed the Amazon Web Services infrastructure deployed to collect, process and catalog the data generated by the security cameras.

In this post we will cover the creation of a serverless REST API.

The image below provides an overview of the serverless web application that will be implemented using this REST API:

Serverless REST API Creation

In the previous post we discussed getting our security videos and images to S3 and processing them via a Lambda function to generate a metadata catalog of the videos in DynamoDB. Now we can create a serverless REST API over the top of the data.

Creating a Lamdba Function for REST Services

As with the Lambda function created in part 2 of the series the first step is to create an IAM role enabling the access required. In order to do that we need to discuss the basic capabilities of the REST API Lambda function.

1. Read the latest security videos – regardless of camera.
2. Read the latest security videos for a particular camera.
3. Generate singed URLs for the videos in S3 to allow playback.
4. Log information to CloudWatch

The first step is to create a new role – I called mine securityMicroserviceRole. Once the role is created you can create and assign each of the following policies.

For all of the policies below you’ll want to replace the “Resource” ARN info with the ARN for your DynamoDB table or S3 Bucket.

Policy #1: Get the latest security videos

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GetOnSecurityAlarmVideos",
            "Effect": "Allow",
            "Action": [
                "dynamodb:GetItem",
                "dynamodb:BatchGetItem",
                "dynamodb:Query"
            ],
            "Resource": "arn:aws:dynamodb:!!Your ARN Info!!:table/security_video_timeline"
        }
    ]
}

 Policy #2: Get the latest security videos by camera

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GetOnSecurityAlarmVideos",
            "Effect": "Allow",
            "Action": [
                "dynamodb:GetItem",
                "dynamodb:BatchGetItem",
                "dynamodb:Query"
            ],
            "Resource": "arn:aws:dynamodb:!!Your ARN Info!!:table/security_alarm_videos"
        }
    ]
}

Policy #3: Generate singed URLs for videos in S3 allowing playback

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:Get*",
        "s3:List*"
      ],
      "Resource": "arn:aws:s3:!!Your ARN Info!!/*"
    }
  ]
}

Policy #4: Log information to CloudWatch

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:!!Your ARN Info!!:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:!!Your ARN Info!!:log-group:/aws/lambda/last5Videos:*"
            ]
        }
    ]
}

Now we can create the Lambda function that will provide our REST API services.

Lambda Function Creation

The process is similar to what we did in part 2 of the series – but in this case we want our Lambda function to be triggered by the AWS API Gateway, not S3. Once again we will select the Blank Template.

In the “Configure Triggers” section we will select API Gateway:

You can use the default “Deployment stage” and AWS IAM in “Security”.

In the “Configure Function” section we will name our function and select the Python runtime:

Similar to in part 2 you will cut and paste the Lambda Function code found in GitHub here into the code box.

In the “Lambda function handler and role” section we will select Choose an existing role and then select the role created earlier in this post.

In the advanced settings I would advise giving your function more execution time – 12 to 14 seconds should be sufficient for testing/debugging. In our case (serverless) we do not need our Lambda function to connect to any EC2 instances. If you do require that select the VPC those instances reside in.

Hit “Next” and “Create Function” and you have created the REST API Lambda Function.

Repeat these same Lambda Function Creation steps for the getCameraNames Lambda function found in the GitHub repository here.

For the moment we will skip testing the Lambda function and move on to configuring API Gateway to field API requests and invoke our function.

Configuring API Gateway for REST API Lambda Function

Before we start configuring our API it is worth an overview of the resources (API capabilities) we will expose:

GET /cameras

The /cameras resource will return a list of the camera names. This is done by walking the S3 bucket and finding all the camera names (see part 2 of the series for details). It is implemented by the following code in the Lambda Function getCameraNames:

""" Gets the current list of cameras by parsing the s3 bucket objects. """
def lambda_handler(event, context):
    """ Lambda Hander """
    import boto3

    s3_client = boto3.client('s3')
    items = s3_client.list_objects(Bucket='security-alarms', Prefix='patrolcams/', Delimiter='/')

    out_list = []
    for obj in items.get('CommonPrefixes', []):
        camera = obj.get('Prefix')
        camera = camera.replace('patrolcams/', '')
        camera = camera.replace('/', '')
        out_list.append(camera)
        # end for

    return out_list

GET /lastfive

The /lastfive resource will return the last 5 (or any other configured number of) security videos regardless of camera name in order from newest to oldest. This method will also support optional GET parameters for:

• The number of videos to return – num_results
• The date the video file was captured – video_date

It is implemented by the following code in the Lambda Function last5Videos:

        # Must be a video timeline request

        # defaults:
        num_results = 10
        video_date = time.strftime('%Y-%m-%d')
        print(video_date)

        if 'querystring' in event['params']:
            if 'video_date' in event['params']['querystring']:
                video_date = event['params']['querystring']['video_date']
            # Fin
            if 'num_results' in event['params']['querystring']:
                num_results = int(event['params']['querystring']['num_results'])
            # Fin
        # Fin

        # Execute the query

        vid_table = dyndb.Table('security_alarm_videos')
        response = vid_table.query(
            TableName="security_video_timeline",
            Select='ALL_ATTRIBUTES',
            KeyConditionExpression=Key('capture_date').eq(video_date),
            ScanIndexForward=False,
            Limit=num_results,
        )

GET /lastfive/{camera}

The /lastfive/{camera} resource will return the last 5 videos for the camera specified in the {camera} path variable (e.g,. /lastfive/office).  It is implemented by the following code in the Lambda Function last5Videos:

        camname = event['params']['path']['camera']
        vid_table = dyndb.Table('security_alarm_videos')
        response = vid_table.query(
            TableName="security_alarm_videos",
            Select='ALL_ATTRIBUTES',
            KeyConditionExpression=Key('camera_name').eq(camname),
            ScanIndexForward=False,
            Limit=5,
        )

To summarize – our API will look like:

http://our.api.host.com/api_name/cameras
http://our.api.host.com/api_name/lastfive
http://our.api.host.com/api_name/lastfive/camera

Create the API

Open the AWS API Gateway console and select “Create API”.

Select “New API” and provide an API name and description then press “Create API”.

Now we can create our API resources – lastfive and cameras – per the specification above.

On the “Create Resource” screen fill in the resource name and resource path:

Also check “Enable API Gateway CORS” to allow API requests from other domains. Then click “Create Resource”.

Initially the resource is created with only the OPTIONS method – we now need to create the GET method:

We then map our Resources GET method to our Lambda Function and press “Save”. You can then test the API resource & method by selecting “Test”:

The test will output the HTTP response and camera list (provided your S3 bucket is created and the getCameraNames has the correct bucket and object path) as well as the logging of the request:

Now repeat these steps for the the lastfive and the GET method for that resource. You should map this to the securityRestAPI Lambda function created above.

Testing this GET method however results in the following error:

{
  "stackTrace": [
    [
      "/var/task/lambda_function.py",
      15,
      "lambda_handler",
      "if 'camera' in event['params']['path']:"
    ]
  ],
  "errorType": "KeyError",
  "errorMessage": "'params'"
}

In order to fix this we need to configure the API Gateway to pass the correct information to our Lambda function.

NOTE: This API was developed and deployed several months ago. Since that time AWS has enhanced the integration of API Gateway and Lambda. All of the methods described in this post are accurate as of the publish date.

In order to do that we will need to configure the “Integration Request” section of the GET method:

At the bottom of this section select “Add mapping template” and type in the content type application/json and select the check mark.

You will see a pop up asking that you secure the integration – click on “Yes, secure this integration”. Now we want to generate a template using “Method Request passthrough”:

The template will auto-generate (you don’t need to fill in the code box). Click “Save”, return to the lastfive resource GET method and test again. The test should be successful.

We can now add the variable {camera} sub-resource of the lastfive resource:

Before testing the child resource – make sure you configure “Integration Request” exactly like we did for the lastfive resource.

When you test the new lastfive/{camera} GET method you’ll be prompted to supply the camera name as follows:

The last step it to configure the GET parameters on the lastfive resource as follows:

Expand the “URL Query String Parameters” section, click “Add query string” and create the num_results and video_date parameters.

Summary

We’ve now created a complete serverless REST API using AWS API Gateway and Lambda functions. This API gives applications access to the IoT device data we ingested into the system using serverless processing in part two of this series.

In part four of the series we will secure the REST API and put it in production so we can build applications using it (in part 5).