API Gateway, Dynamodb Global Tables and Route 53 latency-based routing policy

This is a tutorial about how to deploy an api using Api Gateway, Dynamodb Global Tables and Route 53. The goal is to have low latency for users of the api no matter where they are in the world.

tl;dr

We use AWS SAM to deploy an api with Api Gateway to multiple regions, enable Dynamodb Global Tables for the same regions and create a latency-based routing policy in Route 53 to route traffic to the region with the lowest latency.

"An abstract painting of many computers flying around earth" - DALL-E

"An abstract painting of many computers flying around earth" - DALL-E

Open Table of contents

0. Motivation
1. Prerequisites
2. Setting up S3
3. Setting up DynamodDb
4. Setting up API Gateway
5. Setting up Route 53
6. Conclusion
Addon - How to add another region

0. Motivation

We are already using AWS Api Gateway and DynamoDb for our REST api and persistence layer. So far we have been deploying our application in one region and configured the api as edge-optimized. This brings a latency performance boost for users outside of the origin region, but it is actually slightly slower for users of the origin region, because their requests get routed to the edge network first before they enter the regional network even though they could have entered the regional network straight away.

Luckily, DynamoDb provides multi-region support out of the box with Global Tables. Therefore the only challenge is to deploy our application to all of the desired regions and route traffic based on latency to the nearest region. Our goal is to have one end point https://api.example.com that serves data from our DynamoDb table with the lowest latency depending on where the request originates from.

Before writing this post we asked about this topic on Reddit and Stackoverflow, but got no response or not satisfying responses:

1. Prerequisites

AWS SAM CLI installed

2. Setting up S3

AWS SAM uses S3 to store templates and artifacts of its deployments. Even though the S3 namespace is global, buckets are created in regions. Lambda function need S3 buckets in the region that they are deployed in. Therefore we need a S3 bucket for every region that we want to support. We make our region part of the bucket name, e.g. global-project-infra-[REGION].

us-east-1

aws s3api create-bucket --bucket global-project-infra-us-east-1 --region us-east-1

eu-central-1

aws s3api create-bucket --bucket global-project-infra-eu-central-1 --region eu-central-1   --create-bucket-configuration LocationConstraint=eu-central-1

3. Setting up DynamodDb

DynamoDb provides multi-region support out of the box with Global tables. Global tables are automatically replicated to the configured regions. In our Cloudformation template we can specify our regions in the Replicas list. In order to use global tables we have to also enable DynamoDb streams.

Replication comes with a cost. Every write to a regional Dynamodb table needs to be replicated to all of the other regions. In the us-east-1 region writing costs $1.25 per million write request units. For the same region, replication costs $1.875 per million replicated write request units. That is 50% more than regular WCUs.

table.yaml

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Global project table

Parameters:

  TableName:
    Type: String
    Description: Name of newsletter Dynamodb table

Resources:

  NewsletterTable:
    Type: AWS::DynamoDB::GlobalTable
    Properties: 
      AttributeDefinitions:
        - AttributeName: pk
          AttributeType: "S"
      BillingMode: PAY_PER_REQUEST
      KeySchema:
        - AttributeName: pk
          KeyType: HASH
      Replicas:
        - Region: us-east-1
        - Region: eu-central-1
      StreamSpecification:
        StreamViewType: NEW_AND_OLD_IMAGES
      TableName: !Ref TableName

We build and deploy our template with AWS SAM. We use us-east-1 as region, but since we deploy a global table we could use any region that we want to support.

sam build -t table.yaml && sam deploy --stack-name global-project-table --s3-bucket global-project-infra-us-east-1 --s3-prefix global-project-table --region us-east-1 --capabilities CAPABILITY_IAM --parameter-overrides TableName=global-project-table

4. Setting up API Gateway

We create an api with one end point POST /items.

4.1. Lambda function code

We write a very basic lambda handler in Typescript that writes a (de-facto) unique primary key (pk) and the region to our target Dynamodb table.

src/index.ts

import { APIGatewayProxyHandler } from "aws-lambda";
import * as DynamoDb from "aws-sdk/clients/dynamodb";

const handler: APIGatewayProxyHandler = async () => {
    await new DynamoDb.DocumentClient().put({
        Item: {
            pk: Date.now().toString(),
            region: process.env.REGION
        },
        TableName: process.env.DYNAMODB_TABLE as string
    }).promise();
    return { statusCode: 201, body: "OK" };
};

export default handler;

4.2. Regional custom domain name

The whole purpose of this project is to leverage the capabilities of the individual targeted regions. Therefore we want to configure our api with a regional custom domain name (Please have a look at this Stackoverflow post for a detailed overview of regional vs. edge).

We have to create a regional certificate for api.example.com in every region that we want to support. For this project we have done this manually using the AWS console. We pass the ARN of the certificate as parameter.

4.3. Building api

Since we are using Typescript, we use esbuild for transpiling and bundling our function code.

api.yaml

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Global project api

Parameters:

  TableName:
    Type: String
    Description: Name of newsletter Dynamodb table

  DomainName:
    Type: String
    Description: The domain name that we want to deploy our api to

  RegionalCertificateArn:
    Type: String
    Description: Arn of the regional certificate

Resources:

  ApiGatewayApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: prod

  CustomDomain:
    Type: AWS::ApiGateway::DomainName
    Properties:
      DomainName: !Ref DomainName
      EndpointConfiguration:
        Types:
          - REGIONAL
      RegionalCertificateArn: !Ref RegionalCertificateArn
      SecurityPolicy: TLS_1_2

  CustomDomainMapping:
    Type: AWS::ApiGateway::BasePathMapping
    Properties:
      DomainName: !Ref DomainName
      RestApiId: !Ref ApiGatewayApi
      Stage: !Ref ApiGatewayApi.Stage
    DependsOn:
      - CustomDomain
  
  EmailsEndpoint:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: dist
      Handler: index.default
      Runtime: nodejs16.x
      Environment:
        Variables:
          REGION: !Ref AWS::Region
          DYNAMODB_TABLE: !Ref TableName
      Events:
        ApiEvent:
          Type: Api
          Properties:
            Path: /items
            Method: post
            RestApiId:
              Ref: ApiGatewayApi
      Timeout: 6
      Policies:
        - DynamoDBCrudPolicy:
            TableName: !Ref TableName

Outputs:

  RegionalDomainName:
    Value: !GetAtt CustomDomain.RegionalDomainName

  RegionalHostedZoneId:
    Value: !GetAtt CustomDomain.RegionalHostedZoneId

Our build command looks like this:

esbuild ./src/index.ts --bundle --outdir=dist --platform=node && sam build -t api.yaml

4.4. Deploying api

We need to deploy our api to every region that we want to support.

us-east-1

sam deploy --stack-name global-project-api --s3-bucket global-project-infra-us-east-1 --s3-prefix global-project-api --region us-east-1 --capabilities CAPABILITY_IAM --parameter-overrides TableName=global-project-table RegionalCertificateArn=[REGIONAL_CERTIFICATE_ARN] DomainName=api.example.com

eu-central-1

sam deploy --stack-name global-project-api --s3-bucket global-project-infra-eu-central-1 --s3-prefix global-project-api --region eu-central-1 --capabilities CAPABILITY_IAM --parameter-overrides TableName=global-project-table RegionalCertificateArn=[REGIONAL_CERTIFICATE_ARN] DomainName=api.example.com

Note: Take a look at --s3-bucket. We deploy to buckets that are created in the target region.

5. Setting up Route 53

We want to set up Route 53 so that it routes traffic to the regional end point that has the lowest latency (Latency-based routing). There is not much that we can automate about the routing template. For every region we have to add a AWS::Route53::RecordSet that has four region-specific values:

AliasTarget.DNSName - The regional api endpoint for our custom domain
AliasTarget.HostedZoneId - The host zone id of our regional custom domain
Region - The region where our api is deployed to. Adding the Region field tells Route 53 to apply latency-based routing
SetIdentifier - An identifier that is unique for that record set

We get the dns name and the hosted zone id as output values from our api.yaml files.

routing.yaml

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Global project routing

Resources:

  GlobalProjectRouteUsEast1:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: [HOSTED_ZONE_ID_OF_EXAMPLECOM]
      Name: "api.example.com."
      Type: A 
      AliasTarget:
        DNSName: [DNS_NAME]
        HostedZoneId: [HOSTED_ZONE_ID]
        EvaluateTargetHealth: true
      Region: us-east-1
      SetIdentifier: us-east-1

  GlobalProjectRouteEuCentral1:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: [HOSTED_ZONE_ID_OF_EXAMPLECOM]
      Name: "api.example.com."
      Type: A 
      AliasTarget:
        DNSName: [DNS_NAME]
        HostedZoneId: [HOSTED_ZONE_ID]
        EvaluateTargetHealth: true
      Region: eu-central-1
      SetIdentifier: eu-central-1

We deploy our record sets using AWS SAM;

sam build -t routing.yaml && sam deploy --stack-name global-project-routing --s3-bucket global-project-infra-us-east-1 --s3-prefix global-project-routing --region us-east-1 --capabilities CAPABILITY_IAM

6. Conclusion

Even though we are happy with our results there are a few things to conclude:

If you want to add global tables to n amount of regions your costs for writes will go up at least n times.
Manually deploying to multiple regions requires a lot of operational overhead. Normally we would like to have our entire infrastructure defined in a single template. But since stacks are deployed to a region we need to deploy the same template to different regions. And because some templates depend on the output of stacks from different regions we need to deploy stacks in sequence and in a specific order.
Deploying the same template to different regions can be done with AWS Cloudformation StackSets. We decided to not go with StackSets in the end, because it has an additional overhead to set the right permissions and we lose some of the benefits that AWS SAM provides. If you want a more detailed answer, let us know in the comments

Addon - How to add another region

This tutorial explained how to set up a multi-regional api. But how do we extend our configuration when we want to add another region?

Here are the steps:

Create a S3 bucket in the new region
Add region to Replicas in table.yaml
Create a certificate for api.example.com in the new region
Deploy the api to the new region
Add a AWS::Route53::RecordSet resource to routing.yaml