Building a Production-Ready Email System with AWS SES and Serverless Stack

Open Table of contents

Introduction

In today’s digital landscape, reliable email communication is a cornerstone for any modern application. Whether it’s for user notifications, password resets, or marketing campaigns, having a robust email system is crucial. Amazon Web Services Simple Email Service (AWS SES) offers a cost-effective, scalable solution for email delivery. Coupled with the power of the Serverless Stack (SST), developers can efficiently manage infrastructure while focusing on application logic.

In this blog post, we’ll explore how to build a production-ready email system using AWS SES and SST, leveraging serverless architecture to reduce operational overhead.

Understanding the CAP Theorem

The CAP theorem dictates that a distributed system can achieve only two out of the three guarantees: Consistency, Availability, and Partition tolerance. However, a well-designed system can navigate these constraints effectively, maximizing benefits from each aspect.

Transactional Outbox Pattern Demystified

At its core, the Transactional Outbox Pattern involves storing local events (like database changes) in an “outbox” table. External services or systems can then read from this outbox, ensuring reliable and consistent data propagation.

transactional outbox pattern sequence diagram

DynamoDB Streams: The Outbox Enabler

DynamoDB Streams captures item-level modifications, turning our outbox into a dynamic, real-time feed.

Advantages:

Consistency: Ensures every database change is captured and made available sequentially.
Integration: Seamlessly integrates into the AWS ecosystem, opening doors to services like Kinesis.

Supercharging with Kinesis Streams

Pairing DynamoDB Streams with Kinesis allows for large-scale, real-time processing and analytics. Kinesis subscribes to the outbox events, enabling scalability and flexibility in data processing.

transactional outbox pattern with kinesis

Navigating the CAP Theorem

Using the Transactional Outbox Pattern with DynamoDB and Kinesis offers a harmonized approach to the CAP theorem:

Consistency: Changes are captured reliably, allowing for a consistent view of data.
Availability: AWS’s managed services guarantee high uptime.
Partition Tolerance: The asynchronous nature of DynamoDB Streams feeding into Kinesis ensures system operation even in the face of network partitions.

This combination not only assures robustness but also simplicity in implementation.

Implementation: Crafting a Resilient System

We will be implementing the use case defined in the diagram, showcasing an order creation system. The complete example can be found in the GitHub repository here:

Prerequisites

An AWS account
An AWS profile set (the app configuration assumes a default AWS profile with eu-west-1, though you can go with your specific region).

config(_input) {
  return {
    name: 'transactional-outbox-sst',
    region: 'eu-west-1',
  };
}
// sst.config.ts

SST Overview

SST is a serverless framework built on top of AWS CDK. If you’re unfamiliar, check out its documentation:

SST Documentation

Kinesis Configuration

Create a Kinesis stream and add a Lambda consumer to it:

const stream = new KinesisStream(stack, "Stream", {
  consumers: {
    created: "packages/functions/src/events/created.handler",
  },
});
// stacks/Outbox.ts

DynamoDB Integration

Next, we’ll set up two DynamoDB tables: one for storing orders and another for storing outbox events. We’ll also enable DynamoDB Streams, which will feed the events into our Kinesis stream.

const orderTable = new Table(stack, "orders", {
  fields: {
    orderId: "string",
  },
  primaryIndex: { partitionKey: "orderId" },
});

const eventTable = new Table(stack, "events", {
  fields: {
    eventId: "string",
  },
  primaryIndex: { partitionKey: "eventId" },
  kinesisStream: stream,
  stream: "new_image",
});
// stacks/Outbox.ts

Data processing

Data Processing: define the aws lambda function that processes the data captured by Kinesis, and sends informations to further consumers ensuring system remains responsive and updated.

import { KinesisStreamEvent, KinesisStreamRecordPayload } from "aws-lambda";

export const handler = async (event: KinesisStreamEvent) => {
  for (const record of event.Records) {
    const kinesisRecord: KinesisStreamRecordPayload = record.kinesis;

    const payload: string = Buffer.from(kinesisRecord.data, "base64").toString(
      "utf8"
    );
    const jsonPayload = JSON.parse(payload);

    console.log(
      `Order creation event received: ${JSON.stringify(
        jsonPayload.dynamodb.NewImage,
        null,
        2
      )}`
    );

    // TODO: Process the event data ....
  }
};
//packages/functions/src/events/created.ts

API

Api: create an aws Api gateway with an endpoint that will process the order creation request in transactional manner

const api = new Api(stack, "api", {
  defaults: {
    function: {
      bind: [orderTable, eventTable],
    },
  },
  routes: {
    "POST /": "packages/functions/src/orders/create.handler",
  },
});
//stacks/Outbox.ts

Request processor

Request processor: define the lambda that processes the order creation request

import { ApiHandler } from "sst/node/api";
import { DynamoDB } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocument } from "@aws-sdk/lib-dynamodb";
import { ulid } from "ulid";
import { Table } from "sst/node/table";

export const handler = ApiHandler(async event => {
  const data = JSON.parse(event.body!);
  const orderId = ulid();
  const params = {
    TransactItems: [
      {
        Put: {
          TableName: Table.orders.tableName,
          Item: {
            orderId,
            description: data.description,
          },
        },
      },
      {
        Put: {
          TableName: Table.events.tableName,
          Item: {
            eventId: `evt_${orderId}`,
            timestamp: Date.now(),
            type: "ORDER_CREATED",
            relatedOrderId: orderId,
          },
        },
      },
    ],
  };

  try {
    const client = new DynamoDB({});
    const ddbDocClient = DynamoDBDocument.from(client);

    await ddbDocClient.transactWrite(params);
    console.log("Transaction Successful!");
  } catch (err) {
    console.error("Error executing transaction:", err);
  }

  return {
    statusCode: 200,
    body: `Order #${orderId} created successfully !!`,
  };
});
//packages/functions/src/orders/create.ts

Show Time!

Now that everything is set up, it’s time to see the system in action. Follow these steps to run the project locally:

Prepare the repository:

git clone https://github.com/KBoudich/transactional-outbox-sst.git
cd transactional-outbox-sst
pnpm install

Demo

Here is a small video demoing the creation of orders

Conclusion

By marrying the Transactional Outbox Pattern with DynamoDB Streams and Kinesis Streams, we not only find an effective way to navigate the challenges posed by the CAP theorem but also discover a solution that’s elegant, scalable and resilient, flavoured by a minimal codebase (thanks to SST framework).

the transactional outbox pattern with aws

Table of contents