A Comprehensive Guide on Optimizing Serverless APIs with AWS SQS and SES
When building a serverless API, some work does not need to happen before the API responds to the user.
Sending an email is a good example.
If an API handler creates an account and then waits for a confirmation email to be sent, the user has to wait for both operations to finish.
A better approach is to move the email-sending work into the background using Amazon SQS and Amazon SES.
The basic idea is:
Client
|
| HTTP request
v
API Lambda
|
| 1. Perform the main operation
|
| 2. Send email job to SQS
v
SQS Queue
|
| AWS reads the message
| and invokes the worker Lambda
v
Email Worker Lambda
|
v
Amazon SES
|
v
EmailThis allows the API to respond without waiting for the email to actually be sent.
1. The problem with sending email directly from the API
Suppose we have a simple registration API:
module.exports.handler = async (event) => {
try {
const user = await createUser(event);
await sendWelcomeEmail(user);
return {
statusCode: 201,
body: JSON.stringify({
message: "Account created successfully"
})
};
} catch (error) {
return {
statusCode: 500,
body: JSON.stringify({
message: error.message
})
};
}
};The flow is:
HTTP Request
|
v
Create User
|
v
Send Email
|
v
HTTP ResponseThe API cannot respond until sendWelcomeEmail() finishes.
This creates a few problems:
- The API response takes longer.
- The user has to wait for work that is not directly related to the API response.
- A temporary email/SES failure can affect the API request.
- Email-sending logic becomes tightly coupled to the HTTP request.
- Traffic spikes can create many simultaneous email operations.
The email is usually not something the user needs to wait for. The important operation is the main API operation; the email can happen shortly afterward.
2. What is a queue?
A queue is simply a place where messages or jobs wait to be processed.
Think about a line at a counter:
Job 1
Job 2
Job 3
Job 4A worker takes jobs from the line and processes them.
With AWS SQS:
Producer
|
v
SQS Queue
|
v
Consumer / WorkerThe application that puts a message into the queue is the producer.
The application that processes the message is the consumer or worker.
In this example:
API Lambda = Producer
SQS = Queue
Worker Lambda = Consumer
SES = Email service3. What is Amazon SQS?
Amazon Simple Queue Service (SQS) is a managed message queue provided by AWS.
Instead of one Lambda directly performing slow background work, we can put a message into SQS.
For example:
{
"to": ["user@example.com"],
"templateCode": "WELCOME_EMAIL",
"payload": {
"username": "John"
}
}The message describes the work that needs to be done.
The API does not need to know how the email is actually sent. It only needs to say:
“Here is an email job. Please process it.”
4. What is Amazon SES?
Amazon Simple Email Service (SES) is AWS’s email-sending service.
The worker Lambda can use SES to actually send the email.
SQS and SES have different responsibilities:
SQS
|
| Holds the email job
| Handles delivery to the worker
| Provides retries
| Can move failed messages to a DLQ
|
v
Worker Lambda
|
| Performs the email job
|
v
SES
|
| Actually sends the email
|
v
RecipientSQS does not send the email.
SES does not act as the queue.
The worker connects the two.
5. The optimized API flow
Instead of sending the email directly:
await sendWelcomeEmail(user);we send an email job to SQS:
await sqsClient.send(
new SendMessageCommand({
QueueUrl: process.env.EMAIL_QUEUE_URL,
MessageBody: JSON.stringify({
to: [user.email],
templateCode: "WELCOME_EMAIL",
payload: {
username: user.username
}
})
})
);The API flow now becomes:
HTTP Request
|
v
API Lambda
|
+----> Perform main operation
|
+----> Send email job to SQS
|
v
HTTP ResponseThe email itself happens separately:
SQS Queue
|
v
Worker Lambda
|
v
SES
|
v
EmailThe API does not wait for the entire email process.
6. Sending a message to SQS
Using the AWS SDK:
const {
SQSClient,
SendMessageCommand
} = require("@aws-sdk/client-sqs");
const sqsClient = new SQSClient({
region: process.env.AWS_REGION
});Then:
const message = {
to: ["user@example.com"],
templateCode: "WELCOME_EMAIL",
payload: {
username: "John"
}
};
await sqsClient.send(
new SendMessageCommand({
QueueUrl: process.env.EMAIL_QUEUE_URL,
MessageBody: JSON.stringify(message)
})
);The important property here is:
QueueUrl: process.env.EMAIL_QUEUE_URLThe Queue URL tells SQS:
“Put this message into this particular queue.”
7. Where does the Queue URL come from?
Instead of hardcoding the URL, Serverless/CloudFormation can provide it as an environment variable.
For example:
provider:
environment:
EMAIL_QUEUE_URL:
Ref: EmailQueueAnd the queue can be defined as:
resources:
Resources:
EmailQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: '${self:service}-${sls:stage}-email-queue'For an AWS::SQS::Queue, CloudFormation's Ref gives the queue URL.
So:
EmailQueue
|
| Ref
v
Queue URL
|
v
EMAIL_QUEUE_URL
|
v
process.env.EMAIL_QUEUE_URLThe application therefore does not need to know the actual AWS-generated URL.
8. How does SQS know which Lambda to trigger?
This is where the SQS event source mapping comes in.
With Serverless Framework, we can configure:
emailWorker:
handler: handlers/queues/email-worker.handler
events:
- sqs:
arn:
Fn::GetAtt:
- EmailQueue
- ArnThis tells Serverless/AWS:
“This Lambda is a worker for this SQS queue.”
The queue has an ARN, and the Lambda also has an ARN. AWS uses the event-source configuration to create the relationship between the queue and the Lambda.
Conceptually:
EmailQueue
|
| Queue ARN
v
Event Source Mapping
|
| knows which Lambda to invoke
v
emailWorker LambdaThe ARN itself does not directly call the Lambda. The important concept is the event source mapping created from this configuration.
9. What happens when a message arrives?
Suppose our API sends this message:
{
"to": ["user@example.com"],
"templateCode": "WELCOME_EMAIL",
"payload": {
"username": "John"
}
}The message is now sitting in SQS.
AWS’s SQS-to-Lambda integration reads messages from the queue and invokes the configured worker Lambda.
Get Aryan Chari’s stories in your inbox
Join Medium for free to get updates from this writer.
The worker receives an event containing records:
module.exports.handler = async (event) => {
for (const record of event.Records) {
const message = JSON.parse(record.body);
console.log(message);
// Process the email job
}
};A single Lambda invocation can contain multiple SQS messages. That is why we usually iterate over event.Records.
10. The worker Lambda
The worker can now handle the actual email operation:
const EmailRepository = require("../../repositories/email-repository");
module.exports.handler = async (event) => {
try {
for (const record of event.Records) {
const {
templateCode,
payload,
to,
cc
} = JSON.parse(record.body);
await EmailRepository.publishEmail({
to,
cc,
templateCode,
payload
});
}
} catch (error) {
console.error(
"Error while processing email queue:",
error.message
);
throw error;
}
};The important part is:
throw error;If the email operation fails, the Lambda invocation needs to fail. This allows the message to be processed again instead of treating it as successfully processed.
11. The worker sends the email through SES
The worker can eventually call SES:
const {
SESv2Client,
SendEmailCommand
} = require("@aws-sdk/client-sesv2");
const sesClient = new SESv2Client({
region: process.env.AWS_REGION
});Then:
await sesClient.send(
new SendEmailCommand({
FromEmailAddress: process.env.FROM_EMAIL,
Destination: {
ToAddresses: to
},
Content: {
Simple: {
Subject: {
Data: subject
},
Body: {
Html: {
Data: body
}
}
}
}
})
);So the complete background process is:
SQS Message
|
v
Worker Lambda
|
v
Build/render email
|
v
SES
|
v
Recipient12. What happens when the worker fails?
One of the biggest benefits of using SQS is that a failed message can be processed again.
For example:
Message
|
v
Worker
|
X
Failure
|
v
RetryWe can also configure a Dead Letter Queue (DLQ).
For example:
Resources:
EmailDLQ:
Type: AWS::SQS::Queue
Properties:
QueueName: '${self:service}-${sls:stage}-email-dlq'
EmailQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: '${self:service}-${sls:stage}-email-queue'
RedrivePolicy:
deadLetterTargetArn:
Fn::GetAtt:
- EmailDLQ
- Arn
maxReceiveCount: 3Now, if a message repeatedly fails, it can eventually be moved to the DLQ.
Conceptually:
EmailQueue
|
| Attempt 1 -> failure
|
| Attempt 2 -> failure
|
| Attempt 3 -> failure
|
v
EmailDLQThe DLQ prevents a permanently failing message from being retried indefinitely and gives you a place to inspect or redrive failed jobs.
13. Queue URL vs Queue ARN
Both identify the same queue, but they are used for different purposes.
Queue URL
Used when interacting with the queue through SQS APIs.
For example:
new SendMessageCommand({
QueueUrl: process.env.EMAIL_QUEUE_URL,
MessageBody: JSON.stringify(message)
});Think:
“Where is the queue I want to send this message to?”
Queue ARN
Used to identify the AWS queue resource when configuring AWS relationships.
For example:
events:
- sqs:
arn:
Fn::GetAtt:
- EmailQueue
- ArnThink:
“Which AWS resource is this?”
A useful mental model is:
Queue URL
|
+--> Used to interact with the queue
Queue ARN
|
+--> Used to identify the AWS queue resource
and configure the SQS -> Lambda relationship14. The complete architecture
Putting everything together:
Refer to the architecture diagram. Notice that the API Lambda and email worker are now separated.
The API’s responsibility is:
Receive request
|
v
Perform main operation
|
v
Create email job
|
v
Put job in SQS
|
v
Return responseThe worker’s responsibility is:
Receive email job
|
v
Process job
|
v
Render email
|
v
Send through SESThis is called decoupling.
15. Why is this better than sending the email inside the HTTP Lambda?
1. Faster API responses
Without SQS:
API
|
+-- Database operation
|
+-- Email operation
|
+-- ResponseWith SQS:
API
|
+-- Database operation
|
+-- Put job in SQS
|
+-- ResponseThe API no longer waits for the entire email-sending process. The user gets the response sooner.
2. Email failures are separated from the main API operation
Imagine the main operation succeeds but SES temporarily has a problem.
Without a queue, the API is directly dependent on the email operation.
With SQS:
API
|
+-- Main operation succeeds
|
+-- Email job stored in SQS
|
+-- API respondsThe worker can retry the email independently.
3. Built-in retry behavior
SQS can keep a message available for another processing attempt when the worker fails.
You can also configure a DLQ:
Failed message
|
v
Retry
|
v
Retry
|
v
Retry
|
v
DLQThis provides a more resilient background-processing flow.
4. Better separation of responsibilities
Your HTTP Lambda does not need to contain all of the email logic.
Instead:
API Lambda
|
| "Send this email"
v
SQS
|
v
Email Worker
|
v
SESThe API knows about the job, while the worker knows about the implementation.
This makes the system easier to maintain and extend.
5. Better handling of traffic spikes
Suppose your application suddenly needs to send a large number of emails.
Without a queue, the API layer could end up trying to perform many email operations at the same time.
With SQS:
+-- Job
+-- Job
API ------------>+-- Job
+-- Job
+-- Job
|
v
Worker
|
v
SESThe queue acts as a buffer between the API and the background work.
Workers can process jobs as they become available.
16. Where else can this pattern be used?
The SQS + worker pattern is useful whenever some work can happen asynchronously.
Notifications
API -> SQS -> Notification Worker -> Push/SMS/EmailImage processing
API -> SQS -> Image Worker -> Resize/Compress -> S3Report generation
API -> SQS -> Report Worker -> Generate PDF -> S3Data processing
API -> SQS -> Worker -> Process records -> DatabaseAnalytics
API -> SQS -> Analytics Worker -> Store/process analyticsThe general pattern is:
Synchronous request
|
v
Do the important work
|
v
Put background job in SQS
|
v
Return response
|
|
| Background
| |
| v
| SQS Queue
| |
| v
| Worker
| |
| v
| Actual work
|
v
ClientConclusion
SQS is not an email service.
SES is not a queue.
They solve different problems.
A useful way to remember the architecture is:
SQS = "Do this work later."
Worker = "I'll do the work."
SES = "I'll send the email."For a serverless API, this separation allows the HTTP request to focus on the work that must happen immediately while slower background tasks can be processed independently.
The key flow is:
API Lambda
|
v
Send message to SQS using Queue URL
|
v
SQS stores the job
|
v
AWS SQS/Lambda event source mapping
|
v
Worker Lambda is invoked
|
v
Worker processes the message
|
v
SES sends the emailOnce this mental model is clear, SQS becomes much less mysterious. It is essentially a reliable middle layer that allows your application to say:
“I have some work that needs to be done. Store it, and let a worker handle it.”
