Software Engineering

Message Queue

Send Email

Sqs

Backend Development

Send Email With Message Queue

Jazim Abbas

6 min read

·

Jul 18, 2022

--

2

Listen

Share

When I was in college, I was building an eCommerce Application where I had to sent an email when someone ordered the product.

Problem

The problem is that when over 1000 users orders the product at the same time; they had to wait for a long time. I built this application using a monolithic approach. The way I had implemented first order saved in the database and then sent a confirmation email to the user. Loading animations were running and users had to wait for a long time i.e. 5–10 seconds and sometimes even worse, over 20 seconds. So I was very frustrated, how to solve this problem.

Solution

I came up with a solution using micro-services based architecture. Let me walk through my solution.

There are 3 components in the Architecture Diagram.

• Order Service
• Email Service
• Message Queue

Order Service: Order Service is basically a producer which means it sends messages to queue. You can use any programming language to build this service. I am using nodejs because it's easy to set up.

Email Service: Email service is a consumer, which means it pulls messages from the Queue and sends an email to users. I am using nodejs to build this service.

Message Queue: Message Queue facilitates batching of operations, such as sending out emails or inserting records into a database. Batch instructions can be saved in a queue and all processed at the same time in order instead of being processed one by one, which can be inefficient.
And if something fails, the message will still in the queue so other service resumes that operation and after successfully emailed we will delete the message from the queue so that no other service can pull that message again otherwise there will be duplicate emails we’ll sending to end users which is not good.
I am using AWS SQS as Message Queue. It's an AWS managed service so we don’t have to do anything. AWS will take care of everything. You can use Kafka, RabbitMQ or some other Message Queue. I’ve picked SQS because it's very easy to start with.

Let's go through the implementation step by step.

Setting Up Order Service

mkdir order-service
cd order-servicenpm init -y
npm i express aws-sdk
npm i -D nodemon

Create a helper file for environment variables so that we have only one place where we are putting all env variables.

module.exports = {
  AWS_REGION: "us-east-1",
  AWS_ACCESS_KEY: process.env.AWS_ACCESS_KEY,
  AWS_SECRET_KEY: process.env.AWS_SECRET_KEY,
  AWS_QUEUE_URL: process.env.AWS_QUEUE_URL,
};

Create a new file for SQS and put all the implementation for SQS in that file.

const AWS = require("aws-sdk");
const envVars = require("../env-vars");

const sqsClient = new AWS.SQS({
  apiVersion: "2012-11-05",
  region: envVars.AWS_REGION,
  credentials: {
    accessKeyId: envVars.AWS_ACCESS_KEY,
    secretAccessKey: envVars.AWS_SECRET_KEY,
  },
});

/**
 * @param {{
 *  userEmail:  string
 *  bookName:   string
 * }} bookOrderData
 */
exports.sendMessageToSQS = async function (bookOrderData) {
  return sqsClient
    .sendMessage({
      MessageAttributes: {
        userEmail: {
          DataType: "String",
          StringValue: bookOrderData.userEmail,
        },
        bookName: {
          DataType: "String",
          StringValue: bookOrderData.bookName,
        },
      },
      // MessageDeduplicationId: "TheWhistler",  // Required for FIFO queues
      // MessageGroupId: "Group1",  // Required for FIFO queues
      MessageBody: JSON.stringify(bookOrderData),
      QueueUrl: envVars.AWS_QUEUE_URL,
    })
    .promise();
};

Firstly, create Queue using aws console or aws cli. And it's very easy to set up. You just need two things to create the queue. Name and type. Make other options as default, but you can change later if you need. But for simplicity, make other options as default.

There are 2 options for selecting the Queue type:
Standard
FIFO

If you want, your messages will always process in order, pick FIFO. So the FIFO queue maintains the order of the messages as received and they are also retrieved in the same First-In-First-Out order.

A Standard Queue does not maintain the order of the messages it receives and is better suited for projects that prioritize throughput over the order of events.

Let me walk through the above code snippet.

First, I am creating the SQS client. And in the constructor I am putting the region and IAM credentials. So you need AccessKey & SecretKey for the sdk and make sure you have appropriate permissions i.e. send message to sqs and delete message to sqs. Otherwise this will not work. Because as you can see in the above code snippet I am using sendMessage method, so I need that IAM permission to perform this action. And later in the Email Service, we’ll delete messages so we need delete message IAM permission as well. I hope you get the point.

Lastly, create an entry file named index.js. This is very straightforward. Just creating a post route for book-order. In this method, I am just pushing the detail in SQS which saved that detail so that consumers pull that data and use it.

const express = require("express");
const SQS = require("./utils/sqs");
const app = express();

app.use(express.json());

app.get("/", (_, res) => {
  res.send("Order Service");
});

app.post("/book-order", async (req, res) => {
  try {
    const data = await SQS.sendMessageToSQS(req.body);
    console.log("Order Success: ", data.MessageId);
    res.send("Please check for your inbox");
  } catch (err) {
    console.log("Book Order Error: ", err);
    res.status(400).send("There is some problem.");
  }
});

const PORT = process.env.PORT || 8000;
app.listen(PORT, () => {
  console.log(`App is listening on the port ${PORT} ..`);
});

Setting up an Email Service

mkdir email-service
cd email-servicenpm init -y
npm i aws-sdk nodemailer sqs-consumer

Create a helper js file for environment variables so that we have only file for environment variables.

module.exports = {
  MAIL_HOST: process.env.MAIL_HOST,
  MAIL_PORT: process.env.MAIL_PORT,
  MAIL_USER: process.env.MAIL_USER,
  MAIL_PASSWORD: process.env.MAIL_PASSWORD,  
  AWS_REGION: "us-east-1",
  AWS_ACCESS_KEY: process.env.AWS_ACCESS_KEY,
  AWS_SECRET_KEY: process.env.AWS_SECRET_KEY,
  AWS_QUEUE_URL: process.env.AWS_QUEUE_URL,
};

Next, setting up the email client. I am using nodemailer for sending an email. I am using fake email service i.e. mailtrap.io.
You can create an account in mailtrap, it’s a free service for testing.

const nodeMailer = require("nodemailer");
const envVars = require("./env-vars");

const transport = nodeMailer.createTransport({
  host: envVars.MAIL_HOST,
  port: envVars.MAIL_PORT,
  auth: {
    user: envVars.MAIL_USER,
    pass: envVars.MAIL_PASSWORD,
  },
});

/**
 * @param {{
 *  email:      string
 *  bookName:   string
 * }} data
 */
exports.sendMail = async function (data) {
  try {
    await transport.sendMail({
      from: "jazimabbas@gmail.com",
      to: data.email,
      subject: "Testing Email",
      html: `<p>Your book order (${data.bookName}) has been received </p>`,
    });
    console.log("Successfully send an email");
  } catch (err) {
    console.log("Error while sending an email");
  }
};

Next, create a new file for pulling messages from SQS Queue. I am using a third party package for pulling messages, i.e. sqs-consumer so that we don’t need to write any boilerplate code. It takes care of everything like pulling messages from the queue and after successful pull the messages, it can auto-delete that message from the queue.

const AWS = require("aws-sdk");
const { Consumer } = require("sqs-consumer");
const envVars = require("./env-vars");
const mail = require("./send-email");

const app = Consumer.create({
  queueUrl: envVars.AWS_QUEUE_URL,
  sqs: new AWS.SQS({
    region: envVars.AWS_REGION,
    credentials: {
      accessKeyId: envVars.AWS_ACCESS_KEY,
      secretAccessKey: envVars.AWS_SECRET_KEY,
    },
  }),
  handleMessage: async (message) => {
    const sqsMessage = JSON.parse(message.Body);await mail.sendMail({
      email: sqsMessage.userEmail,
      bookName: sqsMessage.bookName,
    });
  },
});

module.exports = app;

Last, create an entry file

const app = require("./utils/sqs-consumer");

app.on("error", (err) => {
  console.log("Error: ", err.message);
});

app.on("processing_error", (err) => {
  console.log("Processing Error: ", err.message);
});

console.log("Running an Email Service");
app.start();

Conclusions

Converting monolithic to micro-service based architecture is not always a solution. There are many applications exist which are using monolithic architecture. But now many organizations moving towards micro-services. And many companies want their employees to have a deep understanding of distributed systems. But anything comes with its cost.

When we are going to start our journey with coding, for some people it's easy to start, but it's difficult for some people as well. The key to success I think just keep doing what you like. If you are trying and get stuck, don’t frustrate it's a part of the process. And if you get stuck on some problem, then this is a proof that you are trying to do at least something. Most people don’t even try.

Best of luck & thanks for reading. Happy Coding.