VPC Series Part 7: VPC Monitoring with CloudWatch
Today we’ll be setting up network monitoring for our VPCs. We’ll be using VPC Flow Logs and Amazon CloudWatch.
Network monitoring is important because it allows you to proactively identify and fix issues like performance bottlenecks or security threats before they can cause downtime or affect users.
I will be following this guide from NextWork.
This is part 7 of 9 of the AWS VPC series.
Picking off from part 6, this is our current set up:
This is what we want to achieve by the end of today:
What is Amazon CloudWatch?
It’s a monitoring service that watches over your AWS environment in real time.
It visualises these metrics as dashboards and graphs to help you understand your resources’ health and performance.
CloudWatch can also be set to automatically trigger actions based on these metrics, such as shutting down a VPC that is receiving illegitimate traffic
What is a Flow Log?
A flow log is a record of all IP traffic going to and from network interfaces in your VPC. It is used for network monitoring and troubleshooting, capturing details like source and destination IPs, ports, and if the traffic was accepted or rejected.
SET UP A FLOW LOG IAM POLICY AND ROLE
Currently VPC Flow Logs don’t have the permission to write logs and send them to CloudWatch.
In this step we’ll give VPC Flow Logs the authorisation to write logs to CloudWatch by creating an IAM role and policy. This will allow us to finish the flow log setup for the subnet.
- Go to the IAM Console and click on Policies.
- Create policy.
- Choose JSON.
- Replace the default JSON text with the text below:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams"
],
"Resource": "*"
}
]
}
5. Click Next.
6. Policy name: MediumVPCFlowLogsPolicy
7. Create policy.
8. We’ve now created the policy, but we need to create the role.
Roles > Create role.
9. Choose Custom trust policy.
A custom trust policy is a document attached to an IAM role that defines which entities (users, services, or accounts) are trusted to assume that role. It dictates who is granted the authorisation to use the role’s permissions.
10. Replace the JSON code with the code below (the only difference is “Principal”).
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Statement1",
"Effect": "Allow",
"Principal": {
"Service": "vpc-flow-logs.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}11. Next.
12. Search and select MediumVPCFlowLogsPolicy.
13. Next.
14. Role name: MediumVPCFlowLogsRole
15. Create role.
SET UP FLOW LOGS
We are going to set up a way to track all inbound and outbound network traffic for our VPCs and a location to store those records.
- Go to the Cloudwatch console.
- Confirm that the region is the same one that was used to launch your VPCs. This is because CloudWatch is region-specific. The data collected on a VPC in one region is not automatically available in other regions.
- Under Log groups click Logs.
- Create log group.
- Log group name: MediumVPCFlowLogsGroup
- Keep the default settings and click Create.
- Go back to the VPC console and choose Your VPCs.
- Select Medium-1-vpc.
- Actions > Create flow log.
- Name — optional: MediumVPCFlowLog
- Maximum aggregation interval: 1 minute
‘Maximum aggregation interval’ refers to how often traffic flow data will be lumped into a single flow log record. It being set to 1 minute means that flow log records are generated every minute. - Destination log group: MediumVPCFlowLogsGroup
- Service role: MediumVPCFlowLogsRole
We set this up earlier in the ‘SET UP A FLOW LOG IAM POLICY AND ROLE’ section. - Create flow log.
What is a log group?
Imagine a log group as a folder where you keep related logs together.
Usually logs from the same source or application will go into the same log group HOWEVER, as mentioned above, logs are also region-specific. This means that log data gets created and saved in the region it was created. However you can use CloudWatch dashboards to bring together different logs from different regions.
What does the ‘Retention setting’ refer to?
It refers to how long your logs are valid for before they’re deleted.
What does ‘Log class’ mean?
Standard: Means that the logs that get created will get accessed or analysed frequently.
Infrequent Access: Means that your logs are being stored for long-term archiving. You’re charged less for storage, but higher for each time you access the file for analysis.
TESTING CONNECTION BETWEEN VPCS
In this step, we’ll validate if traffic can be sent between these VPCs publicly and privately.
Get Lumenyx’s stories in your inbox
Join Medium for free to get updates from this writer.
We will do this by having Instance 1 send a message to Instance 2, which also confirms that our VPC peering setup is working correctly.
- While on the command line of ‘Instance — Medium VPC 1’, and ping the private IP address of ‘Instance — Medium VPC 2’. For me it looks like ping 10.2.0.159.
If this is successful then it means that the VPC peering connection is successful and traffic can move between these separate VPCs privately. - Now try and ping it using the Instance — Medium VPC 2’s PUBLIC IP address. For me it looks like ping 13.211.211.208.
Receiving ping replies from the public IPv4 address means Instance 2 is correctly configured to respond and that Instance 1 can communicate with it over the public internet.
ANALYSE FLOW LOGS
In this step we’ll review the flow logs for VPC 1’s public subnet and analyse the data to get insights into our network’s activity. We’ll be analysing the logs using Log Insights.
Log Insights is a query service that lets you analyse your CloudWatch logs. It helps you search and visualise log data using a query language to quickly find specific events and troubleshoot issues.
- Go to the CloudWatch Console and select Log groups.
- Click on your Log group.
- Choose your Log stream. This is where your individual logs are.
- Now go to Log insights.
- Within Selection criteria choose MediumVPCFlowLogsGroup.
- Select the Saved and sample queries folder on the right hand side.
- Drop down VPC Flow Logs.
- Drop down Return the tip 10 byte transfers by source and destination IP addresses > Apply.
- Run query.
- Look at the results of the query.
These results show the ten pairs of IP addresses that transferred the most data in your VPC. This allows you to identify and investigate any heavy traffic flows or unusual data transfers within your network.
An explanation of what the log entries mean:
You now know how to look at your VPC’s logs and analyse them using Log Insights!
Here’s what our topology looks like now:
DELETE YOUR RESOURCES — SO YOU DON’T GET CHARGED
Delete these resources:
- CloudWatch Log Group
- Delete the CloudWatch IAM Policy (IAM > Policies > Search for “FlowLogs” > Select MediumVPCFlowLogsPolicy > Delete)
- Delete the CloudWatch IAM Role (IAM > Roles > Search for “FlowLogs” > Select MediumVPCFlowLogsRole > Delete)
- EC2 Instances
- Delete VPC Peering Connections
- Delete your VPCs
The resources below should already be deleted but it’s good practice to check them anyway:
- Subnets
- Route tables
- Internet gateways
- Network ACLs
- Security groups
See you in the next one.
