TryHackMe — AOC2024: Day 22
Hello and Welcome!
Story Overview:
Mayor Malware thought he had successfully implemented a clever scheme, but as the plan unfolded, he realized his name was already on the “naughty list” in Wareville. Unbeknownst to him, his actions would trigger an investigation led by McSkidy and The Glitch. Their task was to follow digital breadcrumbs, uncover hidden evidence, and identify how the webshell got into the system. With limited information due to ephemeral logs and container-based infrastructure, the investigation focused on Kubernetes, log analysis, and container registry data to uncover the truth behind Mayor Malware’s actions.
Learning Objectives:
- Learn about Kubernetes, what it is and why it is used.
- Learn about DFIR, and the challenges that come with DFIR in an ephemeral environment.
- Learn how DFIR can be done in a Kubernetes environment using log analysis.
Introduction To Kubernetes DFIR:
Kubernetes is a popular container orchestration platform that helps manage applications composed of microservices running in containers. With ephemeral workloads in Kubernetes, incident response and digital forensics (DFIR) become more challenging. Containers may spin up and down quickly, making it hard to capture logs and trace attacks. However, Kubernetes provides tools like audit logging to increase visibility, enabling DFIR experts to track events like API requests and container actions. Log analysis plays a critical role in investigating potential breaches and understanding how attackers operate in Kubernetes environments.
How DFIR is Done in Kubernetes:
- Increase Visibility: Use Kubernetes audit logs to capture API requests and events that happen within the cluster.
- Collect Evidence: Analyze logs from various sources (e.g., Apache, Docker) to trace the attacker’s steps.
- Investigate Containers: Track container images and registry interactions to understand how malicious code was introduced.
- Reconstruct the Attack: Use collected data to piece together the attack timeline and identify the root cause.
Challenges:
As companies expand, their service demands often change, which is why many are adopting microservices. This method breaks down applications into smaller, self-contained units, each running in its own container. With potentially thousands of containers running for thousands of services, managing them effectively becomes a challenge.
Get Pradeepbhattarai’s stories in your inbox
Join Medium for free to get updates from this writer.
Kubernetes simplifies this process. It orchestrates these containers by scaling them up or down based on demand, while also ensuring that traffic is distributed evenly. In Kubernetes, containers are organized into pods, which run on nodes, and a collection of nodes forms a Kubernetes cluster.
One problem with Kubernetes is that containers often have a short lifespan, which makes troubleshooting difficult. However, Kubernetes addresses this by enabling audit logging, which captures detailed records of API requests and actions at various points, making it easier to analyze and resolve issues after they occur.
1. What is the name of the webshell that was used by Mayor Malware?
* Start the Machine, Open a Terminal, and type the following command to start the minikube.minikube start* To list the pods type this command —kubectl get pods -n wareville* Type the below command to connect with a specific podkubectl exec -n wareville naughty-or-nice -it — /bin/bash* Once Connected typecat /var/log/apache2/access.log | tailand observe the file at the end.
* Answer: shelly.php
2. What file did Mayor Malware read from the pod?
* Command: Inside /home/ubuntu/dfir_artefactscat pod_apache2_access.logshow some command injected through shelly.php
* Answer:db.php
3. What tool did Mayor Malware search for that could be used to create a remote connection from the pod?
* In the same log of qn2 we see the nc used alongside of db.php
* Answer: nc4. What IP connected to the docker registry that was unexpected?
* Command: cat docker-registry-logs.logs | grep “HEAD” | cut -d ‘ ‘ -f 1 views the docker-registry-logs.logs which cut and grep the portion that starts with 1 which gives the IP address.
* Answer:10.10.130.253
5. At what time is the first connection made from this IP to the docker registry?
* Viewing all the logs from the IP address we got above using command:cat docker-registry-logs.log | grep “10.10.130.253” | headand viewing the first log .
* Answer:29/Oct/2024:10:06:33 +0000
6. At what time is the updated malicious image pushed to the registry?
* Command:cat docker-registry-logs.log | grep “10.10.130.253” |grep "PATCH"shows the logs from there we can grab the time the malicious updated image was pushed to server
* Answer: 29/Oct/2024:12:34:28 +0000
7. What is the value stored in the “pull-creds” secret?
* After investigating, we find that mayor malware’s account had the permission to usepods/exec.
* This allowed the mayor to shell into containers running within the pod. The mayor enumerated roles to find ajob-runnerrole that was bound to a service accountjob-runner-sa.
* After enumerating the pods, he finds that a podmorality-checkerruns with thejob-runner-saservice account attached.
*So heshellsinto the service account which effectively escalates his privileges giving him access to thepull-credssecret.
* The secret here can be found by running the command:kubectl get secret pull-creds -n wareville -o jsonpath='{.data.\.dockerconfigjson}' | base64 --decode
* Answer:{“auths”:{“http://docker-registry.nicetown.loc:5000":{"username":"mr.nice","password":"Mr.N4ughty","auth":"bXIubmljZTpNci5ONHVnaHR5"}}}
Key Takeaways:
- Digital Forensics involves collecting and analyzing digital evidence after an incident, while Incident Response focuses on immediate actions to contain and recover from the threat.
- It helps find Evidences about all the aspects of any incidents like What was the incident about, When it occured, Who caused it, its target, its source, place, goal of that incident and soon.
