Building a Local SOC Lab From Scratch: A Step-by-Step Guide Using Elastic Stack
Introduction
There’s a gap between knowing and doing that no amount of reading can close. You can study SIEM concepts for weeks and still freeze the first time you’re staring at a real alert with no idea where to start. The only way past that is hands-on time in an actual environment, and you don’t need a corporate budget to get one. A laptop, some free software, and a free afternoon are enough.
This guide walks through building a fully working SOC lab using the Elastic Stack on a local Kali Linux VM. By the end, you’ll have a complete log collection pipeline feeding into a searchable platform with a graphical interface on top, the same stack that real security teams use in production.
What We’re Building and Why
Before touching any commands, it’s worth taking a few minutes to understand the architecture. Knowing what each piece does makes troubleshooting much easier and makes you a sharper analyst overall.
The goal is straightforward: your machines generate logs and activity data constantly. You want all of that flowing into one central place where you can search it, correlate events, and investigate. That’s exactly what a SIEM does, and the Elastic Stack is one of the best free ways to build one.
It’s made up of four components, each handling a specific part of the pipeline. Elasticsearch is the storage and indexing engine that makes searching across thousands of events fast. Kibana is the interface where you actually do your work, dashboards, queries, timelines, and visualizations all live here. Elastic Fleet Server manages the agents deployed across your endpoints. Elastic Agent is the lightweight process running on each machine, collecting telemetry and shipping it back into the pipeline.
Four components, one clean pipeline, completely free, and the same tooling you’ll encounter in real enterprise SOC environments.
Before installing anything, make sure your system is fully up to date. This avoids dependency issues and ensures you’re working with the latest package versions.
Setting Up Elasticsearch, The Core of Your SIEM
Step 1: Update Your System
sudo apt update && sudo apt upgrade -yStep 2: Install Required Dependencies
Next, install the packages that the Elastic Stack relies on during setup. These handle secure downloads, repository management, and key verification.
sudo apt install -y curl apt-transport-https gpg
Step 3: Import the Elastic GPG Key and Add the Repository
Before pulling any packages, you need to tell your system where to find them and how to trust them. First, import the Elastic GPG key so your system can verify the authenticity of every package it downloads. Then, add the official Elastic repository to your sources list.
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpgecho "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
Step 5: Install Elasticsearch
With the repository added, update your package list and install Elasticsearch.
sudo apt update
sudo apt install -y elasticsearchOnce the installation finishes, pay close attention to the output in your terminal. The installer will display important security information, including a generated password for the elastic superuser. Save it immediately, because you will need it when setting up Kibana later and there is no easy way to recover it if you miss it.
Step 6: Cap the JVM Heap Memory (Critical for VM Users)
This step is important if you’re running on a virtual machine. By default, Elasticsearch is aggressive with memory usage, and on a VM, it will crash repeatedly if you don’t set a limit manually.
Open the JVM options file:
sudo nano /etc/elasticsearch/jvm.options.d/heap.optionsAdd these two lines:
-Xms1g
-Xmx1gThis caps both the initial and maximum heap size to 1GB, which keeps Elasticsearch stable on a VM without starving the rest of your system. When you’re done, save and exit withCtrl+X, thenY, then Enter.
Step 7: Start and Enable Elasticsearch
Now start the Elasticsearch service and enable it so it launches automatically on every reboot.
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
Step 8: Verify Elasticsearch is Running
Check that the service started successfully.
sudo systemctl status elasticsearchIf everything went smoothly, you should see it active (running) highlighted in green. If it shows a failed or inactive state, double-check your heap memory settings from Step 6 before moving forward.
Step 9: Test the Connection
With the service running, verify that Elasticsearch is actually responding by sending it a quick request.
curl -k -u elastic:YOUR_PASSWORD https://localhost:9200Replace YOUR_PASSWORD with the password that was displayed during installation. If you get a JSON response back with cluster information, you're good to go.
If you missed or lost the password, reset it with:
sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
[Installing and Configuring Kibana]
Step 1: Install Kibana
Since the Elastic repository is already configured from the previous section, you can install Kibana directly without any additional setup.
sudo apt install -y kibana
Step 2: Start and Enable Kibana
Same as before, enable the service so it starts on boot, then start it now.
sudo systemctl enable kibana
sudo systemctl start kibana
Step 3: Generate an Enrollment Token
To connect Kibana to your Elasticsearch instance, you need to generate an enrollment token.
sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibanaThe command will print a long token string in your terminal. Copy it and keep it somewhere handy, you will need it in the next step.
Step 4: Open Kibana in Your Browser
Open the browser on your VM and navigate to:
http://localhost:5601Kibana will greet you with a setup screen asking for an enrollment token. Paste the token you copied in the previous step and click configure. It may take a minute to complete the process, so give it a moment before assuming something went wrong.
Step 5: Generate a Verification Code
After the enrollment token is accepted, Kibana will request a verification code as the final confirmation step. Run this in your terminal:
sudo /usr/share/elasticsearch/bin/kibana-verification-code
Enter the code it gives you into the browser prompt, and you’re in.
Step 6: Log In
Once the verification is complete, you’ll be taken to the login screen. Use the following credentials:
- Username: elastic
- Password: The Elasticsearch superuser password you saved during installation
You should now be looking at the Kibana dashboard for the first time.
[Setting Up Elastic Fleet Server]
Step 1: Open Fleet Management
With Kibana up and running, the next thing to set up is the Fleet Server, which will manage all your endpoint agents. From the Kibana dashboard, navigate to Management and click on Fleet.
Step 2: Add a Fleet Server
In the Fleet section, click Add a new Fleet Server. You’ll be asked to give it a name and a URL.
For the name, use fleet-server. For the URL, you need your Kali VM's local IP address. If you don't know it, run this in your terminal:
ip aLook for the eth0 or ens33 interface and find the inet line, that's your IP address. Then enter it in the Fleet Server URL field using this format:
https://<Your_Machine_IP>:8220Make sure you use https and not http, and the port is always 8220.
Step 3: Generate the Fleet Server Policy
Once you’ve filled in the name and URL, click Generate Fleet Server Policy. This will produce a command that you’ll use to enroll endpoints into your Fleet, essentially telling each machine to start collecting telemetry and shipping it back to your Elastic instance.
In this setup, your Kali VM is playing a dual role. It’s the centralized server running Elasticsearch and Kibana, and at the same time, it’s also an endpoint that will be monitored. So the same machine that collects and stores all the data will also have an agent installed on it, sending its own telemetry into the pipeline. This is completely normal for a home lab setup.
Step 4: Copy the Installation Command
Scroll down, and you’ll see the generated installation command. Before copying it, make sure the selected operating system is set to Linux since that’s what we’re working with on the Kali VM.
Copy the command, then add --insecure at the end of it, before running anything. This flag is necessary because the Elastic Stack uses a self-signed certificate by default, and without it, the agent will reject the certificate and fail to connect.
Step 5: Run the Installation and Enroll the Agent
Paste the command into your terminal and run it. During the installation process, it will ask you a few questions.
Answer Y when prompted to confirm. When it asks for the URL you want to enroll this agent into, enter the Fleet Server URL you configured earlier:
https://<Your_Machine_IP>:8220When it asks for the Fleet enrollment token, go back to Kibana, navigate to Fleet, and open the Enrollment Tokens tab. Find the token, click the eye icon 👁 next to the secret field to reveal it, then copy and paste it into the terminal.
Step 6: Fleet Server Connected
Once the installation completes successfully, you’ll see a confirmation in Kibana showing that the Fleet Server is now connected.
From here, Fleet becomes your central control panel. This is where you manage how agents are deployed across your endpoints, what data integrations are enabled, and what telemetry gets collected and shipped back into your pipeline.
Step 7: Reviewing the Fleet Server Policy
After the Fleet Server connects, you’ll see the Fleet Server Policy listed in the Fleet dashboard, showing your machine as the enrolled host with its associated agent policy.
Clicking on the agent policy gives you a look at what’s being collected by default. Out of the box, Elastic creates two default integration policies that cover the essentials: system logs, notification logs, CPU metrics, and generally everything you need to start running investigations and troubleshooting on your endpoints.
You can add more policies and integrations on top of these, but for most SOC lab work, these two defaults are more than enough to get meaningful telemetry flowing in.
Step 8: Exploring the System Integration
Clicking on System integration opens up its configuration page, where you can add a description and explore exactly what data it collects.
Under the Data Streams section, you’ll see that the System integration collects two types of data: logs and metrics. Logs keep a record of events happening on your machine, things like authentication attempts, service starts and stops, and system activity. Metrics give you a live picture of the machine’s health and state, covering things like CPU usage, memory consumption, and disk activity.
Together, these two streams give you a solid foundation for both investigation and monitoring.
Step 9: The Fleet Server as an Agent
Navigating back and clicking on Fleet Server reveals that it isn’t just a management layer; it’s also an agent in its own right. It actively collects information just like any other endpoint agent would.
This is worth understanding because it means your Fleet Server is fully integrated into the Elastic ecosystem, compatible with both the Elastic Stack and Elastic Agent, and contributing its own telemetry into the pipeline alongside the endpoints it manages.
Step 10: Adding More Endpoints
If you want to monitor additional machines beyond the one you’re already working from, go back to the Fleet dashboard and click Add Agent.
When adding a new agent, you’ll be asked to specify a policy. The policy defines what data you want to collect from that endpoint and how it should be collected.
Expanding the Advanced Options section reveals additional configuration fields, including a description for the policy, spaces, custom fields, and agent monitoring settings, where you can choose to collect agent logs and metrics. Once you’ve configured everything, click Create Policy.
After the policy is created, scroll down, and you’ll see a confirmation that reads “The selected agent policy will collect data for 1 integration”, which is the System integration.
This means the policy will collect basic system information from whichever endpoint you install the agent on.
Get Hussein 404’s stories in your inbox
Join Medium for free to get updates from this writer.
Further down, under the Enroll in Fleet section, you’ll be presented with two options.
The first is Enroll in Fleet, which means the agent will be managed centrally from within this Kibana environment. The second is Run Standalone, which means you’ll manage the agent directly from the endpoint itself. In a real SOC environment, you’ll almost always go with Enroll in Fleet since central management is the whole point.
Step 11: Installing the Agent on Your Endpoint
Scrolling down to the Install Elastic Agent on your host section, select the operating system of the endpoint you want to monitor and run the command shown on screen.
Since our Kali VM is acting as both the server and the monitored endpoint at the same time, there’s no need to run this command again or install a second agent. The agent that was already installed as part of the Fleet Server setup is handling both roles, so we’ll keep it as is and move forward.
Step 12: Verifying Data Is Flowing Into Elasticsearch
Head over to Analytics in the left sidebar and click on Discover.
This is where you can see logs flowing into Elasticsearch in real time. Next to the Data View selector, choose All Logs.
If everything was set up correctly, you’ll see log records already populating the timeline. In the example here, around 8,784 log records were collected in just the last 15 minutes, which confirms that Elasticsearch, Kibana, and the Elastic Agent are all working together successfully.
Note: The number of logs you see will vary depending on the time range you have selected, how active your system is, how many endpoints are enrolled, and which integrations are configured. Don’t worry if your number looks different.
Step 13: Simulating an Event for Investigation
Now that data is flowing, it’s time to put the setup to work. Open your terminal and run the following command to create a new user on the system:
useradd ghost-detected
This is a deliberate test. Adding a user is the kind of action that gets logged at the system level, and it gives us something concrete to search for and investigate in Kibana. In a real SOC environment, unexpected user creation on an endpoint is exactly the type of activity an analyst would want to catch and look into.
Step 14: Investigating the Event in Kibana
Head back to Kibana, refresh the page, and open Discover. In the search bar, enter the following query:
process.name : useraddOn the left panel, find the process.parent.args field and add it as a column to your results.
The process.parent.args field confirms the full command that was executed, including the argument ghost-detected. This shows that Elastic Agent doesn't just capture the process name on its own; it also records the parent process context around it, giving you full visibility into how and from where the command was initiated. That level of detail is exactly what an analyst needs when tracing the origin of a suspicious action on an endpoint.
Step 15: Enriching Your Investigation With Additional Fields
To build a fuller picture of what happened, you can add more fields as columns directly from the left panel. For this event, select event.action, event.dataset, host.ip, and user.id to bring the most relevant context into your view.
With these fields added, the results tell a much more complete story. The event.dataset field shows endpoint.events.process, confirming that this event was captured by Elastic Defend at the endpoint level and that process monitoring is fully operational. The user.id field shows 0, which confirms the command was executed by the root user. As forhost.ip, it may not always populate for local endpoint events; in that case host.name is used instead to identify the source machine.
This is the core of what log investigation looks like in practice: you start with a single event, then layer in fields until you have enough context to understand exactly what happened, who did it, and where it came from.
Step 16: Adding Custom Integrations
So far, we’ve been working with system-level telemetry, but Elastic can go well beyond that. If you’re running a web server on your machine, for example, you can pull its access logs and error logs directly into Elastic as well. This is done through integrations, and Apache is a good example to walk through.
On the left panel, scroll down to Management and click on Integrations.
In the search bar, type Apache and select Apache HTTP Server from the results.
Click Add Apache HTTP Server to begin adding the integration.
Integrations like this one are how you expand your visibility beyond basic system telemetry. Whether it’s Apache, Nginx, or any other application running in your environment, Elastic has a library of ready-made integrations that make it straightforward to start collecting and investigating application-level logs without any custom parsing or configuration from scratch.
Step 17: Configuring and Saving the Apache Integration
Leave the integration settings as they are by default. Make sure both Collect logs from Apache instances and Collect metrics from Apache instances are enabled, as these give you full coverage of both access and error logs alongside performance metrics.
Under Create agent policy, select Existing hosts rather than creating a new policy. For the agent policy, choose Fleet Server Policy from the dropdown, then click Save and continue.
The reason we’re using the Fleet Server Policy here is straightforward. The Apache server we want to monitor lives on the same machine that’s acting as our Fleet Server, so attaching this integration to the existing Fleet Server Policy is the right call rather than spinning up a separate policy for it.
Once saved, the integration will be installed, and you’ll see it listed under the active policies.
Step 18: Verifying the Apache Integration
Head back to Analytics and open Discover. In the search bar, run the following query to filter for Apache events:
event.module : "apache"
No logs will show up at first, and that’s completely expected. Apache only generates access logs when there’s actual traffic hitting the web server, and since no one has visited it yet, there’s nothing to collect.
To simulate some traffic, open the browser on your VM and navigate to:
http://<Your_Machine_IP>
This triggers a request to the Apache web server running on your machine, which generates an access log entry. Go back to Kibana, refresh the Discover view, and run the query again.
The logs will now appear, confirming that the Apache integration is working correctly and that Elastic is successfully picking up application-level logs alongside the system telemetry we set up earlier.
Step 19: Ingesting Custom Log Types
The integrations library covers a wide range of common applications, but you’ll inevitably come across tools or programs whose logs aren’t supported out of the box. When that happens, you handle them as custom logs.
Custom logs usually come in a specific format that Elasticsearch can’t parse automatically, so they need to go through a structured process before they’re stored. That process is called an ingest pipeline, and it’s how you transform raw log data into something clean, structured, and searchable.
To create one, go to the left panel and navigate to Management, then click on Stack Management.
From there, select Ingest Pipelines.
Step 20: Creating the Ingest Pipeline and Generating VPN Logs
From the Ingest Pipelines page, click Create pipeline to start building a new one.
Name the pipeline vpnlogs.
This is the example we’ll work with to demonstrate how custom log ingestion works in practice.
(Adding the Processors)
A processor is what handles the transformation of raw logs into structured, searchable data inside Elasticsearch. For this pipeline, we need two processors.
Click Add processor and search for Grok. Grok works by matching patterns in raw log text and extracting structured values from them.
Set the field to message and paste the following pattern:
%{TIMESTAMP_ISO8601:event.time_string} %{WORD:event.action} %{USER:user.name} %{IP:source.ip} %{IP:vpn.client.ip} %{NOTSPACE:vpn.server.region}This pattern tells Elasticsearch to extract the timestamp, action, username, source IP, client IP, and server region from each log entry. Click Add processor.
Click Add processor again and search for Date. The Date processor converts the raw timestamp extracted by Grok into Elasticsearch’s standard timestamp format. Set the field to event.time_string, the format toISO8601, and the target field to @timestamp. Leave the timezone as is and click Add processor.
With both processors added, the pipeline is ready.
(Generating the VPN Logs)
Before creating the pipeline, generate some sample VPN logs to work with. Run the following commands in your terminal:
Create the scripts folder:
mkdir -p ~/Downloads/scripts
cd ~/Downloads/scriptsCreate the script:
nano vpnlog.pyPaste this code inside nano:
import random
import datetimeactions = ["login", "logout", "failed"]
users = ["alice", "bob", "charlie", "david", "eve", "hussein", "admin", "john"]
source_ips = ["203.0.113.1", "198.51.100.5", "192.0.2.10", "185.220.101.3",
"45.33.32.156", "172.16.0.5", "10.0.0.1", "91.108.4.1"]
client_ips = ["10.8.0.2", "10.8.0.3", "10.8.0.4", "10.8.0.5", "10.8.0.6"]
regions = ["US-East", "US-West", "EU-West", "EU-Central", "Asia-Pacific"]logs = []
base_time = datetime.datetime.utcnow()for i in range(500):
timestamp = base_time - datetime.timedelta(seconds=random.randint(0, 86400))
timestamp_str = timestamp.strftime("%Y-%m-%dT%H:%M:%S")
action = random.choice(actions)
user = random.choice(users)
source_ip = random.choice(source_ips)
client_ip = random.choice(client_ips)
region = random.choice(regions)
log = f"{timestamp_str} {action} {user} @{source_ip} {client_ip} {region}"
logs.append(log)with open("vpnlogs.log", "w") as f:
for log in logs:
f.write(log + "\n")print(f"Generated {len(logs)} VPN log entries in vpnlogs.log")
Save and exit with Ctrl+X, then Y, then Enter.
Run the script:
python3 vpnlog.pyYou should see: Generated 500 VPN log entries in vpnlogs.log
Opening the log file, each entry follows a consistent structure: timestamp, action, username, source IP, client IP, and server region.
These fields map exactly to what we defined in the Grok processor earlier, which is the whole point. The pipeline was built around the structure of these logs so that every entry gets parsed cleanly and consistently when ingested into Elasticsearch.
Now we’re going to create the pipeline by clicking on “Create pipeline”.
Step 21: Pointing Elasticsearch to Your Log File and Verifying Ingestion
After creating the pipeline, the next step is telling Elasticsearch where to find the actual log file. Run the following command in your terminal to get the full path:
find ~ -name "vpnlogs.log"
Copy the full path from the output, you’ll need it shortly.
(Adding the Custom Logs Integration)
Go back to Kibana, navigate to Integrations, and search for Custom Logs Filestream.
This integration is designed exactly for this use case, it lets you ingest logs from any location on your system into Elasticsearch. Click the Add Custom Logs (Filestream) button on the right.
Leave the integration name as filestream-1. Scroll to the bottom of the page, select Existing host, choose Fleet Server Policy, and click Save and continue. The reason we're using the Fleet Server Policy again is the same as before, the log file lives on the same machine.
(Editing the Integration to Point to Your Logs)
Once the integration is added, click the three dots under the Action column to edit it.
Click Change defaults to expand the configuration options and enter the full path to your log file. In this case it’s /home/Hussein/Downloads/scripts/vpnlogs.log. If you're unsure of your path, run find ~ -name "vpnlogs.log" again in your terminal.
Scroll down a little further to the Ingest Pipeline field and enter the name of the pipeline you created earlier, vpn.logs.pipeline.
With the file path and pipeline both configured, Elasticsearch now knows where to find the logs and how to process them. Click Save integration, then Save and deploy changes.
(Verifying the Logs in Discover)
Head over to Discover and select All Logs from the data view. In the top-right corner, set the time filter to Last 24 hours or Last 7 days to make sure you’re casting a wide enough net, since custom logs can sometimes carry timestamps that place them outside the default 15-minute window.
In the search bar, use the following query:
event.dataset : *
This approach is more reliable than filtering by event.module for custom pipelines, since the module field may not always be populated the same way as it is with prebuilt integrations. Querying event.dataset:* confirms that data is actually flowing into Elasticsearch regardless of how the pipeline labels it.
Once ingestion is confirmed, remove the previously selected fields from the Selected fields panel and add the following: host.ip, vpn.client.ip, user.name, source.ip, vpn.server.region, message, and log.file.path. The log.file.path field in particular confirms that the data is being pulled from the exact file you configured.
The VPN logs are now fully ingested, parsed, and visible inside Elasticsearch, exactly as we defined them in the pipeline.
Step 22: Building an Interactive Dashboard
Raw log tables are useful for investigation, but they don’t work well for reporting or presentations. The last step is turning the data we have into an interactive dashboard that tells a visual story.
Start by saving the current VPN logs view. Click Save, name it VPNLOGS, and save it.
Now navigate to Dashboards from the left panel.
Click Create dashboard, then click Add from library and search for VPNLOGS. Add it to the dashboard.
At this point it still looks like the same table from before.
To make it actually useful, click Add, then select Visualization, or if your Elastic version is updated, you’ll see a Create visualization button directly.
On the left side of the visualization editor, choose the fields you want to focus on. On the right, select the chart type. Start with a Line chart, set the horizontal axis to Timestamp and the vertical axis to Count. This gives you a timeline view showing how many connections occurred at each point in time.
The line chart immediately surfaces patterns in the data. In this example, you can see a spike of around 4,981 connection attempts at a specific point in time, exactly the kind of anomaly worth investigating in a real environment.
Clicking on Count of records in the suggestions panel shows you the total number of records across the entire dataset at a glance.
You can also switch to an Area chart to highlight peak regions more clearly, making it easier to spot where connection activity was heaviest.
And that’s the complete setup. From a fresh Kali Linux VM to a fully working SOC lab with log ingestion, custom pipelines, endpoint monitoring, and interactive dashboards, all running on free, enterprise-grade tooling. The same skills and the same stack you’d find in a real security operations environment.
I hope you found this walkthrough helpful.
