How I Reduced an Embedded Linux Boot Time from 35 Seconds to Under 10 Seconds
An engineering case study on measuring the critical boot path, restructuring systemd services, and removing unnecessary delays in a Yocto-based embedded Linux system.
Boot time can easily become an afterthought in embedded Linux projects. During development, new services, initialization scripts, hardware checks, and temporary workarounds are gradually added to the system. Each addition may look harmless on its own, but together they can turn startup into a long chain of unnecessary waits.
I encountered exactly this problem while working on a Yocto-based embedded Linux device. The system took approximately 35 seconds to become operational after power-on. This was too slow for a device expected to initialize its communication interfaces and application services shortly after receiving power.
The first instinct might be to disable anything that appears slow. However, embedded systems often depend on hardware initialization, device nodes, network interfaces, and service ordering. Removing the wrong dependency can make the system appear faster while introducing intermittent failures.
Instead, I approached the problem as a critical-path analysis: measure the startup sequence, identify what was actually blocking progress, and shorten that path without removing required functionality.
The result was a reduction from approximately 35 seconds to under 10 seconds.
The Initial Problem
The target was an ARM-based embedded Linux device running a custom Yocto image and using systemd as its init system. During startup, the device had to initialize several hardware and software components, including:
- CAN interfaces
- A cellular modem
- A wireless network interface
- USB networking
- Device-specific application services
Although all these components were necessary, they did not all need to start sequentially. Over time, some initialization steps had been placed inside a legacy rc.local script, while others were implemented as separate systemd services with broad or unnecessary dependencies.
This created three main problems:
- Some independent operations were executed one after another.
- Hardware-related waits blocked services that did not depend on that hardware.
- The difference between “Linux has started” and “the device is operational” was not clearly defined.
The kernel itself was not responsible for the entire delay. A significant portion of the boot time came from userspace initialization and the way services were ordered.
Before changing anything, I needed to answer a more important question:
What exactly did “boot complete” mean for this device?
Defining “Boot Complete”
Before starting the optimization, I needed a clear definition of when the boot process was actually complete. A short kernel startup time does not necessarily mean that an embedded device is ready to perform its intended function. The kernel may finish booting while network interfaces, communication buses, device nodes, or the main application are still unavailable. For this project, I considered the device operational only when the kernel and userspace initialization had completed, the required CAN and network interfaces were available, and the main application services had started successfully. The goal was therefore not simply to reach a login prompt faster, but to make the complete device ready for use as quickly and reliably as possible.
This definition also prevented misleading improvements. Disabling a slow service could reduce the time reported by systemd, but it would not be a real optimization if an essential interface or application became unavailable afterward. Every change therefore had to satisfy two conditions: it had to shorten the actual startup path, and the device still had to provide the same required functionality after boot.
Measuring the Baseline
I began by collecting a baseline before changing any service or configuration. Running systemd-analyze separated the startup duration into kernel and userspace time, while systemd-analyze blame showed which services and device units spent the most time initializing. The initial results pointed to several notable delays: the legacy rc.local initialization took approximately 6.2 seconds, a filesystem device wait took around 3.8 seconds, random-seed initialization varied between 1.8 and 4.6 seconds, RF-kill initialization took about 2 seconds, the cellular modem power service took approximately 1.7 seconds, and udev device triggering added around 1.4 seconds.
systemd-analyze
systemd-analyze blameThe blame output provided a useful starting point, but it could not be treated as a list of delays that could simply be added together. Multiple systemd units may run in parallel, so a service taking two seconds does not necessarily increase the total boot time by two seconds. A slow unit matters only when another required unit is waiting for it or when it belongs to the critical path. To understand these relationships, I used systemd-analyze critical-chain, which revealed the dependency sequence that was actually delaying the boot target. I also examined the logs with monotonic timestamps using journalctl -b -o short-monotonic. This made it easier to follow the order of events and see how much time had elapsed since power-on without depending on the device’s wall-clock configuration.
systemd-analyze critical-chain
journalctl -b -o short-monotonicThe measurements showed that the problem was not caused by a single unusually slow component. It resulted from a combination of legacy initialization logic, unnecessary service ordering, hardware-related waits, and independent operations being executed sequentially. This changed the direction of the work: instead of focusing only on individual service durations, I began examining how the complete startup sequence was structured.
Finding the Real Bottlenecks
The largest visible delay came from rc-local.service, which took approximately 6.2 seconds. The script had gradually become responsible for several unrelated initialization tasks, including network configuration, hardware setup, and application-specific operations. Because the commands were executed sequentially, a slow hardware initialization step delayed everything that followed it, even when the later operations did not depend on that hardware. This made the script convenient during early development but inefficient as part of the final boot sequence. It also made failures difficult to isolate because systemd could only report the total execution time of rc.local, rather than tracking each operation as an individual service.
Get Selda Turgut’s stories in your inbox
Join Medium for free to get updates from this writer.
Other delays required more careful investigation. For example, the cellular modem needed time to power up and expose its device interfaces, while CAN and USB networking had their own initialization requirements. These operations were necessary, but they did not all depend on one another. Some services also used broad ordering rules or waited for general system targets when they only required a specific device or interface. As a result, independent tasks were unnecessarily serialized. The filesystem device wait, random-seed initialization, RF-kill handling, and udev processing also appeared in the measurements, but their presence in systemd-analyze blame did not automatically mean they were safe to remove. Each one had to be checked against the critical chain and the actual runtime requirements of the device.
Replacing rc.local with Dedicated Services
The most important structural change was moving initialization logic out of rc.local and into dedicated systemd services. Instead of treating the entire device setup as one long script, I separated USB networking, wireless networking, cellular modem power, CAN configuration, and application startup into individual units. This allowed systemd to manage each operation independently, start unrelated tasks in parallel, record their execution times separately, and restart a failed component without repeating the complete initialization sequence.
Each service was then given only the dependencies it actually required. A hardware-dependent service could wait for its corresponding device node or interface, while an unrelated network or application service could continue starting in parallel. This reduced unnecessary serialization and made the startup sequence easier to understand. It also improved maintainability: when one component failed, its status and logs could be inspected directly with systemctl status and journalctl -u, rather than searching through the output of a large initialization script.
A simplified service structure looked like this:
[Unit]
Description=Configure USB network interface
After=sys-subsystem-net-devices-usb0.device
BindsTo=sys-subsystem-net-devices-usb0.device
[Service]
Type=oneshot
ExecStart=/usr/bin/configure-usb-network
RemainAfterExit=yes
[Install]
WantedBy=multi-user.targetThis example illustrates the main idea rather than the exact production configuration. The service waits for the specific usb0 interface it needs instead of delaying unrelated parts of the boot process behind a general initialization script. Applying the same principle to the other components transformed the startup sequence from a mostly sequential chain into a set of smaller services that systemd could schedule according to their real dependencies.
Optimizing Service Dependencies and Hardware Waits
After splitting the initialization tasks into separate services, the next step was to review how those services depended on devices, interfaces, and systemd targets. Some units were waiting for broad targets even though they only required a particular interface or device node. This introduced delays because a service could remain blocked by unrelated initialization work. I replaced these broad dependencies with more specific relationships wherever possible, allowing systemd to start independent services in parallel while still preserving the required startup order.
Hardware availability was especially important for the cellular modem, CAN interfaces, and GNSS device. These components do not necessarily become available at the same moment during every boot, so replacing every wait with a fixed delay would only make the sequence fragile. Instead, hardware-dependent services were connected to the device or interface they actually required. Udev rules were used to provide stable device naming where necessary, and CAN configuration was started only after the relevant device initialization had completed. This reduced unnecessary waiting without creating a race condition between hardware discovery and application startup.
The cellular modem was handled separately because it required a power-up sequence before its interfaces became available. Rather than blocking the complete initialization flow, modem power control was moved into its own service. Other components that did not depend on the modem could continue starting while it initialized. The same principle was applied to USB and wireless networking: each service was responsible for one component, declared only its real dependencies, and could be diagnosed independently if startup failed.
Additional System-Level Improvements
Once the service structure was improved, I examined smaller delays in the remaining boot path. Persistent logging, filesystem behavior, random-seed initialization, RF-kill handling, and filesystem checks all affected startup time to some degree. These were not treated as services that could simply be disabled. Each one was evaluated according to the device’s actual requirements and failure model. For example, reducing persistent writes may be reasonable for an embedded device that sends important logs to another system, but it would be a poor decision if local logs were the only way to investigate field failures.
For this system, journald was configured to use volatile storage, reducing unnecessary writes during startup. The root filesystem mount configuration was also reviewed, and options such as noatime and nodiratime were used to avoid metadata writes caused only by file access. Filesystem checks were removed from the normal boot path only where the storage design and recovery strategy made that acceptable. These changes provided smaller improvements than restructuring rc.local, but together they helped shorten the remaining userspace initialization time.
Random-seed initialization and RF-kill handling required similar care. A delay shown by systemd-analyze blame was not sufficient justification for disabling either component. Removing random-seed initialization without understanding how the device uses randomness could weaken security, while disabling RF-kill handling could affect wireless state management. The final configuration was based on the hardware and software requirements of the target image rather than applying generic boot-optimization recommendations blindly.
The Final Results
After restructuring the initialization process and refining the remaining system configuration, the measured startup time fell from approximately 35 seconds to under 10 seconds. In one of the final measurements, the kernel completed in 2.541 seconds and userspace initialization completed in 7.271 seconds, giving a total reported time of approximately 9.813 seconds.
This corresponds to a reduction of roughly 72% compared with the initial startup time. More importantly, the result was achieved without removing the communication interfaces and application services required by the device. CAN, USB networking, wireless networking, modem initialization, and the main application remained part of the startup process. The improvement came primarily from shortening the critical path, allowing independent work to run in parallel, and eliminating unnecessary serialization.
The final number was not treated as the result of a single successful boot. The device was restarted repeatedly, and the required interfaces and services were checked after startup. Hardware initialization time can vary slightly between boots, particularly when modems and network interfaces are involved. The goal was not to produce the smallest possible number once, but to achieve a consistently faster and still reliable startup sequence.
What I Learned
The most important lesson from this work was that embedded Linux boot optimization is not mainly about disabling services. It is about understanding the critical path. systemd-analyze blame is useful for discovering slow units, but it does not show the complete relationship between them. A service with a long execution time may have little impact if it runs in parallel, while a shorter service can become a major bottleneck if several required components are waiting for it. Combining timing measurements, dependency analysis, and boot logs provided a much more accurate picture of the system.
I also learned that service architecture has a direct effect on both performance and maintainability. Moving unrelated operations out of rc.local did more than reduce boot time. It made each initialization step observable, independently restartable, and easier to debug. Explicit dependencies replaced assumptions hidden inside shell scripts, while hardware-specific waits became localized to the services that actually needed them.
Finally, every optimization had to be evaluated as an engineering trade-off. Disabling a filesystem check, changing logging behavior, or removing a hardware wait may improve a timing result, but it can also reduce reliability, security, or diagnostic capability. The objective was therefore not simply to boot as fast as possible, but to create the shortest reliable path from power-on to an operational embedded system.
Reducing the startup time from approximately 35 seconds to under 10 seconds was the visible outcome, but the more valuable result was a cleaner, more predictable, and more maintainable boot process.
References
- systemd, systemd-analyze — Analyze and Debug System Manager. Documentation for measuring boot performance and inspecting the critical startup path.
- systemd, systemd.unit — Unit Configuration. Reference for unit dependencies and ordering directives such as
After=,Before=,Requires=, andWants=. - systemd, systemd.device — Device Unit Configuration. Documentation for representing kernel devices as systemd units and connecting services to hardware availability.
- systemd, journald.conf — Journal Service Configuration. Reference for journal storage modes, including
Storage=volatile. - Yocto Project, Reference Manual: systemd Class. Documentation for installing, packaging, and enabling systemd service files in Yocto recipes.
