Diagnosing systemd services and restart loops
Author: DEWOLF.IT
Use effective unit configuration, journal messages and explicit dependencies to troubleshoot Linux services.
This English edition covers the main steps. See the Dutch article for the full detail and examples.
Find the first failure and the effective unit
Inspect the loaded unit and any drop-ins. Read the journal around the first failure before focusing on the final restart-limit message.
systemctl status example-app.service
systemctl cat example-app.service
journalctl -b -u example-app.service
Reconstruct the execution context
Check the service user, working directory, environment, executable, mounted paths and sandboxing. Compare with the environment used in a manual test. Starting successfully and becoming ready for traffic can happen at different times.
Model dependencies and recovery deliberately
After= orders units and does not pull them in. Wants= and Requires= describe different dependency strengths. Use RequiresMountsFor= for paths the service needs. A bounded restart policy can recover from transient faults without hiding a persistent one.
[Unit]
StartLimitIntervalSec=300
StartLimitBurst=5
[Service]
Restart=on-failure
RestartSec=10s
Verify service behaviour
Validate the unit, reload configuration, apply the change and verify the service from its consumer's side. Inspect the logs and test restart and boot dependencies separately.
See the systemd.service manual.
