The first 15 minutes of a Linux incident

01. Start with impact

When an alert arrives, the first job is to understand what stopped working. A busy host and an unavailable application are different problems. Write down the affected service, the first known failure time, and the user action that fails.

Try the same request from a second vantage point if one is available. Check whether the issue affects one instance, one network path, or every user. Record the time zone with each observation so application logs and monitoring events can be compared.

02. Take a small host snapshot

Start with a few focused checks. Avoid collecting so much output that the useful signal disappears.

date -Is
uptime
free -h
df -h
df -i
systemctl --failed --no-pager

uptime gives context about recent boots and load. free -h shows memory; pay attention to the available value, not just the free column. df -h checks filesystem space, while df -i checks inode usage. A filesystem can run out of either.

Compare findings with the host’s normal behavior. High load is a clue to investigate, not proof that the CPU is the root cause. A failed unit can also be unrelated to the affected application.

03. Narrow the logs to the service

On an Apache host where the unit is named apache2, inspect its status and recent journal entries. Replace the unit name for your service.

systemctl status apache2 --no-pager
journalctl -u apache2 --since "15 min ago" --no-pager
journalctl -p warning --since "15 min ago" --no-pager

The unit filter helps isolate one service. The priority filter includes warning messages and more severe entries, so it is a useful second view rather than a complete history. Application-specific log files may contain details that never reach the journal. The journalctl manual documents these filters.

Look for a sequence: a connection timeout, then exhausted workers, then failed requests. The first visible error and the underlying cause may be different events.

04. Test one explanation at a time

Connect an observation to a testable explanation. “The database connection failed immediately after a credentials change” gives you a specific next check. “Linux is slow” does not.

Review recent deployments, configuration changes, and dependency incidents. Timing helps prioritize the investigation, but does not establish causation. Coordinate any remediation through your incident process and preserve the evidence needed to evaluate it.

If a known change is the likely trigger, assess whether its documented rollback is appropriate. After a recovery action, verify the same user request that originally failed and watch whether the error rate returns to normal.

05. Leave the next person a useful note

A short incident note prevents repeated investigation. Separate observations from assumptions, and include an owner for the next action.

Impact: which users and which operation?
First observed: timestamp and time zone
Evidence: logs, metrics, and recent changes
Current hypothesis: what explains the evidence?
Action taken: what changed, and when?
Verification: does the user operation work?
Next action: owner and next update time

The goal of the first fifteen minutes is a smaller problem, a clearer timeline, and a defensible next step. Some incidents will take longer; a consistent starting routine makes that time easier to use.