Incident Response for Small Teams: What to Do When Production Breaks at 2am

At 2am, your phone buzzes. Uptime monitor: your site is down. In a large company: the on-call engineer triages, escalates to the relevant team, and a practiced runbook guides resolution. In a small team: you are the on-call engineer, the relevant team, and the person who writes the runbook. While half-asleep.

Here is the incident response process I developed for our small team (3 developers, no dedicated DevOps) that minimizes downtime, maximizes learning, and prevents the same incident from recurring.

Phase 1: Assessment (First 5 Minutes)

Do not immediately start debugging. First: assess severity.

Check: is the site completely down, or partially degraded? How many users are affected (everyone, or a specific segment)? Is data being lost or corrupted? Are financial transactions affected?

Severity levels: Critical (site completely down, data loss, or security breach — all hands immediate response), High (major feature broken, significant user impact — respond within 15 minutes), Medium (degraded performance, non-critical feature broken — respond within 1 hour), and Low (cosmetic issue, minimal impact — address next business day).

For Critical and High: wake up fully. Make coffee. You need clear thinking for the next 30-60 minutes. Rushing while confused makes things worse.

Phase 2: Communication (Minutes 5-10)

Before fixing anything: communicate. Users seeing a broken site with no acknowledgment lose trust faster than users seeing a broken site with "We are aware and working on it."

Public status update: "We are experiencing issues with [specific feature]. Our team is investigating. Updates every 30 minutes." (Use a status page: Statuspage.io, Betteruptime, or even a pinned tweet.)

Internal: notify relevant team members. Even if you can fix it alone: someone else should know (in case it escalates or you need help).

Template: "[Severity]. [What is affected]. [Who is working on it]. [Next update in X minutes]."

Phase 3: Diagnosis (Minutes 10-30)

Systematic diagnosis, not guessing:

Step 1: Check monitoring dashboards. What changed? Error rate spike? Response time increase? Database CPU? Memory usage? Disk space?

Step 2: Check recent deployments. Was anything deployed in the last few hours? (The most common cause of incidents is recent code changes.)

Step 3: Check logs. Error messages often point directly to the cause. Filter by timestamp (when did the problem start?) and severity (errors, not info).

Step 4: Check external dependencies. Is your database provider having issues? Your CDN? Your email service? Check their status pages.

Common causes (in order of frequency): recent code deployment introduced a bug, database connection exhaustion (connection pooling section relevant here), third-party service outage (dependency failure), infrastructure capacity (disk full, memory exhausted), and certificate expiration (HTTPS broken).

Phase 4: Resolution

Once you identify the cause:

If recent deployment caused it: roll back immediately. Do not debug in production under pressure. Roll back, confirm resolution, then debug the issue in a development environment tomorrow.

If infrastructure capacity: scale up temporarily (add instances, increase resources). Investigate root cause later.

If third-party outage: check their status page for ETA. Communicate to users that the issue is upstream. Implement fallback if available.

If data issue: fix carefully. Database operations under pressure are dangerous. Take a backup before making changes. Document every change.

The principle: restore service first, understand the problem second. A rolled-back deployment that we debug tomorrow is better than 2 hours of debugging while users cannot use the site.

Phase 5: Verification

After applying the fix: verify it worked. Do not assume.

Check: monitoring shows normal metrics (error rate dropped, response time normalized), manually test the affected feature (click through the user flow), and confirm users are seeing the fix (check a few specific data points).

Update the status page: "The issue has been resolved. Service is operating normally. We will publish a post-mortem within 48 hours."

Phase 6: Post-Mortem (Next Business Day)

Every significant incident (High or Critical) gets a post-mortem. Not blame — learning. The post-mortem answers:

What happened: timeline from first alert to resolution. What exactly broke and what was the impact.

Why it happened: root cause analysis. Not "the deployment was buggy" — but "the migration script assumed all users had a profile record, but 3% of users created before 2024 do not."

How we resolved it: what fix was applied and how long resolution took.

How we prevent recurrence: specific action items. "Add a database migration that backfills missing profile records. Add a null check in the query. Add a test that verifies the query handles missing profiles."

The Runbook

For recurring incident types: document the resolution steps. A runbook is: specific instructions for diagnosing and resolving a known type of incident. Written simply enough that a stressed person at 2am can follow them.

Our runbooks cover: database connection exhaustion (check connections, restart pooler, identify leaking queries), deployment rollback procedure (specific commands, verification steps), and third-party service failover (switch to backup, update DNS, notify users).

On-Call Rotation for Small Teams

With 3 developers: one week on-call rotation. The on-call person: monitors alerts during their week, responds to Critical/High incidents at any hour, handles Medium incidents during business hours, and documents any incident for the team.

Making on-call sustainable: no more than one week in three (fair rotation), compensate on-call (time off, extra pay, or similar), keep the system reliable (reduce alert frequency through prevention), and respect quiet hours (Low/Medium issues wait until morning).

Alert Quality

Bad alerts: every error triggers an alert (alert fatigue — you stop responding), alerts without actionable information ("something is wrong" — not helpful), and too many alerts for the same issue (first alert is useful, 47th is noise).

Good alerts: alert only when user-facing impact occurs (not every internal error), include context (which service, what error, which users affected), and deduplicate (one alert per incident, not one per occurrence).

Invest in alert quality. An engineer who receives 50 alerts per week treats them all as noise. An engineer who receives 2 alerts per week treats each as significant.

Prevention Is Cheaper Than Response

Every incident is an opportunity to prevent the next one. After each post-mortem: implement the prevention action items, add monitoring for the failure mode (detect it before users do), add the scenario to your test suite (prevent regression), and update runbooks (include what you learned).

Over time: incident frequency decreases, resolution time decreases (better runbooks, better monitoring), and severity decreases (problems caught earlier, before full outage). This is the incident response virtuous cycle: each incident makes the system more resilient to future incidents.