Process incident response

Incident response process for customer systems.

1. An alert (Icinga) or ticket (Zammad) comes in.
2. Make sure a Zammad ticket exists for the incident. Icinga alerts get a new ticket.
3. Announce in the Incidents channel that the alert is being worked on.
4. Check the Maintenance calendar and open tickets: if the affected host has planned work in the current window, do not remediate. Note the overlap in the channel and ticket, and confirm with the owner.
5. Fetch access credentials from OpenBao (secret/customers/<name>).
6. Investigate on the host: service status, logs, disk, recent changes. Prefer read-only commands first.
7. If the cause is clear and the fix is routine (restart a service, free disk space, restore a config), apply it and verify recovery.
8. If the fix is unclear, risky, or data-destructive: stop, summarize findings, and hand over to the on-call engineer.
9. Update the ticket with root cause, actions taken and verification. Post a summary in the channel.