https://wiki.gobyl.cc/index.php?action=history&feed=atom&title=Process_incident_responseProcess incident response - Revision history2026-10-20T18:25:55ZRevision history for this page on the wikiMediaWiki 1.43.11https://wiki.gobyl.cc/index.php?title=Process_incident_response&diff=6&oldid=prevAdmin: seed2026-10-20T15:00:49Z<p>seed</p>
<p><b>New page</b></p><div>Incident response process for customer systems.<br />
<br />
# An alert (Icinga) or ticket (Zammad) comes in.<br />
# Make sure a Zammad ticket exists for the incident. Icinga alerts get a new ticket.<br />
# Announce in the Incidents channel that the alert is being worked on.<br />
# Check the [[Maintenance calendar]] and open tickets: if the affected host has planned work in the current window, do not remediate. Note the overlap in the channel and ticket, and confirm with the owner.<br />
# Fetch access credentials from OpenBao (<code>secret/customers/&lt;name&gt;</code>).<br />
# Investigate on the host: service status, logs, disk, recent changes. Prefer read-only commands first.<br />
# If the cause is clear and the fix is routine (restart a service, free disk space, restore a config), apply it and verify recovery.<br />
# If the fix is unclear, risky, or data-destructive: stop, summarize findings, and hand over to the on-call engineer.<br />
# Update the ticket with root cause, actions taken and verification. Post a summary in the channel.</div>Admin