0.9 The Automation Governance Framework

Why automation degrades without governance

Automation is code. Logic Apps are defined as JSON ARM templates. Automation rules have exportable configurations. KQL queries in custom detections are text files. All of these are code artifacts that follow the same degradation pattern as any unmanaged code: they work on day one, drift from requirements over weeks, break silently over months, and become untouchable legacy over quarters.

The degradation is not theoretical. NE's VirusTotal playbook followed the exact pattern. An analyst built it during a quiet week. It worked. Nobody documented it. Nobody monitored it. The external API changed its rate limits. The playbook started returning HTTP 429 errors on every call. SentinelHealth showed successful launches. AzureDiagnostics (which was never configured) would have shown the internal failures. Six months later, the SOC lead's conclusion was "automation doesn't work" when the accurate conclusion was "unmonitored automation doesn't work."

The governance framework has four pillars. Each one addresses a specific failure mode that kills automation programs. Version control prevents the "rebuild from scratch" scenario when a change breaks something. Testing prevents deploying automation that fails on edge cases the builder didn't anticipate. Monitoring prevents silent failures from accumulating into coverage gaps. Documentation prevents institutional knowledge from walking out the door when the builder changes roles.

Pillar 1: version control

Every playbook exists as an ARM template in a Git repository. When an analyst modifies a playbook in the Logic App designer, they export the updated ARM template, commit it to the repository, and create a pull request. A second analyst reviews the change, checking for logic errors, permission scope creep, and blast radius implications. After approval, the change is merged and deployed.

The repository structure follows infrastructure-as-code conventions. One directory per playbook. Each directory contains the ARM template (the deployable artifact), a README (purpose, trigger, permissions, actions), and a CHANGELOG (what changed, when, and why). Automation rules are exported as workspace-independent ARM template JSON and stored in the same repository under a separate directory.

This workflow catches errors before they reach production. A PR reviewer sees that the updated enrichment playbook now requests Directory.ReadWrite.All when the previous version used User.Read.All, and flags the permission escalation before it is deployed. Without the PR review, the permission change is invisible until the next audit.

For teams where a full Git workflow is not yet practical, the minimum standard is: export the ARM template after every modification and store it in a shared location. Even without PR reviews, having a recoverable copy of each playbook version prevents the scenario where a change breaks something and nobody can get back to the working version.

Pillar 2: testing

Never deploy automation directly to production. The workflow is: build in the Logic App designer, test against sample data, deploy to a staging workspace with test incidents, validate behavior, then promote to production.

The staging workspace is a second Sentinel workspace with minimal data ingestion and manually created test incidents. It does not need to be expensive. A Log Analytics workspace on the free tier, connected to Sentinel, with a handful of test incidents that cover the alert types your playbooks process, is sufficient for validation.

For Tier 1 automation (enrichment), testing confirms that the playbook executes successfully, that entity extraction handles all alert types the playbook will encounter, and that the enrichment data appears in the incident comments. Create one test incident per alert type that the playbook processes and verify the output.

For Tier 3 automation (containment), testing is mandatory. Create dedicated test accounts and test devices that can be safely disabled, isolated, and restored. The test account receives the containment action. The playbook verifies the action succeeded. The rollback playbook restores the account. All of this happens in the staging environment with test entities, never against production users or production devices.

Pillar 3: monitoring

Every playbook must be monitored for success and failure. Logic Apps provide run history where every execution is logged with status (Succeeded, Failed, Cancelled), duration, and error details. Nobody checks run history manually. Monitoring must be automated, and it requires both data sources that Section 0.7 introduced: SentinelHealth and AzureDiagnostics.

SentinelHealth captures whether each automation rule evaluated successfully and whether each playbook was launched. AzureDiagnostics captures the internal execution of each Logic App workflow, including individual action success or failure, execution duration, and error messages. Without both, you see only half the picture. NE's VirusTotal playbook launched successfully every time (SentinelHealth would have shown green). The HTTP action inside it failed every time (AzureDiagnostics would have shown the 429 errors).

The minimum monitoring configuration is a Sentinel analytics rule that alerts when any playbook's internal actions fail. The rule queries AzureDiagnostics for Logic App workflow runs where any action returned a non-success status code. When it fires, the SOC team investigates the failure before it becomes a gap in coverage. SA1 builds the full automation health monitoring dashboard, but the failure detection alert deploys on day one.

Beyond failure detection, track three operational metrics over time. Mean execution duration tells you whether the playbook is getting slower, which often indicates an upstream API degradation. Success rate over 30 days tells you whether reliability is declining, which often indicates a schema change in the data source. Action count tells you how many incidents the playbook processes, which validates that the automation rule is triggering correctly.

Pillar 4: documentation

Every playbook has a runbook. The runbook answers six questions: what does this playbook do (one-paragraph description of the complete workflow), what triggers it (the specific automation rule conditions), what actions does it take (every action including external API calls and containment actions), what can go wrong (known failure modes including API rate limits, timeout on large queries, entity extraction failures, permission expiration), how do I override or disable it (how to temporarily pause without deleting), and who owns it (the person responsible for fixing it when it breaks, including their replacement if they leave).

The runbook is not optional. It is the difference between "the automation broke and we fixed it in 15 minutes" and "the automation broke and we spent three hours reverse-engineering the Logic App JSON before we could diagnose the failure." When Tom Ashworth leaves NE, his enrichment playbook either has a runbook that Marcus Webb can use to maintain it, or it becomes the next dead playbook in the next audit.

The monthly review cycle

Governance is not a one-time activity. The monthly automation review is a standing meeting where the SOC lead and the automation owners review the health of every production playbook. The agenda has four items.

Performance review: for each playbook, examine the 30-day success rate, mean execution duration, and action count. A playbook with a declining success rate needs investigation. A playbook with increasing execution time may indicate an upstream API degradation or a growing data volume that exceeds the query timeout.

Threshold tuning: review the confidence thresholds from Section 0.4 using the 30-day false action rate. If the enrichment playbooks show that a specific detection type consistently scores above the containment threshold but produces false positives on containment, the threshold for that detection type needs to increase. If another detection type consistently scores above the threshold with zero false positives, the threshold for that type could safely decrease.

Retirement assessment: identify automation that no longer delivers value. An enrichment playbook for a threat intelligence feed that was decommissioned. An automation rule that assigns incidents to an analyst who left six months ago. A notification playbook for a Slack channel that the team no longer monitors. Disable, archive, document.

Permission audit: review the managed identity permissions for each playbook against the minimum required scope. Over time, permissions accumulate. A playbook that was originally enrichment-only may have had a "temporary" write permission added for a one-time containment test and never removed. The monthly review is where that permission creep is caught and corrected.

The deploy gate

No playbook enters production without passing the deploy gate. The gate is a checklist that the automation owner completes and the SOC lead verifies before the automation rule is enabled in production. The gate covers all four pillars: ARM template committed to Git, tested in staging with representative test incidents, monitoring analytics rule configured and active, runbook published with all six questions answered, owner assigned with contact information and next-review date, and rollback procedure documented and verified.

The deploy gate is the single control that prevents the dead playbook pattern. Every playbook that has ever degraded silently in any SOC environment would have been caught earlier if a deploy gate had required monitoring configuration before production activation. The gate does not prevent deployment. It ensures that when (not if) the playbook encounters a problem, the team has the tooling and documentation to detect and resolve it quickly.