In this section
0.10 The Automation Maturity Model
Scenario
Rachel Okafor presents NE's automation program to the board. A board member asks: "Where are we now, and where will we be in six months?" Rachel needs a vocabulary that maps SOC automation capability to a measurable scale. "We have some playbooks" is not an answer. "We are Level 1 across all eight dimensions, targeting Level 3 by month six, with specific milestones at each level" is an answer that a board member can track, fund, and hold the team accountable for.
The five maturity levels
The model defines five levels of automation maturity. Each level represents a qualitative shift in how the SOC operates, beyond a quantitative increase in the number of playbooks.
Level 1, Ad Hoc. No intentional security automation. Default Defender AIR may be running unmonitored. A few Sentinel automation rules for severity changes or assignment routing. Possibly one or two dead playbooks. All enrichment, triage, and containment is manual. The defining characteristic: the SOC operates identically whether automation exists or not. NE is Level 1 today. The 847 automated actions in the Action Center run without the SOC's awareness or involvement.
Level 2, Basic Enrichment. Three to five enrichment playbooks running in production and monitored. The most common alert types are automatically enriched with context from threat intelligence feeds, identity risk scores, and device compliance state. Basic notification via automated email for High and Critical severity. Two or three false positive auto-close rules for the noisiest detection patterns. The defining characteristic: automation saves measurable analyst time, but does not take any response actions. This is NE's Month 1 target.
Level 3, Playbook Response. Full enrichment pipeline covering all incident types. Evidence auto-collection at alert time, preserving volatile data before analyst triage delay causes evidence loss. Notification pipeline with severity routing, Teams adaptive cards, and ticket creation. The first containment playbooks are deployed, typically AiTM session revocation and ransomware endpoint isolation, both with confidence thresholds and blast radius safeguards from Sections 0.4 and 0.5. The defining characteristic: automation both enriches and acts, with human judgment preserved for medium-confidence detections. This is NE's Month 3 target.
Level 4, Orchestrated. Cross-environment containment coordinated across identity, endpoint, and network. Sentinel and Defender XDR automation is explicitly coordinated using the framework from Section 0.8. Confidence-based automation tiering is calibrated using measured false positive rates from production data. Full governance is operational: version control, staging workspace, monitoring, runbooks, and monthly reviews from Section 0.9. Metrics dashboard tracks MTTA, MTTR, automation action count, false positive rate, and analyst time saved. The defining characteristic: automation operates as a coordinated system, not a collection of independent playbooks. This is NE's six-month target.
Level 5, Adaptive. The automation improves itself. Confidence thresholds adjust based on rolling false positive rates. New false positive patterns are automatically detected and added to suppression watchlists. Dynamic playbook routing adapts containment actions based on real-time blast radius assessment. The defining characteristic: the system learns from its operational data and requires less manual tuning over time. Level 5 is aspirational for most organizations. Level 4 is the realistic target that this course builds toward.
Figure 0.10a: The five maturity levels mapped to NE's 90-day and six-month targets. Eight assessment dimensions scored 1-5 each produce the overall maturity level. The course progression follows the level sequence: enrichment first, then evidence and notification, then containment, then orchestration.
The eight assessment dimensions
Each dimension is scored 1 through 5, matching the maturity levels. Score honestly. The assessment is for your internal planning, not for a report.
Enrichment coverage. Level 1: no enrichment automation. Level 2: one or two enrichment playbooks covering partial alert types. Level 3: full enrichment pipeline with five or more data sources covering all incident types. Level 4: enrichment with watchlist-driven dynamic logic that adapts queries based on entity type. Level 5: enrichment with automated quality feedback that detects when enrichment data is stale or incomplete.
Evidence collection. Level 1: all manual, evidence captured after analyst opens the incident. Level 2: partial auto-collection for one incident type. Level 3: auto-collection for the top three incident types (AiTM, endpoint, email). Level 4: auto-collection with chain-of-custody metadata (timestamps, hash verification, collection method documented). Level 5: adaptive collection that adjusts scope based on alert severity and entity type.
Notification maturity. Level 1: manual email composition during incidents. Level 2: basic automated email on alert creation. Level 3: Teams adaptive cards plus ticket creation plus severity routing. Level 4: approval gates for containment plus escalation timeouts plus stakeholder-specific formatting. Level 5: context-aware notification that adapts content per audience automatically.
Containment automation. Level 1: all manual containment. Level 2: false positive auto-close rules only. Level 3: one containment playbook with confidence thresholds and blast radius safeguards. Level 4: cross-environment containment coordinated across identity, endpoint, and network. Level 5: adaptive containment with self-tuning thresholds based on rolling false positive rates.
Governance. Level 1: no governance. Level 2: some monitoring configured. Level 3: version control plus monitoring plus runbooks. Level 4: full governance with staging workspace, testing pipeline, and monthly review cycle. Level 5: automated governance with CI/CD for playbook deployment and automated testing.
Platform coordination. Level 1: no coordination between Sentinel and Defender XDR. Level 2: awareness of both platforms but no explicit coordination. Level 3: playbooks check entity state before acting to avoid duplicate actions. Level 4: unified dashboard monitoring both platforms with explicit coordination rules. Level 5: automated conflict resolution and dynamic action routing between platforms.
Metrics. Level 1: no automation metrics. Level 2: basic counts (playbook executions, success/fail). Level 3: MTTA and MTTR tracking with before/after comparison. Level 4: full dashboard with false positive rate, rollback frequency, time saved, and cost impact. Level 5: predictive analytics on automation performance trends.
Continuous improvement. Level 1: no improvement process. Level 2: reactive improvement (fix when broken). Level 3: quarterly review. Level 4: monthly review with threshold tuning cycle from Section 0.9. Level 5: continuous data-driven improvement with automated anomaly detection on automation performance.
The progression path
The maturity model maps directly to course modules. This is not abstract: each level advancement corresponds to specific playbooks and configurations that you build, test, and deploy.
Level 1 to Level 2 is the enrichment sprint. SA1 builds the first enrichment playbooks. SA2 expands enrichment coverage to all incident types. The deploy gate from Section 0.9 applies to every playbook. Monitoring is configured on day one. By the end of Month 1, analysts open incidents that already contain IP reputation, user risk score, device compliance state, and alert correlation history. The MTTA improvement is immediate and measurable.
Level 2 to Level 3 is the response expansion. SA3 adds evidence collection playbooks that capture volatile data at alert time. SA4 builds the notification pipeline with Teams adaptive cards, email routing, and ticket creation. SA5 through SA7 build the first containment playbooks with the confidence thresholds from Section 0.4 and the blast radius safeguards from Section 0.5. The transition from Level 2 to Level 3 is the moment where automation starts taking actions, where previously it only provided context. The governance framework from Section 0.9 is mandatory before any containment playbook enters production.
Level 3 to Level 4 is the orchestration phase. SA8 through SA10 coordinate cross-environment response, align Sentinel and Defender XDR actions, and build the operational metrics dashboard. The monthly review cycle begins. Confidence thresholds are tuned using production data. The automation program transitions from a project (build and deploy) to an operational capability (monitor, measure, improve).
Anti-Pattern
Starting with containment because the CISO asked for it
The CISO's first question is always "can we automatically contain threats?" The correct answer is yes, but not first. Containment is Level 3 capability. Deploying containment playbooks without the enrichment pipeline (Level 2) means the containment decision relies on the raw alert severity alone, without the context that confidence scoring requires. Deploying containment without governance (Section 0.9) means no rollback procedure when the playbook makes a wrong decision. Start with enrichment. It has zero blast radius, builds organizational trust in automation, and creates the data pipeline that the confidence thresholds need to function. The CISO gets containment in Month 3, built on a foundation that makes the containment decision reliable.
Scoring your own environment
Run the assessment against your own SOC. Score each dimension from 1 to 5 based on current capabilities, not aspirations. Add all eight scores and divide by eight. The result is your current maturity level.
The most common scoring error is overrating governance. Teams that have one or two monitoring alerts configured rate themselves at Level 3 governance, but Level 3 requires version control, monitoring, and runbooks operating together. A single analytics rule that detects playbook failures is Level 2 monitoring. Level 3 governance means the ARM template is in Git, the monitoring rule is active, and the runbook is published and current. Score the dimension based on the weakest component, not the strongest.
The second common error is conflating Defender XDR platform automation with SOC-built automation. Attack disruption and AIR are Microsoft-built capabilities that run independently of your automation program. They contribute to the platform coordination dimension but do not raise your enrichment, collection, or containment scores. NE has 847 automated actions in the Action Center, but those actions are platform automation, not SOC automation. NE's SOC-built automation score is still Level 1.
The score is useful in two ways. First, it identifies the dimensions with the lowest scores, which are your highest-priority improvement areas. If enrichment is at Level 1 and governance is at Level 2, enrichment is the priority because it blocks advancement across every other dimension. You cannot build reliable containment without enrichment data feeding the confidence scoring model. Second, the score produces the before/after measurement that demonstrates program value. Your pre-course assessment and your post-course assessment, compared on the same eight dimensions, quantify the impact of the automation program you built.
Set realistic targets. Advancing from Level 1 to Level 2 in one dimension in 30 days is achievable for enrichment and notification. Advancing from Level 1 to Level 4 in 90 days is not achievable for any dimension. The course is structured for one level advancement per month across the primary dimensions (enrichment, collection, containment), with governance and metrics advancing in parallel. An organization that starts at Level 1 and reaches Level 2.4 in 90 days has made significant, measurable progress. An organization that aims for Level 4 in 90 days and reaches Level 2.0 has made the same progress but perceives it as failure because the target was unrealistic.
Automation Principle
Maturity is measured, not declared. Score your SOC on the eight dimensions before starting the automation build, and score it again after each major milestone. The delta between assessments is the evidence that the program is delivering value. A SOC at Level 3 with measured improvement from Level 1 is more credible to auditors, leadership, and the team than a SOC that claims Level 4 without a documented baseline.
Get weekly detection and investigation techniques
KQL queries, detection rules, and investigation methods — the same depth as this course, delivered every Tuesday.
No spam. Unsubscribe anytime. ~2,000 security practitioners.