DE0.2 Why Organizations Need Detection Engineers

The organizational investment gap

Most security programs invest in three layers. Prevention — firewalls, endpoint protection, email filtering, Conditional Access, MFA. Operations — a SOC team or a managed SOC partner to monitor alerts, triage incidents, and escalate threats. Response — incident response plans, forensic tooling, maybe a retainer with an IR firm for major incidents.

These three layers receive budget, headcount, and executive attention. The fourth layer — the system that determines what the SOC can actually see — typically receives none. The detection library is treated as a byproduct of the SIEM deployment rather than an engineered system that requires dedicated skill, dedicated time, and ongoing maintenance. The assumption is that deploying Sentinel and enabling template rules means threats are detected. It doesn't. It means alerting infrastructure is deployed. What that infrastructure alerts on — and what it misses — depends entirely on the quality of the detection rules. And in most organizations, nobody is responsible for that quality.

This creates an unusual situation. The organization pays for the SIEM. It pays for the data ingestion — potentially tens of thousands of dollars per month for Sentinel's Log Analytics consumption. It pays for the SOC analysts who triage the alerts. It pays for the managed SOC partner who monitors after hours. It pays for the IR retainer in case a breach reaches containment stage. But it doesn't pay for the person who determines whether the rules that generate the alerts the SOC triages actually detect the attacks the organization faces. The most expensive link in the chain — the human judgment that decides which techniques get detected and which don't — receives no investment.

This is not a niche problem affecting under-resourced organizations. It's the default state across the industry. Organizations with mature SOC teams, substantial security budgets, and experienced leadership still operate detection libraries that were configured during the SIEM deployment, augmented reactively after incidents, and never systematically evaluated, tested, or maintained. The structural gap is not about budget or talent — it's about the absence of a function that treats detection as an engineering discipline rather than a deployment task.

Estimated time: 35 minutes.

What organizations look like without detection engineering

The consequences of the missing function compound over time. Each one makes the others worse.

The detection library is frozen in time

Rules were deployed during the Sentinel implementation — typically 12-24 months ago. A few custom rules were added after major incidents. Nobody has added a new rule in months.

Meanwhile, the threat landscape has shifted. AiTM phishing kits that bypass MFA became commodity tooling. Cloud-native persistence through OAuth consent grants became a standard technique. Adversaries began abusing workload identities and AI agent permissions.

The detection library still looks for last year's threats. The gap between what the rules detect and what attackers actually do widens every month, invisibly, because nobody measures it.

Run this query to see how long your rules have been operating without modification:

// How stale is your detection library?
SentinelAudit
| where TimeGenerated > ago(365d)
| where Description has "analytics rule"
| summarize
    LastChange = max(TimeGenerated),
    Changes = count()
    by SentinelResourceName
| extend DaysSinceChange = datetime_diff('day', now(), LastChange)
| sort by DaysSinceChange desc

SentinelResourceName                     LastChange            Changes  DaysSinceChange
───────────────────────────────────────  ────────────────────  ───────  ───────────────
Suspicious inbox rule creation           2024-11-12T14:22Z     1        554
Impossible travel activity               2024-11-12T14:23Z     1        554
New country sign-in                      2024-11-12T14:24Z     1        554
MFA failure threshold                    2025-01-08T09:15Z     2        497
Password spray detected                  2024-11-15T10:41Z     1        551

Changes = 1 means the rule was created and never modified. DaysSinceChange above 365 means nobody has tuned, reviewed, or updated the rule in over a year. If most of your rules show a single change event from the same day — that was the implementation day. Everything since has been unmanaged.

Coverage is a mystery

Nobody has mapped the active rules to ATT&CK techniques. The CISO reports to the board that "we have 35 active analytics rules" — because rule count is the only number available.

Whether those 35 rules cover 5% or 50% of the attack techniques relevant to the organization is unknown. Worse, nobody knows which techniques are relevant. The threat modeling that would identify priority techniques hasn't been done because the function that does it doesn't exist.

The board hears "35 rules" and assumes protection. The number means nothing without coverage context.

False positives destroy trust

Template rules produce false positives because they're calibrated for a generic enterprise, not for this specific organization. The impossible travel rule fires on field engineers who fly between sites. The suspicious PowerShell rule fires on admin automation. The MFA failure rule fires on users mistyping OTP codes.

The SOC learns which rules to ignore. Over six months, the ignore list expands to include rules that occasionally catch real threats.

Real attacks that arrive as low-to-medium severity alerts from "usually noisy" rules get auto-closed.

Knowledge lives in people's heads

The SOC lead who wrote the custom rules leaves the organization. The rules remain active, but nobody understands them — what they detect, why the thresholds are set as they are, which exclusions exist and why.

The new SOC lead doesn't touch the rules because modifying a rule you don't understand risks breaking something that works. Six months later, a data source schema changes and two rules start returning zero results. Nobody notices because nobody monitors rule health.

Post-incident revelations repeat

The pentest report lists twelve techniques the red team used. The detection library covers two.

The incident post-mortem reveals that telemetry existed at every phase — SigninLogs captured the compromised authentication, DeviceProcessEvents captured the lateral movement, OfficeActivity captured the data staging — and no rule examined any of it.

The organization adds one or two rules in response. The next pentest reveals the same structural problem with different techniques. Without a systematic approach, the organization is permanently one incident behind.

The investment case can't be made

The CISO wants to hire a detection engineer, but they can't quantify the gap because the measurement framework doesn't exist.

"Our rules might not be good enough" is not a board-level argument. "We cover 8% of the 145 ATT&CK techniques relevant to our industry" is a board-level argument. But producing that sentence requires the exact function the CISO is trying to fund.

Detection engineering solves this circular problem by making its own value measurable from day one.

What changes when the function exists

The contrast is not theoretical. It's measured.

The detection library evolves

New threat advisories, hunt findings, and incident post-mortems feed a prioritized backlog. Detection rules are developed on a sprint cadence — two to three new rules per week, four to six tuning passes per month, coverage reviewed quarterly.

When Mandiant publishes a report on a campaign targeting manufacturing companies, the detection engineer evaluates it against the existing library, identifies the uncovered techniques, and has production rules deployed within days. The organization's detection capability adapts at the speed of threat intelligence.

Coverage is measured and communicated

ATT&CK coverage percentage is calculated monthly. The board report says "coverage improved from 10% to 48% over six months" — a statement that leadership understands without technical translation.

The gaps are documented as accepted risk with a remediation timeline and cost estimate. For the first time, the organization can answer "are we protected?" with something other than "we have a SIEM."

False positive rates trend downward

The monthly tuning cadence reviews every false positive from the previous 30 days. Each one is classified by root cause: threshold too broad, legitimate pattern not excluded, detection logic matches an unintended behavior, data quality issue.

Each root cause gets a targeted fix. Over six months, the false positive rate drops from the typical 40-50% of an untuned library to 15-20%. The SOC's trust in the alert queue recovers.

Detection is proactive

The detection engineer doesn't wait for an incident to reveal a gap. They model threats proactively and build rules before the attack arrives.

Consider the operational difference. A threat advisory drops on Tuesday morning: attackers are using OAuth consent grants to establish persistent mailbox access that survives password resets.

Without detection engineering: The advisory gets read, maybe discussed in a meeting, possibly added to a risk register. Nobody writes a new rule. The technique becomes exploitable indefinitely.

With detection engineering: By Tuesday afternoon, the engineer has reviewed the ATT&CK mapping (T1098.003) and written a hypothesis. By Wednesday, the rule is built, tested, and deployed in report-only mode. By the following Monday, it's promoted to production. Advisory to deployed detection in five business days.

The program's value is self-evident

Four metrics tracked monthly with trend lines. The board report: "Coverage increased from 10% to 48%. False positive rate decreased from 43% to 22%. Median detection time is 8 minutes. The program costs one FTE. The industry average breach cost for our sector is $4.88 million."

Detection engineering's measurement framework turns security investment from a cost-of-fear conversation into a coverage-gap conversation. The first is hard to fund. The second is straightforward.

Knowledge stays in the system

Every detection rule has a written specification. The detection-as-code pipeline stores rules in version control with commit history, code reviews, and deployment records. The tuning log documents every adjustment with the evidence that prompted it.

When the detection engineer leaves, their successor inherits a documented system — not a collection of undocumented queries that only the original author understood.

The career dimension

Detection engineering is one of the most in-demand specializations in cybersecurity today. The demand exists because the organizational gap described above exists nearly everywhere, and organizations are starting to recognize it. The role sits at the intersection of three high-value skill sets: threat intelligence (understanding adversary behavior), security engineering (building production systems), and data engineering (working with large-scale telemetry). The combination is rare. Each skill set is valuable independently. Someone who can model threats, engineer production detection systems, and work with the data at scale commands significant market value.

The career paths into detection engineering are varied, but they share a common pattern: each begins in a role that touches detection from one angle, and detection engineering is the step that integrates all angles.

SOC analysts who want to build rather than triage move upstream. They already understand the alert queue from the consumer's perspective — what makes an alert useful, what makes one noise, what entity enrichment an analyst needs to triage quickly. That consumer understanding is invaluable for building rules that produce good alerts. The analyst who becomes a detection engineer writes rules with better entity mapping, clearer severity rationale, and more useful response procedures because they know what it's like to receive a bad alert.

Threat hunters who want to codify their findings move sideways. They already think in hypotheses and work with the same telemetry. The difference between a hunt finding and a detection rule is persistence — the hunt finding is a one-time discovery, the detection rule is that discovery automated and running every five minutes. The hunter who becomes a detection engineer turns their best hypotheses into production rules that catch the technique continuously instead of once.

Security engineers who want to specialize move inward. They already build production systems, understand CI/CD pipelines, and operate infrastructure. The detection-as-code practice — Git, automated testing, versioned deployment — is natural territory. What's new is the adversarial dimension: the detection rule isn't just a deployed configuration, it's a detection that an attacker will actively try to evade.

Incident responders who want to prevent rather than react move left in the timeline. They already understand attack chains, evidence analysis, and the consequences of late detection. The responder who spent three weeks investigating a BEC compromise and discovered that the telemetry existed at every phase — that responder becomes the detection engineer who writes the seven rules that catch the next BEC in the first hour.

Whatever your starting point, the transition to detection engineering is a shift from reactive to proactive. From responding to what the rules catch to deciding what the rules catch. From hoping the detection works to knowing it works because you built it, tested it, and measured it yourself. That shift — from consumer to architect — is the transformation this course delivers.