TH0.3 The Detection Pyramid

Three categories of threat

Not all threats are the same, and the operational response to each category is fundamentally different. Confusing these categories — treating all threats as if they were the same kind of problem — is one of the most common strategic errors in security operations. It leads to organizations that invest heavily in detection engineering and then cannot understand why sophisticated attacks still succeed.

Known-known threats. The attack technique is documented. The telemetry that records it is identified. A detection rule exists and is deployed. When the attack occurs, an alert fires. This is the domain of detection engineering — and it is the only category that automated detection covers.

Examples: phishing email delivery matching a known URL pattern. Malware execution matching a known file hash. Sign-in from an IP address on a threat intelligence blocklist. Credential stuffing producing a volume of failed authentications that exceeds the rule threshold.

Known-unknown threats. The attack technique is known — it appears in MITRE ATT&CK, in threat intelligence reports, in conference presentations. But no detection rule exists in your environment for this specific technique, or the rule that exists has been evaded by a variant the rule author did not anticipate. You know this category of attack is possible. You suspect it may be occurring. But you have no automated mechanism to confirm or deny it.

Examples: OAuth consent phishing where the malicious application name does not match your rule’s pattern list. AiTM session hijacking from infrastructure that is not yet in any threat intelligence feed. Lateral movement via pass-the-ticket where the Kerberos anomaly is too subtle for your current detection threshold. Data exfiltration via SharePoint download that falls just below your volume alert trigger.

This is the domain of threat hunting. The technique is known. The hypothesis is formable. The data exists in your logs. What is missing is someone looking.

Unknown-unknown threats. The attack technique has not been documented. No one has published detection logic for it. No one has written an ATT&CK entry for it. It is a novel technique, a zero-day exploitation of a platform behavior, or a creative combination of legitimate operations that no one has previously identified as an attack path. You cannot hunt for something you cannot hypothesize.

Examples: a novel method of abusing a legitimate M365 feature that has not been reported. An attacker using a combination of Graph API calls that individually look benign but collectively constitute a data exfiltration pipeline. A supply chain compromise through an application integration that security researchers have not yet analyzed.

This is the domain of anomaly detection — statistical baselining that identifies deviation from normal without requiring a predefined hypothesis. The system does not know what the attack looks like. It knows what normal looks like, and it flags when something deviates.

Figure TH0.3 — The detection pyramid. Detection engineering covers the base (known-known). Hunting covers the middle (known-unknown). Anomaly detection covers the apex (unknown-unknown). An organization that only invests in the base has no visibility into the two categories that produce the most damaging intrusions.

The paradox at the top

The pyramid narrows as you move up — there are fewer unknown-unknown techniques than known-unknown techniques, and fewer known-unknown techniques than known-known techniques. But the damage distribution is inverted. The known-known threats at the base are high-volume but relatively low-impact individually — commodity phishing, opportunistic credential stuffing, known malware families. They produce alerts, they consume SOC analyst time, and they are important to detect. But they rarely produce the intrusions that make headlines.

The threats at the top of the pyramid — the novel techniques, the carefully planned intrusions, the adversaries who study your defenses before acting — are low-volume but catastrophically high-impact. These are the nation-state actors with 90-day dwell times. The ransomware affiliates who spend two weeks mapping your environment before encrypting it. The insider who finds a data exfiltration method that no one has thought to monitor. They succeed specifically because they operate outside the known-known layer.

A SOC that invests exclusively in detection engineering is optimizing for the base of the pyramid. It will catch the most threats by volume. It will produce the busiest incident queue. And it will be completely blind to the threats that cause the most damage — because those threats are designed to operate above the detection rule layer.

How the three layers interact

The three layers are not independent. They feed each other in a cycle that makes the entire security operation smarter over time — but only if all three are active.

Hunting feeds detection engineering. Every successful hunt that identifies a threat technique in the known-unknown layer produces at least one new detection rule. The hunt validates that the technique occurs in your environment, confirms which telemetry records it, and produces tested KQL logic that can be deployed as an analytics rule. The technique moves from the known-unknown layer (hunting) to the known-known layer (detection). The detection gap shrinks. This is the hunt-to-detection pipeline that runs through every campaign module in this course.

Anomaly detection feeds hunting. When a behavioral baseline flags an anomaly — a user whose data access pattern deviates from their 90-day normal — it does not fire an alert the way a detection rule does. It produces a lead. That lead becomes a hunting hypothesis: “User X’s SharePoint download volume tripled this week. Is this a legitimate project or data exfiltration?” The hunt investigates the anomaly, confirms or refutes the hypothesis, and either escalates to IR or documents the legitimate explanation. The anomaly moves from the unknown-unknown layer (baseline deviation) to the known-unknown layer (hunting investigation).

Detection engineering feeds hunting prioritization. Detection rules that fire produce investigation data. Investigation findings reveal attack patterns. Those patterns identify gaps — techniques the attacker used that were not detected, or were detected late. Each gap becomes a hunting hypothesis. The detection layer’s failures are the hunting layer’s backlog. TH3 formalizes this process through ATT&CK coverage analysis.

Hunting feeds anomaly detection. Hunts that examine behavioral patterns — authentication baselining in TH4, download volume analysis in TH8, process frequency analysis in TH9 — produce the baselines that anomaly detection uses. Without hunting to validate what “normal” looks like, anomaly detection has no calibrated reference point. And without hunting to investigate what anomaly detection flags, the anomalies are just numbers in a workbook that no one acts on.

The cycle works only if all three layers are staffed and active. Remove hunting, and the known-unknown layer is unmonitored — detection rules only improve when someone happens to notice a gap. Remove anomaly detection, and novel techniques are invisible. Remove detection engineering, and the known-known threats overwhelm the SOC with uninvestigated compromises. The three layers are complementary, not competing.

Where most SOCs stop

Most M365 security operations invest in one layer: detection engineering. They build analytics rules. They tune alert thresholds. They subscribe to threat intelligence feeds that automatically create detection indicators. They review Microsoft’s recommended detections and deploy the ones relevant to their environment. This is important work, and they should keep doing it.

But it is only one-third of the visibility architecture. The other two-thirds — hunting the known-unknown and baselining the unknown-unknown — are either unstaffed, unscheduled, or treated as “nice to have if we get time.” They never get time, because the known-known layer generates enough alerts to consume every available analyst hour.

This is the operational trap that hunting breaks. Hunting is not additional work layered on top of an already overloaded SOC. It is the work that discovers the threats no amount of alert triage will surface — and produces the detection rules that make the alert layer more effective, reducing the noise that overloads the SOC in the first place.

TH14 (Building a Hunt Program) addresses the practical question of how to allocate hunting hours within a resource-constrained SOC. The answer is not “hire a dedicated hunt team” — most organizations cannot. The answer is structured allocation: a defined number of hours per week or month, protected from the alert queue, dedicated to executing hunt campaigns from a prioritized backlog. The detection rules those hunts produce improve the alert layer, which reduces the analyst hours spent on false positives, which frees capacity for more hunting. The flywheel turns.

Anomaly detection in M365

This course focuses on hunting — the known-unknown layer. But the relationship to anomaly detection is important because some of the KQL techniques taught in TH2 serve both purposes.

series_decompose_anomalies() in KQL is a hunting tool when you use it to analyze a specific user’s download volume during a hunt campaign. It is an anomaly detection tool when you deploy it as a scheduled query that runs daily against all users and surfaces deviations for review.

autocluster() is a hunting tool when you use it to identify patterns in hunt results. It is an anomaly detection tool when you deploy it against sign-in logs to find authentication events that do not fit any established cluster.

The boundary between hunting and anomaly detection is operational, not technical. The same KQL function serves both purposes. The difference is intent: hunting starts with a hypothesis and uses the tool to test it. Anomaly detection starts with a baseline and uses the tool to flag deviations that may generate hypotheses.

This course teaches both applications, but the primary focus is the hypothesis-driven hunting that lives in the middle layer of the pyramid — because that is where the largest gap exists in most M365 security operations.

Measuring your pyramid investment

The following query categorizes your recent security alerts by detection source — giving you a rough view of which pyramid layers are producing findings in your environment:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

// Which pyramid layers are producing findings?
SecurityAlert
| where TimeGenerated > ago(90d)
| extend DetectionLayer = case(
    ProviderName == "ASI Scheduled Alerts", "Known-Known: Custom Rules",
    ProviderName has "MCAS", "Known-Known: Defender for Cloud Apps",
    ProviderName has "MDATP", "Known-Known: Defender for Endpoint",
    ProviderName has "IPC", "Unknown-Unknown: Identity Protection (UEBA)",
    ProviderName has "Azure Sentinel" and AlertName has "Anomal",
        "Unknown-Unknown: Anomaly Detection",
    "Known-Known: Built-in Detection")
| summarize AlertCount = count(), DistinctAlerts = dcount(AlertName)
    by DetectionLayer
| sort by AlertCount desc
// If "Unknown-Unknown" layers return zero or near-zero results,
// your anomaly detection layer is either not deployed or not producing
// If all results are "Known-Known" — only the base of the pyramid is active
// The Known-Unknown layer (hunting) will never appear here because
//   hunting does not produce automated alerts — it produces findings
//   that are documented in hunt records, not in the SecurityAlert table

References Used in This Subsection

• MITRE Corporation. “MITRE ATT&CK — Enterprise Matrix.” https://attack.mitre.org
• Microsoft. “Microsoft Sentinel UEBA — User and Entity Behavior Analytics.” Microsoft Learn. https://learn.microsoft.com/en-us/azure/sentinel/identify-threats-with-entity-behavior-analytics
• Microsoft. “Kusto Query Language — series_decompose_anomalies().” Microsoft Learn. https://learn.microsoft.com/en-us/kusto/query/series-decompose-anomalies-function
• Sqrrl (now Amazon). “A Framework for Cyber Threat Hunting.” — foundational hunting methodology reference