DE0.6 The Microsoft Detection Surface

Why the Microsoft stack is different

Most detection engineering courses teach a fragmented model: endpoint data in one tool, identity data in another, email data in a third, cloud activity in a fourth. The detection engineer writes rules in multiple query languages against multiple data stores and manually correlates across systems.

The Microsoft security stack consolidates this. Every major telemetry source — endpoint process execution, email metadata, identity authentication, cloud application activity, on-premises Active Directory, firewall logs (via CEF/Syslog), and custom application logs — lands in a single queryable surface. KQL queries span all of it. A single Sentinel analytics rule can join SigninLogs (identity) with EmailEvents (email) with DeviceProcessEvents (endpoint) in one query. That cross-source join — “show me users who had a risky sign-in AND received a phishing email AND then ran an encoded PowerShell command on their endpoint within 60 minutes” — is the detection power that catches multi-phase attacks.

This is not theoretical. The CHAIN-HARVEST detection rule for Phase 5 (BEC from compromised account) requires exactly this: join SigninLogs (identity risk on the sender) with EmailEvents (internal email with financial keywords to executive recipients). That join is one KQL query. In a fragmented stack, it requires exporting data from two separate tools and correlating in a spreadsheet — or building a custom integration pipeline. In the Microsoft stack, it is a join operator.

Figure DE0.4 — The Microsoft detection surface. Four telemetry domains feed into a unified KQL query surface. Detection rules deploy through two mechanisms: Sentinel analytics rules and Defender XDR custom detections.

KQL: the detection language

KQL is not just a query language — it is the detection language for the Microsoft security stack. Every analytics rule in Sentinel, every custom detection in Defender XDR, every hunt query in Advanced Hunting, and every workbook visualization is written in KQL. Learning KQL once applies everywhere.

For detection engineering specifically, KQL provides capabilities that most SIEM query languages lack. The join operator correlates across tables (identity + email + endpoint in one query). The make-series operator creates time-series baselines for anomaly detection. The arg_max and arg_min functions extract the most recent or earliest event per entity. The bag_unpack and parse_json functions handle the nested JSON structures common in Microsoft telemetry. The materialize function optimizes complex queries that reference the same filtered dataset multiple times.

This course assumes KQL proficiency. You need to write where, summarize, join, let, extend, project, and time-windowed aggregations without reference material. If you need to build that proficiency first, our Mastering KQL course provides the foundation. If you can write a multi-table join with time-windowed correlation and entity extraction, you are ready to build detection rules.

Two detection mechanisms

The Microsoft stack provides two mechanisms for deploying detection rules: Sentinel analytics rules and Defender XDR custom detections. They serve different purposes and the detection engineer uses both.

Sentinel analytics rules are the primary detection mechanism for this course. They run KQL queries against the Sentinel workspace on a configurable schedule (every 5 minutes to every 24 hours) or in near-real-time (NRT, every 1 minute). They can query any table in the workspace — Microsoft telemetry, third-party logs, custom tables. They produce alerts that are grouped into incidents and assigned to the SOC. They support entity mapping, MITRE ATT&CK tagging, and automation rules for response. The detection engineer’s core output is Sentinel analytics rules.

Defender XDR custom detections run KQL against the Advanced Hunting schema specifically — the Device, Email, Identity, and CloudApp tables. They run on fixed schedules (every 1, 3, 12, or 24 hours). They are best suited for endpoint-specific detections where the response action is Defender-native: isolate device, quarantine file, block URL, disable user. They cannot query third-party data (CommonSecurityLog, Syslog, custom tables). Use them when the detection and response are both in the Defender ecosystem.

When to use which: If the detection requires third-party data (firewall logs, Linux syslog, custom application logs) — Sentinel. If the detection requires near-real-time response (NRT rule at 1-minute cadence) — Sentinel. If the detection and response are both endpoint-native (detect process, isolate device) — Defender custom detection is faster to deploy. If the detection requires cross-source correlation (identity + email + endpoint) — Sentinel, because it can join all sources in one query. This course builds primarily in Sentinel with Defender custom detections noted where they offer an advantage.

The unified data model advantage

The single most valuable property of the Microsoft detection surface is the unified data model. Every Defender product writes to tables that share common entity fields: AccountUpn, DeviceName, IPAddress, Url, FileName. When you build a detection rule that joins SigninLogs with DeviceProcessEvents, the AccountUpn field in both tables refers to the same entity. No field mapping. No data normalization. No ETL pipeline.

In a heterogeneous environment (Splunk for SIEM, CrowdStrike for EDR, Okta for identity), building a cross-source detection rule requires: extracting the username from CrowdStrike’s process telemetry (field name: UserName), extracting the username from Okta’s sign-in events (field name: actor.alternateId), normalizing both to a common format, and joining on the normalized field. That normalization is a detection engineering tax that every cross-source rule must pay.

In the Microsoft stack, the tax is zero. SigninLogs | join DeviceProcessEvents on AccountObjectId works immediately. The entity is the same across all tables. This is not a minor convenience — it is a structural advantage that makes cross-source detection rules practical to build and maintain. The detection engineer spends their time on the detection logic, not on data engineering.

Troubleshooting: Common platform questions

“We have Sentinel but not all Defender products. Can we still do detection engineering?” Yes. Detection engineering works with whatever data sources are connected to your workspace. If you have Defender for Endpoint and Entra ID but not Defender for Office 365, you build endpoint and identity detection rules. Your email detection coverage will be limited to what you can extract from OfficeActivity (audit logs) rather than EmailEvents (full email metadata). The methodology is the same — the data sources constrain which techniques you can detect, not whether you can do detection engineering.

“We use Defender XDR but not Sentinel. Can we still take this course?” Partially. Defender XDR Advanced Hunting provides the query surface for building and testing detection rules, and custom detections provide a deployment mechanism. You lose Sentinel-specific capabilities: NRT rules, third-party data correlation, automation rules with Logic Apps playbooks, and workbook-based coverage reporting. The core detection engineering methodology still applies, but DE9 (testing), DE10 (detection-as-code), and DE11 (capstone) are built for Sentinel deployment.

“Our Sentinel ingestion is expensive. Adding more rules will increase cost.” Analytics rules query existing data — they do not ingest new data. Running a KQL query against SigninLogs that is already in your workspace costs zero additional ingestion. The cost increase comes from connecting NEW data sources (if your coverage assessment reveals gaps in telemetry), not from building rules against existing data. Detection engineering is one of the highest-ROI activities in security operations precisely because it extracts more value from data you already pay to ingest.

References used in this subsection

• Course cross-references: DE1 (Sentinel analytics rule architecture), DE3-DE8 (rule building across all domains), Mastering KQL course (KQL proficiency prerequisite)

The Microsoft detection surface: data source inventory

1
2
3
4
5
6
7
8

// DATA SOURCE INVENTORY: What tables are ingesting data and at what volume?
Usage
| where TimeGenerated > ago(1d)
| where IsBillable == true
| summarize DailyGB = round(sum(Quantity) / 1024, 2) by DataType
| sort by DailyGB desc
// This shows your ACTUAL data surface — every table with data is a potential detection source
// Tables with 0 volume are connected but not receiving events (investigate)