DE1.8 Defender XDR Custom Detections

Defender XDR custom detection architecture

Defender XDR custom detections share the same KQL language as Sentinel but differ in several architectural ways. The query runs against the Advanced Hunting schema — the Device*, Email*, Identity*, CloudApp*, and Alert* tables. The frequency is fixed to preset options: every 1 hour, every 3 hours, every 12 hours, or every 24 hours. There is no 5-minute option and no NRT equivalent.

The primary advantage of Defender custom detections is native response integration. When a custom detection fires, it can automatically execute Defender response actions on the entities identified in the query results: isolate a device from the network, quarantine a file, block a URL across all managed endpoints, disable a user account, or soft-delete a malicious email. These actions execute within the Defender platform without requiring a Logic Apps playbook or Sentinel automation rule — reducing the implementation complexity for endpoint-centric detections.

The primary limitation is scope. Defender custom detections cannot query: CommonSecurityLog (firewall data), Syslog (Linux data), WindowsEvent (on-prem DC events via AMA), custom tables (_CL), or the Anomalies table. Any detection that requires data from these sources must be built in Sentinel. Cross-source correlation between Microsoft telemetry and third-party data is Sentinel-only.

Figure DE1.8 — Sentinel analytics rules vs Defender XDR custom detections. Sentinel is the default for detection engineering. Defender custom detections serve the narrow case of endpoint-only detection with native Defender response actions.

When to use each: the NE decision framework

For Northgate Engineering, the decision is straightforward for most rules:

Use Sentinel for: AiTM token theft detection (requires SigninLogs + AADNonInteractiveUserSignInLogs join), BEC detection (requires SigninLogs + EmailEvents join), configuration drift monitoring (requires AuditLogs with CA policy filtering), cross-site lateral movement (requires DeviceLogonEvents + CommonSecurityLog join), USB exfiltration with manufacturing exemptions (requires DeviceFileEvents + watchlist join), and any detection involving Linux server logs (Syslog).

Use Defender custom detection for: Known malicious process execution where the response is device isolation (e.g., CHAIN-ENDPOINT Cobalt Strike beacon — detect the encoded PowerShell command and immediately isolate m.thompson’s workstation). File hash matching where the response is file quarantine (known malware hash detected on endpoint — quarantine the file across all devices). URL blocking where the response is network-level block (known C2 URL — block across all managed endpoints).

The dual-deployment pattern: For critical detections, deploy BOTH. A Sentinel NRT rule detects the ransomware pre-encryption and fires the alert for the SOC. A Defender custom detection detects the same pattern and automatically isolates the device. The Sentinel rule provides visibility and investigation context. The Defender custom detection provides automated containment. They are complementary, not competing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// DEFENDER CUSTOM DETECTION EXAMPLE
// Detect Cobalt Strike beacon + auto-isolate device
DeviceProcessEvents
| where Timestamp > ago(1h)
| where FileName in~ ("powershell.exe", "pwsh.exe")
| where ProcessCommandLine has "-encodedcommand"
    and ProcessCommandLine has "-nop"
    and ProcessCommandLine has "-w hidden"
// In Defender: configure "Impacted entities" → DeviceId for auto-isolation
// In Sentinel: build companion NRT rule with entity mapping for investigation

Defender response actions in detail

The primary value of Defender custom detections is the native response actions — actions that execute directly in the Defender platform when the detection fires. Understanding each action and its scope is essential for the dual-deployment decision.

Isolate device. Disconnects the device from the network while maintaining communication with the Defender cloud service. The device can still receive commands from the SOC (run live response, collect forensic data) but cannot communicate with internal resources or the internet. This is the most common automated response for endpoint detections. Use for: ransomware pre-encryption, active C2 beacon, confirmed credential dump on a device. Reversible: the SOC reconnects the device after investigation.

Quarantine file. Moves the detected file to a quarantine location on the device, preventing execution. Applies to the specific file on the specific device — does not affect other copies on other devices. Use for: known malware hashes, suspicious executables in staging locations, tools downloaded by an attacker (Mimikatz, PsExec). Reversible: the SOC can restore the file if it is a false positive.

Block URL. Adds the URL to a network block list that applies across all managed endpoints. Any endpoint attempting to reach the blocked URL is prevented. Use for: confirmed C2 domains, phishing URLs that bypass email security, exfiltration endpoints. This is a GLOBAL action — it affects all devices, not just the detecting device. Exercise caution: blocking a URL used by a legitimate SaaS application (misidentified as C2) disrupts the entire organization.

Disable user. Disables the Entra ID account associated with the detection. The user cannot sign in to any M365 service or Entra ID-integrated application. Use for: confirmed account compromise where session revocation alone is insufficient (the attacker may have registered new MFA methods or created app registrations). Reversible but high-impact — the user is completely locked out until re-enabled.

Soft-delete email. Removes emails matching the detection criteria from user mailboxes. Use for: phishing emails that bypassed Safe Links and were delivered to multiple recipients. The email is moved to the Deleted Items folder (soft delete) or permanently removed (hard delete). Use with extreme caution: deleting a legitimate email is difficult to recover from.

NE dual-deployment candidates

For Northgate Engineering, 3 detection rules justify dual deployment — Sentinel for detection and investigation, Defender for automated containment:

The remaining 40+ detection rules are Sentinel-only — they either require cross-source correlation (Sentinel-only capability), third-party data (firewall, Linux), or their detection patterns are too nuanced for Defender’s fixed frequency schedules.

Defender custom detection configuration walkthrough

Creating a custom detection in the Defender portal:

1. Start in Advanced Hunting. Write and test the KQL query. Verify it returns the expected results against recent data. The query must use Timestamp (not TimeGenerated — Defender’s schema uses a different time field).

2. Click “Create detection rule.” This converts the Advanced Hunting query into a custom detection rule. Select the frequency (1, 3, 12, or 24 hours).

3. Configure impacted entities. Map the query output fields to entity types: DeviceId for device-level actions, AccountObjectId for user-level actions, SHA256 for file-level actions, Url for URL-level actions. Without entity mapping, the response actions cannot identify their targets.

4. Configure actions. Select the response action per entity type: isolate device, quarantine file, block URL, disable user, collect investigation package. Multiple actions can be chained — “isolate device AND collect investigation package.”

5. Set alert metadata. Alert title, severity, MITRE ATT&CK mapping, description. This metadata appears in the Defender incident queue and, if forwarded via the data connector, in Sentinel incidents.

6. Review and activate. The rule begins executing at the selected frequency after activation. Monitor the rule’s output in Defender → Custom Detection Rules → [rule name] → Detection history.

Troubleshooting: Defender custom detection issues

“Custom detection created but no alerts appear in Sentinel.” Defender custom detection alerts are not automatically forwarded to Sentinel unless the Microsoft Defender XDR data connector is configured with “Incidents and alerts” enabled. Check: Sentinel → Data connectors → Microsoft Defender XDR → ensure alert forwarding is active.

“Custom detection runs but the automated response does not execute.” Verify the “Impacted entities” configuration in the custom detection. The response action (isolate device) requires a mapped DeviceId in the query output. If the query does not return DeviceId (or returns it with a different column name), the response action cannot identify the target device.

“Defender custom detection query works in Advanced Hunting but returns 0 results as a custom detection.” Check the time reference. Advanced Hunting uses Timestamp while Sentinel uses TimeGenerated. Also verify the query does not use take, sample, or other non-deterministic operators that produce variable results between manual execution and scheduled execution.

References used in this subsection

• Microsoft Defender XDR custom detections documentation
• Course cross-references: DE1.1 (rule types), DE1.6 (NRT as alternative for speed), DE1.10 (Sentinel automation as alternative for response), DE8 (ransomware dual-deployment pattern)