DE0.10 Tools and Capabilities

The detection engineering stack on Microsoft

Detection engineering on the Microsoft stack uses a specific set of tools. Each tool serves a distinct purpose in the lifecycle. Understanding what each tool does — and what it doesn't do — prevents the common mistake of treating one tool as the entire detection engineering practice.

Estimated time: 30 minutes.

Microsoft Sentinel

Sentinel is the SIEM and SOAR platform. It is where detection rules live, where alerts are generated, where incidents are created, and where the SOC operates. For detection engineering, Sentinel provides the analytics rule engine — the infrastructure that runs your KQL queries on a schedule, evaluates the results against trigger conditions, maps entities, assigns severity, and creates alerts.

Sentinel analytics rules come in several types. Scheduled rules run KQL at a configurable frequency (5 minutes to 24 hours) with a configurable lookback window — they're the workhorses, and approximately 90% of the rules in this course are scheduled. Near-Real-Time (NRT) rules run every minute against the most recent data — lowest latency, highest compute cost, reserved for high-impact techniques where detection speed directly affects blast radius (ransomware pre-encryption indicators, active credential dumping, data exfiltration in progress).

Anomaly rules use machine learning to detect statistical outliers without explicit KQL — useful for behavioral baselines but not customizable in the way scheduled rules are. Fusion rules correlate multiple lower-confidence signals into higher-confidence multi-stage attack detections — Microsoft's version of attack chain correlation, but limited to the correlation patterns Microsoft has defined. Microsoft Security rules pass through alerts from other Microsoft security products.

For detection engineering, the scheduled rule is your primary tool. You control the KQL, the scheduling, the entity mapping, the severity, and the response automation.

NRT rules are the same but with a fixed one-minute frequency and some KQL operator restrictions (no joins in the same way, limited lookback). You'll learn the NRT constraints in Module 1 and use NRT rules for specific high-impact detections in DE4 (credential attacks) and DE8 (ransomware indicators).

The Sentinel workspace architecture matters for detection engineering because it determines what data is available for rules to query. A single workspace that receives all five data source families gives detection engineers the ability to write cross-family joins — a rule that correlates identity telemetry with email telemetry in one query.

A multi-workspace architecture (common in large enterprises with data residency requirements) complicates cross-family detection because joins across workspaces require the cross-workspace query syntax and have performance implications. This course assumes a single workspace — the developer tenant setup in Section 7 creates one.

Cost is a consideration. Sentinel charges per GB of data ingested into the Log Analytics workspace. The five data source families produce different volumes: endpoint telemetry (DeviceProcessEvents especially) is the highest volume, followed by identity telemetry, then cloud app events, then email events, then infrastructure logs.

The developer tenant's free tier (5 GB/day for 31 days) is more than sufficient for the course's data volume. In production, Sentinel costs scale with the number of endpoints, the breadth of data connectors, and the retention period. The detection engineering decision — "should I connect this data source?" — is partly a coverage question and partly a cost question. Module 2 (threat modeling) addresses this trade-off.

Defender XDR Advanced Hunting

Advanced Hunting is a query interface within the Microsoft Defender XDR portal (security.microsoft.com). It provides access to the same endpoint, email, identity, and cloud app tables that Sentinel accesses — but with some differences.

Advanced Hunting includes tables and fields that Sentinel doesn't have (particularly detailed endpoint telemetry from MDE). It also supports Defender XDR custom detection rules — a parallel detection mechanism that operates within the Defender ecosystem rather than Sentinel.

For detection engineering, Advanced Hunting serves two purposes. First, it's an exploration environment — you can prototype KQL queries against the data before deploying them as Sentinel analytics rules.

The query editor has autocompletion, schema exploration, and result visualization that make iterative query development faster than working in the Sentinel Logs blade. Second, it's the source of email and endpoint data — the Microsoft 365 Defender data connector brings Advanced Hunting tables into Sentinel.

The relationship between Sentinel and Advanced Hunting is complementary, not competitive. Detection rules that need to join across data families (identity + email + endpoint) deploy in Sentinel because Sentinel has all the tables.

Detection rules that need deep endpoint telemetry or that benefit from Defender's built-in response actions can deploy as Defender custom detections. Most detection programs use both, with Sentinel as the primary detection platform and Defender custom detections for endpoint-specific rules that benefit from MDE's automatic response capabilities.

Verify your workspace is ready for detection engineering. This PowerShell confirms the workspace exists, shows retention settings, and lists the data tables with content:

# Workspace health check — confirm your environment is ready
$ws = Get-AzOperationalInsightsWorkspace -ResourceGroupName "rg-sentinel"
Write-Host "Workspace: $($ws.Name)"
Write-Host "Retention: $($ws.RetentionInDays) days"
Write-Host "SKU: $($ws.Sku)"
Write-Host "Daily cap: $($ws.WorkspaceCapping.DailyQuotaGb) GB"
# Count tables with data in the last 7 days
$tables = Get-AzOperationalInsightsTable -ResourceGroupName "rg-sentinel" `
    -WorkspaceName $ws.Name |
    Where-Object { $_.TotalRetentionInDays -gt 0 }
Write-Host "Tables with data: $($tables.Count)"

KQL — Kusto Query Language

KQL is the query language for both Sentinel and Advanced Hunting. It's the implementation language for detection engineering on the Microsoft stack. Every rule in this course is written in KQL.

KQL is a pipeline language — data flows through a series of operators, each transforming the dataset. A typical detection query reads a table, filters by time range, filters by specific field values, joins with another table, projects the fields needed for the alert, and summarizes the results.

Each operator appears on its own line with a pipe character at the start. The pipeline model makes complex queries readable because each step does one thing.

This course assumes basic KQL competency — you can read a query, understand what each operator does, and modify field names and filter values. If you've written KQL for investigation or hunting, you have the foundation you need.

If KQL is new, the KQL for Security skill on this platform teaches the language. Module 1 (rule architecture) teaches the Sentinel-specific KQL patterns — entity mapping functions, time-series operators, cross-table joins, and the performance considerations specific to scheduled rules that run against large datasets.

Explore your available tables and their schemas before building rules. This KQL shows which tables have data and how much:

// Which tables have data? What's the volume?
Usage
| where TimeGenerated > ago(7d)
| where IsBillable == true
| summarize DailyGB = round(sum(Quantity) / 1024 / 7, 2)
    by DataType
| sort by DailyGB desc
| take 10

This tells you which data families are active and which produce the most volume. The tables you'll query most in detection rules — SigninLogs, DeviceProcessEvents, OfficeActivity, EmailEvents — should all appear in the top 10. If a table is missing, the corresponding data connector needs configuration before you can build rules against that data family.

The lab pack

The course lab pack contains four components that support the detection engineering lifecycle.

Evidence data

— approximately 3,500 events across 8 Sentinel tables containing all six attack chains buried in 14 days of legitimate background noise. When you build a rule, you import the relevant evidence data and test whether the rule fires on the attack events without firing on the noise.

Rule templates

— every rule built in the course is available in ARM template format (for deployment to Sentinel), KQL format (for the query itself), and YAML format (for detection-as-code pipelines). The templates include entity mapping, MITRE ATT&CK assignments, severity rationale, and rule specifications. You can deploy templates directly — but the learning happens when you build the rules yourself.

Sysmon configuration

— a baseline configuration that captures the process, network, file, and registry events needed for endpoint detection rules. If you're using a lab environment without MDE, Sysmon provides equivalent telemetry through the SecurityEvent table.

Program templates

— the rule specification template (DE1), the threat model template (DE2), the tuning log (DE9), the coverage report template (DE10), and the board report template (DE11). These are the operational documents that turn individual rules into a program.

AI integration

AI is not a standalone module in this course — it's a tool integrated throughout the lifecycle. Starting in DE2, you'll use AI for hypothesis generation from threat advisories.

In DE3-DE8, AI assists with KQL drafting, rule specification generation, and false positive pre-analysis. In DE9, AI accelerates monthly tuning reviews by classifying alert patterns. In DE10, AI assists with documentation and coverage gap analysis.

The course teaches when AI helps and when human judgment is non-negotiable. AI generates competent first-draft KQL from a well-written hypothesis.

It does not produce hypotheses — that requires understanding the organization's threat landscape, technology stack, and data patterns. AI pre-identifies likely false positive sources from a rule specification. It does not make tuning decisions — that requires understanding the environment's specific operational patterns and the SOC's tolerance for noise.

The practical reality: AI makes a competent detection engineer faster. It does not make a novice into a detection engineer. The discipline — the thinking, the methodology, the adversarial mindset — must be learned by doing it. AI accelerates the doing.

Git and CI/CD

Module 10 teaches the detection-as-code pipeline. The tools: Git for version control (GitHub free tier is sufficient), a CI/CD platform for automated testing and deployment (GitHub Actions), and the Sentinel API for programmatic rule management. The pipeline stores rules as code, runs automated tests on every pull request, and deploys validated rules to Sentinel on merge.

You don't need Git experience before starting the course. Module 10 teaches the pipeline from scratch, including repository structure, branch strategy, PR workflow, and CI/CD configuration. If you already use Git, you'll recognize the patterns — detection-as-code follows the same development practices as application code.

Atomic Red Team and testing tools

Testing detection rules requires generating the attack behavior the rule is supposed to detect. You can do this manually — run the specific command, create the specific inbox rule, generate the specific sign-in pattern — but manual testing is slow and inconsistent.

Atomic Red Team is an open-source framework that provides automated test cases ("atomics") mapped to ATT&CK techniques. Each atomic generates the specific telemetry for a specific technique — T1003.001 has an atomic that accesses LSASS memory the same way Mimikatz does, producing the exact DeviceProcessEvents records that a credential dumping detection rule should fire on.

Atomic Red Team is optional for this course — the lab pack's evidence data contains pre-generated attack telemetry that you can use for testing without running the atomics yourself. But for ongoing detection engineering after the course, Atomic Red Team becomes the testing framework: before deploying a new rule, you run the relevant atomic in your test environment and verify the rule fires.

Other testing tools include MITRE Caldera (a full adversary emulation platform) and Infection Monkey (automated breach and attack simulation). These are more comprehensive than Atomic Red Team but also more complex to operate. For detection engineering, Atomic Red Team's individual technique tests align most directly with the rule-by-rule development process.

The ATT&CK Navigator

Mentioned in Section 8, the Navigator is the visualization tool for coverage mapping. It's a web application that renders ATT&CK as a colored heatmap.

You create layers representing your detection coverage, overlay them with threat group technique profiles, and export the result as the visual artifact in your board report. The course integrates Navigator usage starting in DE2 (threat modeling) and throughout DE10 (detection-as-code coverage reporting).

Community resources and detection content

Detection engineering has an active open-source community that produces detection rules, frameworks, and research. Key resources you'll reference throughout the course:

Sigma

— a generic signature format for detection rules that can be translated to any SIEM query language. Sigma rules are platform-agnostic — a Sigma rule for credential dumping translates to KQL for Sentinel, SPL for Splunk, and EQL for Elastic. The Sigma rule repository on GitHub contains thousands of community-contributed rules.

In this course, you'll write KQL directly because Sentinel-specific features (entity mapping, alert grouping, NRT scheduling) don't translate through Sigma. But Sigma is a valuable source of detection hypotheses — each Sigma rule represents a hypothesis that someone tested.

The detection engineering community

— practitioners share detection rules, research, and methodology through blogs, conference talks, and GitHub repositories. SpecterOps publishes detection research through their blog and training programs. The SANS Reading Room contains detection-focused papers.

Individual practitioners publish detection rules and analysis on Medium, Substack, and personal blogs. These are sources for detection hypotheses — when someone publishes a detection for a technique you don't cover, that's a hypothesis you can evaluate, adapt for your environment, and add to your backlog.

Microsoft's own detection content

— the Sentinel content hub, the Microsoft Threat Intelligence blog, and the Microsoft Defender Threat Intelligence portal all publish detection-relevant content. Template rules in the content hub are starting points (as discussed in Section 3), but the underlying logic — the tables, fields, and patterns the templates query — is useful for understanding what's detectable in the Microsoft telemetry model.

The threat intelligence blog publishes technique analysis that maps directly to detection hypotheses.