Sign In
In this section

DE1.7 The Rule Specification Template and Common Mistakes

6-8 hours · Module 1 · Free
What you already know

You've configured a test rule through every stage of the pipeline — query, threshold, entity mapping, severity, ATT&CK classification, and alert grouping. Each section taught a configuration parameter as a design decision. This section assembles those decisions into the rule specification template — the engineering document you'll produce for every rule you build in this course and in production. You'll also see the most common design mistakes that detection programs make, so you know what to avoid.

Scenario

The detection engineer who built your top 5 detection rules left the organization 3 months ago. A rule called "Suspicious PowerShell Activity" fires 12 times per week. The SOC triages each alert, but 9 out of 12 are false positives from the admin team's maintenance scripts. Nobody knows whether the threshold should be adjusted, what the original false positive profile looked like, or whether the rule was designed to detect encoded PowerShell specifically or any PowerShell execution above a certain frequency. The rule's entire design context exists in one person's memory — and that person is gone. A rule specification would have documented all of this: the detection hypothesis, the expected false positive sources, the tuning parameters, the analyst response procedure. The next detection engineer could have tuned the rule in 15 minutes instead of spending 3 hours reverse-engineering the KQL to understand the original intent.

The rule specification is the engineering document that makes every design decision reviewable, auditable, and maintainable. It exists outside the Sentinel rule wizard — in a Git repository, a wiki, a shared document — alongside the rule's KQL. The wizard captures the configuration. The specification captures the reasoning.

Estimated time: 40 minutes.

THE RULE SPECIFICATION — SEVEN SECTIONS HYPOTHESIS What does this rule detect? CONTEXT Why does this technique matter? DATA Table, fields, latency CONFIGURATION Entities, severity, timing, grouping FP PROFILE Expected false positive sources RESPONSE Analyst triage procedure MAINTENANCE Tuning cadence and metrics The wizard captures the configuration. The specification captures the reasoning. Every rule you build from DE3 onward includes a specification. This template is the format.

Figure DE1.7 — The rule specification has seven sections: detection hypothesis, threat context, data requirements, configuration rationale, false positive profile, analyst response procedure, and maintenance cadence. Together they make every design decision reviewable by anyone on the team.

The template

Here is the complete rule specification for your brute force test rule, serving as the template you'll use for every rule in this course:

1. Detection hypothesis

Rule name: Failed sign-ins followed by success

Hypothesis: Detects 5 or more failed authentication attempts followed by at least one successful sign-in for the same user account within a 20-minute window, indicating a credential compromise where the attacker eventually obtained the correct password through brute force or password spraying.

ATT&CK mapping: Credential Access (TA0006) → T1110 Brute Force

What this is NOT: This rule does not detect password spraying across many accounts with few attempts each (T1110.003 — that requires a separate rule with different aggregation logic). It does not detect credential stuffing using leaked password databases (similar KQL but different threshold calibration). It detects single-account brute force where the attacker succeeded.

2. Threat context

Brute force remains one of the most common initial access techniques. M-Trends reports it consistently among the top 5 initial access vectors. For organizations without MFA enforcement, a successful brute force grants full account access. For organizations with MFA, the risk depends on whether the attacker has also compromised the MFA factor (SIM swap, MFA fatigue, AiTM proxy). This rule is a baseline indicator — the first signal that an account's credentials are compromised, even if MFA prevents immediate account takeover.

3. Data requirements

Table: SigninLogs

Required fields: TimeGenerated, UserPrincipalName, ResultType, IPAddress

Ingestion latency: SigninLogs typically ingests within 2-3 minutes. Rule lookback extended to 20 minutes (vs 15-minute frequency) to compensate for occasional delays.

Licensing: Requires Entra ID P1 or P2 for full SigninLogs. Entra ID Free provides limited sign-in data.

4. Configuration rationale

Rule type: Scheduled (requires summarize aggregation — not possible in NRT)

Frequency: 15 minutes — fast enough for credential compromise detection, low enough compute cost for a rule that runs 96 times per day.

Lookback: 20 minutes — 5-minute overlap with previous execution to catch ingestion-delayed events.

Threshold: Greater than 0 (the KQL where FailedCount >= 5 handles the detection threshold; the alert threshold fires on any matching result).

Severity: Medium — confirmed credential compromise with moderate impact. Dynamic override raises to High when FailedCount exceeds 20 or DistinctIPs exceeds 5.

Event grouping: AlertPerResult — each compromised user is an independent investigation.

Alert grouping: Entity-based on Account, 24-hour window. All alerts for the same user consolidate into one incident.

Entity mapping: Account → FullName → UserPrincipalName (strong identifier). IP → Address → IPAddress (strong identifier for public IPs).

Custom details: FailedAttempts → FailedCount, SuccessfulLogins → SuccessCount, DistinctSourceIPs → DistinctIPs.

Alert name template: Brute force — {UserPrincipalName} from {IPAddress} ({FailedCount} failures)

5. False positive profile

Expected FP sources:

  • Service accounts with password rotation: Automated password rotation generates multiple failed attempts followed by success. Mitigation: maintain a service account watchlist and exclude via where UserPrincipalName !in (ServiceAccountWatchlist).
  • Shared workstations with multiple users: Multiple users failing before one succeeds can aggregate into a threshold match. Mitigation: the per-user summarize handles this — each user is aggregated independently.
  • Legacy applications with cached credentials: Applications that retry with old credentials after a password change. Mitigation: filter by AppDisplayName if specific legacy apps are identified.

Expected FP rate: 10-15% initially, dropping to under 5% after the first tuning cycle with watchlist exclusions.

6. Analyst response procedure

  1. Open the incident. Check the Account entity — is this a privileged account, a regular user, or a service account?
  2. Check the custom details: how many failures, how many source IPs, was there a successful sign-in?
  3. If successful sign-in occurred: check the sign-in details (MFA method, device compliance, location) to determine whether the success was MFA-protected.
  4. If MFA was not present on the successful sign-in: treat as confirmed compromise. Revoke sessions, reset password, check for persistence (inbox rules, OAuth apps, registered devices).
  5. If MFA was present and verified: the attacker obtained the password but couldn't complete authentication. Reset password as a precaution. Monitor for MFA fatigue attacks.
  6. Check the source IPs against threat intelligence. Investigate whether other accounts were targeted from the same IPs.

7. Maintenance cadence

Monthly review: Pull FP rate from the SecurityIncident table. If FP rate exceeds 20%, investigate the top FP source and add to the exclusion watchlist. If FP rate drops below 2%, consider lowering the threshold from 5 to 3 to catch lower-volume attacks.

Quarterly review: Re-evaluate severity calibration against actual incident outcomes. Check whether the ATT&CK mapping should include sub-techniques. Review the analyst response procedure against actual triage patterns.

Common rule design mistakes

The eight most common mistakes in rule design

1. No entity mapping. The alert fires but the investigation graph is empty. The analyst manually extracts entities from raw data. Fix: map Account, IP, and Host with strong identifiers on every rule.

2. Default severity on every rule. Everything is Medium. The SOC has no priority signal. Fix: calibrate severity using the three-factor model (impact, confidence, SOC capacity).

3. No ATT&CK mapping. The rule fires but doesn't register in coverage metrics. Fusion can't correlate it. Fix: map tactics and techniques — including sub-techniques where applicable.

4. Alert grouping disabled. Every execution creates a new incident. The queue fills with duplicates. Fix: enable entity-based grouping with a 24-hour window.

5. Lookback shorter than expected ingestion delay. Events arrive after the lookback window closes. The rule misses them. Fix: extend lookback beyond frequency and use ingestion_time() for deduplication.

6. Threshold too low (or too high) for the environment. The template threshold doesn't match your traffic baseline. Fix: run the query manually for 30 days, analyze the result distribution, and set the threshold at the point that separates normal from suspicious.

7. No false positive documentation. The SOC doesn't know which alerts are expected benign positives. Every alert gets full investigation. Fix: document known FP sources and the verification steps for each.

8. No maintenance plan. The rule was deployed and never revisited. The environment changed, the threshold became stale, and the FP rate climbed from 5% to 40%. Fix: monthly FP review, quarterly severity and coverage re-evaluation.

Write your test rule specification

Using the template above, write the complete specification for your brute force test rule. If you're following along with a different detection hypothesis, adapt the template to your rule.

The specification should exist as a document alongside your rule's KQL — in a Git repository, a wiki, a shared OneNote, or whatever documentation system your team uses. The format matters less than the completeness: every section filled, every design decision documented, every false positive source identified.

From DE3 onward, every rule you build includes a specification. The specification is not optional documentation — it's the engineering artifact that makes the rule maintainable. A rule without a specification is a query that works today but becomes unmaintainable when the environment changes, the team changes, or the threat landscape evolves.

The same rule specification in machine-readable JSON — the format used in detection-as-code pipelines where the spec lives alongside the KQL in a Git repository:

JSON
{
  "rule_id": "DE1-TEST-001",
  "name": "Failed sign-ins followed by success",
  "technique": "T1110",
  "tactic": "TA0006",
  "severity": "Medium",
  "frequency": "PT15M",
  "lookback": "PT20M",
  "threshold": 0,
  "event_grouping": "AlertPerResult",
  "alert_grouping": {
    "enabled": true,
    "method": "Selected",
    "entities": ["Account"],
    "window": "PT24H"
  },
  "entity_mapping": [
    { "type": "Account", "field": "UserPrincipalName" },
    { "type": "IP", "field": "IPAddress" }
  ],
  "fp_sources": [
    "Service accounts with password rotation",
    "Shared workstations with multiple users",
    "Legacy apps with cached credentials"
  ],
  "expected_fp_rate": "10-15%",
  "tuning_plan": "Monthly: review FP rate, add to watchlist. Quarterly: re-evaluate threshold."
}

And the equivalent detection logic expressed as a Sigma rule — the vendor-agnostic format that compiles to KQL for Sentinel, SPL for Splunk, and Lucene for Elastic:

Sigma
title: Brute Force — Failed Sign-Ins Followed by Success
id: de1-test-001
status: test
description: Detects 5+ failed auth attempts followed by success
  for the same user within 20 minutes
author: Detection Engineering Course
tags:
  - attack.credential_access
  - attack.t1110
logsource:
  product: azure
  service: signinlogs
detection:
  failed:
    ResultType|not: '0'
  success:
    ResultType: '0'
  timeframe: 20m
  condition: failed | count(ResultType) by UserPrincipalName > 5
    and success
level: medium

Three representations of the same detection: the prose specification documents the reasoning, the JSON enables automation, and the Sigma enables sharing across platforms. An intermediate detection engineer works in all three. The course uses KQL as the primary implementation language, but the specification and the Sigma representation ensure the detection logic is portable and maintainable regardless of which SIEM platform the student operates.

Detection Engineering Principle

A detection rule without a specification is a query that only the author understands. The specification captures every design decision — hypothesis, data requirements, configuration rationale, false positive profile, response procedure, maintenance cadence — so that any detection engineer can review, tune, or decommission the rule without reverse-engineering the KQL. The specification is the engineering document. The KQL is the implementation. Both are required.

Next

Section 8 summarizes what you've learned in this module and previews what comes next in DE2 — threat modeling and prioritization, where you build the detection backlog that determines which rules you build first.

© 2026 Ridgeline Cyber Defence™ Ltd. Content may not be reproduced or redistributed. Worked artifacts may be adapted for use within your organization.
Unlock the Full Course See Full Course Agenda
← Previous Next →