TH1.15 Quality Assurance: Peer Review and Hunt Validation

The error you do not catch

A hunt for inbox rule manipulation queries CloudAppEvents for New-InboxRule operations. The query returns zero results. The analyst concludes: “No malicious inbox rules in the last 30 days.” The hunt record is filed. The detection rule is deployed.

But the attacker used the Graph API to create the inbox rule, not the Outlook client. The Graph API creation path produces a different operation name in CloudAppEvents — or may appear only in MicrosoftGraphActivityLogs. The query was correct for one creation method and blind to another. The conclusion was false negative. The detection rule has the same blind spot.

A 5-minute peer review would have caught this. “Did you check all inbox rule creation paths — Outlook, OWA, PowerShell, EWS, and Graph API?” The question identifies the gap. The analyst runs an additional query. The conclusion changes.

Three review points

Review point 1: Before the hunt — hypothesis and scope review. Before the first query runs, a second analyst (or the SOC lead) reviews the hypothesis and scope definition for 5 minutes.

Review questions: Is the hypothesis testable with the scoped data sources? Are all relevant data tables included? Is the time window appropriate for the technique? Is the population correctly defined? Are there technique variants that the scope should cover but does not?

This review catches the inbox rule example above — the reviewer asks about creation paths before the hunt starts, not after it concludes.

Review point 2: Before closing — hunt record review. Before the hunt is marked complete, a second analyst reviews the hunt record for completeness and methodology adherence.

Review questions: Were all four query funnel steps executed (orientation, indicator, enrichment, pivot)? Were false positives analyzed and documented? Were exclusions justified with specific evidence? Is the conclusion supported by the analysis? If the conclusion is “refuted” (no finding), does the scope cover the technique adequately — or could the technique have been present but invisible due to scope gaps?

Review point 3: Before deploying — detection rule review. Before the hunt-derived detection rule goes into production, a second analyst reviews the rule for correctness, exclusion appropriateness, and threshold calibration.

Review questions: Does the rule’s time window and frequency avoid gaps? Are the exclusions from the FP analysis documented and justified? Could any exclusion be exploited by an attacker (e.g., excluding an IP range that an attacker could route through)? Is the entity mapping correct? Is the severity appropriate?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// Detection rule validation: does the new rule cover
// what the hunt query covered?
// Compare the hunt query results against the rule's results
// over the same time window
// Hunt query result count (from hunt record): ____
// Run the detection rule query manually over the hunt's time window:
// If the rule returns fewer results than the hunt query,
// the rule's filters or exclusions may be too aggressive
// If the rule returns MORE results, the rule is less precise
// and may need additional exclusions from the FP analysis

The peer review checklist

Figure TH1.15 — Three review points for hunt quality assurance. Total time: approximately 20 minutes per hunt. The investment prevents false negatives that undermine the entire program.

For solo hunters

If your team has only one analyst who hunts, peer review is not available. Three adaptations:

Self-review with a checklist. Before closing a hunt, walk through the checklist below. Check each item honestly. The checklist compensates for the absence of a second perspective.

Time-delayed review. Complete the hunt. Wait 24 hours. Re-read the hunt record with fresh eyes. The overnight gap provides cognitive distance that catches errors you missed during the hunt session.

Periodic batch review. Every 3 months, review the last quarter’s hunt records as a batch. Look for patterns — are you consistently missing certain data sources? Are your exclusions getting more permissive over time? Are your conclusions consistently reaching one outcome (always refuted, never confirmed)? Patterns in your own work reveal systematic biases that individual self-review misses.

References Used in This Subsection

• Course cross-references: TH1.2 (scope — technique variant coverage), TH1.7 (hunt record template), TH1.6 (detection rule conversion), TH15 (mature review practices)