AI-Assisted Detection Rule Development

60-90 minutes Modules: Module 3

Objective

Use AI to convert a threat intelligence report into a deployable detection rule. You will prompt the AI to analyse the TI, formulate a detection hypothesis, generate KQL, assess false positive conditions, and produce a rule specification document.

Required: Access to Claude. A threat intelligence report (use the one provided below, or bring your own from CISA, your ISAC, or a vendor blog).

Step 1: Analyse the threat report with AI

TI Report (provided):

“Threat group STORM-1567 is targeting financial services organisations with a new AiTM phishing kit hosted on Cloudflare Workers. The kit uses Turnstile CAPTCHA challenges to evade automated URL scanning. Phishing emails impersonate the target organisation’s IT department and claim a mandatory security update requires re-authentication. The phishing URL redirects through a legitimate-looking landing page before proxying to the Microsoft login page. Post-compromise, the adversary establishes persistence via inbox forwarding rules and OAuth application consent (requesting Mail.ReadWrite and Files.ReadWrite.All permissions).”

Prompt:

Role: You are a detection engineer analysing a threat intelligence 
report to develop detection rules.

Context: [Paste the TI report above]

Task:
1. Extract all observable indicators (IOCs and behavioral indicators)
2. Map each indicator to a MITRE ATT&CK technique
3. For each indicator, identify the M365 log source that would 
   record it
4. Assess which indicators are most detectable with lowest 
   false positive rate
5. Recommend the top 2 detection rule candidates with hypotheses

Evaluate the output:

Did it extract both tactical indicators (Cloudflare Workers, Turnstile) and behavioral indicators (inbox forwarding, OAuth consent)?
Are the ATT&CK mappings correct?
Are the log source recommendations accurate?

Step 2: Generate the KQL query

Take the top detection rule candidate from Step 1 and prompt for KQL:

Role: You are a KQL detection engineer for Microsoft Sentinel.

Context: Based on the analysis above, develop a KQL detection rule 
for: [description of the selected detection candidate]

Task: Write a complete KQL analytics rule query that:
1. Queries the correct log table(s)
2. Filters for the specific behavioral pattern
3. Uses appropriate time windows
4. Includes entity extraction for incident mapping
5. Has comments explaining each section

Constraints:
- The query must be a valid Sentinel scheduled analytics rule
- Use a 1-hour lookback window
- Include a threshold that minimises false positives
- Use watchlists for tuning (reference watchlist names)

Evaluate:

Is the KQL syntactically valid?
Does it query the correct table?
Are the filters specific enough to avoid excessive false positives?
Are entities extracted for incident mapping?

Important: Do not deploy AI-generated KQL without testing. Paste it into the Sentinel Logs query editor and run it against your historical data to verify it works.

Step 3: Assess false positives with AI

Role: You are tuning a detection rule for production deployment.

Context: [Paste the KQL query from Step 2]

Task: Identify the likely false positive sources for this detection:
1. What legitimate activity matches the same pattern?
2. For each FP source, suggest a specific exclusion or tuning 
   adjustment
3. Estimate the expected FP rate in a typical enterprise 
   (500-5000 users)
4. What watchlists should be created for ongoing tuning?

Step 4: Generate the rule specification

Role: You are documenting a detection rule for the SOC detection 
rule library.

Context: [Paste the KQL and FP analysis from Steps 2-3]

Task: Produce a complete detection rule specification document with:
1. Rule ID, name, and description
2. ATT&CK mapping (tactic, technique, sub-technique)
3. Data sources and required connectors
4. The KQL query with inline comments
5. Scheduling (frequency, lookback, threshold)
6. Entity mapping specification
7. Known false positive conditions and exclusions
8. Triage and response guidance for the SOC analyst
9. Testing procedure

Step 5: Review and refine

Review the complete output (TI analysis → hypothesis → KQL → FP assessment → specification). Ask yourself:

Would I deploy this rule as-is, or does it need modifications?
What did the AI get right that saved me time?
What did the AI get wrong that I needed to correct?
How would I refine my prompts to get better output next time?

Document your refinements — this feedback improves your prompt library for future detection engineering sessions.

Verification checklist

TI report analysed with AI — indicators and ATT&CK mappings extracted
KQL query generated and validated (or corrections noted)
False positive assessment completed with tuning recommendations
Full rule specification document produced
AI output reviewed against your expert judgment
Prompt refinements documented for future use

← All Labs