0.4 The Confidence Threshold Problem

Why severity labels are not confidence scores

Most analytics rules produce alerts with a severity: Informational, Low, Medium, High. Severity describes the potential impact if the alert is a true positive. It does not describe the probability that the alert is a true positive. A High-severity AiTM alert means "if this is real, the impact is significant." It does not mean "this is probably real."

This distinction is the core of the confidence problem. An analyst reading a High-severity alert applies implicit confidence judgment: they evaluate the alert details, the user context, the enrichment data, and their own experience to decide whether the alert is real. That judgment is invisible, undocumented, and inconsistent across analysts and shifts. Two analysts evaluating the same alert may reach different conclusions.

Automated containment cannot use implicit judgment. A playbook needs a number, a quantified probability that the alert represents a genuine compromise, to decide whether to execute a containment action or route the incident for human review. That number is the confidence score, and the threshold at which the playbook acts is the confidence threshold.

The true positive rate as foundation

The simplest and most reliable measure of detection confidence is the analytics rule's true positive rate (TPR) over a historical window. TPR answers a direct question: of the alerts this rule generated in the last 90 days, what percentage were classified as true positives by the SOC?

// Calculate true positive rate per analytics rule (90-day window)
SecurityIncident
| where TimeGenerated > ago(90d)
| where Status == "Closed"
| extend RuleName = tostring(AdditionalData.alertProductNames)
| summarize
    TotalClosed = count(),
    TruePositives = countif(Classification == "TruePositive"),
    FalsePositives = countif(Classification == "FalsePositive"),
    BenignTP = countif(Classification == "BenignPositive")
    by Title
| extend TPR = round(TruePositives * 100.0 / TotalClosed, 1)
| where TotalClosed >= 20
| sort by TPR desc
// Filter: minimum 20 closed incidents for statistical reliability
// TPR: percentage of closed incidents classified as true positive
// Rules with TPR >= 90% are candidates for automated containment

Run this query against your workspace. Rules with a TPR above 90% are strong candidates for Tier 3 automation with session revocation. Rules below 70% need tuning before any containment automation is appropriate. Rules between 70% and 90% are candidates for semi-automated response with approval gates.

The TotalClosed >= 20 filter matters. A rule that fired 3 times and all 3 were true positives has a 100% TPR, but the sample size is too small to be statistically meaningful. Twenty closed incidents is the minimum for reasonable confidence in the rate. Fifty is better. For high-impact containment actions like account disablement, you want 100+ data points before trusting the TPR as a basis for fully automated action.

The base rate problem

TPR alone can mislead when the base rate of true incidents is low relative to total alert volume. Consider a workspace generating 500 incidents per day. If 10 of those are genuine compromises and 490 are benign or false positives, the base rate of true incidents is 2%. Even with a rule that has a 90% TPR, the 10% false positive rate applied to the high-volume benign population produces more false actions than true actions in absolute numbers.

This is the base rate fallacy applied to security automation. A rule with 90% TPR sounds reliable, and it is reliable for its own alerts. But if that rule generates 50 alerts per day and only 5 are true positives, the 90% TPR means the rule correctly classifies 45 of those 50. The remaining 5 misclassifications include 4-5 false positives that your automation would incorrectly act on. For session revocation, 5 unnecessary re-authentications per day might be acceptable. For account disablement, 5 business disruptions per day is not.

The solution is not to abandon TPR but to combine it with enrichment signals that increase confidence for individual alerts. A rule-level TPR of 90% tells you the rule is generally reliable. Adding alert-level signals tells you whether this specific alert is in the 90% or the 10%.

Understanding the base rate also explains why different alert types have different automation ceilings. A rule that generates 5 alerts per day with a 95% TPR produces roughly 1 false positive per week. Automating session revocation for that rule means one unnecessary re-authentication per week, which is easily tolerable. A rule that generates 100 alerts per day with the same 95% TPR produces 5 false positives per day. Automating the same containment action for that rule means five disrupted users daily, which may exceed your tolerance. The threshold for high-volume rules should be higher than for low-volume rules because the absolute number of false actions scales with volume, even when the rate stays constant.

Composite confidence scoring

A composite confidence score combines the rule's baseline TPR with alert-specific enrichment signals to produce a per-incident confidence value. Each enrichment signal either increases or decreases confidence from the baseline.

The composite model is additive with a cap. Start with the rule's TPR as the baseline. Add positive signals (TI match, elevated risk, impossible travel, post-compromise indicators). Subtract negative signals (IP on known-safe list, user recently traveled, sign-in from corporate VPN range). Cap at 99% because no signal combination produces absolute certainty. Floor at the rule's TPR minus a maximum adjustment to prevent enrichment noise from dropping confidence below a reasonable level.

The signal weights are calibrated through observation, not calculation. Start with conservative weights (2-5% per signal). Deploy the composite scoring in reporting mode for 30 days, where the score is calculated and logged but no automation fires. Review incidents where the composite score disagreed with the analyst's classification. Adjust weights based on which signals most reliably distinguish true positives from false positives in your environment.

Some signals have high discriminative power across most environments. Impossible travel (sign-in from two countries within minutes) is a strong positive signal for AiTM. Post-compromise indicators (inbox rule creation, OAuth consent grant within minutes of the alert) are even stronger because they indicate the attacker is already acting on the compromised session. IP reputation is useful but noisier because TI feeds have their own false positive rates and because legitimate users occasionally connect from VPN exit nodes that share IP ranges with known-malicious infrastructure.

Other signals are environment-specific. A sign-in from a country where your organization has no offices is a strong positive signal in an organization that operates in two countries. In a multinational with offices in 40 countries, it is weaker. A device compliance failure is a strong negative signal in an organization that enforces compliance universally, but weaker in an organization where BYOD is common and non-compliant devices are expected. The 30-day reporting mode exists specifically to identify which signals carry weight in your environment before you trust them with automated decisions.

Setting the threshold

Different containment actions require different confidence thresholds because the cost of a false action differs by action type. Section 0.3 established this principle. Here is how to translate it into specific numbers.

Session revocation (lowest impact): False positive cost is a forced re-authentication across devices, roughly 30 seconds of disruption. Semi-automated threshold: 85%. Fully automated threshold: 95%. At 95%, you accept that 1 in 20 automated revocations will be a false action, producing a brief disruption for a legitimate user.

Account disablement (high impact): False positive cost is total M365 access loss until an admin re-enables the account. Semi-automated threshold: 92%. Fully automated threshold: 97%, and only for non-VIP accounts. VIP accounts always route to approval regardless of confidence.

Device isolation (highest impact): False positive cost is complete loss of device functionality until IT releases the isolation. Semi-automated threshold: 95%. Fully automated: restricted to confirmed malware detections from Defender for Endpoint with high-fidelity EDR correlation, which typically have TPR above 98%.

These numbers are starting points, not universal rules. Your thresholds depend on your organization's risk tolerance, your help desk capacity to reverse false actions, and the business criticality of your user population. A SOC supporting a trading floor where 60 seconds of downtime has regulatory implications will set higher thresholds than a SOC supporting a university where re-authentication is a minor inconvenience.

Calibrating thresholds is an iterative process. Start with the thresholds above. Run the composite scoring in reporting mode for the first 30 days, logging the score for every incident but not executing any automated action. After 30 days, review the data: how many incidents exceeded the threshold? Of those, how many were true positives? How many were false positives? The false-action rate you observe in reporting mode is the false-action rate you will experience in production. If that rate is unacceptable for the containment action, raise the threshold until the observed false-action rate drops to a tolerable level.

Document the calibration decision. Record the rule, the threshold, the sample size, the observed TPR, and the false-action rate. This documentation is the evidence that justifies the automation to your SOC manager, your CISO, and any auditor who asks why a playbook is disabling user accounts without human approval.

When confidence is not enough

Confidence answers one question: is this alert probably a true positive? It does not answer a second question: if I act on this alert and I am wrong, what is the damage? A 96% confidence score on an alert affecting a standard user account justifies automated session revocation. The same 96% confidence score on an alert affecting the CEO's account during the annual board meeting might not, because the business impact of a false action on that specific entity at that specific time exceeds the security benefit of 30 seconds faster containment.

This is why confidence alone does not determine automated action. SA0.5 introduces the blast radius assessment, which evaluates the second question: what is the cost of being wrong? Together, confidence threshold and blast radius assessment form the complete decision framework for Tier 3 automation. Both must pass before any containment action executes automatically.