1.1 What AI Actually Does (and Does Not Do)

What AI Actually Does (and Does Not Do)

Every vendor in the security industry now claims their product is “AI-powered.” Firewalls are AI-powered. SIEMs are AI-powered. Endpoint detection is AI-powered. The term has become so diluted that it communicates almost nothing about what a product actually does. This subsection strips the marketing away and examines what AI — specifically, large language models (LLMs) — can and cannot do in a security operations context.

Understanding these mechanics is not academic. It directly determines how you use AI tools, what you trust them for, and where you apply verification. An analyst who does not understand why an LLM hallucinates will trust fabricated output. An analyst who understands the mechanism catches it before it causes damage.

Three categories of “AI” in security products

When a security vendor says “AI-powered,” they mean one of three things. These are fundamentally different technologies with different strengths and failure modes.

Rule-based systems with conditional logic. The oldest form. If-then rules, threshold-based alerting, signature matching. When a vendor calls this AI, they are using the term loosely. Examples: YARA rules, Sigma rules, static threshold alerts in any SIEM. These are deterministic — given the same input, they always produce the same output. They do not learn, adapt, or generalize. They do exactly what the rule specifies. Their failure mode is false positives (rule too broad) or false negatives (rule too narrow).

Traditional machine learning (ML). Statistical models trained on historical data to classify new observations. Examples: anomaly detection in user behavior analytics (UEBA), malware classification in endpoint protection, spam filtering. These models learn patterns from training data and generalize to new inputs. Their failure mode is distribution shift — when the production data looks different from the training data, the model’s accuracy degrades. A UEBA model trained on pre-pandemic office behavior produces false positives when everyone starts working from home. ML models are good at classification and anomaly detection. They are poor at generation, explanation, and reasoning.

Large language models (LLMs). Neural networks trained on massive text corpora that generate human-like text output. Examples: Claude, ChatGPT, Gemini, Copilot. These models predict the next token (word fragment) based on the preceding context. They are fundamentally text generation systems — they produce statistically probable output, not factually verified output. Their failure mode is hallucination: generating confident, well-structured output that is factually wrong. LLMs are good at generation (queries, reports, scripts, analysis), pattern recognition across text, and structured transformation (reformatting, summarizing, translating between formats). They are poor at factual verification, real-time data access, mathematical precision, and maintaining state across sessions.

Why the distinction matters for security operations: When your SIEM vendor says “AI-powered detection,” they almost certainly mean traditional ML (anomaly detection) or rule-based systems — not LLMs. When this course teaches AI-assisted investigation, it means LLMs — specifically, using conversational AI to generate queries, analyze logs, draft reports, and build detection rules. These are different tools solving different problems. Do not conflate them.

How large language models work — the minimum you need to know

An LLM processes input (your prompt plus any uploaded context) through a neural network and generates output one token at a time, where each token is selected based on its probability of following the preceding tokens. The model does not “understand” your query in the way a human analyst does. It identifies patterns in the input that match patterns it learned during training and generates a statistically probable response.

Context window. The model can only process a fixed amount of text at once — the context window. For Claude, this is approximately 200,000 tokens (roughly 150,000 words). Everything the model needs to know must fit in this window: your system prompt, uploaded documents, conversation history, and your current query. Information outside the context window does not exist to the model. This is why Claude does not remember your conversation from yesterday unless you provide it again.

Training data cutoff. The model’s knowledge comes from its training data, which has a fixed cutoff date. For Claude, this is currently May 2025. The model does not know about vulnerabilities disclosed after this date, Microsoft portal changes made after this date, or threat actor campaigns discovered after this date — unless it uses web search or you provide the information in the context window. In security operations, where the threat landscape changes weekly, this is a significant limitation.

Temperature and randomness. LLM output is not deterministic. The same prompt can produce slightly different output each time because the token selection process includes a randomness factor (temperature). For security work, this means: if you ask for a KQL query twice, you may get two slightly different queries. Both may be valid. One may be better than the other. This is not a bug — it is a fundamental property of how LLMs generate text.

The five hallucination patterns in security contexts

Hallucination is the most dangerous failure mode for security professionals using AI. The model generates output that reads authoritatively, is syntactically correct, and is factually wrong. These patterns are predictable.

Pattern 1: Invented table and column names. The model generates a query referencing AADUserRiskEvents — a table that does not exist in any Sentinel workspace. The real table is AADRiskyUsers or the risk information is in SigninLogs. The query is syntactically perfect. The table name is statistically probable (it sounds like a real Microsoft table). It does not exist.

How to catch it: Run every generated query in your lab environment before production. If the query fails with a “table not found” or “column not found” error, the reference is hallucinated. Check the Sentinel schema browser or the Defender advanced hunting schema reference for the correct table.

Pattern 2: Outdated feature descriptions. The model describes a Defender for Endpoint feature using its pre-cutoff name or portal location. Microsoft renames features, moves portal navigation, and restructures settings frequently. The description may have been accurate in the training data but is no longer correct.

How to catch it: When the model describes a portal location (“Navigate to Settings → Endpoints → Advanced Features”), verify it in the actual portal. If the menu path does not match, check the current Microsoft documentation.

Pattern 3: Fabricated references. The model cites a CVE number (CVE-2025-XXXXX), a MITRE technique, or a vendor documentation URL that does not exist. The citation is formatted correctly and sounds plausible. It is fabricated. This is especially common when the model generates hypothetical examples and invents supporting references to make them more concrete.

How to catch it: Verify every CVE against the National Vulnerability Database (nvd.nist.gov). Verify every MITRE technique against the ATT&CK matrix (attack.mitre.org). Follow every URL. If the reference does not resolve, it is fabricated.

Pattern 4: Overconfident analysis. The model analyzes a sign-in log entry and concludes “this is definitively token replay” when the evidence is ambiguous. The analysis reads authoritatively. The confidence level is not justified by the evidence. The model does not express appropriate uncertainty — it generates confident conclusions regardless of evidence strength.

How to catch it: For any analytical conclusion, ask: “What evidence supports this conclusion? What alternative explanations exist?” If the model cannot cite specific log fields and values that support its conclusion, the analysis is overconfident.

Pattern 5: Plausible but incorrect remediation steps. The model recommends revoking refresh tokens using a PowerShell cmdlet that has been deprecated or renamed. The recommendation sounds correct. The cmdlet name is plausible. It either does not exist or does not do what the model claims.

How to catch it: Verify every PowerShell cmdlet, API endpoint, and portal action against the current documentation before executing it in production. Use Get-Help and -WhatIf parameters.

Knowledge boundaries — what the model does not know

Your environment. The model does not know your Sentinel table names, your watchlist contents, your conditional access policies, your network topology, your organizational structure, or your incident history. Every query and recommendation must be adapted to your specific environment. Generic output is the starting point, not the finished product.

Current events. Threat actor TTPs, new vulnerabilities, Microsoft product changes, and regulatory updates from after the training cutoff are unknown unless the model uses web search — and web search results are not guaranteed to be accurate or current.

Operational context. The model does not know that the CEO is traveling, that the finance team processes payments on Tuesdays, that the new VPN changed everyone’s external IP last week, or that three employees just returned from parental leave and are generating anomalous sign-in patterns. This operational context — the context that determines whether an alert is a true positive or a false positive — must come from you.

Confidential information. The model does not have access to classified threat intelligence, internal incident reports, or proprietary detection logic unless you provide it in the context window. When you do provide it, the data handling implications depend on the platform and plan you are using (covered in subsection 1.5).

AI across the security field — beyond SOC operations

The examples above focus on SOC operations because that is where most security teams first adopt AI. But AI’s applicability extends across the full security discipline:

Application security (AppSec). Claude Code Security (launched February 2026) performs reasoning-based vulnerability scanning — reading codebases the way a human security researcher would, tracing data flows, and identifying context-dependent vulnerabilities that pattern-based SAST tools miss. Beyond scanning, Claude Code assists with security code review, threat modeling (STRIDE analysis of application architectures), security regression test generation, and secure coding guidance. Module 5.3 covers this in depth.

Vulnerability management. AI assists with vulnerability triage and prioritization (enriching scanner findings with exploitability context and business impact), patch impact analysis (assessing what breaks when you patch), and vulnerability report generation for management. The volume problem in vulnerability management — thousands of findings, limited remediation capacity — is where AI’s prioritization assistance has the highest impact.

DevSecOps. Claude Code integrates into CI/CD pipelines for automated PR security reviews, infrastructure-as-code security analysis, container configuration scanning, and dependency vulnerability auditing. The shift-left model works only if security review scales with development velocity — AI provides that scale.

Cloud security. Claude assists with cloud configuration posture assessment (comparing your Azure/AWS/GCP configuration against CIS Benchmarks), cloud IAM policy analysis (identifying overprivileged accounts and excessive permissions), and multi-cloud governance documentation. Module 6.5 covers cloud compliance workflows.

Privacy engineering. AI accelerates the documentation that privacy regulations require: DPIAs, ROPAs, DSAR response compilation, and privacy impact assessments. Module 6.5 covers privacy engineering with Claude.

GRC and executive reporting. Policy drafting, compliance gap analysis, risk register maintenance, and board-level security reporting — all documentation-intensive tasks where AI compresses weeks of work into hours. Module 6 covers the full compliance workflow.

Each of these domains has specific Claude workflows covered in later modules. The key principle applies uniformly: AI generates and assists. Humans verify, decide, and act.