DE0.7 The Data You Already Have

The ingestion-detection gap

Your SIEM ingestion bill pays for data. Your analytics rules determine how much of that data produces security value. If you ingest 20 tables and query 6, you are paying full price for 14 tables of data that generates zero alerts. That data is not wasted from a retention perspective — you can hunt through it manually, you can query it during incident response — but from a detection perspective, it is idle.

The ingestion-detection gap is the ratio of tables with at least one analytics rule to total ingested tables. Northgate Engineering’s gap: 6 / 20 = 30%. Seventy percent of their ingested data has no automated detection.

This is the detection engineer’s opportunity. Every new rule built against a previously unqueried table extracts additional value from data the organization already pays to ingest. No new data connector needed. No new ingestion cost. Just KQL against existing tables.

Figure DE0.7 — Northgate Engineering data source audit. Green tables have at least one analytics rule. Red tables are ingested and paid for but have zero detection rules — including the highest-volume table (AADNonInteractiveUserSignInLogs at 4.8 GB/day).

The highest-value unqueried tables

AADNonInteractiveUserSignInLogs (4.8 GB/day). The highest-volume table in the workspace and zero rules query it. This table records every non-interactive authentication: token refreshes, app-based authentication, service principal sign-ins. AiTM session token theft (CHAIN-HARVEST Phase 2) produces a signature pattern in this table — a non-interactive token refresh from a different IP than the preceding interactive sign-in. Token replay attacks, application-level compromise, and service principal abuse are all detectable here. DE4 builds rules against this table.

DeviceNetworkEvents (2.4 GB/day). Every network connection from every managed endpoint. RDP lateral movement (CHAIN-MESH Phases 3 and 5), C2 beaconing (CHAIN-ENDPOINT Phase 7), DNS tunneling, and data exfiltration over unusual ports — all visible here. This is the primary table for lateral movement and exfiltration detection. DE8 builds rules against it.

DeviceFileEvents (1.1 GB/day). Every file creation, modification, and deletion on managed endpoints. Bulk file reads from file shares (CHAIN-ENDPOINT collection), USB file writes (CHAIN-FACTORY exfiltration), ransomware mass encryption (CHAIN-MESH Phase 7), and document staging for exfiltration — all visible here. DE7 builds rules against it.

CommonSecurityLog (1.2 GB/day). Palo Alto firewall logs — allow and deny events across all sites. Inter-site traffic patterns, egress to suspicious destinations, large data transfers, and VPN session metadata. This is the only visibility into network-layer activity for devices without Defender for Endpoint. DE8 uses this table for cross-site lateral movement correlation.

DeviceRegistryEvents (0.6 GB/day). Registry modifications on managed endpoints. Persistence mechanisms (Run keys, services), security tool tampering (Defender configuration changes), and malware configuration writes. DE5 builds persistence detection rules against this table.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

// Identify your unqueried tables
// Run in your Sentinel workspace to find tables with data but no rules
let queriedTables = SecurityAlert
    | where TimeGenerated > ago(90d)
    | where ProviderName == "ASI Scheduled Alerts"
    | extend RuleQuery = tostring(ExtendedProperties.["Query"])
    | mv-expand TableName = extract_all(@"(\w+)\s*\|", RuleQuery)
    | summarize by tostring(TableName);
Usage
| where TimeGenerated > ago(30d)
| where IsBillable == true
| summarize DailyGB = round(sum(Quantity) / 1024 / 30, 2) by DataType
| where DailyGB > 0.01
| join kind=leftanti queriedTables on $left.DataType == $right.TableName
| sort by DailyGB desc
// Output: tables you pay for but have zero detection rules against

Data sources NOT connected

The detection gap includes not only unqueried tables but also data sources that are not connected to Sentinel at all. Northgate has five significant telemetry gaps:

AWS CloudTrail — the engineering team uses AWS S3 for some file storage. No visibility into API calls, IAM activity, or data access in AWS. A cross-cloud attack that pivots from Microsoft to AWS is invisible.

OT network logs — the manufacturing floor network at Bristol and Sheffield is air-gapped (correctly). Sunderland’s OT network is connected to the corporate network (a design flaw). No OT logs are forwarded to Sentinel from any site.

RHEL auditd — the 4 RHEL CAD/CAM servers have auditd configured but the logs are reviewed manually, not forwarded to Sentinel. The crown jewels have no real-time detection.

VPN session logs — Prisma Access session metadata is available in Panorama but not forwarded to Sentinel. VPN authentication goes through Entra ID (visible in SigninLogs) but VPN session duration, bandwidth, and disconnect events are not.

DNS query logs — internal DNS resolvers do not log queries. DNS tunneling, DGA domain resolution, and C2 over DNS are invisible.

Connecting these sources increases ingestion cost. The detection engineer must justify each connection with specific detection rules it enables — the detection-first principle from the course blueprint. DE2 includes this cost-benefit analysis in the 90-day roadmap.

Troubleshooting: “We do not know what tables we have”

Run the Usage table query. Usage | where TimeGenerated > ago(30d) | where IsBillable == true | summarize DailyGB = round(sum(Quantity) / 1024 / 30, 2) by DataType | sort by DailyGB desc shows every table with data and its volume. This is the starting point for any data source audit.

Check the data connectors page. Sentinel → Data connectors shows all configured connectors and their status. Connected connectors with “last log received” timestamps confirm active data flow. Connectors that show “not connected” or stale timestamps indicate data sources that were configured but stopped working — a common issue after tenant changes or API key rotations.

References used in this subsection

• NE Training Universe blueprint (data source inventory)
• Course cross-references: DE2 (cost-benefit analysis for new data sources), DE4 (AADNonInteractiveUserSignInLogs rules), DE7 (DeviceFileEvents rules), DE8 (DeviceNetworkEvents and CommonSecurityLog rules)