10.10 ASIM Parsers and Data Normalisation

ASIM Parsers and Data Normalisation

Introduction

Required role: Microsoft Sentinel Contributor for analytics rules. Sentinel Responder for incident management.

Every security product names the same thing differently. A source IP address is IPAddress in SigninLogs, SourceIP in CommonSecurityLog, IpAddress in SecurityEvent, and SourceIPAddress in BehaviorAnalytics. A user account is UserPrincipalName in SigninLogs, Account in SecurityEvent, and SourceUserName in CommonSecurityLog.

This inconsistency means an analytics rule written for SigninLogs does not work against SecurityEvent data without modification. A brute-force detection rule must be written separately for each data source — multiplying the number of rules and the maintenance burden.

ASIM (Advanced Security Information Model) solves this by normalising data from multiple sources into a common schema. One analytics rule written against the ASIM schema detects brute-force attacks across Entra ID, Windows, Linux, and any other source with an ASIM parser — without source-specific KQL.

How ASIM works

ASIM has three components: schemas, parsers, and normalised content.

Schemas define the standardised column names for each event category. The Authentication schema defines: TimeGenerated, SrcIpAddr (source IP), TargetUsername (target user), EventResult (Success/Failure), EventProduct (the product that generated the event), LogonMethod, and many more. Every authentication event from any source is mapped to these standard columns.

Parsers are KQL functions that translate source-specific data into the ASIM schema. The vimAuthenticationSigninLogs parser reads SigninLogs and maps IPAddress → SrcIpAddr, UserPrincipalName → TargetUsername, and ResultType → EventResult. The vimAuthenticationSecurityEvent parser reads SecurityEvent and maps IpAddress → SrcIpAddr, Account → TargetUsername, and derives EventResult from EventID (4624 = Success, 4625 = Failure).

Unifying parsers combine all source-specific parsers into a single function. imAuthentication is the unifying parser for all authentication data — it internally calls vimAuthenticationSigninLogs, vimAuthenticationSecurityEvent, vimAuthenticationSyslog, and any other authentication parsers you have deployed. Querying imAuthentication returns normalised authentication events from all sources in a single result set.

ASIM schema categories

ASIM defines schemas for the most common security event categories.

Authentication (imAuthentication) — sign-ins, logons, MFA events. Sources: SigninLogs, SecurityEvent (4624/4625), Syslog (auth).

Network Session (imNetworkSession) — network connections, firewall events. Sources: CommonSecurityLog, DeviceNetworkEvents, AzureNetworkAnalytics.

DNS (imDns) — DNS queries and responses. Sources: DnsEvents, DeviceDnsEvents, Syslog (DNS).

Process (imProcessCreate, imProcessTerminate) — process creation and termination. Sources: SecurityEvent (4688), DeviceProcessEvents, Syslog (auditd).

File (imFileEvent) — file creation, modification, deletion. Sources: DeviceFileEvents, SecurityEvent (4663).

Web Session (imWebSession) — HTTP requests. Sources: CommonSecurityLog (web proxy), W3CIISLog.

Deploying ASIM parsers

ASIM parsers are deployed through Content Hub solutions or from the Microsoft Sentinel GitHub repository.

Content Hub deployment. Install the relevant Content Hub solution (e.g., “ASIM — Authentication” or “ASIM — Network Session”). The solution deploys the unifying parser and all source-specific parsers as KQL functions in your workspace.

Verification after deployment:

1
2
3
4
5

// Test the Authentication unifying parser
imAuthentication
| where TimeGenerated > ago(1h)
| summarize EventCount = count() by EventProduct, EventResult
| order by EventCount desc

If this returns results from multiple products (e.g., “Azure Active Directory” for Entra ID, “Microsoft Windows” for SecurityEvent, “Linux” for Syslog), the parsers are working correctly — data from all sources is normalised into the same schema.

Writing analytics rules with ASIM

The power of ASIM is that one rule covers all sources. Here is the brute-force detection rule rewritten for ASIM:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

// ASIM-normalised brute-force detection — works across ALL authentication sources
let FailureThreshold = 20;
imAuthentication
| where TimeGenerated > ago(1h)
| where EventResult == "Failure"
| summarize
    FailureCount = count(),
    TargetUsers = make_set(TargetUsername, 10),
    Products = make_set(EventProduct, 5),
    FirstFailure = min(TimeGenerated),
    LastFailure = max(TimeGenerated)
    by SrcIpAddr
| where FailureCount > FailureThreshold
| join kind=inner (
    imAuthentication
    | where TimeGenerated > ago(1h)
    | where EventResult == "Success"
    | project SuccessTime = TimeGenerated, SrcIpAddr,
        CompromisedUser = TargetUsername, EventProduct
) on SrcIpAddr
| where SuccessTime > LastFailure
| project SrcIpAddr, FailureCount, TargetUsers, Products,
    CompromisedUser, SuccessTime

This single rule detects brute-force across Entra ID sign-ins, Windows RDP logons, Linux SSH authentication, and any future authentication source that has an ASIM parser deployed. Without ASIM, you would need three separate rules (one per source) with three separate KQL queries.

Entity mapping for ASIM rules: Map SrcIpAddr to the IP entity and CompromisedUser to the Account entity. These column names are consistent across all ASIM outputs — the entity mapping works regardless of the underlying data source.

Additional ASIM analytics rule examples

Cross-source network anomaly detection using imNetworkSession:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

// Detect unusual outbound connections across all network data sources
imNetworkSession
| where TimeGenerated > ago(1h)
| where DstPortNumber !in (80, 443, 53, 22, 25, 587)
| where not(ipv4_is_private(DstIpAddr))
| where EventResult == "Success"
| summarize
    ConnectionCount = count(),
    UniquePorts = dcount(DstPortNumber),
    Ports = make_set(DstPortNumber, 10),
    Products = make_set(EventProduct, 5)
    by SrcIpAddr
| where ConnectionCount > 50 or UniquePorts > 10
| project SrcIpAddr, ConnectionCount, UniquePorts, Ports, Products

This detects potential C2 communication or port scanning across firewall CEF data, Defender for Endpoint network events, and any other source with a Network Session ASIM parser — all in a single rule.

Cross-source process execution detection using imProcessCreate:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

// Detect suspicious process execution across Windows and Linux
imProcessCreate
| where TimeGenerated > ago(15m)
| where TargetProcessName has_any (
    "powershell", "cmd", "wscript", "cscript", "mshta",
    "certutil", "bitsadmin", "curl", "wget", "python", "bash")
| where ActingProcessName has_any (
    "outlook", "winword", "excel", "powerpnt", "onenote",
    "thunderbird", "chrome", "firefox", "msedge")
| project TimeGenerated, DvcHostname, ActorUsername,
    ActingProcessName, TargetProcessName, TargetProcessCommandLine,
    EventProduct

This detects Office applications or browsers spawning suspicious child processes — a classic initial access pattern — across both Windows (SecurityEvent, DeviceProcessEvents) and Linux (Syslog auditd) simultaneously.

ASIM parser deployment and management

Deploying from Content Hub. Navigate to Sentinel → Content Hub → search “ASIM.” Install the relevant schema packages: “ASIM Authentication,” “ASIM Network Session,” “ASIM Process Event,” “ASIM DNS.” Each package deploys the unifying parser, all source-specific parsers, and example analytics rules.

Verifying parser deployment. After installation, verify each parser is working:

1
2
3
4

// Test each unifying parser — should return data from multiple products
imAuthentication | where TimeGenerated > ago(1h) | summarize count() by EventProduct
imNetworkSession | where TimeGenerated > ago(1h) | summarize count() by EventProduct
imProcessCreate | where TimeGenerated > ago(1h) | summarize count() by EventProduct

If a parser returns zero results for a product you have connected, check: is the source-specific parser deployed? Does the source table have data? Is the parser function correctly registered in the workspace?

Parser versioning. ASIM parsers are updated periodically by Microsoft (new source support, bug fixes, schema updates). When a Content Hub solution update is available for an ASIM package, review the changelog before updating — schema changes can break existing analytics rules that reference removed or renamed columns.

ASIM-powered investigation patterns

ASIM normalisation enables investigation patterns that span multiple data sources in a single query.

Cross-source IP investigation. Given a suspicious IP, find all authentication attempts AND all network connections from that IP — regardless of which product captured them.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

let InvestigateIP = "203.0.113.47";
let AuthEvents = imAuthentication
| where TimeGenerated > ago(24h)
| where SrcIpAddr == InvestigateIP
| project TimeGenerated, Schema = "Authentication",
    User = TargetUsername, Result = EventResult,
    Product = EventProduct;
let NetworkEvents = imNetworkSession
| where TimeGenerated > ago(24h)
| where SrcIpAddr == InvestigateIP or DstIpAddr == InvestigateIP
| project TimeGenerated, Schema = "Network",
    User = SrcUsername, Result = DvcAction,
    Product = EventProduct;
union AuthEvents, NetworkEvents
| order by TimeGenerated asc

This produces a unified timeline of all activity from a single IP across sign-in logs, Windows events, Linux SSH, firewall logs, and endpoint network events — one query, one result set, one timeline.

Cross-source user investigation. Given a compromised user, find all their authentication events, process executions, and file accesses across all sources.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

let InvestigateUser = "j.morrison";
let Auth = imAuthentication
| where TimeGenerated > ago(24h)
| where TargetUsername has InvestigateUser
| project TimeGenerated, Activity = "Authentication",
    Detail = strcat(EventResult, " from ", SrcIpAddr), Product = EventProduct;
let Processes = imProcessCreate
| where TimeGenerated > ago(24h)
| where ActorUsername has InvestigateUser
| project TimeGenerated, Activity = "ProcessCreate",
    Detail = strcat(TargetProcessName, " ", TargetProcessCommandLine),
    Product = EventProduct;
union Auth, Processes
| order by TimeGenerated asc

This unified timeline shows the user’s activity across all platforms — authentication events from any source plus process execution from any endpoint — in a single chronological view. Without ASIM, building this timeline requires writing and maintaining separate queries for each data source and manually merging the results.

ASIM vs raw table queries: a practical comparison

To illustrate the value of ASIM, compare the same detection written both ways.

Detection: failed authentication from external IP exceeding threshold.

Raw table approach (3 separate rules needed):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

// Rule 1: Entra ID
SigninLogs | where ResultType != "0"
| where not(ipv4_is_private(IPAddress))
| summarize count() by IPAddress | where count_ > 20

// Rule 2: Windows
SecurityEvent | where EventID == 4625
| where IpAddress != "" and not(ipv4_is_private(IpAddress))
| summarize count() by IpAddress | where count_ > 20

// Rule 3: Linux SSH
Syslog | where Facility == "auth" | where SyslogMessage has "Failed"
| parse SyslogMessage with * "from " SourceIP " port " *
| where not(ipv4_is_private(SourceIP))
| summarize count() by SourceIP | where count_ > 20

ASIM approach (1 rule covers all):

1
2
3
4

imAuthentication
| where EventResult == "Failure"
| where not(ipv4_is_private(SrcIpAddr))
| summarize count() by SrcIpAddr | where count_ > 20

The ASIM version is: shorter (4 lines vs 12), easier to maintain (one rule vs three), automatically covers new sources (deploy a new parser, the rule starts detecting), and uses consistent entity names (SrcIpAddr works for entity mapping regardless of source). The trade-off is the minor performance overhead of the union operation inside the parser.

ASIM adoption roadmap

Adopt ASIM incrementally rather than converting all rules at once.

Phase 1: Deploy parsers (Week 1). Install ASIM Content Hub solutions for all relevant schemas (Authentication, Network Session, Process). Verify parsers return data.

Phase 2: Enable ASIM-based Content Hub rules (Week 2). Many Content Hub analytics rules are written in ASIM. Enable these first — they are pre-tested and optimised.

Phase 3: Convert high-value custom rules (Month 1-2). Identify your top 10 custom analytics rules by true positive volume. For each, evaluate: does the detection apply to multiple sources? If yes, convert to ASIM. If no (source-specific detection), keep the raw table query.

Phase 4: Write new rules in ASIM by default (Ongoing). For all new rule development, use ASIM unless there is a specific reason to use raw tables (source-specific fields needed, NRT performance requirements). This ensures your rule library becomes increasingly normalised over time.

ASIM parser troubleshooting

Parser returns zero results. Check: is the source table populated? (SigninLogs | count — if zero, the connector is not delivering data, not a parser issue). Is the parser function deployed? (imAuthentication | getschema — if it fails, the function is not installed). Was the parser recently updated and a column name changed?

Parser returns results from only some sources. Check which source parsers are installed: run each source-specific parser individually (vimAuthenticationSigninLogs | count, vimAuthenticationSecurityEvent | count). Missing sources need their parser installed from Content Hub.

Parser returns incorrect data (wrong EventResult for some events). The source-specific parser has a mapping error — it may be misinterpreting a result code. Check the parser’s KQL logic: navigate to the workspace → Functions → find the parser → examine the mapping logic. Report issues to Microsoft via the Sentinel GitHub repository.

Parser is slow (> 30 seconds for 1-hour query). The unifying parser queries all source tables. If some source tables are very large, the union is expensive. Optimise by: adding | where TimeGenerated > ago(...) as early as possible in your analytics rule query (the parser passes this filter down to each source), or use the source-specific parser when only one source is relevant.

ASIM and entity mapping: a natural fit

ASIM schemas use consistent column names across all sources — which means entity mapping is also consistent.

When you write an analytics rule using ASIM and map SrcIpAddr to the IP entity, that mapping works regardless of whether the underlying data came from SigninLogs (where the raw column is IPAddress), SecurityEvent (where it is IpAddress), or Syslog (where it was parsed from the message). The parser handles the translation — the entity mapping is always SrcIpAddr → IP → Address.

This consistency means: your entity mapping standard (subsection 10.4) is automatically enforced for all ASIM-based rules. Cross-incident correlation works across all data sources. And new data sources added in the future automatically integrate with existing entity mappings when you deploy their ASIM parser.

Building an ASIM hunting query library

ASIM is ideal for hunting because investigation queries naturally span multiple sources.

Hunt: external IP accessing multiple users.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

imAuthentication
| where TimeGenerated > ago(24h)
| where EventResult == "Success"
| where not(ipv4_is_private(SrcIpAddr))
| summarize UniqueUsers = dcount(TargetUsername),
    Users = make_set(TargetUsername, 20),
    Products = make_set(EventProduct, 5)
    by SrcIpAddr
| where UniqueUsers > 3
| order by UniqueUsers desc

Hunt: rare process execution across all endpoints.

1
2
3
4
5
6
7

imProcessCreate
| where TimeGenerated > ago(7d)
| summarize ExecutionCount = count(),
    UniqueHosts = dcount(DvcHostname)
    by TargetProcessName
| where ExecutionCount < 5 and UniqueHosts < 3
| order by ExecutionCount asc

Rare processes (executed fewer than 5 times across the entire environment) are high-value hunting targets — they may be legitimate but unusual tools, or they may be attacker tools that have not been seen before.

When NOT to use ASIM

ASIM is not always the right choice. Use source-specific queries when:

You need source-specific fields. ASIM normalises to common columns. Source-specific fields (e.g., ConditionalAccessStatus in SigninLogs, LogonType in SecurityEvent) are not in the ASIM schema. If your detection requires these fields, query the source table directly.

Performance is critical. ASIM unifying parsers union multiple tables — each execution queries all configured sources. For NRT rules (subsection 10.3), this overhead may push detection latency beyond acceptable limits. Use source-specific queries for NRT.

Only one source is relevant. If your detection only applies to one data source (e.g., Entra ID sign-in risk events), using imAuthentication adds unnecessary overhead by querying SecurityEvent and Syslog tables that cannot contain the relevant data. Query SigninLogs directly.

Creating custom ASIM parsers

If your environment has a data source without an existing ASIM parser, you can write one.

Custom parser structure: An ASIM parser is a KQL function that reads a specific table and maps its columns to the ASIM schema.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

// Custom ASIM Authentication parser for a custom application table
// Save as a KQL function named: vimAuthenticationCustomApp
let CustomAppAuth = (starttime:datetime=datetime(null), endtime:datetime=datetime(null)) {
    CustomAppAuth_CL
    | where (isnull(starttime) or TimeGenerated >= starttime)
        and (isnull(endtime) or TimeGenerated <= endtime)
    | extend
        EventProduct = "CustomHRApp",
        EventVendor = "Northgate",
        EventType = "Logon",
        EventResult = case(
            Action == "login_success", "Success",
            Action == "login_failed", "Failure",
            "NA"),
        SrcIpAddr = IPAddress,
        TargetUsername = UserPrincipalName,
        TargetUsernameType = "UPN",
        EventResultDetails = FailureReason
};
CustomAppAuth

After deploying this parser, register it with the imAuthentication unifying parser. Now the ASIM brute-force rule detects attacks against your custom application alongside Entra ID, Windows, and Linux — without modifying the rule.

ASIM schema reference: key column mappings

Understanding which source columns map to which ASIM columns is essential for writing and debugging parsers.

Authentication schema key mappings:

Network Session schema key mappings:

ASIM parser performance considerations

ASIM unifying parsers internally union multiple tables and apply transformations. This is computationally more expensive than querying a single table directly.

For NRT rules: Consider using source-specific queries rather than ASIM parsers. The additional processing time of ASIM may add seconds to detection latency — acceptable for scheduled rules but potentially significant for NRT.

For high-frequency scheduled rules (every 5 minutes): Test the ASIM query execution time in the Logs blade. If the query takes more than 30 seconds, consider optimising: add time filters early in the query (| where TimeGenerated > ago(10m) as the first operator), or use the source-specific parser (vimAuthenticationSigninLogs) instead of the unifying parser (imAuthentication) if only one source is relevant.

For hunting queries: ASIM is ideal — the slight performance overhead is acceptable for interactive hunting, and the cross-source normalisation dramatically accelerates investigation.

Knowledge check