6.5 Custom Logs and API Ingestion

Custom Logs and API Ingestion

By the end of this subsection, you will understand when to use custom log tables, how the Logs Ingestion API works, and be able to choose the right custom ingestion approach for a given scenario.

Not every data source has a built-in connector. Custom applications, SaaS platforms without native Sentinel integration, and proprietary systems require custom ingestion.

Three approaches — choose by scenario

Logs Ingestion API — the most common path

The Logs Ingestion API accepts JSON payloads over HTTPS and writes them to your workspace through a DCR. This is the foundation for all custom data ingestion.

How it works:

1. Create a custom table in your workspace (e.g., CustomWebApp_CL). Define the schema: column names and data types.
2. Create a DCR that maps the incoming JSON fields to the table columns and optionally transforms the data.
3. Create an Entra ID app registration for authentication (client ID + secret).
4. POST JSON from your application to the DCR endpoint URL with a bearer token.

Example: custom web application sending auth failures

Your internal web application logs authentication failures. You want to detect brute force against this app the same way you detect it against Entra ID.

The application sends a JSON payload:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

-- This is the JSON structure your app POSTs to the DCR endpoint
-- {
--   "TimeGenerated": "2026-03-21T14:32:00Z",
--   "EventType": "AuthFailure",
--   "SourceIP": "203.0.113.88",
--   "UserName": "j.morrison",
--   "AppName": "InternalHRPortal",
--   "FailureReason": "InvalidPassword",
--   "UserAgent": "Mozilla/5.0..."
-- }

Once ingested, query it like any standard table:

1
2
3
4
5
6
7

CustomWebApp_CL
| where TimeGenerated > ago(1h)
| where EventType_s == "AuthFailure"
| summarize FailCount = count(), Users = dcount(UserName_s)
    by SourceIP_s
| where FailCount > 10
| sort by FailCount desc

Codeless Connector Platform (CCP)

CCP creates a full Sentinel data connector — with a connector page in the gallery, health monitoring, and configuration UI — without writing code. It uses an ARM template to define API polling.

When to use: You want a reusable, maintainable connector for a SaaS platform with a REST API. The platform does not support webhooks (you must poll). You plan to share the connector with the community or use it across multiple workspaces.

When NOT to use: For one-off data sources, scripts, or applications that can POST JSON, the Logs Ingestion API is simpler and requires no ARM template knowledge.

Azure Functions

An Azure Function runs on a schedule (e.g., every 5 minutes), calls a third-party API, transforms the response, and sends the transformed data to the workspace via the Logs Ingestion API.

When to use: The data source has a REST API but no webhook support. The transformation is too complex for a DCR alone (e.g., enriching events with external data, flattening nested JSON, deduplicating).

Example scenario: Your threat intelligence provider has a REST API that returns new IOCs. An Azure Function polls every 5 minutes, transforms the response into ThreatIntelligenceIndicator format, and ingests it. Now Sentinel matches these IOCs against all incoming data automatically.