SA1.9 Monitoring Automation Health

Prerequisite: enable diagnostic logging

Logic Apps do not send diagnostic data to Log Analytics by default. You must enable it for each Logic App.

Navigate to the Logic App → Monitoring → Diagnostic settings → Add diagnostic setting. Name: “SendToLogAnalytics.” Check “WorkflowRuntime” (captures run-level success/failure) and “allMetrics.” Destination: Send to Log Analytics workspace → select your Sentinel workspace. Save.

After enabling, diagnostic data appears in the AzureDiagnostics table within 5-10 minutes. The key fields are: resource_workflowName_s (playbook name), status_s (Succeeded/Failed/Cancelled), startTime_t and endTime_t (timing), error_code_s and error_message_s (failure details).

Enable this on every Logic App as part of the deployment checklist. If you have 10 playbooks and forget to enable diagnostics on 2 of them, those 2 are invisible to monitoring — exactly the blind spot that causes silent failures.

Layer 1: Failure detection (alert within 1 hour)

Create a Sentinel analytics rule that fires when any playbook fails:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.LOGIC"
| where Category == "WorkflowRuntime"
| where OperationName == "Microsoft.Logic/workflows/workflowRunCompleted"
| where status_s == "Failed"
| where TimeGenerated > ago(1h)
| project
    TimeGenerated,
    PlaybookName = resource_workflowName_s,
    ErrorCode = error_code_s,
    ErrorMessage = substring(error_message_s, 0, 500),
    Duration = datetime_diff('second', endTime_t, startTime_t)
| summarize
    FailureCount = count(),
    LastFailure = max(TimeGenerated),
    Errors = make_set(ErrorMessage, 3)
    by PlaybookName

Configure the analytics rule: frequency = every 1 hour, lookback = 1 hour, trigger when results > 0, severity = Medium.

Critical: do NOT trigger a playbook from this alert. If your monitoring detects a playbook failure and triggers another playbook to notify the team, and THAT playbook also fails, you have an infinite loop of failing playbooks detecting failing playbooks. Instead, use a simple automation rule that sends an email notification directly (the Sentinel automation rule can send email without a Logic App) or changes the incident severity and assigns it to the automation owner.

Layer 2: Health metrics (daily dashboard)

Build a Sentinel workbook or saved queries for daily review. The key metrics:

Success rate per playbook:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.LOGIC"
| where OperationName == "Microsoft.Logic/workflows/workflowRunCompleted"
| where TimeGenerated > ago(7d)
| summarize
    Total = count(),
    Succeeded = countif(status_s == "Succeeded"),
    Failed = countif(status_s == "Failed")
    by PlaybookName = resource_workflowName_s
| extend SuccessRate = round(todouble(Succeeded) / todouble(Total) * 100, 1)
| project PlaybookName, Total, Succeeded, Failed, SuccessRate
| order by SuccessRate asc

Target: 99%+ success rate for all playbooks. Below 95% indicates a persistent issue requiring investigation.

Mean execution time per playbook (trend):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.LOGIC"
| where OperationName == "Microsoft.Logic/workflows/workflowRunCompleted"
| where status_s == "Succeeded"
| where TimeGenerated > ago(30d)
| extend DurationSeconds = datetime_diff('second', endTime_t, startTime_t)
| summarize AvgDuration = avg(DurationSeconds) by
    PlaybookName = resource_workflowName_s,
    Day = bin(TimeGenerated, 1d)
| render timechart

This chart shows whether playbooks are getting slower over time. A gradual increase in execution time may indicate: growing data volumes (KQL queries take longer), API degradation (external TI service is slower), or permission token issues (authentication retries increasing latency).

Daily execution volume:

1
2
3
4
5
6
7
8

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.LOGIC"
| where OperationName == "Microsoft.Logic/workflows/workflowRunCompleted"
| where TimeGenerated > ago(30d)
| summarize DailyRuns = count() by
    PlaybookName = resource_workflowName_s,
    Day = bin(TimeGenerated, 1d)
| render timechart

A sudden drop in execution volume means either incident volume decreased (check Sentinel) or the automation rule trigger is broken (the rule stopped firing the playbook). A sudden spike means alert volume increased or a misconfigured rule is triggering the playbook on incidents it should not match.

Layer 3: Operational metrics (monthly review)

Monthly, compile the metrics that prove automation value and identify improvement opportunities:

Incidents auto-enriched: count of incidents with the “enriched” tag. Compare to total incident count for enrichment coverage percentage.

Containment actions executed: count of incidents with the “auto-contained” tag. Break down by containment type (session revoke, endpoint isolate, account disable). Track false positive rate (incidents where auto-containment was followed by a rollback).

Estimated analyst hours saved: for each playbook, multiply execution count by estimated time saving per execution. Enrichment: 5 minutes saved × execution count. FP auto-close: 3 minutes saved × execution count. Containment: 15 minutes saved × execution count.

Automation health summary: average success rate, number of failure incidents, mean downtime (time between failure and fix), playbooks requiring tuning or retirement.

These metrics form the monthly automation review document in SA11 and the business case for continued automation investment in SA12.