TH2.4 series_fir() and Time Series Smoothing

4-5 hours · Module 2 · Free

Operational Objective

Raw time series data from M365 logs is noisy. A user's hourly sign-in count jumps between 0, 3, 1, 0, 5, 2, 0 — is there a pattern, or is it random variation? Smoothing filters remove noise to reveal underlying trends. The series_fir() function applies finite impulse response filters — including moving averages — that make patterns visible for hunting analysis.

Deliverable: The ability to apply moving average smoothing to M365 time series data for hunting, and to use the smoothed series alongside the raw series to identify genuine behavioral shifts versus random noise.

⏱ Estimated completion: 20 minutes

Noise hides patterns

When you run make-series on sign-in data with hourly bins, the raw values fluctuate. Some hours have 5 sign-ins, the next has 0, the next has 8. Visually and statistically, this noise obscures whether the overall level has increased, decreased, or stayed the same over days or weeks.

A moving average smooths the series by replacing each data point with the average of itself and its neighbors. A 24-hour moving average on hourly data replaces each hour’s value with the average of the surrounding 24 hours — revealing the daily trend while eliminating hourly fluctuations.

Moving average with series_fir()

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
// 7-day moving average on daily sign-in data
SigninLogs
| where TimeGenerated > ago(30d)
| where ResultType == 0
| make-series DailyCount = count()
    on TimeGenerated
    from ago(30d) to now()
    step 1d
    by UserPrincipalName
| extend SmoothedCount = series_fir(
    DailyCount,
    dynamic([1,1,1,1,1,1,1]),  // 7-element filter = 7-day average
    true,  // normalize (divide by sum of filter coefficients)
    false) // center the filter
// DailyCount = raw daily values (noisy)
// SmoothedCount = 7-day moving average (trend visible)
// Compare: if SmoothedCount is trending upward over the last week
//   while the baseline period was flat, the user's behavior has changed
| project UserPrincipalName, TimeGenerated, DailyCount, SmoothedCount

The filter array dynamic([1,1,1,1,1,1,1]) defines a 7-point equal-weight moving average. Each element contributes equally. For a weighted moving average that emphasizes recent data, use decreasing weights: dynamic([1,2,3,4,3,2,1]) — the central point and its immediate neighbors are weighted more heavily than distant points.

Hunting application: detecting sustained behavioral shifts

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
// Detect users whose smoothed download volume increased by 2x+
// over the last 7 days compared to the prior 23 days
CloudAppEvents
| where TimeGenerated > ago(30d)
| where ActionType == "FileDownloaded"
| make-series Downloads = count()
    on TimeGenerated
    from ago(30d) to now()
    step 1d
    by AccountId
| extend Smoothed = series_fir(
    Downloads, dynamic([1,1,1,1,1,1,1]), true, false)
// Extract baseline average (first 23 days) and recent average (last 7 days)
| extend BaselineAvg = avg_of(
    array_slice(Smoothed, 0, 22))
| extend RecentAvg = avg_of(
    array_slice(Smoothed, 23, 29))
| where BaselineAvg > 0
| extend ShiftRatio = RecentAvg / BaselineAvg
| where ShiftRatio > 2.0
// Users whose smoothed download volume doubled in the last week
// compared to their 3-week baseline — sustained increase, not a spike
| project AccountId, BaselineAvg, RecentAvg, ShiftRatio
| sort by ShiftRatio desc

The difference between this and TH2.3 (anomaly detection): series_decompose_anomalies() finds individual data points that spike. series_fir() smoothing reveals sustained shifts in the overall level. Both are useful. Anomaly detection catches acute events (a single day of massive downloads). Smoothing catches gradual changes (download volume slowly increasing over two weeks as an attacker methodically exfiltrates data).

Figure TH2.4 — Raw versus smoothed time series. Noise in the raw data obscures a clear upward trend that the 7-day moving average reveals.

Try it yourself

Exercise: Smooth your sign-in data

Run the 7-day moving average query against your organization-wide sign-in data (single series, not per-user). Add `| render timechart` to visualize.

Observe: is the smoothed line flat, trending up, or trending down? A flat smoothed line means stable activity. An upward trend during a period when no new users were onboarded could indicate increased automated activity (legitimate or malicious). A downward trend during business-as-usual could indicate a data ingestion problem.

⚠ Compliance Myth: "Smoothing removes important data — raw values are always more accurate"

The myth: Smoothing filters destroy information. Analysis should always use raw data to ensure nothing is missed.

The reality: Smoothing does not remove data — it separates the signal (trend, pattern) from the noise (random variation). The raw data is preserved in the original series. The smoothed series is a second view that highlights different patterns. Use raw data for acute anomaly detection (individual spikes). Use smoothed data for trend analysis (sustained behavioral shifts). Both are needed. Neither is sufficient alone.

Extend this approach

series_fir() supports custom filter designs beyond simple moving averages. An exponential weighting `dynamic([1,2,4,8,4,2,1])` emphasizes the center point more heavily. A Gaussian-like filter produces smoother output with less edge distortion. For most hunting applications, the equal-weight moving average is sufficient. Custom filters become useful when you need to distinguish between specific frequency components — for example, separating weekly business cycles from daily patterns in download volume data.

📋 Operational Artifact — series_fir() Quick Reference

Syntax: | extend Smoothed = series_fir(Series, Filter, Normalize, Center)
Common filters: 7-day MA: dynamic([1,1,1,1,1,1,1]) — weekly smoothing on daily data 24-hour MA: dynamic([1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]) — daily smoothing on hourly data Weighted: dynamic([1,2,3,4,3,2,1]) — center-weighted average
Normalize: true = divide by sum of filter (produces average). false = raw sum. Center: false = causal filter (only uses past values). true = symmetric.

References Used in This Subsection

Microsoft. “KQL series_fir().” Microsoft Learn. https://learn.microsoft.com/en-us/kusto/query/series-fir-function
Course cross-references: TH2.2 (make-series), TH8 (sustained exfiltration detection), TH12 (beacon trend analysis)

You're reading the free modules of this course

The full course continues with advanced topics, production detection rules, worked investigation scenarios, and deployable artifacts. Premium subscribers get access to all courses.

View Pricing See Full Syllabus

← TH2.3 series_decompose_anomalies() TH2.5 top-nested for Frequency Analysis →