Catch late security events without replaying every alert

Give each clock a job

Late data needs a wider event-time search, but widening that search alone can repeatedly select the same activity. For an existing Microsoft Sentinel scheduled analytics rule, separate when an event occurred, when it became queryable and when the rule ran. Then decide which clock controls historical eligibility and which controls the current evaluation slice.

Microsoft's ingestion-delay guidance presents this distinction directly: extend the event-time lookback to include delayed records and constrain ingestion time to the rule's recent interval. That pattern addresses a specific timing problem. It is not an exactly-once guarantee for every rule, join, retry or outage. ^[1]

Scheduled analytics rules also have a built-in execution delay. Current documentation describes a five-minute offset and scheduling and lookback ranges from five minutes to fourteen days. An execution offset is not a substitute for understanding a particular source's arrival behavior. A source can still arrive after the event window used by the query. ^[2]

Picture a hypothetical sign-in-related event that becomes searchable after its event-time window has passed. The source event is not necessarily wrong, and the rule is not necessarily disabled. The query may simply ask for a historical interval that no longer contains it. A second rule that repeatedly searches a much broader interval can create the opposite problem: repeated selection without a clear reconciliation policy.

Before editing a query, write down the rule family, source tables, event timestamp field, recurrence and current lookback. Include the intended decision, such as whether the event should create a new alert or enrich an existing investigation. Timing repairs become easier to evaluate when the detection's purpose is explicit rather than hidden behind a dashboard count.

Source. Original timing model informed by Microsoft documentation. ^[1][2][3]

Method. Conceptual sequence without invented delays or scale.

Accessible table and figure data

Figure 1 accessible table

Clock	Meaning	Review question
Event time	Reported source occurrence	Is the activity historically eligible?
Ingestion time	Approximate queryable arrival	Which recent arrival slice applies?
Rule execution	Evaluation opportunity	Did the run succeed?

Figure 1 accessible table

Clock	Meaning	Review question
Event time	Reported source occurrence	Is the activity historically eligible?
Ingestion time	Approximate queryable arrival	Which recent arrival slice applies?
Rule execution	Evaluation opportunity	Did the run succeed?

Download imageDownload table CSV

Measure delay at the table boundary

Do not borrow an ingestion allowance from another team's rule merely because both sources arrive in the same workspace. Define a measurement plan for the table and connector involved. Identify the collection period, event timestamp semantics, arrival timestamp availability and exceptional records. This article has no tenant measurements and supplies no fabricated delay percentile.

Kusto's ingestion_time function returns an approximate ingestion timestamp. It can return null when the relevant ingestion-time policy was not enabled, and it does not establish a total order across overlapping ingestion operations. Treat it as a useful query boundary with documented limitations, not a precise source clock or universal sequence number. ^[3]

A review should inspect the long tail as well as routine arrivals. Ask whether delayed records share a source, connector condition or incident period. Preserve exceptional cases rather than deleting them to make an average look healthy. The relevant decision is which delayed events the rule should still evaluate and how events outside that policy will be handled.

Timestamp meaning matters before arithmetic. A field might represent when the originating system created an event, when another system forwarded it or when a record was normalized. Confirm the table's semantics and preserve raw fields. A negative or surprising difference deserves investigation; it should not automatically be reinterpreted as network latency.

The proposed measurement record should include the exact query, table, timestamp fields, timezone treatment and collection dates, with restricted references to any sensitive results. Record whether missing ingestion timestamps were excluded and how many records that exclusion affected if a real measurement is performed. A reported distribution without its population and exclusions is difficult to use responsibly.

Use the result to propose an allowance, then test its detection consequences. A wider window may increase query work and include more records eligible for correlation. Those are workload-specific tradeoffs to observe in the reader's environment. No performance benefit or cost reduction is inferred here from a longer or shorter interval.

Change lookback without replaying the window

The documented repair has two dimensions. Event time defines how far back relevant activity may have occurred; a recent ingestion slice identifies newly arrived material for this run. A record can be old by event time and new by ingestion time. Keeping those meanings separate is the core of the pattern, not the particular variable names used in a sample. ^[1]

The simplified query below illustrates the shape for an existing scheduled rule that uses SecurityEvent and TimeGenerated. Its intervals are hypothetical configuration values, not measured source allowances or recommended defaults. The reader must replace them using the actual rule schedule and source evidence, confirm ingestion-time availability and add the real detection logic.

Apply the final selection to the records that should trigger the detection, not indiscriminately to every input used for context. Historical records may be needed to explain a new arrival. If an early filter removes that context, the query can look efficient while changing the security decision. Review the logical role of each input before deciding where its time predicates belong.

Avoid claiming that an overlapping event-time window is itself a duplicate bug. Overlap can be necessary to admit late arrivals. The question is whether the final output repeatedly represents the same decision without an intended update or reconciliation behavior. Incident grouping is a downstream behavior and should not be used to conceal an unclear event-selection contract.

Also state the boundary of the pattern. If no run evaluates the recent ingestion slice, those records can age out before the next successful run. A historical replay needs a deliberately selected interval and its own duplicate-handling policy. Extending the ordinary lookback does not automatically recover every missed execution.

Keep the old and proposed queries with a short explanation of the changed semantics. A reviewer should be able to say which previously missed case is now admitted, which already evaluated records remain excluded and which context is still available. That is more meaningful than approving a change solely because a query editor accepts it.

// Simplified KQL shape, not a validated production rule.
let recurrence = 5m;
let arrival_allowance = 15m;
SecurityEvent
| where TimeGenerated >= ago(recurrence + arrival_allowance)
| extend ReceivedAt = ingestion_time()
| where isnotnull(ReceivedAt) and ReceivedAt > ago(recurrence)
// Apply the reviewed detection and output logic here.

Source. Original before-and-after model based on Microsoft's documented pattern and rule health guidance. ^[1][3][4]

Method. Conceptual comparison, not a measured reduction in missed or duplicate alerts.

Accessible table and figure data

Figure 2 accessible table

Boundary	Before	After
Late event	Outside narrow event window	Eligible under reviewed allowance
Previously seen data	Repeated by broad lookback alone	Recent ingestion slice limits ordinary output
Missed execution	Implicit gap	Explicit backfill and reconciliation

Figure 2 accessible table

Boundary	Before	After
Late event	Outside narrow event window	Eligible under reviewed allowance
Previously seen data	Repeated by broad lookback alone	Recent ingestion slice limits ordinary output
Missed execution	Implicit gap	Explicit backfill and reconciliation

Download imageDownload table CSV

Treat joins as a second arrival problem

A join creates another timing boundary. The event that triggers interest and the event that supplies context may become queryable in different runs. A rule can correctly admit the first record yet fail to produce a match because its partner has not arrived. That is a different problem from a single late record falling outside its own window.

Test both arrival orders in an explicitly hypothetical fixture. In one case the contextual record exists first; in the other the triggering record arrives first. Decide whether the rule should wait, produce a partial result or revisit the decision after the partner arrives. Without that policy, a timing filter can silently choose behavior the detection author never intended.

The same reasoning applies to repeated context. A reference record may legitimately support several new triggering events. Deduplicating it globally could remove necessary context, while treating every joined row as a separate incident could inflate the output. Define the unit of detection before choosing a deduplication key or grouping behavior.

Do not assume that placing the ingestion predicate on both sides solves the problem. It can require the two sources to arrive within the same slice, which may be narrower than the intended correlation relationship. Conversely, leaving both sides unbounded can create excessive or irrelevant matches. The correct structure depends on the detection question and the tables' supported semantics.

A practical review artifact is a small case table with triggering event, partner availability, evaluation run and expected decision. Use non-sensitive synthetic identities and clearly mark every row as a planned expectation. The purpose is logical validation, not a chart of invented detection accuracy. Preserve cases that remain unresolved for a separate reconciliation workflow.

Separate late data from a missed run

Rule execution health needs its own evidence. Microsoft documents scheduled-rule retries after failures, including an initial attempt and five retries. The page contains inconsistent shorthand elsewhere, so this guide uses that explicit attempt description rather than saying there are six retries. Inspect actual run health instead of inferring success from the rule remaining enabled. ^[4]

A missed run and delayed source data require different repairs. A delayed event may be selected by a subsequent healthy run under the intended window policy. A skipped ingestion slice may require backfill even when every source arrived normally. The incident record should say which condition was observed, and whether both occurred.

For backfill, define the historical interval, rule version and output destination before execution. Decide how a result will be associated with an existing alert or investigation. A replay of old records can be useful, but it should not masquerade as newly observed activity merely because the replay ran today.

Keep the original event time and the backfill execution context together. Reviewers need to understand both the historical activity and the later detection. An operational summary that uses only alert creation time can make recovered evidence appear to describe a current event, changing urgency and potentially confusing response decisions.

Assign ownership of gaps. A connector owner may address source delivery while a detection owner addresses execution and reconciliation. If each team assumes the other owns the missed period, the rule can return to a healthy state without anyone evaluating the missing evidence. A successful new run does not close that historical obligation.

Choose the current rule family deliberately

This guide's lookback discussion is scoped to existing Sentinel scheduled analytics. Near-real-time analytics have a different execution model and documented constraints. They should not be described as the same query with a smaller recurrence value, and a timing fix should not quietly change rule family without a separate behavior review. ^[5]

Current Microsoft guidance also distinguishes analytics rules from custom detections. The comparison documentation helps identify differences in execution, data and management rather than assuming that similarly named controls are interchangeable. Retain the rule family in the test receipt so a later editor does not transfer timing assumptions between systems. ^[7]

For new rules, Microsoft's current custom-detections overview recommends that workflow, while existing analytics remain relevant. Custom detection frequency and lookback depend on the data source. Review the current source-specific table and required outputs instead of carrying the scheduled-analytics ranges into a new implementation. ^[6][8]

A migration decision should begin with behavioral parity, not portal location. List the event population, timestamp semantics, correlation context, output identity, grouping and failure handling required by the existing detection. Then establish what the destination rule family supports and how any differences will be accepted or mitigated.

Do not let a product transition deadline become a substitute for this analysis. Such dates change, and this article intentionally avoids relying on an old portal retirement date. The durable requirement is to validate the detection under the execution model it actually uses, with current documentation and a clearly recorded review date.

Build a timing validation record

Create a bounded acceptance set before deployment. Include timely arrival, delayed arrival within the intended allowance, arrival beyond that allowance, duplicate delivery, missing ingestion timestamp, reversed join arrival and a missed execution. These are proposed cases, not tests performed for this article and not empirical measures of detection quality.

For each case, state the expected decision and why. A duplicate might update existing context, be suppressed or remain visible as a distinct record depending on the actual event schema. Do not label all repeated actors and timestamps as duplicates. Two legitimate operations can share those values, so the chosen identity must preserve meaningful distinctions.

Preserve both positive and negative expectations. The rule should admit the intended delayed event and exclude unrelated old material that only happens to share a broad time window. A test that checks only whether any alert appears can miss substantial changes in the detected population.

Keep logical validation separate from production performance validation. A small fixture can reveal a timing mistake without establishing runtime, cost or scale behavior. Conversely, a fast query can still make the wrong security decision. Record each check according to what it actually demonstrates, with no claim of a live benchmark here.

After an authorized deployment, compare observed rule health and selected output with the acceptance record. If the source schema or timestamp policy changes, revisit the assumptions rather than preserving the old query as a timeless solution. The fixture should be maintained alongside the detection logic, not left as an unexplained attachment.

Include exact-boundary cases in that fixture. An event on the edge of a lookback or ingestion interval should have an intentional expected result, not an accidental one produced by changing greater-than to greater-than-or-equal. Record the operators used in each predicate and check how adjacent runs divide responsibility. Also test a replay using the same historical interval after an ordinary run has already evaluated part of it. The expected result should describe how existing detections are recognized and how new evidence is attached. This is particularly important when the alert's identity is derived from grouped records rather than a single source event.

Set an owned late data policy

An effective late-data policy states how old activity can be and still receive automated evaluation, what happens beyond that boundary and who owns historical review. It also distinguishes a new alert from an update to an existing investigation. This is an editorial recommendation for operational clarity, not a universal Sentinel setting.

The policy should preserve exceptional evidence even when ordinary automation does not process it. A very late record may be important to an ongoing incident despite falling outside the normal schedule. Route that case deliberately rather than claiming that a bounded query has eliminated every possible gap.

Close a timing repair with the actual behavior demonstrated: the rule family, source semantics, accepted arrival cases, skipped-run handling and remaining limitations. Do not promise exactly-once detection from an approximate timestamp and a recurring query. A precise, owned boundary is more reliable than a broad assurance that the window is now large enough.

Give replay output a distinct operational route when necessary. A retrospective detection may need to enrich an existing case rather than page an on-call responder as though the activity just occurred. Define that routing with the response team before running backfill. Preserve the original event time and the replay context so an analyst can still judge the historical activity's significance without confusing collection repair with a new security event.