Every Rule Is a Snapshot of the Environment That Created It
SPL Requires a Rule for Every Hypothesis You Have Not Written Yet
Detection Engineering Requires Dedicated Capacity Most Teams Don’t Allocate
When an Alert Closes, the Decision Logic Disappears With It
The Coverage Metric That Actually Answers the Question
Run This Check on Your Environment Today
Ten alerts a day reads like a stable environment. For most SOC teams, low alert volume is one of the better signals available: manageable workload, detection engineering doing its job, nothing requiring escalation. What it measures is how often the current detection logic, with its current exception lists, in its current state, produces a signal. Those are two different measurements.
A detection engineering library built two years ago and never systematically tested against a live attack scenario can generate ten alerts a day and miss the intrusion that matters. The dashboard looks identical in both cases. The number of active rules hasn’t changed, and the ATT&CK coverage percentage hasn’t changed either, as well as the alerts-per-day figure. What has changed, silently, is what those rules would actually catch.
Here is what this article covers and who it is written for:
- Who: CISOs, SecOps Directors and detection engineering leads running Splunk in production
- What: Five structural reasons SIEM correlation rules lose effectiveness without surfacing in SOC metrics
- Why it matters: Standard dashboards measure rule existence. Whether those rules fire on a real attack is a separate question.
Every Detection Engineering Rule Is a Snapshot of the Environment That Created It
An untested detection rule is indistinguishable from a working one. Both appear active in the dashboard, both run on every cycle and count the activity toward your coverage percentage. The difference only becomes visible when something tries to trigger them.
A correlation rule captures the behavior pattern the author understood on the day they wrote it, in the environment that existed then. Over the following months, both change.
A new authentication system gets deployed, a vendor tool starts generating events that match the rule’s logic but represent legitimate behavior. The rule becomes noisy: someone adds an exclusion like a service account, an IP range, a known process, and the alert volume drops. The rule continues running, and the dashboard still marks it active.
Eighteen months later, the exclusion list has grown. Consider what that looks like in SPL:
index=auth action=failure
| stats count AS failure_count by user, src_ip
| where failure_count > 5
AND NOT user IN (“svc_backup”,”svc_monitor”,”svc_deploy”,”svc_scanner”,”svc_ldap”)
AND NOT cidrmatch(“10.10.5.0/24“, src_ip)
AND NOT cidrmatch(“192.168.100.0/24“, src_ip)
AND NOT cidrmatch(“172.16.50.0/24“, src_ip)
This rule was written to detect brute-force authentication attempts. It still runs on every cycle. But it will not fire on a credential stuffing attempt originating from a subnet added to the exception list twelve months ago for a vendor access pattern. Nothing in your alert volume or rule count tells you that. Ten alerts a day is entirely consistent with this rule covering a fraction of what it was designed to cover.
.
The CardinalOps 2025 State of SIEM Detection Risk report analyzed over 13,000 production detection rules across Splunk, Microsoft Sentinel, QRadar, and other SIEMs. It found that 13% of rules in production environments are broken and will never fire an alert. A previous year’s dataset put that figure at 18%. These are active security programs with regular investment, carrying the same blind spot. Rules break when log source formats change, when field names shift after a vendor update, or when exception lists have grown to eliminate the coverage they were meant to provide. In every case, the rule still appears in the active count.
An AI system running continuously against the same environment evaluates each rule’s behavior against live traffic, surfacing rules that stopped firing, exceptions that eliminated coverage, and logic that no longer matches the environment it was written for. The role is critical: checking existing detection logic for blind spots the original author could not see from a single point in time.
SPL Requires a Rule for Every Hypothesis You Have Not Written Yet
Splunk’s query language is well-suited for finding patterns in high-volume log data. Its defining structural characteristic, and the source of a specific detection gap, is that it requires an explicit rule for every behavior pattern you want to detect. It finds what you specifically modeled.
This works for known attack patterns. It creates a measurable gap for behaviors that span multiple data sources across time windows longer than standard lookback periods.
A realistic scenario: a user account generates three MFA failures on Monday morning, each individually below your alert threshold. Six hours later, your EDR records unusual process behavior on a workstation that the same account accessed. Your vulnerability scanner has an open finding on that workstation from two weeks prior: a known exploit, unpatched. Each of these events, reviewed independently, sits below the threshold for an alert. Correlated across three sources and a six-hour window, they describe a credential-based intrusion attempt with an identified target.
Detecting this requires an explicit rule that joins authentication telemetry, endpoint data, and vulnerability findings across a multi-day window. That rule can be written. The practical question is whether it was, and how many similar cross-source, multi-day hypotheses exist in the backlog of detection logic that has never been built.
Most in-house detection libraries grow in response to incidents the team has already encountered. Rules get written for attack patterns the SOC has seen. Systematic expansion toward techniques not yet observed requires dedicated time for hypothesis generation that competes directly with day-to-day operations.
Splunk built Risk-Based Alerting specifically because this structural limit exists. RBA accumulates risk scores across sub-threshold events from multiple sources, so that combinations of weak signals become visible where no single signal would cross an alert threshold. A team operating without RBA is missing a detection mechanism the platform was designed to provide. Covering that gap manually would require writing an explicit rule for every multi-source behavioral chain worth detecting, including the ones not yet imagined.
Research from Anvilogic and ESG found that 86% of security professionals report it takes a week or more to build, test, and deploy a single new detection rule. At that pace, systematic coverage expansion against the full ATT&CK matrix is difficult to sustain alongside active operations.
The math creates a structural ceiling for teams relying on explicit rules alone. A detection engineer actively tracking thirty or forty behavioral hypotheses is working at full capacity. An AI system evaluates thousands simultaneously, across every data source, without a queue. The cross-source, multi-day patterns that take a week to formalize into a rule get evaluated continuously before anyone writes the rule. The difference is the surface area being watched.
Detection Engineering Requires Dedicated Capacity Most Teams Don’t Allocate
Done properly, detection engineering follows a continuous cycle: write a rule, simulate the attack scenario it is intended to catch, verify it fires correctly, verify it does not fire on known-good activity, version and document the logic, revisit it when the environment changes. For a mid-size enterprise environment, this is a dedicated function: one or two engineers whose primary responsibility is the quality and coverage of the detection library, with time reserved specifically for simulation and testing.
In most in-house SOC teams, detection engineering belongs to the same person running SIEM administration. The same person who tunes rules also maintains log sources, manages integrations, and responds to active incidents. The SANS 2024 Detection and Response Survey found that 73% of organizations report difficulty crafting quality, reliable detection rules. Detection engineering at the cadence the discipline requires does not fit alongside full SIEM administration in a single role.
In practice, it is always simulation and testing that give first.
If this reflects how your detection engineering function is currently staffed, we are available for a first conversation.
What changes when AI handles continuous hypothesis evaluation is the removal of the bandwidth ceiling. The detection engineer still owns the environment: the exception context, the architectural decisions, the institutional knowledge of why each rule exists. The AI runs hypothesis evaluation at a scale and cadence that the engineer’s available hours alone cannot reach. The rule library keeps running. The gaps between the rules get watched.
The outcome is predictable: the detection library grows reactively, adding rules for techniques the team has already encountered, while systematic coverage expansion happens slowly or not at all.
The second risk is more acute. Every rule in a detection library was added for a reason. Every exception was created in a context. The person who built those rules knows which ones were written for a specific incident, which thresholds were chosen based on observed behavior in that environment, and which exclusions were added for a vendor tool that no longer exists. That knowledge is in the person who wrote the rules, and nowhere else.
When that person is unavailable for a month (a medical situation, a resignation, a leave), whoever covers for them inherits a library they cannot fully read. The exception for the IP range looks like any other exception. The threshold on the authentication rule looks like it was set deliberately. The logic behind each decision is gone. The next person rebuilds it from scratch or, more commonly, works around it without rebuilding it at all.
This is the default outcome when detection engineering knowledge is never written down as part of the engineering process itself.
When an Alert Closes, the Decision Logic Disappears With It
Every time an analyst closes an alert, whether escalated, resolved as a true positive, or closed as a false positive, the investigation logic that produced that decision leaves with it. What data sources were checked? What hypothesis was tested? Why did this pattern qualify as a false positive? That reasoning exists in the analyst’s memory, and in most environments, nowhere else in structured form.
The consequence compounds over time. The same patterns get re-triaged repeatedly because the previous resolution is not accessible in a way that speeds the next investigation. Playbooks do not evolve from real decisions because the decisions themselves are not captured in a format that feeds back into the process. When a new analyst joins the team, the institutional knowledge they need to operate effectively does not exist in any document. It has to be rebuilt from experience.
Alert management systems capture the outcome of an investigation but not the reasoning behind it. The result is a SOC that closes alerts at volume without accumulating investigative intelligence from them. The library of resolved alerts grows, but the institutional knowledge those resolutions represent does not.
An AI agent investigating the same alert captures the hypothesis it tested, the sources it queried, and the reasoning behind the resolution in structured form. When the same pattern surfaces again, that record is accessible. The decision logic survives the closure because it was never stored only in memory. Over time, those records become the foundation for playbooks that evolve from real decisions rather than from assumptions about what an attack might look like.
The Coverage Metric That Actually Answers the Question
A MITRE ATT&CK coverage percentage tells you which techniques have at least one detection rule associated with them. What it does not tell you is whether those rules would fire on a real attack against your current environment, with your current exception lists, against your current log sources.
The CardinalOps research found that enterprise SIEMs cover an average of 21% of MITRE ATT&CK techniques with at least one detection rule. When narrowed to the ten techniques most commonly observed in actual attacks, organizations cover an average of four. The same research found that the telemetry needed to detect 90% of ATT&CK techniques is already flowing into most SIEMs. The data is already there. The detection logic that uses it is the missing piece.
A coverage percentage built on broken rules, or on rules whose exception lists exclude the accounts most relevant to the technique being modeled, does not measure what it appears to measure.
The measurement that answers the real question is attack simulation: generate live log events that match real attack scenarios in your specific environment, then check which correlation rules produce alerts and which do not. This requires testing rule behavior against current log sources, current field names, and current exception logic, on a regular cadence across the full rule lifecycle.
Most SOC teams run this irregularly. Purple team exercises require coordination and dedicated time. Attack simulation against a live environment competes for the same engineering bandwidth as everything else in the operational queue. The result is that coverage is assumed rather than tested, and the gap between assumed and actual tends to be wider than the dashboard suggests.
The Question Worth Asking
Standard SOC metrics measure the existence of detection infrastructure. They do not measure whether that infrastructure works on a live attack against today’s environment. An untested rule and a working rule look exactly the same until the moment they need to perform differently.
Every SOC team has a number they use to describe their coverage. The more useful question is when that number was last verified against a live attack in the current environment: current log sources, current exception lists, current field schemas. If the answer is not recent, or not sure, the gap already exists.
One category of tool addresses this structurally. AI-based detection systems operate as parallel analysts, holding thousands of behavioral hypotheses simultaneously and evaluating them continuously against live data. The detection engineer’s SPL rules keep running. The institutional knowledge they represent stays intact. The AI evaluates the hypotheses the rule backlog hasn’t reached yet.
The practical difference is in what the AI analyst does that a human cannot sustain at volume: it does not lose context after a long shift, does not close an alert as a false positive because a user “always does this,” does not miss the connection between an event on Monday and an anomaly on Wednesday, and holds ninety days of activity across every host and account simultaneously.
Two AI analysts and two people outperform four people without them, because each does what the other cannot. People carry business context, accountability, and judgment. AI carries coverage and memory at a scale human bandwidth cannot reach.
Run This Check on Your Environment Today
Before any external conversation, these four checks give you an accurate picture of where your coverage actually stands:
1. Pull your top 20 rules by volume and audit their exceptions. For each rule, count the exceptions added in the last 12 months. If any exception covers a service account or IP range with privileged access, flag it. That account now has reduced detection coverage and the dashboard will not show it.
2. Map your exception lists against your highest-risk accounts. Pull a list of accounts with administrative or elevated privileges. Cross-reference against your SIEM exception lists. Any overlap is a detection gap against the accounts most relevant to insider threat and credential abuse scenarios.
3. Pick three ATT&CK techniques your rules claim to cover. Find the accounts they exclude. Choose techniques relevant to your threat model (T1078 Valid Accounts, T1021 Remote Services, T1003 OS Credential Dumping are common starting points). For each, identify which accounts and subnets your rules exclude. Check whether those exclusions cover accounts that would realistically be used in an attack.
4. Find the last date your team ran an attack simulation against live rules. If it was more than six months ago, your coverage number reflects your rule library at that point in time. Environment changes since then are untested. If a simulation has never been run, the coverage percentage is an estimate based on rule existence rather than rule effectiveness.
If what these checks surface is larger than expected, a structured external review is a practical next step.
UnderDefense runs this kind of detection coverage assessment as a starting point, with findings delivered before any longer engagement is proposed.
1. Why do Splunk correlation rules degrade over time?
Correlation rules degrade because environments change faster than detection logic is reviewed. When a rule generates noise after a system change, engineers add exceptions to reduce false positives. Those exceptions accumulate over months and narrow the rule’s detection aperture without any dashboard metric surfacing the change. A rule written to catch brute-force authentication attempts may, after eighteen months of exception additions, no longer fire on the exact behavior it was designed to detect, while still appearing active and healthy in the SIEM console.
2. Why is MITRE ATT&CK coverage percentage not an accurate measure of detection effectiveness?
ATT&CK coverage percentage counts techniques that have at least one active detection rule associated with them. It does not measure whether those rules would fire on a real attack against the current environment. A rule that covers a technique on paper but contains exception logic excluding the most relevant accounts or subnets counts toward coverage without contributing to real detection. According to CardinalOps research, enterprise SIEMs cover an average of 21% of ATT&CK techniques, and of those, rules may be broken, degraded, or misconfigured without the coverage number reflecting it.
3. How does exception accumulation affect Splunk detection rule coverage?
Each exception added to a correlation rule narrows its detection aperture. Individually, exceptions are reasonable responses to noise. Cumulatively, they can reduce a rule’s effective coverage to a fraction of its original design. An environment with 150 active rules and 600 accumulated exceptions distributed across them has a substantially different detection surface than 150 clean rules, but both look identical in a dashboard. Exception lists are rarely audited systematically because doing so competes with active detection engineering work.
4. What percentage of Splunk correlation rules in production environments are broken?
According to the CardinalOps 2025 State of SIEM Detection Risk report, which analyzed over 13,000 production detection rules across Splunk, Microsoft Sentinel, QRadar, and other SIEMs, 13% of rules in active production environments are broken and will never fire an alert. The previous year’s dataset from the same research program put that figure at 18%. Rules break when log source formats change, when field names shift after vendor updates, or when exception logic grows to the point of eliminating the coverage the rule was built to provide.
5. What is the difference between claimed SIEM coverage and measured coverage?
Claimed coverage is what a SIEM dashboard reports: active rule count, ATT&CK percentage, daily alert volume. Measured coverage is what those rules would actually detect against a real attack in the current environment, accounting for broken rules, accumulated exceptions, and untested detection logic. The gap between the two grows predictably over time in any environment that changes faster than its detection logic is reviewed. The only way to measure actual coverage is through attack simulation: generating live log events and verifying which rules produce alerts and which do not.
6. Should a Splunk SOC use Risk-Based Alerting?
Yes. Splunk built Risk-Based Alerting specifically to address a structural limit in correlation-rule-based detection: behaviors that span multiple data sources across time windows produce no alert unless a specific rule was written for that exact pattern. RBA accumulates risk scores across sub-threshold events from multiple sources, making combinations of weak signals visible where no individual event would cross an alert threshold. A Splunk environment running without RBA is missing a detection mechanism the platform was designed to provide, particularly relevant for credential abuse chains, lateral movement, and multi-stage attack patterns.
