Stay Ahead With Linux Security HOWTOs

security practiceslinuxmonitoring tools

Detection-as-Code for Linux: Building Security Rules That Last

One of the easiest mistakes to make in detection engineering is assuming a rule keeps working simply because nobody has touched it. Most of the time, nobody removes the rule. Nobody disables it. It just gets forgotten. . A few months later, someone upgrades a distribution, changes how SSH keys are managed, replaces a logging agent, or migrates an application to containers. The detection is still enabled, but it's watching an environment that no longer exists. The alert everyone expected never comes because the rule quietly stopped matching reality. I've seen teams spend hours debugging a SIEM only to discover nothing was technically broken. The detection simply hadn't kept up with the Linux systems it was supposed to monitor. Detection-as-Code grew out of that problem. Instead of treating detections as something you configure once inside a security platform, they're managed like any other engineering project. Rules live in Git, changes are reviewed, tests run automatically, and every modification has a history. The technology isn't the interesting part. The discipline is. What Is Detection-as-Code? Detection-as-Code moves the detection out of the SIEM and into a repository. The rule becomes another file that changes alongside the systems it depends on, rather than being edited directly in the platform and forgotten until an alert goes missing. That matters because detection rules depend on assumptions about the environment. A rule looking for changes to ~/.ssh/authorized_keys assumes those file events are being logged exactly the way they were six months ago. If someone replaces auditd with eBPF or moves those workloads into containers, those assumptions break. Keeping your rules in Git doesn't stop those infrastructure changes, but it forces them to be visible. When the logging pipeline changes, you update the detection logic in the same PR as the infrastructure change. You stop fixing "broken" detections reactively and start updating them as part of your standard operationalrhythm. Many teams now use Sigma as their primary language for this. You treat the Sigma rule as your "source of truth." You deploy it, and let your pipeline convert it into the specific dialect your SIEM (Splunk, Elastic, Sentinel) needs. If you ever switch SIEMs, your core logic stays safe in Git. How Does Detection-as-Code Work on Linux? Imagine you notice attackers commonly establish persistence by adding a new SSH key. You don't just hack together a query in your SIEM console. Author: You write a Sigma rule that alerts whenever ~/.ssh/authorized_keys is modified by a process other than your approved configuration management tool. You commit it to your /detections/sigma/ folder. Pull Request (PR): You open a PR. A teammate reviews it, not just for accuracy, but for potential noise. CI Validation: Your pipeline kicks off. It replays historical logs and exercises the detection using an Atomic Red Team test that emulates the targeted persistence behavior, confirming the rule doesn't trigger during normal daily backups. Merge & Deploy: Once the tests pass and a human reviewer signs off, the pipeline automatically pushes the rule to your production SIEM or runtime security tool ( Falco or Tetragon ). How Should You Organize Detection Rules? If you’re going to do this, keep it tidy. A disorganized repository is just as bad as a messy SIEM. I usually set it up like this: /security-detections ├── detections/ │ ├── sigma/ # The core logic │ └── falco/ # Runtime rules ├── tests/ │ └── mock_logs/ # Logs for validation ├── ci/ # Your CI/CD glue ├── docs/ # Explaining the "why" └── README.md Which Linux Data Sources Should You Monitor? You can’t detect what you can’t see. Relying solely on generic logs isn't going to cut it anymore. Auditd: The reliable workhorse for traditional Linux file integrity and system calls. Journald/Systemd: Use this for catching service-level persistence. If a service is created in a weird way, you’ll find it here. Sysmon for Linux : Great if you’re already used to Windows telemetry; it helps bridge the visibility gap between OSes. eBPF (Falco, Tetragon): This is the modern standard for containers and cloud-native workloads. When traditional logs disappear, eBPF sees everything happening at the kernel level. How Do You Test Detection Rules? Testing is the engine of DaC. Without it, you are just automating the deployment of broken rules. The goal is to build a "gatekeeper" in your CI/CD pipeline that validates every detection before it ever reaches production. To move beyond basic syntax checks, your pipeline should implement a multi-layered testing strategy: Replaying historical attack logs: Don't guess if a rule works—prove it. Feed recorded logs from past incidents through your rule to ensure it would have fired correctly. This is the best way to verify your detection logic against real-world adversary behavior. Validating Sigma rules: Use tools like sigma-cli to validate syntax and ensure the rules comply with your schema before they are committed. This catches typos and logic errors before they trigger a CI failure. Unit testing detections: Treat every rule like a piece of application code. Create small, "unit" test files containing only the specific log events required to trigger the rule. If the rule doesn't fire, the test fails. Regression testing: As your library grows, new rules can inadvertently interfere with old ones or create logic overlaps. Run a full suite of regression tests against your entire rule set to ensure that a change to one detection doesn't silence another. Performance testing: Heavy regex or complex sub-queries can grind a SIEM to a halt. Test your rules against a high-volume log stream in a staging environment to ensure they don't introduce unacceptable latency or consume excessive compute resources. False-positive benchmarking: Use a "known-good" set of logs—capturing routine updates, system reboots, and administrative activity—to ensure your rule remains silent on benign noise. If your rule fires during these tests, it’s not ready for production. The automation platform matters less than the workflow. Whether you use GitHub Actions, GitLab CI, Jenkins, or another system, the goal is the same: treat detection testing with the same rigor you apply to your application code. Testing Method Objective Log Replay Does it catch threats from our history? Unit Testing Do specific inputs trigger the expected output? Regression Testing Did I break any of our older rules? Performance Testing Does this query cripple our SIEM? False Positive Analysis Is it going to wake up the team for nothing? Common Detection-as-Code Mistakes to Avoid When starting with DaC, avoid these common traps: Treating Git as a backup: Nope. Git is the only place the rule lives. If it isn't in Git, it shouldn't be in your SIEM. Period. Deploying without automated testing: A rule that hasn't been tested against mock logs is a gamble, not a control. Neglecting tuning: Rules need maintenance. If a rule produces false positives, treat it like a bug and fix it in the repo. Retire obsolete rules: Detection repositories should shrink as often as they grow. If a rule is obsolete, kill it. Removing noise is as important as finding threats. Dependency blindness: Don't write a rule that depends on a log format the infra team is planning to delete next week. When Should You Use Detection-as-Code? Detection-as-Code is not a silver bullet. If you manage five rules in a lab, don't bother. The overhead of setting up a CI/CD pipeline will eat you alive. But if you’re managing hundreds of nodes, have multiple analyststouching the rules, and you're tired of alerts breaking every time someone runs an update—it’s time. The complexity is only justified as you scale. Conclusion Linux infrastructure never stays the same. Containers are rebuilt, kernels are patched, and attackers are always changing their playbook. Detection-as-Code doesn't make your rules "smarter." It makes them maintainable. And in a mature Linux environment, maintainability is the difference between a security program that actually works and a forgotten rule that stopped catching attackers six months ago. . Explore the importance of Detection-as-Code for maintaining effective security rules in a dynamic Linux environment.. Detection as Code, Linux Security, Monitoring Tools, Detection Rules, Security Practices. . Dave Wreski

Jul 07, 2026 • Dave Wreski How to Learn Tips and Tricks