How to Respond After Detecting a Compromised Linux Server

Step 1: Verify the Server Is Actually Compromised

Start by confirming the server is actually compromised. Broken applications, failed deployments, and unstable services can create symptoms that look malicious at first.

Early triage is mostly about identifying unauthorized access, suspicious processes, and unexpected network activity before making changes to the system. 

who

w

last -a

Look closely at:

• Unknown usernames
• Source IP addresses you do not recognize
• Logins at unusual hours
• Simultaneous sessions from different locations
• Sessions that remain active longer than expected

An SSH login from another country at 3 AM does not automatically confirm compromise, but it should immediately raise questions. The same goes for service accounts logging in interactively when they normally do not.

Once login activity looks suspicious, move directly into process inspection. Attackers often leave behind running malware, reverse shells, persistence scripts, or crypto miners.

ps auxf

top

pstree -p

This stage is less about memorizing process names and more about spotting behavior that feels wrong for the system.

Things worth investigating include:

A compromised web server spawning /bin/bash underneath an Apache or Nginx process deserves immediate attention.

After checking processes, inspect network activity. A server talking to unfamiliar systems or listening on unexpected ports may already be under remote control.

ss -plant

lsof -i -P -n

Pay attention to:

• Unknown listening ports
• Outbound connections to unfamiliar IPs
• Established sessions with cloud VPS providers
• Reverse shell behavior
• Services binding to unusual interfaces

A reverse shell often appears as a process maintaining an outbound connection to a remote host over a strange high-numbered port. If the server normally only handles web traffic but suddenly maintains persistent outbound sessions to external infrastructure, that is not normal operational behavior.

Step 2: Do Not Reboot the Server Yet

In most cases, rebooting a compromised Linux server too early destroys valuable evidence. Memory-resident malware disappears. Active network connections vanish. Temporary files get deleted. Shell history stored only in memory may never be recoverable again.

A reboot can also terminate the exact malicious process you were trying to investigate. That sounds helpful until you realize it removes the ability to understand what the attacker was doing, how they got in, and whether they still have access somewhere else.

The following artifacts may disappear after a reboot:

• RAM-based malware
• Active reverse shells
• Running attacker processes
• Unsaved shell history
• Temporary files under /tmp
• Active network sessions
• Evidence tied to process memory 

Avoid rebooting the server unless the situation is actively getting worse. Preserve logs first. Save process information. Capture active network connections before changing services or killing processes.

In ransomware or destructive attacks, emergency shutdowns may still become necessary. Outside of those situations, immediate reboots usually create more problems for the investigation.

Step 3: Isolate the Server Carefully

Most compromised servers should be isolated quickly, but investigators still need access to the system. The goal is to limit attacker movement without cutting off visibility completely.

Good containment limits external access while keeping the server reachable for evidence collection and review.

Good Containment Methods

A cloud workload might be isolated by changing security groups. An on-premises server might move into a quarantine VLAN. The exact method changes between environments, but the objective stays the same.

Bad Containment Methods

Some responses create more problems than the attackers themselves.

• Pulling the power cable
• Wiping the virtual machine immediately
• Deleting suspicious files before analysis
• Restarting services blindly
• Running cleanup scripts without evidence collection

Deleting malware immediately feels productive, but it often destroys indicators investigators need later. The same malicious file may reveal persistence methods, attacker tooling, or lateral movement paths across the environment.

Containment should slow the attacker down without destroying the evidence trail.

Step 4: Capture Volatile Evidence Immediately

Volatile evidence disappears fast. Running processes terminate. Network connections close. Logs rotate. This is the stage where responders start preserving everything they may need later for investigation, reporting, or legal review.

Begin with process information.

Save Process Information

These commands create snapshots of currently running processes.

ps auxf > /root/processes.txt

pstree -p > /root/pstree.txt

The redirected output preserves the process tree exactly as it existed during the investigation. That matters because attackers often terminate malware once they realize they have been detected.

Move next into network evidence.

Save Network Connections

ss -plant > /root/network_connections.txt

lsof -i -P -n > /root/open_sockets.txt

This helps identify:

• Remote command-and-control infrastructure
• Reverse shells
• Internal lateral movement
• Suspicious listening services
• Unexpected outbound sessions

A single established connection to an unfamiliar VPS provider can become the clue that ties the compromise together later.

Then, preserve user activity.

Save Logged-In User Activity

who > /root/logged_in_users.txt

last -a > /root/login_history.txt

Historical login records often reveal how long the compromise existed before detection. Attackers regularly use valid credentials after initial access, especially when moving laterally.

Logs should also be preserved before rotation occurs.

Preserve Logs Before Rotation

cp -r /var/log /root/log-backup

journalctl > /root/journalctl.txt

Linux systems rotate logs automatically. A busy server may overwrite critical authentication or application records quickly, especially during active attacks.

Useful logs commonly include:

At this stage, documentation matters almost as much as evidence collection. Record timestamps. Save commands used during the investigation. Keep notes on every change made to the server.

Step 5: Check for Persistence Mechanisms

Attackers rarely compromise a server and leave voluntarily. Persistence mechanisms help them regain access after reboots, password resets, or partial cleanup efforts. This is why compromised systems often appear clean at first and then become compromised again days later.

Cron jobs are one of the most common persistence methods on Linux systems.

Inspect Cron Jobs

crontab -l

ls -la /etc/cron*

Look for:

• Scripts executing from /tmp
• Obfuscated shell commands
• Base64-encoded payloads
• Unknown scheduled jobs
• Tasks running as root unexpectedly

Attackers like cron because it survives reboots and blends into normal system administration.

SSH keys are another common persistence method, especially after credential theft.

Review SSH Keys

cat ~/.ssh/authorized_keys

Unauthorized SSH keys allow attackers to reconnect without passwords. Compare keys against known administrators and remove anything unexplained only after preserving copies for investigation.

Persistence also appears inside systemd services.

Inspect Systemd Services

systemctl list-units --type=service

systemctl list-unit-files

Suspicious services often:

• Use vague names
• Restart automatically
• Execute scripts from unusual directories
• Launch hidden shells or networking tools

A fake service named system-update.service launching a binary from /tmp/.cache should not be ignored.

Startup scripts deserve inspection too.

Check Startup Scripts

Common locations include:

/etc/profile

/etc/rc.local

/etc/init.d/

/etc/bash.bashrc

Attackers sometimes inject commands directly into startup scripts so malware launches every time a user logs in or the server boots.

Kernel-level persistence is harder to detect and far more dangerous. If root access was obtained, trust in the operating system itself becomes questionable.

Step 6: Determine How the Attackers Got In

Understanding the initial entry point matters because cleanup without fixing the original weakness usually leads to reinfection. Attackers rarely use complicated techniques when simple exposure works.

The most common Linux compromise paths include:

• Exposed SSH services
• Weak or reused passwords
• Vulnerable web applications
• Public-facing admin panels
• Credential theft through phishing
• Unpatched software packages
• Default credentials left unchanged

Authentication logs provide a good starting point.

lastb

grep "Failed password" /var/log/auth.log

Large numbers of failed SSH logins may indicate brute-force activity. A successful login immediately after repeated failures deserves investigation.

Web servers should also be reviewed carefully. Exploited applications often leave traces in access logs.

Things to look for include:

Outdated packages create another common entry point. Attackers routinely target publicly known vulnerabilities within days of disclosure. The longer internet-facing systems remain unpatched, the higher the risk becomes.

The investigation may eventually show that the compromise began outside the server itself. Stol

Final Thoughts

A compromised Linux server can turn into a larger incident quickly if the response is rushed. Reboots, cleanup attempts, and configuration changes made too early often remove the evidence investigators still need.

Early response work is mostly about preserving visibility long enough to understand the scope of the compromise. Save process data. Preserve logs. Identify active connections. Check whether the attacker established persistence or moved into other systems.

A server may also appear stable after basic remediation while the original access path still exists. Unpatched applications, exposed admin services, stolen SSH keys, and reused credentials commonly lead to repeated compromise.

In higher-severity incidents, rebuilding from a trusted image is often safer than attempting partial cleanup on a compromised host. Especially after root-level access or suspected persistence inside the operating system.

FAQ

Should I disconnect a compromised Linux server from the network immediately?

Usually, yes, but isolation should be controlled. The goal is to limit attacker movement while preserving evidence and maintaining investigator access.

Instead of powering the server off immediately, restrict inbound and outbound traffic carefully. Remove the system from production traffic if possible while preserving secure access for forensic review.

Should I reboot a hacked Linux server?

Usually no. Rebooting destroys volatile evidence, including memory artifacts, active attacker sessions, temporary malware files, and live network connections.

Exceptions exist during destructive ransomware events or situations where active compromise threatens other systems immediately.

What logs should I check after a Linux server compromise?

Start with authentication and system logs.

Common sources include:

• /var/log/auth.log
• /var/log/secure
• journalctl
• Apache or Nginx access logs
• Cron logs
• Audit logs
• Application logs

These logs help identify login activity, exploitation attempts, persistence, and lateral movement.

How do attackers maintain persistence on Linux servers?

Attackers commonly maintain persistence through:

• SSH authorized keys
• Cron jobs
• Systemd services
• Startup scripts
• Web shells
• Kernel modules
• Hidden user accounts

Persistence mechanisms are designed to survive reboots and partial cleanup.

Can a compromised Linux server ever be trusted again?

Sometimes, but root-level compromise changes the situation significantly.

If attackers gained administrative access, complete trust in the operating system may be impossible to restore confidently. Many organizations rebuild compromised systems entirely instead of attempting a deep cleanup.

What is the first thing to collect during incident response?

Volatile evidence should be collected first.

That usually includes:

• Running processes
• Active network connections
• Logged-in users
• Memory artifacts
• Open sockets
• Authentication logs

This information disappears quickly once systems reboot or attackers detect investigation activity.

Related Reading

• What Is SELinux? A Practical Take for Linux Admins
• Understanding Log Management and Analysis Tools for Linux Systems
• Top Linux Vulnerability Scanners in 2026: A Guide to Open-Source Security Tools
• Why CI/CD Pipelines Became Targets in Software Supply Chain Attacks
• RubyGems Attack Highlights Open Source Supply Chain Risks for Linux Teams