Discover LinuxSecurity Features
Real World Linux Security: Bob Toxen's Perspective
In this interview, Bob introduces his new book, discusses the "seven deadly sins" of Linux security, and outlines the benefits of the open source software model. He also points out the pitfalls that many system administrators fall into and how to avoid them.
Recently I got the opportunity to speak to Bob Toxen, the author of "Real World Linux Security: Intrusion Prevention, Detection and Recovery." He is a long time contributor to many Linux/Unix projects and currently works as an independent consultant. When Bob isn't securing Linux systems, writing articles/books, he is riding around Atlanta in his Rolls-Royce Silver Shadow proudly displaying his front "Linux" license plate. You will find that Bob's approach to security is one that is simple but effective. I encourage you to take the time and read what he has to say. You'll come away with a new perspective on security.
LinuxSecurity.com- Please give our readers a brief introduction to your new book, "Real World Linux Security: Intrusion Prevention, Detection and Recovery." Who does the book appeal to? Where can it be purchased?
Bob Toxen- Rather than simply providing a guide to crackers or serving as a theoretical treatise, it is a very "hands on" step-by-step guide to securing almost any Linux or Unix system to the extent desired. Along the way I explain why the various things must be done to secure a system, in a way that does not require a PhD in computer science.
It helps any system administrator understand how and why Linux and Unix can be made secure and enables administrators to gain a deeper understanding of security and why Linux and Unix do things the way that they do. I mention a number of real incidents to give people a "feel" for what they might encounter in a real break-in and to avoid the "this cannot happen to me" mentality. I have used humor throughout the book to make it fun.
One of the unique features is Chapter 2, "Quick Fixes For Common Problem". It allows a system administrator to take care of the most severe problems without a large investment in time. I realize that most system administrators are overworked and that "if it takes too long" they will not bother. There are jokes or puns behind almost every example. Many of the examples are complete and ready for use.
I also explain the most common attacks and some of the most subtle. This is done to smash common misconceptions that prevent people for solving security problems. One of the most common misconceptions is that throwing a firewall up between the a company network and the Internet magically will solve all of the security problems. I did this by explaining how easily a cracker can tunnel through a firewall or do an end run around it.
In later chapters, various subsystems, problems, and techniques are examined in depth. The use of security tools are explained in detail. Methods of detecting attacks, locking the would-be cracker out automatically, and paging and emailing the SysAdmin are shown. Ways to recover quickly from successful attacks are discussed in depth.
Many of the ideas, such as the technique that can be used by an E-Commerce site for protecting customers' credit card data and the "Adaptive TCP Wrappers" and "Cracker Trap" are my own ideas. The reader will not see them elsewhere. Rather than their being "crackpot" ideas, though, the best security experts could not break my credit card solution. While my Adaptive TCP Wrappers and Cracker Trap solutions may not be appropriate for large sites, they are very effective for home systems and small to medium shops and have proven themselves to be very effective in a year of continuous use at several sites. The book offers complete IP Chain examples, including the use of a DMZ and how to separate services on different servers for maximum security.
There is extensive cross-referencing, including page numbers, that also allow the book to be used as a reference to look up related problems. There is an extensive index and appendices listing online and other resources. The accompanying CD-ROM contains the security programs and scripts I created to easily detect open ports, implement the "Adaptive TCP Wrappers", "Cracker Trap", automatic paging and light flashing alert, defaced web page detector, implement firewall rules, etc., as well as many popular security-related tools.
The book can be purchased online now at www.softpro.com, Borders.com, Amazon.com, bn.com, and fatbrain.com. It is stocked in SoftPro, Borders, and MicroCenter stores and at some Barnes & Noble stores. Some stores have been having trouble keeping it in stock; readers may want to phone ahead and have a copy reserved. Almost any bookstore can order it for pickup in a few days. The ISBN is 0-13-028187-5. The publisher is Prentice Hall.
Bob mentions TCP Wrappers and IPChains. If you would like to further research these topics, here are a few resources that may be helpful. Both TCP Wrappers and IPChains are instrumental in the development of a secure network.
LinuxSecurity.com- What lead you to writing a book? How did you find a publisher and approximately how long did it take to write.
Bob Toxen- The number one reason for writing it was to help Linux win the war to provide quality software to the world. Even my non-computer friends are trying Linux and finding it productive. I do not know of anyone who has replaced a Linux system with another operating system unless forced to by someone who does not have to maintain it.
I saw the greatest weakness with Linux being that almost nobody knows how to secure it. I think that it is capable of being orders of magnitude more secure than Windows, including 2000 and NT, and my goal with the book was to provide this knowledge to everyone. I think that this will help turn the tide. The royalties I stand to make on book sales are far less than the value of the time that I took away from my Linux and Unix consulting business to write it. I consider this to be my major contribution to Linux.
A publisher found me and asked me to write a book on one of several Linux topics. I picked security. I thought that my 25 years of Unix with a continuing interest in security, 5 years of Linux, and 10 years of network programming gave me a unique ability to write on security.
Unfortunately, that publisher subsequently stopped returning phone calls, possibly due to a competing project. Having already done the basic book design and having started writing it, I went to the 1999 Atlanta Linux Showcase and met Miles Williams of Prentice Hall. While a quick but poor quality product can be more profitable, Miles allowed me to "do it right".
I already had worked part time for almost a year. After that, I worked almost all of my waking moments, seven days a week for six months writing and editing the book. My six excellent technical reviewers cut me no slack. Vulnerabilities that I did not originally talk about were discussed. Unnecessary material was removed. Unclear passages were rewritten, sometimes several times. Miles had me alter the organization a bit for better clarity.
I am very grateful to have had two of the world's leading Linux and Unix security experts as technical reviewers to help me make it complete and accurate. These were Kurt Seifried of SecurityPortal.com and Mike Warfield, Wizard of Internet Security Systems.
Two other reviewers were experienced Linux system administrators and helped me make rough parts more understandable. One newbie friend asked to read drafts. While targeted for the experienced SysAdmin, she understood almost all of it. We worked to add clarity to the remaining bits. The sixth reviewer has a strong security bent and had lots of suggestions.
Eric Raymond wrote the Foreword. Steve Bourne, creator of the Bourne Shell, provided a cover quote.
LinuxSecurity.com- Please explain the "Seven Deadly Sins of Linux Security" and how you cover this topic in your book.
Bob Toxen- These are the most common problems that allow a Linux or Unix system to be broken into. Larry Gee wrote this part of the book as well as the section on Samba. These sins are:
1. Weak passwords 2. Open network ports 3. Old software versions 4. Poor Physical Security 5. Insecure CGIs [web server helper programs] 6. Stale and unnecessary accounts 7. Procrastination
We explain how each problem can be avoided. I explain how good passwords can be created. I discuss how a system can be scanned for open network ports and I provide a program that I wrote on the accompanying CD-ROM that does this more easily than netstat and which also recognizes the most common cracker server ports. It also flags any port above 1023 in the listen state that is not known to be legitimate and which may be a cracker server waiting for the word to attack.
I explain how to get on mailing lists to be notified when security fixes are available for various Linux and Unix distributions. I explain why CGIs tend to be insecure and that this endangers a web server's most important data even though they do not run as root.
While procrastination may be obvious, many break-ins are at least partially due to it. By listing this as one of the deadly sins, I hope to get many system administrators to resolve the other problems NOW.
LinuxSecurity.com offers many resources that are helpful in reducing the level of system vulnerability due to the "seven deadly sins." Here I have outlined articles that provide additional information on each of the topics above.
- Choosing Secure Passwords
- Scanning and Defending Networks with Nmap Scanning Tools
- Keeping Up to Date
- Physical Security
- User, System, and Process Accounting
- Security is an Interactive Sport
LinuxSecurity.com- I understand that your book provides solutions and further explanations of problems mentioned in Bruce Schneier's book, "Secrets and Lies: Digital Security in a Networked World." What are some examples of the problems and your proposed solutions?
Bob Toxen- Bruce's book should be required reading for all CEOs and managers who think that a firewall and Windows solves their computer security problems. Bruce seems to take the attitude now that because security never can be perfect, we ought to simply give up. My words are my own opinions.
Both Bruce and I discuss the problems of laptop computers containing confidential data being stolen, risking the data's exposure. My book explains how to keep data on the disk in an encrypted form using the Free Software Foundation's GPG program that implements PGP security. I also suggest that the reader use the PPDD encrypted disk driver that encrypts all data before writing it to the disk. He says that many laptops are stolen at airports. I explain the technique and how to prevent it from being used against you.
Bruce talks about the problems of buffer overflow attacks as if it was a never ending unsolvable problem. He says that in 1998 over 66% of all CERT advisories were for buffer overflow vulnerabilities and that during two weeks in 1999, 18 buffer overflow security flaws were found in Windows NT.
The latter problem can be avoided by replacing those NT systems with Linux. His statistics for 1998 (which include non-Linux systems) are, of course, obsolete now. In 2000 there were few buffer overflow vulnerabilities discovered and most of these could have been blocked by one of several techniques I discuss in the book. My favorite is installing Libsafe. It's easy to install and use. It blocks approximately 50% of buffer overflow vulnerabilities.
It works by being invoked instead of some standard dynamic library routines that crackers commonly take advantage of. It provides hardened versions of these routines used to overflow buffers, such as gets() and strcpy(). These hardened versions determine -- in real-time and without affecting performance -- if any attempted operation would cause overwriting of the stack. They will cause the program to terminate rather than risk an overflow. Libsafe was created by Bell Labs, which also created Unix.
I also discuss techniques for causing commonly attacked programs, such as named, sendmail, Apache, and CGIs to be invoked with sufficiently limited privileges that most vulnerability found by crackers will not allow much harm. I also discuss separating out different services onto different systems so that, say, a successful CGI attack cannot be used to attack web pages, mail, DNS, or other services. I explain a technique that prevents even a successful CGI, Apache, or network sniffing attack from allowing a cracker to get a company's customer credit card data. This technique is impervious to virtually all attacks. When reading Bruce's book after reading mine, other examples will occur to a reader.
For additional information on Libsafe, please refer to the Secure Programming for Linux and Unix HOWTO.
Libsafe: Protecting Critical Elements of Stacks "The libsafe library protects a process against the exploitation of buffer overflow vulnerabilities in process stacks. Libsafe works with any existing pre-compiled executable and can be used transparently, even on a system-wide basis. The method intercepts all calls to library functions that are known to be vulnerable. A substitute version of the corresponding function implements the original functionality, but in a manner that ensures that any buffer overflows are contained within the current stack frame. Libsafe has been shown to detect several known attacks and can potentially prevent yet unknown attacks. Experiments indicate that the performance overhead of libsafe is negligible. Copyright (C) 1999 Bell Labs, Lucent Technologies."
LinuxSecurity.com- What are some of the major pitfalls Linux Administrators fall into? How can your book help to avoid these problems?
Bob Toxen- Many think that Linux is secure out of the box, which it is not. I've seen shocking weaknesses in Red Hat, Slackware, and Debian. I'm sure that if I did out of the box installs of most other Linux and Unix distributions, I'd see problems. Even Mandrake's "pick your security level" is not a cure-all and it is not clear what each security level actually does.
Many think that break-ins are rare and so they should worry about other problems instead. Perhaps 50% of networks connected to the Internet have suffered break-ins. Any medium to large entity is guaranteed to suffer break-ins.
Some think that just throwing money at a firewall or router company or just running an intrusion detection system will solve their problems. I debunk this by explaining how these "solutions" can and do fail.
Some think that it will take too long and that they do not have time. Some have a fatalistic attitude. I offer Chapter 2's "quick fixes" for the most commonly exploited vulnerabilities.
Many articles have been written to help administrators secure their "out of the box" Linux install. Some of them include, "Post Installation: Is it secure out of the box?" "Securing and Optimizing Linux: Red Hat Edition" and "Securing the Linux Environment Part One: Installation Issues."
LinuxSecurity.com- You mention, "Almost all steps can be done on production systems without rebooting or even disrupting current services." Please explain your reason for taking this approach and give a few examples of problems that can be fixed quickly and easily without rebooting the system.
Bob Toxen- Most large companies have policies against taking their large systems down any more than absolutely necessary. One of my clients forbids scheduled downtime between 8 am and 9 pm during the week. Most SysAdmins would rather be elsewhere at other times and this causes some not to do what is necessary to secure their systems. Linux has an advantage here over Windows security fixes that often require a reboot.
Some of the problems that can be fixed quickly and easily without rebooting including using Crack to discover weak passwords, implementing shadowed MD5 passwords to make it harder for crackers to break whatever passwords are used, using the "find" program to find and fix files with inappropriate permissions, e.g. world-writable or inappropriate set-UID bits, and finding unnecessary open network ports (services) and shutting them off.
Even updating sendmail can be done by moving the new version to the correct place and issuing a single command line to kill the sendmail process listening on port 25 and restarting it. During this 20 millisecond window, any remote systems that try to connect simply will try again shortly. This relies on a unique Linux and Unix feature when the mv command (or similar command) moves the new version to where the old version of a program was. If some user is running the old version at the time, it will continue to run until finished even though this old version was removed from its directory on disk. This works for similar daemons too.
LinuxSecurity.com- What do you feel is the most common Linux system vulnerability? What can be done to prevent this?
Bob Toxen- NFS and its cohorts mountd, portmap, statd, etc. is the most common vulnerability. If at all possible, turn them off! Ssh is a secure alternative for copying files between systems or doing remote and no reboot is required. If one cannot live without NFS, set up an isolated network containing the NFS servers and clients. Protect this network from the Internet and an entity's large internal network with a Linux firewall. Be sure that the firewall that protects one's entire network from the Internet blocks these ports.
If the NFS systems are physically separated, link them with a Linux VPN (Virtual Private Network)/firewall. In the book, I cover an innovative solution using PPP over a ssh secure encrypted connection. I have implemented this solution for a client's network of Unix systems that suffered some break-ins.
NFS's design prevents it from being made secure but it can be so useful. This is why it is such a problem. Many Linux distributions and and some Unix versions enable it by default and that is a really bad idea.
Next to NFS, buffer overflows seem to be the greatest problem. The FTP daemon and Sendmail most frequently are attacked this way. Libsafe is an excellent innovative solution; keeping one's software up-to-date is critical. My clients have demanded that I provide a security patch update service for them so that they do not have to worry about it. Weak passwords are a major problem too.
Security and NFS from the NFS HOW-TO.
LinuxSecurity.com- Do you believe the open source nature of Linux provides a superior vehicle to making security vulnerabilities easier to spot and fix?
Bob Toxen- Absolutely! This allows hundreds of people to look for vulnerabilities in any piece of code instead of the half dozen or so that might have access to a proprietary piece of code. Those half dozen may be under time pressure to finish a project. When a programming project is behind schedule, as almost all are, security and testing are the first parts to be cut.
While the white hats will not have access to proprietary code, the black hats usually will get a copy. This recently happened to Microsoft's "crown jewels". It has happened to Cisco's code. No doubt, most cases of illegally obtained code never are publicized. A good cracker could just disassemble the executable program instead and some do and find vulnerabilities.
I also find open source programs to be of a much higher quality than proprietary code. As cutting edge technology, Linux, and Unix before it, has attracted the world's best programmers. It is a matter of pride and integrity to do the best job possible.
LinuxSecurity.com- Where do you see IT security and Linux going in the future? What will it take to make vendors and users more security conscience?
Bob Toxen- I see the various Linux distributions being much more secure "out of the box" due to competitive pressure. I also see them "walking the SysAdmin" through the process of securing their systems. I see the statefull and content-based features of some of the best proprietary firewalls become standard open source solutions on Linux.
I see monitoring of networks, with humans to backup up intrusion detection programs, to respond to suspicious activity. I offer this service as do a number of other firms. Only the largest entities have the in-house capability to do this. Many sites will have "Security Backup" systems that can be deployed within minutes or seconds in case of a security breach, possibly automatically. This is discussed in the book.
Most entities will do remote monitoring of their web sites for defacement and will do periodic security scans of their important systems for signs of successful break-ins and planted Trojans. I'm already doing this for clients and myself.
Already, many system administrators select Linux and BSD Unix due to their excellent security and ease of use. Those that do not accept this will lose market share or even go bankrupt. The recent breach of Egghead's IIS server comes to mind. Cracker tools have gotten so good that everyone will be forced to maintain good security or get cracked.
LinuxSecurity.com- What are your future plans? Do you plan on writing any other books/articles?
Bob Toxen- I plan to help my clients secure their Linux and Unix systems and networks. I have opened the eyes of many and take pleasure in helping others see. I hope to turn more people from what some consider to be the Dark Side of the Force. I have no future book plans, other than revising this book in a year or two. I have been invited to write magazine articles, give security classes, and speak at events. I look forward to doing this.
LinuxSecurity.com- Tell us a little bit about the history behind your car. Have you installed any type of security system?
Bob Toxen- I purchased my Rolls-Royce Silver Shadow at auction on eBay at the end of March 1999. I could not resist telling all of my friends about my acquisition on April 1. To my amazement two actually believed me. Some didn't for months. It may be seen around Atlanta with a "Linux" front license tag and at:
My principal client at the time was growing rapidly. To help everyone get to know each other better, the office manager sent email to everyone asking that we tell all what our favorite dessert was, what our fantasy car would be, etc. When I said that my fantasy car was a Rolls-Royce, she said that one was being auctioned off on eBay.
My bluff was called! With much trepidation and questioning of my sanity I started bidding. I knew next to nothing about them other than that they were beautiful, built to last a lifetime, and fit for royalty. As the bids grew, I did more research and got a friend who is a mechanic with some Rolls-Royce experience to agree to fly out to Reno to see it.
While it does have an alarm system, Rolls-Royces are so rare and instantly recognizable that few professional thieves would steal one and amateurs probably would not get far. I am careful where I park it. To avoid carjacking, I always drive with the doors locked and carry arms. This would be excellent practice for anyone.
LinuxSecurity.com- Can you offer a brief description of your background? How long have you been working with Linux and Security?
Bob Toxen- I spent four years at the University of California, Berkeley majoring in Computer Science. My Freshman year was the one that Ken Thompson came on sabbatical to create and teach the Operating Systems class and to port Unix to the PDP 11/70. When I took the class two years later, it was the identical class and I greatly benefited from it. Berkeley's strengths were in operating systems, compiler theory and construction, and structured programming; these skills have served me very well ever since.
Like many great universities, though, at Berkeley the professors were gods, graduate students their priests, and undergraduates the peons. Because of this, undergraduates were denied the opportunity to improve the Unix kernel or utilities. The solution taken by myself and a few friends was to break into the system and make improvements in secret.
Doug Merritt and I worked to enhance the terminal driver so that a control-H would blank out the character deleted and control-U would blank out the entire line on the screen. To my knowledge this was the first implementation of this feature on Unix in the world. We made other improvements, some in secret and some in the open. We also influenced development of many of the utilities, especially vi and Berkeley Mail.
I was the first person at Berkeley to start enhancing the shell. I was able to do this only when a security lapse allowed me to read and copy its source. If I had it to do again, instead I would have spent more effort to be allowed to do my research in the open. I created the Berkeley Unix lock program overtly. Variations of lock now are universal on Unix, Linux, and Windows.
Following school, I consulted for a few years and then worked at Onyx porting Unix to their microcomputers, the first micros to run Unix. On the side, I gave seminars on Unix and wrote magazine columns and articles on it. Steve Bourne then hired me at Silicon Graphics to be part of the four person team to port Unix to their first workstation. I then spent five years at Stratus Computer creating a fault-tolerant Unix and later a fault-tolerant NFS server, both running on top of a non-Unix native operating system.
Since 1990 I have been an independent consultant and I now have people working for me.
Bob, I would like to thank you for having this interview. We look forward to seeing your future projects, articles, and contributions to Linux and security.