Msyslog has the ability to log syslog messages to a database. This allows for easier monitoring of multiple servers and the ability to be display and search for syslog messages using PHP or any other programming language that can communicate with the database. . "Since the beginning, life has relied upon the transmission of messages. For the self-aware organic unit, these messages can relay many different things. The messages may signal danger, the presence of food or the other necessities of life, and many other things. In many cases, these messages are informative to other units and require no acknowledgement. As people interacted and created processes, this same principle was applied to societal communications. As an example, severe weather warnings may be delivered through any number of channels - a siren blowing, warnings delivered over television and radio stations, and even through the use of flags on ships. The expectation is that people hearing or seeing these warnings would realize their significance and take appropriate action. In most cases, no responding acknowledgement of receipt of the warning is required or even desired." I never would have guessed that this message came from the Introduction of RFC 3164 The BSD Syslog Protocol . Reviewing and maintaining the system logs on dozens of servers is a daunting task. Logging into each one and running grep or awk on each one can be very tedious and time-consuming. Luckily, there are programs like Logwatch and Logdog that can parse syslog files (and other files) and filter out keywords and send email or pager alerts. Fortunately, Syslog has a feature that allows for remote logging to a central server or servers. This feature allows virtually any unix syslog daemon to send syslog messages to a remote server that is configured to accept syslog messages. On a Linux system, for example, the syslogd daemon can be started with the " -r " option which tells the daemon to listen for incoming syslog messages. The port it listens on is 514 and theprotocol it accepts is UDP. # /usr/sbin/syslogd -r -m 0 " -m 0 " disables the timestamp mark in the syslog file, /var/log/messages on Linux systems. The configuration below is the only client configuration needed. The rest of the article pertains only to the central syslog server. Each client's syslog.conf file is then configured to send alerts to the central syslog server, by adding the line: *.* @julie *.* means to send all syslog messages to the remote syslog server, in this article named julie. Then refresh the syslog daemon: # /sbin/service syslogd restart End client configuration. Now on julie if you run tail -20 /var/log/messages (show the last 20 lines), you should now see the alerts sent with the hostname of the client's that have been configured to send alerts to julie. NOTE: has a program, called NTSyslog, that enables Windows Event Logs to be sent to a Unix syslog server. The process of reviewing multiple servers can be a lot easier using grep, awk, or perl, now that you have a central location where all the messages are sent. To take this one stop further, it can be incorporated with MySQL and PHP . The first thing we need to do is to get the syslog messages to the MySQL server. This is where Msyslog comes into play. Msyslog is a replacement for the standard syslog daemon that comes installed with most unix systems. Msyslog also has a nice feature of cryptographically signing syslog messages to let an admin know if their syslog files have been altered. Using the cryptographic features will be discussed in the next article. For now, the focus is on sending syslog messages to a remote server, logging to a database, and viewing the logs over a web interface using PHP. Finally, you will need to compile Apache, MySQL, and PHP support. Depending on your OS you may have a package manager that will do this work for you. Oh! You know me...I am not going to leave youhanging, my fellow readers. Here is a tutorial that explains how to setup Apache, PHP, MYSQL, and SSL. Just get the latest versions. Configuring Msyslog This was setup by downloading and installing the rpm from the msyslog website at: https://sourceforge.net/projects/msyslog/ The tarball install compiled cleanly on Red Hat 7.0-7.3. # cd msyslog-x.xxx # ./configure # make # make install rpm install: # rpm -ivh msyslog-xxx.rpm The rpm install added a startup script named " msyslogd " to the /etc/rc.d/init.d/ directory. If you installed from source, here is a startup script you can add to your OS's startup directory. The line: # daemon msyslogd $CONFIG $DEBUG $MARK $IM_BSD $IM_DOORS $IM_LINUX $IM_STREAMS $IM_TCP $IM_UDP $IM_UNIX in the " msyslog " startup script was changed by adding the switches " -i udp -p 514 -i om_mysql " # daemon msyslogd $CONFIG $DEBUG $MARK $IM_BSD $IM_DOORS $IM_LINUX $IM_STREAMS $IM_TCP $IM_UDP $IM_UNIX -i udp -p 514 -i om_mysql -i udp -p 514 - Listen on the standard port 514 for incoming syslog messages via udp -i om_mysql - load the mysql support module for logging to a mysql database This was done before the existing syslog daemon is shutdown so that when it is stopped, the settings above will immediately take affect and remote logging will continue. The normal syslog daemon was shutdown and myslogd started up immediately: # /sbin/service syslogd stop ; /sbin/service msyslogd start To ensure everything is still working run " tail -f " on the /var/log/messages file to see if logs from remote servers were being received: # /usr/bin/tail -f /var/log/messages ^C " tail -f " allows data to be viewed while a file is being appended. The logging to mysql was setup by first creating a database called " logd ": # /usr/bin/mysqladmin -p -u root create logd Then the script supplied in the man page for the om_mysql module was loaded into the database. # /usr/bin/mysql -p -u root logd < syslog.sql The syslog.sql file contained this, I modified the supplied sql file to index the host, date, and message fields.: mysql> CREATE TABLE syslogTB ( facility char(10), # OPTIONAL field for facility priority char(10), # OPTIONAL field for priority date date, # date of this log message time time, # time of this message host varchar(128), # host logging, If you have a host with # 128 characters you probably # have other issues to worry about than #someone being l33t. 8-) message text, INDEX host_index (host), INDEX date_index (date), INDEX message_index (message (50)) , #Index the first 50 characters seq int unsigned auto_increment primary key # optional sequencenumber ); #Table to import host names mysql> CREATE TABLE sysloghosts ( hostname varchar(128) # host logging, Same principles as # above for a 128-character hostname. 8-) ); The "sysloghosts" table is used as a dropdown list on the PHP search form. This is only run if new hosts are configured to log to julie. I retrieved the list from the /var/log/messages file with this command: # /bin/awk ' { print $4 } ' /var/log/messages | sort | uniq > /tmp/hosts.tmp # /bin/chown mysql:mysql /tmp/hosts.tmp The mysqld owner must hsve permissions to import the file into the database. Log into the mysql logd database as a root user (not system root), delete the current hosts, and add new hosts file: # /usr/bin/mysql -p -u root logd Enter Password: mysql> DELETE FROM sysloghosts; mysql> LOAD DATA INFILE '/tmp/hosts.tmp' INTO TABLE sysloghosts LINES TERMINATED BY 'n'; mysql> exit Delete the temporary file: # rm -f /tmp/hosts.tmp The user " mysql " is used to insert thesyslog data into the database. Also, the mysql user will be used to select data from the database using PHP. Log into the database as the admin user and grant the user " mysql " rights to edit and update the " logd " database. # /usr/bin/mysql -u root -p logd Enter Password: mysql> GRANT SELECT, INSERT on logd.* TO mysql@localhost IDENTIFIED BY 'dahbadahba'; mysql> FLUSH PRIVILEGES; the mysql user is allowed to select and insert data for the logd database (GRANT SELECT,INSERT on logd.*) from the localhost (TO mysql@localhost) with the password "dahbadahba" (IDENTIFIED BY ' dahbadahba ';) and then the privileges are enabled (Flush privileges;) . Syslog configuration file In order for this to work the password for the database has to be kept in the syslog.conf file. A few changes were made to prevent normal users from viewing the syslog.conf file; thus, revealing the database password. (NOTE: Never use the system's root password for a database password) First, the default permissions, on some unix systems, for /etc/syslog.conf are readable-writeable by root and readable by the group "root" and by the world (644). This was changed to 600: # /bin/chmod 600 /etc/syslog.conf Now it is only readable and writeable by root. Test it by trying to "cat" the file as a normal user : # /bin/cat /etc/syslog.conf Hopefully, the following message will be displayed: # cat: /etc/syslog.conf: Permission denied The options for logging to the mysql database can be added to the bottom of the /etc/syslog.conf file: *.* %mysql -s localhost -u mysql -p dahbadahba -d logd -t syslogTB -D *.* -- log all syslog messages to the mysql database -s - hostname -u - user to log into the database as -p - the database password -d - the database name -t - the database table name -D - delay logging to the database (preventsoverloading the mysql daemon if large numbers of syslog messages are received). Restart the myslogd daemon: # /sbin/service msyslogd restart Watch the directory where your mysql databases are located and see if the file grows. Restricting access to julie: EnGarde Secure Linux EnGarde has everything necessary to create thousands of virtual Web sites, manage e-mail, DNS, and firewalling for an entire organization, and supports high-speed broadband connections all using a Web-based front-end. [ View Screenshots ] [ Buy Online ] [ Feature List ] By default, the syslog daemon will accept syslog messages from any server. Be sure to use firewall rules to only allow syslog messages from the servers that should be logging to it. If your firewall supports it, use a threshold for logging to prevent a Denial-of-Service (DoS) attack. Also, the clients will listen on port 514 when sending log messages to the syslog server so be sure to firewall incoming requests to the client's syslog port, as well. No one should be connecting to the client's syslog port. Only the system adminstrators should have access to julie. One reason is that root passwords and other user's passwords could be echoed into the syslog files because of those with fast fingers may type the password in the "username" or "Login:" field and hit "Enter". Yes I am guilty. . That's the only time I have stopped the syslog daemon, opened /var/log/messages and mnaully deleted an entry. (NOTE: For sanity be sure your syslog files are chmod 600 and owned by root.) The following rules restrict access to particular hosts: (NOTE: the policy is to DENY ALL) # This restricts access to the entire web directory on julie # You can configure as you like Options Includes FollowSymLinks AllowOverride None Order deny,allow deny from all # > 8-) # allow from (space delimited list of allowed hosts or networks) allow from 192.168.0.2 192.168.0.3john.server.com clint.server.com The logs should also be reviewed via the web on julie over a secure connection. Your firewall can block or redirect incoming port 80 requests so access to the server is granted only over a secure connection. Viewing syslog messages Logs are viewed on julie using php to extract the data from the mysql database. Create a directory under your root directory called " web-syslog " # /bin/mkdir /var/www/html/web-syslog Place these .php files there: (syslog-index.txt and syslog-search.txt). Create an include outside of your web directory: # touch /var/www/gsyslog.php Be sure this file is owned by the owner of the httpd daemon and readable-writeable by only that user. If you put this file somewhere else on your filesystem, be sure the owner of the httpd daemon has read access to the directory where it is located. This file will contain the password for the logd database. Add the following: You can also find this script at . A side note (special thanks to Steve Reed ): For those of you running older versions of php, you might need to make the following modifications to syslog-search.php: -$host = trim(addslashes(htmlspecialchars($_GET['host']))); -$message = trim(addslashes(htmlspecialchars($_GET['message']))); -$date = trim(addslashes(htmlspecialchars($_GET['date']))); -$numresults = $_GET['numresults']; -$start = $_GET['start']; +$host = trim(addslashes(htmlspecialchars($host))); +$message = trim(addslashes(htmlspecialchars($message))); +$date = trim(addslashes(htmlspecialchars($date))); +$numresults = $numresults; +$start = $start; Point your browser to: . Hopefully you should see a page where you can select the hosts to retreive records for and an optional date and message field. Other Notes: Disable the standard syslog daemon from starting during bootup. You'll have to check your OS documentation forstarting a stopping services during bootup. On Red Hat you can use the " chkconfig " program or " ntsysv" : # /sbin/chkconfig --level 345 syslog off or # /usr/sbin/ntsysv Uncheck the syslog option. If you upgrade your system then you will have to be sure that the msyslogd daemon starts up and not the standard syslog daemon on julie. Conclusion: Centralized logging and viewing of logs is very valuable, especially when you have dozens of servers to monitor. It can also serve as a place to look for errors if a server has crashed and can't be broke back on line. Take care to ensure that only necessary people are allowed to view the logs and if at all possible view the logs only over a secure connection. A database backend whether mysql, postgres, or whatever database gives you more power and control of how to display and manipulate the data that is being stored. Duane Dunston is a Computer Security Analyst at STG Inc. for the National Climatic Data Center in Asheville, NC. He received his B.A. and M.S. degrees from Pfeiffer University and he has his G SEC certification from SANS. He hangs out at Old Europe Cafe, Early Girl's eatery, Anntony's, and any place with good tea and hot choc olate. Duane has been working in security for 5 years and wishes he had the funding for a "Basic Security Tour" so he could provide the wo rld with hands-on training on how to implement the security recommendations from the Sans Top 20 List of the most common vulnerabiliti es. He knows that applying these recommendations to any network can minimize the most common types of attacks. Not only does he enjoy his work in computer security, he also likes to get involved in its ever-growing technologies. Duane says, "Security is one of those jobs where you have to stay abreast of new technologies and new ways that attackers are compromising computer systems. Security keeps evolving and the industry has to keep up with it, that is why we need well-trained, evolvingsecurity professionals supportive manag ers to help us with this ongoing process". . Centralized logging with MySQL simplifies syslog management across servers, enhancing security and monitoring.. msyslog, ability, syslog, messages, database, allows, easier, monitoring. . Duane Dunston
The syslog daemon is a very versatile tool that should never be overlooked under any circumstances. The facility itself provides a wealth of information regarding the local system that it monitors.. However, what happens when the system it's monitoring gets compromised? When a system becomes compromised, and the intruder obtains elevated root privileges, he now has the ability, as well as the will, to trash any and all eviden ce leading up to the intrusion, on top of erasing anything else thereafter, including other key system files. That's where remote system logging comes in, and it's real super-easy to set up. This example uses Slackware Linux 8.1 as the base to begin, but can be applied easily to other distros using the syslog daemon that was ported to Linux by Mar tin Schulze (
A Comprehensive Guide to Building Encrypted, Secure Remote Syslog-ng Servers with the Snort Intrusion Detection System . Introduction The precursor to this article, Creating Secure Remote Log Servers, was the first in a series of papers focused on walking readers through configuring and deploying secure remote log servers. This second paper in the series offers a much more robust alternative to first generation SYSLOG servers; providing a much more reliable remote logging facility that is effective for use within Honeynets ( ) and Intrusion Detection System deployments. Remote log servers can provide centralized logging capability for IDS' spread across large network environments. I have proposed this approach for centralized logging in large IDS deployments on government networks that typically consist of multiple CLASS A networks. What this paper hopes to accomplish is to walk its readers through building next generation secure remote log servers to use in any environment, more specifically those wanting to utilize this form of logging with the Snort Intrusion Detection System ( https://www.snort.org/ ). For those of you who follow my papers regularly, you know that my writing style is that of precise detail without any real expectations from its readers of intimate knowledge on how to configure and use the utilities I write about. This proves the same for this paper as well. I will walk you through installing and configuring the Snort IDS as well as downloading, installing, and configuring Syslog-ng (Syslog, Next Generation). I will detail how to configure Snort to log to syslog for alerts to be generated locally and remotely to the offsite Syslog-ng server over an encrypted SSL tunnel. This will be the most comprehensive paper available to the community; offering a step-by-step guide to configuring Secure Remote Log Servers and interaction with Intrusion Detection Systems. Preparing Your Systems Ok, the first thing you're going to want to do is setup both systems and identify which one willbe the (CLIENT); the Snort box running Syslog-ng that will send its logs to the (SERVER); the system that listens for incoming connections for logs from the Syslog-ng client. Let's lay out a few ground rules to set the foundation for this paper. I will be referring to each system accordingly as outlined above. The IP addresses for each system are: CLIENT 192.168.0.1 SERVER 192.168.0.2 The Client The first task we'll accomplish is downloading and configuring Syslog-ng for use as the client. Now if you remember, the client must be configured to send the Syslog alerts remotely to the other server. Syslog-ng stands for Syslog Next Generation. As the name implies, Syslog-ng was designed to meet higher standards of stability in logging as well as added security and encryption functionality. A unique feature of Syslog-ng is its capability to offer TCP logging, which all of you should hopefully already know is different from first generation SYSLOG as its predecessor utilizes UDP only. To download Syslog-ng hop over in your favorite lynx browser ;) to and download. As of this writing, the current version is 1.4.15. You will also need to download libol as the instructions imply. pa-obsd01# pwd /export/syslog-ng-1.4.14 pa-obsd01# ./configure && make && make install Alright, for some reason when I installed and configured Syslog-ng, it didn't create the /etc/syslog-ng directory, nor provide me any default configuration files. So for obvious reasons you won't have to worry about that because I'll be providing the configurations for you. Go ahead and mkdir the /etc/syslog-ng directory and untar Syslog-ng after installing libol. Once completed you should have a binary for syslog-ng in your /usr/local/sbin directory. The following configuration information should be stored as /etc/syslog-ng/syslog-ng.conf ################################################### # # This is a working Syslog-ng file for a Syslog-ng # CLIENTsystem only. # # Refer to the comments below for some of the # syntax being used. # File: /etc/syslog-ng/syslog-ng.conf # # Syslog-ng configuration file created by # Eric "Loki" Hines # Email: loki@fatelabs.com # # Syslog-ng is # Copyright (c) 1999 Balazs Scheidler # #################################################### # This identifies the source machine (gateway) and # gives it a name. You can name the identifier anything # you want, e.g. source barney.localhost # { unix-dgram("/dev/log"); internal(); }; or whatever you # want. Have fun, but make sure to remember what name # you give it for the log statement. source gateway { # If you are not using OpenBSD, you will need to change # this to your specific syslog device file. # The different options for each OS is provided at # https://www.oneidentity.com unix-dgram("/dev/log"); internal(); }; # What I've done here (thanks Jason Ish), is configured # Syslog-ng to log locally to our /var/log directory as well as # remotely to the remote Syslog-ng SERVER. This is an awesome # idea as it creates 2 locations for log files to eliminate # single points of failure. (Also an awesome idea with # honeynets, dig? J destination localhost { file("/var/log/syslog-ng.all"); }; destination shell { tcp("192.168.0.2" port(514)); }; # This ties our source and destination together, think of it # this way (src + dst = logging) log { source(gateway); destination(localhost); source(gateway); destination(shell); }; You should now have a working configuration file for the sylsog-ng client, let's go ahead and setup Snort for logging to the Syslog server. This will actually be more trivial than you might think. Go ahead and download Snort from https://www.snort.org/ . As of this writing, the current version is 1.8.6. If you are worried about the new fragroute IDS evasion tool and protectingagainst these types of attacks, currently, Snort offers a stable-snapshot release for download. The next release of Snort will evidently provide these enhancements, so choose your poison. Go ahead and untar Snort and let's walk through the configuration. 192.168.0.1 pa-obsd01# pwd /export/snort-1.8.6 pa-obsd01# ./configure && make && make install I feel kind of ridiculous pasting in those ./configure commands but some of you would be surprised with the kind of emails I get after writing a paper :D So that table is for some of you that don't yet know how to compile and install a program. Then again, if you already didn't know that I'd question your idea of building a secure remote log server at this early in the game ;) But we've all got to start somewhere right? Moving on. We're going to go ahead and make a quick modification to the Snort configuration file. # or you can specify the variable to be any IP address # like this: var HOME_NET 192.168.0.1 # Set up the external network addresses as well. # A good start may be "any" var EXTERNAL_NET any Lets go ahead and start up Snort to log to syslog. The Snort development team made this extremely simple for us. Because we've configured Syslog-ng to log remotely for us, Snort doesn't have to do ANYTHING but log locally to syslog. This is accomplished merely by using the following syntax 192.168.0.1 pa-obsd01# adduser Snort pa-obsd01# passwd Snort pa-obsd01# ./snort –D –A full –c snort.conf –d –D –e –u snort –g snort –s (Please don't run snort as root.) The other flags can be omitted without any problems, but making sure to leave the –s flag in tact as that is what enables Snort logging to Syslog. Upon initiation of Snort, our Syslog-ng will now be trapping those alerts and sending them over the wire to the remote Syslog-ng server. However, because it isn't yet configured those alerts will be lost,maybe I should have done this step last :D, hah, man I crack myself up. The Server Let's go ahead and configure the remote Syslog-ng server now for receipt of those alerts. For obvious reasons, go ahead and download Syslog-ng again for the server and run through the configure and make install again. After doing so, we'll go ahead and configure Syslog-ng to accept alerts from the Client. 192.168.0.2 source shell { unix-dgram("/dev/log"); internal(); # Listen on public interface, port 514 for incoming connections tcp(ip(192.168.0.2) port(514) max-connections(1)); }; destination localhost { file("/var/log/syslog-ng.all")); }; # Again, we tie both statements together with the log function. log { source(shell); destination(localhost); }; To start up Syslog-ng we'll go ahead and execute /usr/local/sbin/syslog-ng. Oh, go ahead and start up Syslog-ng on the CLIENT as well. You should now be successfully logging Snort alerts from the remote system as demonstrated below. 192.168.0.2 pa-obsd01# tail –f /var/log/syslog-ng.all May 14 02:37:18 localhost/localhost/192.168.0.1 Snort: [1:1002:4] WEB-IIS cmd.exe access [Classification: Web Application Attack] [Priority: 1]: {TCP} 192.168.0.2:3434 -> 192.168.0.1:80 May 14 02:37:19 localhost/localhost/192.168.0.1 Snort: [1:1002:4] WEB-IIS cmd.exe access [Classification: Web Application Attack] [Priority: 1]: {TCP} 192.168.0.2:3458 -> 192.168.0.1:80 May 14 02:37:19 localhost/localhost/192.168.0.1 Snort: [1:1002:4] WEB-IIS cmd.exe access [Classification: Web Application Attack] [Priority: 1]: {TCP} 192.168.0.2:3474 -> 192.168.0.1:80 May 14 02:37:20 localhost/localhost/192.168.0.1 Snort: [1:1002:4] WEB-IIS cmd.exe access [Classification: Web Application Attack] [Priority: 1]: {TCP} 192.168.0.2:3496 -> 192.168.0.1:80 May 14 02:37:20 localhost/localhost/192.168.0.1 Snort: [1:1002:4] WEB-IIS cmd.exe access[Classification: Web Application Attack] [Priority: 1]: {TCP} 192.168.0.2:3515 -> 192.168.0.1:80 May 14 02:37:20 localhost/localhost/192.168.0.1 Snort: [1:1002:4] WEB-IIS cmd.exe access Priority: 1]: {TCP} 192.168.0.2:3547 -> 192.168.0.1:80 May 14 02:37:21 localhost/localhost/192.168.0.1 Snort: [102:7:1] (spp_http_decode) Overlong Unicode character received {TCP} 192.168.0.2:3565 -> 192.168.0.1:80 May 14 02:37:21 localhost/localhost/192.168.0.1 Snort: [1:1002:4] WEB-IIS cmd.exe access [Classification: Web Application Attack] [Priority: 1]: {TCP} 192.168.0.2:3565 -> 192.168.0.1:80 May 14 02:37:21 localhost/localhost/192.168.0.1 Snort: [102:7:1] (spp_http_decode) Overlong Unicode character received {TCP} 192.168.0.2:3585 -> 192.168.0.1:80 The Firewall Now we will want to install a firewall on the remote Syslog-ng server. This will allow us to specify what systems are and are not allowed to connect to our system as well as specify an ACL for what IP's are allowed to log to our Syslog port. We will be accomplishing this through a simple PF (Packet Filter) config file. I have provided mine below. For you other users of IPF, the syntax should work the same. 192.168.0.2 #### #### SET VARIABLES. CHANGE THIS TO YOUR NIC INTERFACE ID #### ifconfig -a EXT="de0" #### #### BLOCK IN ALL RFC 1918 #### block in quick on $EXT inet from 192.168.0.0/16 to any block in quick on $EXT inet from 172.16.0.0/12 to any block in quick on $EXT inet from 10.0.0.0/8 to any block out quick on $EXT inet from any to 192.168.0.0/16 block out quick on $EXT inet from any to 172.16.0.0/12 block out quick on $EXT inet from any to 10.0.0.0/8 #### #### EXPLICITY ALLOW ONLY 192.168.0.1 TO PORT 514 (syslog-ng) #### IF YOU USE THIS FIREWALL CONFIG FOR STUNNEL, CHANGE IT TO #### THE INCOMING STUNNEL PORT WE SET, 5140 pass in quick on $EXT inet proto tcp from { 192.168.0.1/32 } to any port = 514 #### ####EXPLICITY BLOCK ALL OTHER TRAFFIC AND LOG #### ALLOW ALL OUTGOING #### block in log quick on $EXT from any to any pass out quick on $EXT from any to any keep state Stunnel I have decided to break this paper up into (2) two sections. The following section and configuration files for Syslog-ng will only be for those of you who want to encrypt the syslog data over SSL. For those of you who have your own ways of handling the encryption (vpn, etc), feel free to ignore this section and only use the configuration files provided previously. Client Your first task, should you choose to accept it, is to download and configure Stunnel J. You can download Stunnel from . Now, for some reason I keep getting the same compile errors when compiling on OpenBSD 3.0. So for those of you who are experiencing the same problems, simply install stunnel from ports, RPM, or whatever alternative or binary distribution your platform offers. I did in fact install from ports, so aside from the ./configure and make install, I think all of you can pretty much handle this on your own. After the installation has completed, you will want to configure Syslog-ng to log to LOCALHOST to a port where Stunnel will be awaiting connections. Stunnel will then be the carrier of the data over to the SERVER where another Stunnel daemon will be waiting for connections. Use the following configuration file for Syslog-ng located in /etc/syslog-ng/syslog-ng.conf 192.168.0.1 ########################################################################## # # This is a working Syslog-ng file for a Syslog-ng CLIENT system using # STUNNEL only. # Refer to the comments below for some of the syntax being used. # File: /etc/syslog-ng/syslog-ng.conf # # Syslog-ng configuration file created by Eric "Loki" Hines # Email:
In this feature story we present a very comprehensive guide to setting up a secure log server, by Eric Hines. Mr. Hines goes into very good detail and covers everything from building and configuring syslogd to securing the remote log server. . AUTHORS BIOS Eric Hines As a 7 1/2 year member of the network security industry, Eric currently provides security consulting to many Fortune 500 to Fortune 50 Corporations across the US. Eric also gives keynote speakings on information security at numerous different E-Commerce and E-Banking seminars. Eric has deployed several VPN Extranets from British Columbia to the United States, as well as plays active roles in contributions to many security-related mailing lists. Eric enjoys creating custom attack signatures using packet analyzers for Intrusion Detection Systems such as SNORT, and holds strong interest in Intrusion Detection technology. Fyodor Yarochkin Works for TRG ( ). Personal information available at: . INTRODUCTION Welcome to the first issue of a series of security-related HOWTO papers. In these documents both Fyodor and I aimed to give various tips, hints and tricks on various aspects of Internet security. Comments and suggestions are welcome and can be emailed to us directly at the contact information provided at the bottom of this paper. We hope you find it useful and recommend hammering away at the README files for all utilities being referenced in this paper. In this paper we make references to and as headers. In this, SERVER, refers to nothing more then the machine that is configured to receive logs from remote machines (CLIENTS). CLIENT, referring to the machine that is configured to send its logs to the REMOTE LOG SERVER. - Eric Hines A. WHAT IS A REMOTE LOG SERVER A remote log server is nothing more then a system preconfigured at install-time to provide hard drive space for other systems tolog to. This system must be completely secured and locked down. No unencrypted remote access should be allowed, all RPC Daemons and other misc. services should be turned off as well. The only data allowed to the machine should be UDP/Port 514. We will be walking you through a step-by-step process that details how to configure, install, and deploy a remote log server. Utilizing some of the most renowned security experts across the globe for input, I've compiled a comprehensive, and easy to understand guide on ensuring this to be a successful launch. B. SYSLOGD Recompiling syslogd The first step in turning on remote logging capabilities on your machine is recompiling syslogd to read a completely different syslog.conf file. This is for security reasons, which we will detail more later in this paper. NOTES This was an idea derived from the "To Build a Honeypot" whitepaper written by Lance Spitzner. Thanks for your contributions to the security community, Lance. The beautiful idea behind this strategy is to fool the intruder into thinking there is no remote logging setup on the machine. By leaving the old /etc/syslog.conf file in tact, also helps in tricking the unsuspecting cracker. STEP 1: COMPILING OUR NEW VERSION In our first step, we face somewhat of a problem in that syslogd ships with the operating system by default, so no source code is provided with it. You will need to download the source code from an outside party, such as Red Hat, or even Joey's FTP site (the creator of syslogd). As of this writing, Joey's ftp site is down so I have provided an alternative resource to download the needed file. Your distribution however, would be in violation of Open Source policies by not providing the source code to you. So check your particular OS vendor for the source code if you do not want to download it from any of the links specified in this paper. SITE 1: (currently down as of thiswriting) or SITE 2: Hat/Red Hat/Red Hat-6.2/SRPMS/SRPMS/sysklogd-1.3.31-16.src.rpm After 'rpm -iv -v sysklogd-1.3.31-16.src.rpm, the source code should have been installed to /usr/src/Red Hat/SOURCES/sysklogd-1.3-31 (that is, if you are running Red Hat) STEP 2: CHANGING THE DEFAULT CONFIG FILE LOCATION Here is the fun part. This is used to fool an intruder once he has compromised the system into thinking your log files are stored locally. We will be recompiling syslogd to read a completely separate configuration file, leaving the old default one in tact, just in case the intruder runs a 'cat' on the /etc/syslogd.conf file to check if remote logging has been enabled. By leaving the old configuration there proceeding our recompile of the new binary, ensures that we can fool the cracker into thinking otherwise. Lets modify the default location of /etc/syslog.conf [ehines@myhost sysklogd-1.3-31]$ cd /usr/src/Red Hat/SOURCES/sysklogd-1.3-31 [ehines@myhost sysklogd-1.3-31]$ vi syslogd.c do a /grep for syslog.conf This will bring us to #define _PATH_LOGCONF "/etc/syslog.conf" Change the above location value to something you prefer. I suggest something like /etc/.sys/CORE.conf or something ingenious like that. Copy the new syslogd binary over the old (typically /sbin/syslogd) BACKUP old version before doing this. As an admin, you should have already known that, right? :) NOTES The reason why we do not simply just start the old syslogd with the (-f conffile) switch is because if an attacker were to grab a 'ps' list of processes, the switch you used will show up, telling the intruder that we are specifying a completely different configuration file. Ultimately, defeating this purpose. I recommend leaving the old /etc/syslog.conf file in tact as it completely throws off the intruder. Also, keep local logs, as if the /var/logdirectory is empty, well, use your imagination on why. I would have to make the suggestion to use a different filename, other then syslog.conf when specifying the new location. Understand that an intruder can also just do a find / -name "syslog.conf" and locate another copy of the configuration file. STEP 3: MODIFY THE (REAL) SYSLOG CONFIG FILE For example, to forward ALL messages to a remote host use the following syslog.conf entries: # Sample syslogd entry to forward all messages to a remote host. *.* @hostname # To forward all kernel messages to a remote host the configuration file would be as follows: kern.* @hostname # Remote Logging which includes local logging as well (for fault tolerance) kern.crit @finlandia kern.crit /dev/console .. and so on.. repeat for all log files you wish to keep offsite. This includes third party applications. C. THE CLIENT (REMOTE LOG SERVER) With the client server, (or the machine accepting all of the remote logs), all you have to do is launch syslogd with the -r switch, specifying remote logging capabilities. Make sure that you remember to ADD: syslog 514/udp to the /etc/services file as well, on the server, or it will not listen to the specified port. NOTES If the network consists of different domains you don't have to complain about logging fully qualified names instead of simple hostnames. You may want to use the strip-domain feature -s of this server. You can tell the syslogd to strip off several domains other than the one the server is located in and only log simple hostnames. ADD 'syslog 514/udp'to the /etc/services file. If this entry is missing syslogd neither can receive remote messages nor send them ADD '*.* @hostname' to the /etc/syslog.conf file. To forward all Kernel logging messages to a remote host, Make sure that if you do notspecify an IP ADDRESS that you add the host to the /etc/hosts file to avoid DNS RESOLUTION errors. If the remote host is located in the same domain as the host, syslogd is running on, only the simple hostname will be logged instead of the whole fqdn. (fully qualified domain name) D. LOCKING DOWN THE REMOTE LOG SERVER One of the most crucial steps in this project is to fully lock down the machine that is receiving all of the logs, or referred to as the remote log server. 1. Turn off (ALL SERVICES) Remember, this is ONLY a log server, so all other services should NOT be turned on. Lets walk through that process right now. [root@myhost /etc]# cd /etc [root@myhost /etc]# vi inetd.conf 1a. TURN OFF ALL INETD SERVICES You will want to put a '#' comment in front of EVERY service listed in this file. The inetd.conf file is IMHO one of the largest mistakes of the Linux operating system :). But the ability to disable the inetd services makes it all better :) You will want to comment out everything: a. echo b. discard c. daytime d. chargen e. time f. ftp g. telnet h. shell i. login j. exec k. comsat l. talk m. ntalk n. dtalk o. pop-2 p. pop-3 q. imap r. uucp s. tftp t. bootps u. finger v. cfinger w. systat x. netstat y. auth z. linuxconf zz. swat 1b. DISABLE ALL RPC SERVICES [root@myhost /etc]# cd /etc/rc.d Anything listed in the rc. directories that start with a capital "S" means that the service will start at boot time. In order to disable it from starting, you will want to 'mv to (a lowercase 's'). e.g. mv S11portmap s11portmap // This disables portmapper from starting at boot time. You will want to do this with all unneeded services in this directory. These include: nfslock, apmd, netfs, identd, autofs, portmap, atd, pcmcia, I would evendisable the inetd startup script as well, isdn, sendmail, gpm, httpd, vmware, xfs,linuxconf, local, You are basically (other then the obvious services) are going to want to disable all of these rc controlled services.You will want to go through all of the rc0.d-rc6.d directories looking for these same files to disable. 2. DISABLE ACCOUNTS [root@myhost /etc]# vi /etc/passwd Go through your password file and remove practically every account that isn't being used. I would suggest downloading the tool, /bin/noshell (https://www.cerias.purdue.edu/about/history/coast/archive/data/categ50.html) and use this with all accounts in the password file that should not have remote login access. This is a very informative program that provides admins additional information on who is attempting to login with disabled accounts. 3. INSTALL SSH Since the release of OpenSSH, giving the GNU opensource community free access to secure shell, telnet should now be obsolete in a security centric environment. GET AWAY from using telnet. With packet sniffers getting easier and easier to use (tools such as dsniff, 'www.dsniff.org') allows even the biggest nitwit to snarf packet data. So case-in-point, don't use telnet. I bet you are so happy, because guess what? I'm going to walk you through the installation of OpenSSH! (ohh goody).. 3a. DOWNLOADING OPENSSH Point your web browser to https://www.openssh.org/ as of this writing they are on version 2.1.1 which has support for both SSH1 and SSH2 protocols. The release date was June 08, 2000. After downloading the source tarball, go ahead and do the following: [root@myhost ]# gzip -d openssh-2.1.1p2.tar.gz [root@myhost ]# tar -xvf openssh-2.1.1p2.tar I can't realistically walk you through everything, so please save me some heartache, as well as yourself, and just read the INSTALL file before continuing. I have no idea what particular requirementsyour system has. I will however, walk you through my install. First you will need working installations of both Zlib and OpenSSL (yes there is an OpenSSL project as well) OpenSSL 0.9.5a or greater: /index.html Or, if you're even more lazy then the audience this paper was written for, (God forbid), then grab the RPM's from the link below. RPMs of OpenSSL are available at Alternatively, Jim Knoble has written an excellent X11 passphrase requester. This is maintained separately at: https://ntrnet.net/ If you wish to build the GNOME passphrase requester, you will need the GNOME libraries and headers. GNOME: GNOME – Simple, beautiful, elegant. Also, grab GNUmake if you do not yet have it, OpenSSH has only been tested with this program. [root@myhost ]# ./configure && make && make install This will install the binaries in /opt/{bin,lib,sbin}, but will place the config files in /etc/ssh If you are using PAM, you will need to manually install a PAM control file as "/etc/pam.d/sshd" (or wherever your system prefers to keep them). A generic PAM configuration is included as "contrib/sshd.pam.generic", you may need to edit it before using it on your system. If you are using a recent version of Red Hat Linux, the config file in contrib/Red Hat/sshd.pam should be more useful. Failure to install a valid PAM file may result in an inability to use password authentication. I like to run configure with '--without-pam' to disable PAM support. PAM is automatically detected and switched on if found. This is because I opt to use RSA Authentication instead. [root@myhost ]# tar -xvf zlib.tar [root@myhost ]# cd zlib-1.1.3 [root@myhost zlib-1.1.3]# ./configure && make && make install You shouldn't get any errors, if you do, sorry to do this, but refer to the README file. What I like to do when compiling something and something goes wrong, I justgrep a keyword from the error on the provided README file(s) to see if there is anything specifically mentioned about my error. (Just a suggestion) To install OpenSSL, you will need: Perl 5 an ANSI C compiler a supported Unix operating system [root@myhost ]# tar -xvf openssl-0.9.5a.tar [root@myhost ]# ./config && make && make test && make install Now lets build OpenSSH! [root@myhost ]# cd openssh-2.1.1p2 [root@myhost ]# ./configure --without-PAM && make && make install NOTES Because I opt for RSA authentication, I run configure with the --without-PAM option as I am not a big fan of basic password authentication. With the advancements in Cryptography and more secure forms of authentication, PAM and other password-type authentication mechanisms IMHO will become obsolete within a few years When you have successfully compiled and installed OpenSSH, you will receive the following information. Key generation complete. Your identification has been saved in /usr/local/etc/ssh_host_key. Your public key has been saved in /usr/local/etc/ssh_host_key.pub. The key fingerprint is: d5:74:83:d0:3f:c4:b4:d6:c5:39:1d:94:ee:9b:a8:61 root@soc1.priv.nuasis.com Generating DSA parameter and key. Your identification has been saved in /usr/local/etc/ssh_host_dsa_key. Your public key has been saved in /usr/local/etc/ssh_host_dsa_key.pub. The key fingerprint is: ed:58:65:b9:8b:fe:05:81:c2:8c:06:c9:cb:ac:bb:e6 root@soc1.priv.nuasis.com 3b. CONFIGURING OPENSSH Even though the default installation of OpenSSH should work, I would take a quick stab at the ssh config files to see if you want to change anything. Because we configured OpenSSH without PAM support, we want to enable RSA Authentication. Uncomment it in the /usr/local/etc/ssh_config file respectively (this is for the system-wide client configuration). Then go ahead and edit the SSHD configfile, /usr/local/etc/sshd_config Lets go ahead and disable PermitRootLogin (do this by changing 'yes' to 'no' Go ahead and save the file and quit. After installation, a host key and DSA key should have been created for you. So lets go ahead and move to the next step. NOTES ON SSH v2 THE SSH PROTOCOL v2 SSH protocol version 2 In SSH v2, each host has a host-specific DSA key used to identify the host. However, when the daemon starts, it does not generate a server key. Forward security is provided through a Diffie-Hellman key agreement. This key agreement results in a shared session key. The rest of the session is encrypted using a symmetric cipher, currently Blowfish, 3DES or CAST128 in CBC mode or Arcfour. The client selects the encryption algorithm to use from those offered by the server. Additionally, session integrity is provided through a cryptographic message authentication code (hmac-sha1 or hmac-md5). Protocol version 2 provides a public key based user authentication method (DSAAuthentication) and conventional password authentication. AUTHORIZED_KEYS FILE FORMAT The $HOME/.ssh/authorized_keys file lists the RSA keys that are permitted for RSA authentication in SSH protocols 1.3 and 1.5 Similarly, the $HOME/.ssh/authorized_keys2 file lists the DSA keys that are permitted for DSA authentication in SSH protocol 2.0. Each line of the file con tains one key (empty lines and lines starting with a `#' are ignored as comments). Each line consists of the following fields, separated by spaces: options, bits, exponent, modulus, comment. The options field is optional; its presence is determined by whether the line starts with a number or not (the option field never starts with a number). The bits, exponent, modulus and comment fields give the RSA key; the comment field is not used for anything (but may be convenient for the user to identify the key). Note that lines in this fileare usually several hundred bytes long (be cause of the size of the RSA key modules). You don't want to type them in; instead, copy the identity.pub file and edit it. CREATING USER SPECIFIC KEY FILES With each user that you want to have access to the SSH server for, run /usr/local/bin/ssh-keygen on the machine you are going to be ssh'ing from. After running ssh-keygen, goto $HOME/.ssh/ and cp identity.pub -> authorized_keys KEY LOCATION OVERVIEW The system you want to login to (the SSH Server) is where you store the $HOME/.ssh/authorized_keys file. This is the public key file for your user. The server will require this to decrypt your private key on your host machine. The system you are logging in FROM is where you store the $HOME/.ssh/identity file. This is the private key file you generated. NOTES For those of you wishing to use SSH v2 (be careful, it's a pretty new version. I'd stick with v1 to be stable and safe). [root@myhost /usr/local/bin]# ./ssh-keygen -d This -d flag will create the DSA Public Keyfile needed for DSA Authentication. (When generating host keys, you can not specify a password) Once you have generated your key files, go ahead and plop your sshd module file into the /etc/pam.d/ directory. You are probably wondering, jeese, will Eric provide me with a working sshd module? Well the answer is YES :) See below ---- snip ----- #%PAM-1.0 auth required /lib/security/pam_unix.so shadow nodelay auth required /lib/security/pam_nologin.so account required /lib/security/pam_unix.so password required /lib/security/pam_cracklib.so password required /lib/security/pam_unix.so shadow nullok use_authtok session required /lib/security/pam_unix.so session required /lib/security/pam_limits.so ----- snap ----- Remember, this is being used on my RH 6.2 box only. If it does not work, consult the genericcopy provided with the SSH distribution. OVERVIEW Now you should have ALL services turned off and SSH as the only means of connecting remotely to the machine for administration purposes. The next step we want to do is setup that infamous host-based Firewall! 4. THE FIREWALL Packet filtering doesn't really make sense on your logging box until you've got your tcp stack vulnerable to some fragmentation issue, since you have got all the services down anyway and you can control who is permitted to establish secure shell connections to the machine using ssh access control features. It might be reasonable though to filter out syslog port (514/udp) but it doesn't really stop people from sending garbage to your syslog server, however it does make their job harder since it's udp which could be easily spoofed. This is at least until you have a multihomed machine where all logs come to internal interface so you can safely filter out everything that comes from the outside. However here are basic recommended filtering rules for machine. For this purpose you can use either ipchains package if you're using Linux (last time I checked ipfilter didn't compile cleanly there) or try to use ipfilter package if you use something else (works perfectly on *BSD and Solaris machines at least). For ipchains I recommend the following shell script. #!/bin/sh PATH=/usr/sbin:/sbin:/bin:/usr/sbin LOCAL_INTERFACE="192.168.1.1/32" # put your real IP address LOCAL_NETWORK="192.168.1.0/24" # put your local net IP address/mask here SSH_PERMITTED="192.168.1.2/32 192.168.2.3/32" # who allowed to ssh SYSLOG_PERMITTED="192.168.1.5/32 192.168.5.2/32" # who allowed to log syslog messages # deny everything ipchains -P input DENY ipchains -P output DENY ipchains -P forward DENY ipchains -F #permit ssh for ipaddr in $SSH_PERMITTED; do ipchains -A input -p tcp -s $ipaddr -d0/0 22 -i $LOCAL_INTERFACE -j ACCEPT done # permit outgoing tcp ipchains -A output -p tcp -i $LOCAL_INTERFACE -j ACCEPT ipchains -A input -p tcp ! -y -i $LOCAL_INTERFACE -j ACCEPT # permit syslog for ipaddr in $SYSLOG_PERMITTED; do ipchains -A input -p udp -s $ipaddr -d $LOCAL_INTERFACE 514 -i $LOCAL_INTERFACE -j ACCEPT done # if you would like to log all the other connection attempts, # uncommend these... #ipchains -A input -p tcp -i $LOCAL_INTERFACE -l -j DENY #ipchains -A input -p udp -i $LOCAL_INTERFACE -l -j DENY #ipchains -A input -p icmp -i $LOCAL_INTERFACE -l -j DENY For IP filter package, rules would look approximately like this: # close everything on local interface block in all on le0 from any to any # pass secureshell pass in on le0 proto tcp from 192.168.1.2/32 to 192.168.1.1/32 port = 22 pass out on le0 proto tcp from 192.168.1.2/32 to 192.168.1.1/32 port = 22 # or you can replace these two rules with #pass in on le0 proto tcp from 192.168.1.2/32 to 192.168.1.1/32 port = 22 keep state #pass SYSLOG pass in on le0 proto udp from 192.168.1.2/32 to 192.168.1.1/32 port = 514 5. LOG REPORTING What good would a remote log server be if you weren't monitoring the log files. I would recommend several different programs. Logcheck - Swatch - Below we've included a simple shell script (Courtesy of Mr. Bill Pennington) Which will archive the log files every day/hour/minute to an offsite location (using scp of course). #!/bin/bash #Simple script to rotate the log files on a daily basis #Bill Pennington 1/19/2000 #Set the date variable date=`date +%m-%d-%Y` #Rename the messages file mv /var/log/messages /var/log/messages.$date #HUP the syslog daemon so it writes to a new file killall -HUP syslogd #Compress the file /bin/gzip /var/log/messages.$date #Rename the secure file mv /var/log/secure/var/log/secure.$date #HUP the syslog daemon so it writes to a new file killall -HUP syslogd #Compress the file /bin/gzip /var/log/secure.$date #Rename mail file mv /var/log/maillog /var/log/maillog.$date #HUP the syslog daemon so it writes to a new file killall -HUP syslogd #Compress the file /bin/gzip /var/log/maillog.$date #Then scp them somewhere 6. TIME It is vital to ensure that the logserver has the correct time/date. Get xntpd install it, then "ntpdate timeservername" at least once a day, I do it twice a day. If you want to use your logs in any type of legal proceeding you are going to need accurate time. You should be doing this on all your servers of course. 7. OTHER SYSLOG DEVICES Their are many other devices including CISCO that can be setup to log to syslog. Having all your cool *nix boxes logging to a central log server is way cool, but what about those other boxes on your network? To get a complete view of your network and servers, setup all your devices to log to syslog. Depending on the number of devices you have, you might want to dedicate a box for each type of device or server. i.e. unix log server, netgear log server, NT log server... Cisco All Cisco devices support logging to syslog. The commands to set your Cisco device to log to log messages to syslog vary from device to device. Some examples are provided here but you should check your Cisco documentation for more complete instructions. Routers In config mode... logging logging facility defaults to LOCAL7 Pix Firewall In config mode... logging host logging facility defaults to LOCAL4 logging trap from emergencies to debug , be carefull with debug you will get a ton of traffic! Switches In config mode... set logging server enable set logging server set logging session enable setlogging level all 1 default Again you will want to look at your Cisco docs or Cisco's web site for complete details but the above examples should get you started. Windows NT You can make NT servers forward events to central syslog servers as well. There are several shareware/freeware packages available to accomplish this. Search https://www.bhs.com/ for syslog to find one that is right for you. D. RESOURCES Swatch - OpenSSH - https://www.openssh.org/ LogCheck - xntp and list of public NTP servers - https://www.eecis.udel.edu/~ntp/ Cisco - https://www.cisco.com/ Windows NT software - https://www.bhs.com/ Robert Graham's sniffing FAQ (hints to build `receive-only device' E. CONTRIBUTIONS Lance Spitzner (Renowned security whitepaper author of the 'Know Your Enemy' series. Thank you for all your support Lance. Martin Roesch (Creator of the popular Intrusion Detection System (SNORT)). Had a blast with you at Defcon! Hope to see you next year. https://www.snort.org/ Bill Pennington (Thanks for all the help with SNORT and your contributions to this paper!) Good luck with everything! Frank Heidt (@Stake Security Team - Enjoyed the cigar Frank! Hope to see you at Defcon next year as well. Sorry I didn't end up implementing your syslog design into this paper. I didn't think I could explain as perfectly as you could;)) Fyodor Yarochkin (This paper wouldn't have been possible without you bud! Thanks!) F. SHOUTS Lamagra, Safety, #RootHat, zen-parse, lockdown, Crimson, Invader, LoA, Fyodor, Faisal Jawdat, Art Stine, Vince Pereira, Ted Barban, Malverian, Frank Heidt, Bill Pennington, Marty, Lance.. and all the rest. "TIMMMMMY!" G. CONTACT INFORMATION Eric Hines
Get the latest Linux and open source security news straight to your inbox.