Complete Reference Guide to Creating a Remote Log Server
12 - 23 min read
Aug 22, 2000
AUTHORS BIOS
Eric Hines
As a 7 1/2 year member of the network security industry, Eric
currently provides security consulting to many Fortune 500 to Fortune
50 Corporations across the US. Eric also gives keynote speakings on
information security at numerous different E-Commerce and E-Banking
seminars. Eric has deployed several VPN Extranets from British Columbia
to the United States, as well as plays active roles in contributions to
many security-related mailing lists. Eric enjoys creating custom attack
signatures using packet analyzers for Intrusion Detection Systems such
as SNORT, and holds strong interest in Intrusion Detection technology.
Fyodor Yarochkin
Works for TRG ( ). Personal information available at: .
INTRODUCTION
Welcome to the first issue of a series of security-related HOWTO papers. In these documents both Fyodor and I aimed to give various tips, hints and tricks on various aspects of Internet security. Comments and suggestions are welcome and can be emailed to us directly at the contact information provided at the bottom of this paper.
We hope you find it useful and recommend hammering away at the README files for all utilities being referenced in this paper.
In this paper we make references to <SERVER> and <CLIENT> as headers. In this, SERVER, refers to nothing more then the machine that is configured to receive logs from remote machines (CLIENTS). CLIENT, referring to the machine that is configured to send its logs to the REMOTE LOG SERVER.
- Eric Hines
A. WHAT IS A REMOTE LOG SERVER
A remote log server is nothing more then a system preconfigured at install-time to provide hard drive space for other systems to log to. This system must be completely secured and locked down. No unencrypted remote access should be allowed, all RPC Daemons and other misc. services should be turned off as well. The only data allowed to the machine should be UDP/Port 514. We will be walking you through a step-by-step process that details how to configure, install, and deploy a remote log server. Utilizing some of the most renowned security experts across the globe for input, I've compiled a comprehensive, and easy to understand guide on ensuring this to be a successful launch.
B. SYSLOGD
Recompiling syslogd
The first step in turning on remote logging capabilities on your
machine is recompiling syslogd to read a completely different
syslog.conf file. This is for security reasons, which we will detail
more later in this paper.
NOTES
This was an idea derived from the "To Build a Honeypot" whitepaper
written by Lance Spitzner. Thanks for your contributions to the
security community, Lance. The beautiful idea behind this strategy is
to fool the intruder into thinking there is no remote logging setup on
the machine. By leaving the old /etc/syslog.conf file in tact, also
helps in tricking the unsuspecting cracker.
STEP 1: COMPILING OUR NEW VERSION
In our first step, we face somewhat of a problem in that syslogd ships with the operating system by default, so no source code is provided with it. You will need to download the source code from an outside party, such as Red Hat, or even Joey's FTP site (the creator of syslogd). As of this writing, Joey's ftp site is down so I have provided an alternative resource to download the needed file. Your distribution however, would be in violation of Open Source policies by not providing the source code to you. So check your particular OS vendor for the source code if you do not want to download it from any of the links specified in this paper.
SITE 1: (currently down as of this writing)
or
SITE 2: Hat/Red Hat/Red Hat-6.2/SRPMS/SRPMS/sysklogd-1.3.31-16.src.rpm
After 'rpm -iv -v sysklogd-1.3.31-16.src.rpm, the source code should have been installed to /usr/src/Red Hat/SOURCES/sysklogd-1.3-31 (that is, if you are running Red Hat)
STEP 2: CHANGING THE DEFAULT CONFIG FILE LOCATION
Here is the fun part. This is used to fool an intruder once he has compromised the system into thinking your log files are stored locally. We will be recompiling syslogd to read a completely separate configuration file, leaving the old default one in tact, just in case the intruder runs a 'cat' on the /etc/syslogd.conf file to check if remote logging has been enabled. By leaving the old configuration there proceeding our recompile of the new binary, ensures that we can fool the cracker into thinking otherwise.
Lets modify the default location of /etc/syslog.conf
[ehines@myhost sysklogd-1.3-31]$ cd /usr/src/Red Hat/SOURCES/sysklogd-1.3-31 [ehines@myhost sysklogd-1.3-31]$ vi syslogd.c
do a /grep for syslog.conf
This will bring us to
#define _PATH_LOGCONF "/etc/syslog.conf"
Change the above location value to something you prefer. I suggest something like /etc/.sys/CORE.conf or something ingenious like that.
Copy the new syslogd binary over the old (typically /sbin/syslogd)
BACKUP old version before doing this. As an admin, you should have already known that, right? :)
NOTES
The reason why we do not simply just start the old syslogd with the
(-f conffile) switch is because if an attacker were to grab a 'ps' list
of processes, the switch you used will show up, telling the intruder
that we are specifying a completely different configuration file.
Ultimately, defeating this purpose. I recommend leaving the old
/etc/syslog.conf file in tact as it completely throws off the intruder.
Also, keep local logs, as if the /var/log directory is empty, well, use
your imagination on why.
I would have to make the suggestion to use a different filename, other then syslog.conf when specifying the new location. Understand that an intruder can also just do a find / -name "syslog.conf" and locate another copy of the configuration file.
STEP 3: MODIFY THE (REAL) SYSLOG CONFIG FILE
For example, to forward ALL messages to a remote host use the following syslog.conf entries:
# Sample syslogd entry to forward all messages to a remote host. *.* @hostname # To forward all kernel messages to a remote host the configuration file would be as follows: kern.* @hostname # Remote Logging which includes local logging as well (for fault tolerance) kern.crit @finlandia kern.crit /dev/console
.. and so on.. repeat for all log files you wish to keep offsite. This includes third party applications.
C. THE CLIENT (REMOTE LOG SERVER)
With the client server, (or the machine accepting all of the remote logs), all you have to do is launch syslogd with the -r switch, specifying remote logging capabilities. Make sure that you remember to ADD: syslog 514/udp to the /etc/services file as well, on the server, or it will not listen to the specified port.
NOTES
If the network consists of different domains you don't have to
complain about logging fully qualified names instead of simple
hostnames. You may want to use the strip-domain feature -s of this
server. You can tell the syslogd to strip off several domains other
than the one the server is located in and only log simple hostnames.
ADD 'syslog 514/udp'to the /etc/services file.
If this entry is missing syslogd neither can receive remote messages nor send them
ADD '*.* @hostname' to the /etc/syslog.conf file.
To forward all Kernel logging messages to a remote host,
Make sure that if you do not specify an IP ADDRESS that you add the host to the /etc/hosts file to avoid DNS RESOLUTION errors.
If the remote host is located in the same domain as the host, syslogd is running on, only the simple hostname will be logged instead of the whole fqdn. (fully qualified domain name)
D. LOCKING DOWN THE REMOTE LOG SERVER
One of the most crucial steps in this project is to fully lock down the machine that is receiving all of the logs, or referred to as the remote log server.
1. Turn off (ALL SERVICES)
Remember, this is ONLY a log server, so all other services should NOT be turned on. Lets walk through that process right now.
[root@myhost /etc]# cd /etc [root@myhost /etc]# vi inetd.conf
1a. TURN OFF ALL INETD SERVICES
You will want to put a '#' comment in front of EVERY service listed in this file. The inetd.conf file is IMHO one of the largest mistakes of the Linux operating system :). But the ability to disable the inetd services makes it all better :) You will want to comment out everything:
a. echo
b. discard
c. daytime
d. chargen
e. time
f. ftp
g. telnet
h. shell
i. login
j. exec
k. comsat
l. talk
m. ntalk
n. dtalk
o. pop-2
p. pop-3
q. imap
r. uucp
s. tftp
t. bootps
u. finger
v. cfinger
w. systat
x. netstat
y. auth
z. linuxconf
zz. swat
1b. DISABLE ALL RPC SERVICES
[root@myhost /etc]# cd /etc/rc.d
Anything listed in the rc.
directories that start with a capital "S" means that the service will
start at boot time. In order to disable it from starting, you will want
to 'mv
You will want to do this with all unneeded services in this directory. These include: nfslock, apmd, netfs, identd, autofs, portmap, atd, pcmcia, I would even disable the inetd startup script as well, isdn, sendmail, gpm, httpd, vmware, xfs,linuxconf, local, You are basically (other then the obvious services) are going to want to disable all of these rc controlled services.You will want to go through all of the rc0.d-rc6.d directories looking for these same files to disable.
2. DISABLE ACCOUNTS
[root@myhost /etc]# vi /etc/passwd
Go through your password file and remove practically every account that isn't being used. I would suggest downloading the tool, /bin/noshell (https://www.cerias.purdue.edu/about/history/coast/archive/data/categ50.html) and use this with all accounts in the password file that should not have remote login access. This is a very informative program that provides admins additional information on who is attempting to login with disabled accounts.
3. INSTALL SSH
Since the release of OpenSSH, giving the GNU opensource community free access to secure shell, telnet should now be obsolete in a security centric environment. GET AWAY from using telnet. With packet sniffers getting easier and easier to use (tools such as dsniff, 'www.dsniff.org') allows even the biggest nitwit to snarf packet data. So case-in-point, don't use telnet.
I bet you are so happy, because guess what? I'm going to walk you through the installation of OpenSSH! (ohh goody)..
3a. DOWNLOADING OPENSSH
Point your web browser to www.openssh.com, as of this writing they are on version 2.1.1 which has support for both SSH1 and SSH2 protocols. The release date was June 08, 2000. After downloading the source tarball, go ahead and do the following:
[root@myhost ]# gzip -d openssh-2.1.1p2.tar.gz [root@myhost ]# tar -xvf openssh-2.1.1p2.tar
I can't realistically walk you through everything, so please save me some heartache, as well as yourself, and just read the INSTALL file before continuing. I have no idea what particular requirements your system has. I will however, walk you through my install.
First you will need working installations of both Zlib and OpenSSL (yes there is an OpenSSL project as well)
Zlib:
freesoftware
OpenSSL 0.9.5a or greater:
/index.html
Or, if you're even more lazy then the audience this paper was written for, (God forbid), then grab the RPM's from the link below. RPMs of OpenSSL are available at
Alternatively, Jim Knoble
If you wish to build the GNOME passphrase requester, you will need the GNOME libraries and headers.
GNOME:
GNOME – Simple, beautiful, elegant.
Also, grab GNUmake if you do not yet have it, OpenSSH has only been tested with this program.
[root@myhost ]# ./configure && make && make install
This will install the binaries in /opt/{bin,lib,sbin}, but will place the config files in /etc/ssh
If you are using PAM, you will need to manually install a PAM control file as "/etc/pam.d/sshd" (or wherever your system prefers to keep them). A generic PAM configuration is included as "contrib/sshd.pam.generic", you may need to edit it before using it on your system. If you are using a recent version of Red Hat Linux, the config file in contrib/Red Hat/sshd.pam should be more useful. Failure to install a valid PAM file may result in an inability to use password authentication.
I like to run configure with '--without-pam' to disable PAM support. PAM is automatically detected and switched on if found. This is because I opt to use RSA Authentication instead.
[root@myhost ]# tar -xvf zlib.tar [root@myhost ]# cd zlib-1.1.3 [root@myhost zlib-1.1.3]# ./configure && make && make install
You shouldn't get any errors, if you do, sorry to do this, but refer to the README file. What I like to do when compiling something and something goes wrong, I just grep a keyword from the error on the provided README file(s) to see if there is anything specifically mentioned about my error. (Just a suggestion)
To install OpenSSL, you will need:
- Perl 5
- an ANSI C compiler
- a supported Unix operating system
[root@myhost ]# tar -xvf openssl-0.9.5a.tar [root@myhost ]# ./config && make && make test && make install
Now lets build OpenSSH!
[root@myhost ]# cd openssh-2.1.1p2 [root@myhost ]# ./configure --without-PAM && make && make install
NOTES
Because I opt for RSA authentication, I run configure with the
--without-PAM option as I am not a big fan of basic password
authentication. With the advancements in Cryptography and more secure
forms of authentication, PAM and other password-type authentication
mechanisms IMHO will become obsolete within a few years
When you have successfully compiled and installed OpenSSH, you will receive the following information.
Key generation complete. Your identification has been saved in /usr/local/etc/ssh_host_key. Your public key has been saved in /usr/local/etc/ssh_host_key.pub. The key fingerprint is: d5:74:83:d0:3f:c4:b4:d6:c5:39:1d:94:ee:9b:a8:61This email address is being protected from spambots. You need JavaScript enabled to view it. Generating DSA parameter and key. Your identification has been saved in /usr/local/etc/ssh_host_dsa_key. Your public key has been saved in /usr/local/etc/ssh_host_dsa_key.pub. The key fingerprint is: ed:58:65:b9:8b:fe:05:81:c2:8c:06:c9:cb:ac:bb:e6This email address is being protected from spambots. You need JavaScript enabled to view it.
3b. CONFIGURING OPENSSH
Even though the default installation of OpenSSH should work, I would take a quick stab at the ssh config files to see if you want to change anything.
Because we configured OpenSSH without PAM support, we want to enable RSA Authentication. Uncomment it in the /usr/local/etc/ssh_config file respectively (this is for the system-wide client configuration).
Then go ahead and edit the SSHD config file, /usr/local/etc/sshd_config
Lets go ahead and disable PermitRootLogin (do this by changing 'yes' to 'no' Go ahead and save the file and quit.
After installation, a host key and DSA key should have been created for you. So lets go ahead and move to the next step.
NOTES ON SSH v2
THE SSH PROTOCOL v2 SSH protocol version 2
In SSH v2, each host has a host-specific DSA key used to identify
the host. However, when the daemon starts, it does not generate a
server key. Forward security is provided through a Diffie-Hellman key
agreement. This key agreement results in a shared session key. The rest
of the session is encrypted using a symmetric cipher, currently
Blowfish, 3DES or CAST128 in CBC mode or Arcfour. The client selects
the encryption algorithm to use from those offered by the server.
Additionally, session integrity is provided through a cryptographic
message authentication code (hmac-sha1 or hmac-md5).
Protocol version 2 provides a public key based user authentication method (DSAAuthentication) and conventional password authentication.
AUTHORIZED_KEYS FILE FORMAT
The $HOME/.ssh/authorized_keys file lists the RSA keys that are
permitted for RSA authentication in SSH protocols 1.3 and 1.5
Similarly, the $HOME/.ssh/authorized_keys2 file lists the DSA keys that
are permitted for DSA authentication in SSH protocol 2.0. Each line of
the file con tains one key (empty lines and lines starting with a `#'
are ignored as comments). Each line consists of the following fields,
separated by spaces: options, bits, exponent, modulus, comment. The
options field is optional; its presence is determined by whether the
line starts with a number or not (the option field never starts with a
number). The bits, exponent, modulus and comment fields give the RSA
key; the comment field is not used for anything (but may be convenient
for the user to identify the key).
Note that lines in this file are usually several hundred bytes long (be cause of the size of the RSA key modules). You don't want to type them in; instead, copy the identity.pub file and edit it.
CREATING USER SPECIFIC KEY FILES
With each user that you want to have access to the SSH server for,
run /usr/local/bin/ssh-keygen on the machine you are going to be
ssh'ing from. After running ssh-keygen, goto $HOME/.ssh/ and cp
identity.pub -> authorized_keys
KEY LOCATION OVERVIEW
The system you want to login to (the SSH Server) is where you store
the $HOME/.ssh/authorized_keys file. This is the public key file for
your user. The server will require this to decrypt your private key on
your host machine. The system you are logging in FROM is where you
store the $HOME/.ssh/identity file. This is the private key file you
generated.
NOTES
For those of you wishing to use SSH v2 (be careful, it's a pretty new version. I'd stick with v1 to be stable and safe).
[root@myhost /usr/local/bin]# ./ssh-keygen -d
This -d flag will create the DSA Public Keyfile needed for DSA Authentication. (When generating host keys, you can not specify a password)
Once you have generated your key files, go ahead and plop your sshd module file into the /etc/pam.d/ directory. You are probably wondering, jeese, will Eric provide me with a working sshd module? Well the answer is YES :) See below
---- snip ----- #%PAM-1.0 auth required /lib/security/pam_unix.so shadow nodelay auth required /lib/security/pam_nologin.so account required /lib/security/pam_unix.so password required /lib/security/pam_cracklib.so password required /lib/security/pam_unix.so shadow nullok use_authtok session required /lib/security/pam_unix.so session required /lib/security/pam_limits.so ----- snap -----
Remember, this is being used on my RH 6.2 box only. If it does not work, consult the generic copy provided with the SSH distribution.
OVERVIEW
Now you should have ALL services turned off and SSH as the only
means of connecting remotely to the machine for administration
purposes. The next step we want to do is setup that infamous host-based
Firewall!
4. THE FIREWALL
Packet filtering doesn't really make sense on your logging box until you've got your tcp stack vulnerable to some fragmentation issue, since you have got all the services down anyway and you can control who is permitted to establish secure shell connections to the machine using ssh access control features.
It might be reasonable though to filter out syslog port (514/udp) but it doesn't really stop people from sending garbage to your syslog server, however it does make their job harder since it's udp which could be easily spoofed. This is at least until you have a multihomed machine where all logs come to internal interface so you can safely filter out everything that comes from the outside.
However here are basic recommended filtering rules for machine. For this purpose you can use either ipchains package if you're using Linux (last time I checked ipfilter didn't compile cleanly there) or try to use ipfilter package if you use something else (works perfectly on *BSD and Solaris machines at least).
For ipchains I recommend the following shell script.
#!/bin/sh
PATH=/usr/sbin:/sbin:/bin:/usr/sbin
LOCAL_INTERFACE="192.168.1.1/32" # put your real IP address
LOCAL_NETWORK="192.168.1.0/24" # put your local net IP address/mask here
SSH_PERMITTED="192.168.1.2/32 192.168.2.3/32" # who allowed to ssh
SYSLOG_PERMITTED="192.168.1.5/32 192.168.5.2/32" # who allowed to log syslog messages
# deny everything
ipchains -P input DENY
ipchains -P output DENY
ipchains -P forward DENY
ipchains -F
#permit ssh
for ipaddr in $SSH_PERMITTED;
do
ipchains -A input -p tcp -s $ipaddr -d 0/0 22 -i $LOCAL_INTERFACE -j ACCEPT
done
# permit outgoing tcp
ipchains -A output -p tcp -i $LOCAL_INTERFACE -j ACCEPT
ipchains -A input -p tcp ! -y -i $LOCAL_INTERFACE -j ACCEPT
# permit syslog
for ipaddr in $SYSLOG_PERMITTED;
do
ipchains -A input -p udp -s $ipaddr -d $LOCAL_INTERFACE 514 -i $LOCAL_INTERFACE -j ACCEPT
done
# if you would like to log all the other connection attempts,
# uncommend these...
#ipchains -A input -p tcp -i $LOCAL_INTERFACE -l -j DENY
#ipchains -A input -p udp -i $LOCAL_INTERFACE -l -j DENY
#ipchains -A input -p icmp -i $LOCAL_INTERFACE -l -j DENY
For IP filter package, rules would look approximately like this:
# close everything on local interface block in all on le0 from any to any # pass secureshell pass in on le0 proto tcp from 192.168.1.2/32 to 192.168.1.1/32 port = 22 pass out on le0 proto tcp from 192.168.1.2/32 to 192.168.1.1/32 port = 22 # or you can replace these two rules with #pass in on le0 proto tcp from 192.168.1.2/32 to 192.168.1.1/32 port = 22 keep state #pass SYSLOG pass in on le0 proto udp from 192.168.1.2/32 to 192.168.1.1/32 port = 514
5. LOG REPORTING
What good would a remote log server be if you weren't monitoring the log files. I would recommend several different programs.
- Logcheck - www.psionic.com
- Swatch - www.swatch.org
Below we've included a simple shell script (Courtesy of Mr. Bill Pennington) Which will archive the log files every day/hour/minute to an offsite location (using scp of course).
#!/bin/bash #Simple script to rotate the log files on a daily basis #Bill Pennington 1/19/2000 #Set the date variable date=`date +%m-%d-%Y` #Rename the messages file mv /var/log/messages /var/log/messages.$date #HUP the syslog daemon so it writes to a new file killall -HUP syslogd #Compress the file /bin/gzip /var/log/messages.$date #Rename the secure file mv /var/log/secure /var/log/secure.$date #HUP the syslog daemon so it writes to a new file killall -HUP syslogd #Compress the file /bin/gzip /var/log/secure.$date #Rename mail file mv /var/log/maillog /var/log/maillog.$date #HUP the syslog daemon so it writes to a new file killall -HUP syslogd #Compress the file /bin/gzip /var/log/maillog.$date #Then scp them somewhere
6. TIME
It is vital to ensure that the logserver has the correct time/date. Get xntpd install it, then "ntpdate timeservername" at least once a day, I do it twice a day. If you want to use your logs in any type of legal proceeding you are going to need accurate time. You should be doing this on all your servers of course.
7. OTHER SYSLOG DEVICES
Their are many other devices including CISCO that can be setup to log to syslog.
Having all your cool *nix boxes logging to a central log server is way cool, but what about those other boxes on your network? To get a complete view of your network and servers, setup all your devices to log to syslog. Depending on the number of devices you have, you might want to dedicate a box for each type of device or server. i.e. unix log server, netgear log server, NT log server...
Cisco
All Cisco devices support logging to syslog. The commands to set
your Cisco device to log to log messages to syslog vary from device to
device. Some examples are provided here but you should check your Cisco
documentation for more complete instructions.
Routers
In config mode...logging <server ip address or name>
logging facility <facility> defaults to LOCAL7
Pix Firewall
In config mode... logging host <ip address of syslog server> logging facility <facility> defaults to LOCAL4 logging trap <level> from emergencies to debug , be carefull with debug you will get a ton of traffic!
Switches
In config mode...set logging server enable
set logging server <ip address of log server>
set logging session enable
set logging level all 1 default
Again you will want to look at your Cisco docs or Cisco's web site for complete details but the above examples should get you started.
Windows NT
You can make NT servers forward events to central syslog servers as
well. There are several shareware/freeware packages available to
accomplish this. Search https://www.bhs.com/ for syslog to find one that is right for you.
D. RESOURCES
- Swatch -
- OpenSSH - http://www.openssh.com
- LogCheck -
- xntp and list of public NTP servers - https://www.eecis.udel.edu/~ntp/
- Cisco - https://www.cisco.com/
- Windows NT software - https://www.bhs.com/
- Robert Graham's sniffing FAQ (hints to build `receive-only device'
E. CONTRIBUTIONS
Lance Spitzner (Renowned security whitepaper author of the 'Know Your Enemy' series. Thank you for all your support Lance.
Martin Roesch (Creator of the popular Intrusion Detection System (SNORT)). Had a blast with you at Defcon! Hope to see you next year. https://www.snort.org/
Bill Pennington (Thanks for all the help with SNORT and your contributions to this paper!) Good luck with everything!
Frank Heidt (@Stake Security Team - Enjoyed the cigar Frank! Hope to see you at Defcon next year as well. Sorry I didn't end up implementing your syslog design into this paper. I didn't think I could explain as perfectly as you could;))
Fyodor Yarochkin (This paper wouldn't have been possible without you bud! Thanks!)
F. SHOUTS
Lamagra, Safety, #RootHat, zen-parse, lockdown, Crimson, Invader, LoA, Fyodor, Faisal Jawdat, Art Stine, Vince Pereira, Ted Barban, Malverian, Frank Heidt, Bill Pennington, Marty, Lance.. and all the rest. "TIMMMMMY!"
G. CONTACT INFORMATION
Eric Hines
