Cylant's "0wn this box challenge" is a research effort providing a good challenge to skilled crackers. Dave Wreski and Benjamin Thomas of interview Cylant Technology, Inc. seeking to find more information regarding the work to create this contest, and the experiences gained from holding it.

The victim system has been setup using a Red Hat 6.2 machine with every possible service and package installed.

Unique about this contest is that Cylant is not a security company. They are a behavioral measurement and control company. They believe they have a product that can monitor the behavior of a system, and use this information to counteract the actions of an intruder not determined to be normal behavior.

Behavior is monitored using the reports generated by CylantSecure for Linux. The successful cracker will receive the actual victim box which sports a 850 MHz Athlon, 256 MB Ram, 20 GB disk, and ATI Rage 128 video card. Good luck to those who try! What is Cylant trying to show by allowing people to attack your server?

Cylant: We are demonstrating one of the capabilities presented through execution behavioral measurement and control. Other security systems rely on either specific or general knowledge about attacks, our approach is very different. We calibrate the system for its normal behavior. In the case of, the normal behavior of a web server and some standard system administration activity are what constitute normal behavior. The behavior of the system is compared in real-time against the behavior expressed by the baseline.

CylantSecure, which is the only thing securing victim, is a demonstration of the behavioral measurement technology that we have developed. Victim demonstrates that this technology is not only viable, but directly applicable when it comes to rejecting intrusion attempts.

The core technology is described here: Do you see this as another 'Hack This Box' contest?

Cylant: To a certain extent this is exactly another 'Hack This Box' contest. The nature of the target box sets this apart. First, victim represents every security admin's nightmare. Most of the running services have known exploits. Further, victim is running many unnecessary services. This doesn't seem to be the nature of most target systems in 'Hack this Box' challenges.

Secondly, the system securing the box knows nothing about security or the vulnerabilities in the software running on victim. This system is the first we know of that uses execution behavioral measurement. This approach is not dependent on signatures or rule-sets that must be constantly updated to keep current with newly discovered vulnerabilities.

There is one fundamental similarity, eventually somebody will succeed. Given the massive misconfiguration of the box, it is inevitable that somebody will successfully own victim. Our interest is in discovering the resolution limitations in CylantSecure's default settings in a very hostile environment.


Victim was hacked by some of my old co-workers at EarthLink/Mindspring. They succeeded in part because of a bug we found today in CylantSecure. We have fixed the bug and issued round two of the challenge. Part of the challenge for us, was watching this attack proceed over a period of hours. Not deleting a setuid shell for hours was kind of hard to do.

Same challenge, same goal, but this time the CylantSecure isn't having suicidal tendencies. For details on the bug, see: What kernel modifications have you made?

Cylant: First, let me state that the source code for the modified kernel is available on our web site. Hat/kernels/ The kernels for the Advanced and SOHO versions are contained in the downloads for those products. Activation for SOHO and Advanced requires a license key.

We changed the kernel source in four ways. First, we inserted about 3300 points of instrumentation throughout the source for the kernel. These instrumentation points are the source of the execution behavioral data that we both store into the baseline and measure. Secondly, we have added a few elements to the task_struct and sk_buff structures in the kernel. We use these to identify the cause of the behavior we are seeing at the instrumentation points. Thirdly, we have added code to start the profiling process when a process is execed or an IP packet is handed from the device driver to ip_rcv(). And finally, we have added a quick check in ip_rcv() against a table of banned IP addresses. This lets us drop packets from banned hosts very early. What other changes have been made?

Cylant: We load kernel modules that gather the instrumentation data into profiles and ship it on to our statistical engine. What are the other components of your product?

Cylant: We have a user land daemon that is essentially a multiplexor. It takes incoming behavioral profiles and passes them to the statistical engine for measurement against the baseline. There is also a management console which provides running reports and configures the CylantSecure core. Communication with the console happens through SSL. Why did you choose Linux?

Cylant: We chose Linux for a number of reasons. The principle reason was that it was a sufficiently large and complex piece of software. The core technology is applicable to any piece of software. The Linux kernel provided a good demonstration ground. Implicit in this is the fact that the source was available for us to instrument. We chose Linux over other open source operating systems because it has the most mind share.

We chose to instrument a kernel because doing so imposed a high reliability requirement on the instrumentation and profiling process we developed. Debugging kernel crashes might be a real pain, but you have a real hard time labeling the bug a "feature" and going on with the day. When instrumenting other applications, such as apache, we didn't have to deal with such a high reliability requirement.

Another benefit of choosing a kernel was that it reduced the amount of options we had in terms of behavioral profiling. This refers to the number of different ways of attributing behavior to its cause. For example, activity in the IP stack can be "blamed" on the source IP and port in the packet traversing the stack. Behavior underneath the system calls can be tagged with the process ID in the task structure passed in and out of these functions. Naturally, you can spoof IP addresses and have the system not know the real cause of the behavior. I don't know of a good way around that other than getting access points to fix their ACLs. Your page indicates that if your software detects anomalous behavior, the process associated with it will be killed. What is the definition of anomalous behavior?

Cylant: Anomalous and normal behavior are a function of the role a system has been deployed in. For victim, the normal behavior is serving web pages. Additionally, it is normal for system administrator to be tailing log files, and building the auto-generated portions of the victim site. That is pretty much the extent of "normal" behavior. Abnormal behavior is anything else.

The difference between normal and abnormal is not a bold black line though. What is really of interest is the difference between normal behavior, and the behavior currently occurring. This is where the behavioral measurement aspect of our approach applies. We set thresholds on the allowed difference from normal; anything over the thresholds is stopped. For victim we have set these thresholds rather low. How much overhead is associated with your approach?

Cylant: Typically the overhead is less than 1%. At times, the overhead can have as much as a 5% impact on the performance. The advanced version has a 40 MB memory footprint. The memory footprint is large because that is where the profiles are generated by the kernel. In the SOHO version of watcher, the memory footprint is less than 10 MB since it profiles less concurrent behavior. What other applications do you think your software has?

Cylant: Our background is in the area of software reliability. The behavioral measurement technology was developed to measure the reliability of running software. By generating the baseline during testing, you can tell if the software is being used in a tested manner. This lets you create a set of certified behavior for a system, and know when the system departs from this behavioral set in the field. Additionally, if the cause of the behavior can be determined, the behavior could be stopped. Ideally a truly hardened software system would actively work to keep the execution behavior within both a defined normal behavioral set, and a defined certified behavioral set. What is the average amount of time that you believe a person will spend trying to compromise your system?

Cylant: It depends on how determined the attacker is. We have had some people try for several hours, and others try for multiple days. On the other hand, I suspect that some people have given up after their first try. A full analysis of the logs would be required to really know for sure. The spoofable nature of IPv4 makes even the log analysis less than 100% reliable for determining this. How closely do you monitor their actions?

Cylant: The answer sort of depends on what is meant by "you". CylantSecure monitors essentially all the behavior their actions induce on the kernel. Humanly, we don't pay that much attention to victim. We tend to watch the page just to see what people have been trying to do. Periodically we check to see if somebody has created an /etc/owned_ file. That is about all we are doing in the way of monitoring attacks against victim.

This lack of observation really stems from our goal of wanting to create a simple means for people to protect their servers from attacks. If we had to watch victim like a hawk, then we would not really have accomplished anything with CylantSecure. How much detailed information will be made publicly available if the system is compromised?

Cylant: We are going through the log files right now to put together a trace from the successful compromise by the EarthLink/MindSpring group. The information they provided about their method has helped us track their activity through the log files. We will be releasing all of the information we have garnered from the log files relating to their successful attack. What do you believe the most common method people will use to find vulnerabilities on your system?

Cylant: We have spent a fair amount of time thinking about how to circumvent our techniques. The first way that comes to mind is loading kernel modules that overwrite the instrumentation functions implemented by our kernel modules. Having our modules be the last ones loaded and disallow any further module loads would get around this vulnerability.

The other mechanism which would work would be to kill the watcher daemon. The mathematical comparisons are done in watcher, so killing it would removed the protection provided by CylantSecure. In fact, a bug in our policy code which caused watcher to kill itself is what allowed the EarthLink/MindSpring team to successfully compromise the first victim.

The CylantSecure kernel modules capture a certain amount of behavioral data before the profile is shipped on to watcher for measurement against the behavioral baseline. An attack that does not induce enough behavior to cause a profile to be emitted would be undetected. The dump interval is configurable to reduce this window. However, the current settings seem to be sufficiently low to spot and stop attacks. Do you have any experience with previous intrusions, if so, how did it occur?

Cylant: So far, intrusions have only happened when CylantSecure was exhibiting suicidal tendencies. That is the limit of our experience with successful intrusions. What are some of the most creative ways people have tried to exploit your system, or any other that you may have had experience with?

Cylant: Our way of securing systems doesn't really know "how" people are trying to exploit the system. Our view of the attacks is in terms of their "distance" from the normal behavior baseline. During this challenge, we have seen some seriously large distances. The normal behavior has a distance that is under .001; most of the attacks have a distance a full order of magnitude greater than that. On a couple of occasions we have seen attacks that have distances over 2; three full orders of magnitude greater than the distances for normal behavior. Would you be willing to make the configuration changes that have been made from the default install available so one might set up a similar environment locally to test?

Cylant: Certainly.
We did an "Everything" install of Red Hat 6.2.
Installed CylantSecure:LKP Advanced with the 2.2.18 kernel.
Put a test web site on victim.
Request pages from the test web site.
Turn on nearly every service listed in inetd.conf.
Setup cron jobs to restart services that get killed.
Setup cron jobs to build log summary files.
Setup a cron job to force httpd and inetd to be restarted. What modifications are necessary to be made when new software is added or removed from the system?

Cylant: If the new software uses the system in a different manner, then its behavior will need to be added to the calibration set, or a separate sub policy will need to be made for it. We do not yet have a tool for adding behavioral profiles into the calibration data without putting the resource back into profile gathering mode. We are working on tools to do this. How much administrative overhead is involved?

Cylant: For systems that are not constantly having new software installed on them there is very little maintainance. For systems that have new software installed on them regularly, the administrator will have to spend time updating the calibration set with the behavior for the new software.

We feel that victim is representative of some of the worst possible conditions for an administrator, and our sys admin is not spending very much time working with CylantSecure. He does, however, counsel against running a hacking challenge. Especially one where the target box is massively vulnerable. Did your old co-workers at EarthLink/Mindspring that cracked the box the first time have any previous knowledge about the configuration of the system?

Cylant: No. They simply knew about a known password problem with Pirahna in the default Red Hat 6.2 configuration. They were the first of multiple people who attempted access victim this way. After they got in, they promptly changed the password. This kept them from being bothered, and resulted in more than one person asking if we had changed the password to the 'pirahna' account. Would it have done any good had they had a map of the product configuration changes?

Cylant: Not likely. We disclose the relevant configuration information on the web site on victim. People can see what services are running on victim. We have added the output from `rpm -qa` to the site just to help people more.

At it's heart, CylantSecure is a tool for spotting and stopping new exploits and attacks. It does this by effectively representing the allowed normal system behavior. Therefore, the existence of vulnerabilities in installed software is something we take for granted. We simply enforce the behavior exhibited during calibration.

As a consequence of CylantSecure being more of a behavioral measurement and control product than a security product, have you gained any additional knowledge about the thought process and methods of the typical attacker?

We have observed that the processes used by attackers tend to be very noisy from a behavioral measurement point of view. Mostly what we have learned is that the current methodologies are not designed with circumventing behavioral measurement in mind. Since we are the first people doing behavioral measurement, this is not real surprising. would like to thank Cylant, Inc. for having this interview. We look forward to seeing your future projects, articles, and contributions to Linux and security.