Open Source Security Testing Methods
6 - 12 min read
May 02, 2001
Copyright © 2001 by Richard C. Jankowski and Dave Wreski
Copyright license terms available at
Originally written for /
The OSSTMM homepage is .
Interview with Pete Herzog, OSSTMM creator
[LinuxSecurity.Com] Pete, could you describe your security background and how you got started with this project?
[Pete Herzog] It's hard for me to think of my security background because in a way I think that's all that I've ever done. My move to doing it in the corporate/professional sense really began with my involvement in IBM's Emergency Response Service European start-up in 1997. Before that I did the project/consulting thing around America trying different institutions like education and government but never felt personally involved. Since I left IBM in the middle of 1999, I have felt involved in some way or another in the European Internet security growth-- much of it in banking.
This project came about in an idea to teach my wife the finer points of security testing. We had moved to Barcelona, Spain for the birth of our daughter and she wanted to be able to work from home. We had so much going on and dealing with too much political red tape concerning Visas and Working Papers that I was constantly commuting to a Consulate or Embassy. On a train ride back home one day I scribbled a couple flow charts on scrap paper (which will be included in the manual). I was hoping to find the key to splitting the tasks of security testing in a way that my wife could lighten my work load by doing the investigative portions of information security which she has a knack for. I don't remember it being a big deal but my wife says it was. She says I got off the train and the first thing I told her was that I figured out a methodology for security testing and this could be important. She says also that I said immediately that I would give it away by publishing it online. I may have said these things but at that point I know I didn't have the details worked out like GNU licensing and such. A month later, I took over the ideahamster.org domain name my brother was sitting on. This week we posted version 1.0.
[LinuxSecurity.Com] What made you decide that there was a need for an Open Source testing standard?
[Pete Herzog] My first real understanding on the need was in the first wave of feedbacks I got from the people who downloaded the very pathetic first draft. It was incredibly positive. In the beginning, it wasn't so much the public need but my need for a methodology. Which is why I was so surprised that it was so well received-- I wasn't the only person who kept thinking that a methodology existed in some secret corporate safe somewhere that was so great that it was guaranteed to find all security problems in an Internet presence. But one doesn't exist. And the only way to make one exist is to open it up to everyone as open-source. If enough eyes look at it, then maybe ALL the bases can be covered. Maybe it can truly be thorough. And I see it really happening now.
[LinuxSecurity.Com] Why do you think that the companies performing security assessments will follow the manual?
[Pete Herzog] There is no reason not to be compliant with the manual at least as a comparison. If I was a tire repairman and the world's tire repairmen put together a standard for tire repair that will keep the customer happy and ensure safety, I would incorporate into my way of repairing tires. And if my way was better because I exceed all the requirements then I can still use the standard as an example of the minimum.
As it is, security testers are an innovative group who need to be both methodical and radical to perform their job well. This manual works with them, guiding their hand, not forcing it. The manual focuses on the method and not the analysis which means that the security tester can still apply his/her own security concepts to the final report since everybody seems to have a different opinion on what is secure.
[LinuxSecurity.Com] What does an open source standard provide to them as opposed to in-house methodologies?
[Pete Herzog] A methodology can be seen as intellectual capital for any company. They take time and money to develop and maintain. This is why they are guarded so closely from the competition. Many of the security departments and companies that I know are on a strict budget. All have a limited number of security experts. An open source standard does not have these problems. Theoretically, an open source project can consist of ALL the experts. It can include the radicals that help make the advancements because there is no restriction on who can supply the good ideas. And there is no financial issue to consider when developing.
[LinuxSecurity.Com] What do you hope to accomplish with your project?
[Pete Herzog] I want to make a thorough security test methodology for Internet security testing. I want this to lead the way to other open security standards that have been obscured for far too long. When Victor Rodriguez asked if he could supplement the manual with a methodology for secure coding, I really saw the potential this could have. The concept of a document written by many rather than a select team or individual as was the way of the whitepaper may have a future in writing technical documentation. The ideahamster site will soon sport a roster of open source documents.
[LinuxSecurity.Com] Why is it important to do regular security auditing/testing?
[Pete Herzog] It's hard to answer this question because it's so obvious to me and I only got a C in marketing. So please excuse the cliche in the answer where it seems like I'm repeating the "security is a process" line from those who are in the business of selling the security process.
[LinuxSecurity.Com] How often is it necessary to perform a security snapshot? Can incremental changes be made?
[Pete Herzog] The manual doesn't answer these questions and my personal opinion falls under the realm of "paranoid" according to my wife. It's up to the tester and the target organization to decide what frequency of testing is best.
[LinuxSecurity.Com] Does user training fit into this in any way? There's no overlooking the end-user element.
[Pete Herzog] End-users, or "people" as they are refered to in the non-computer world, can be the cause of Internet security problems. Often it is necessary to remind them not to execute e-mail attachments or send their passwords home to themselves in a clear-text e-mail so they can have a back-up. Administrators avoid training them by building safeguards like server-based virus checking and proxy servers because it is easier to mechanically restrict them than to get continuous compliance. The manual leaves end-user training to the analysis part of the reporting process and a good analyst could tell you if the users need better security training or not.
[LinuxSecurity.Com] From what perspective do you run the tests? From the cracker/hacker perspective, or from a more knowledgeable system administrator perspective?
[Pete Herzog] The beauty of the methodology is that you need to work like a hacker and act like a professional to be successful. Which is why the training supplement concentrates on exactly those elements for success. A thorough test can only be performed by those who understand the expected results and the tasks which need to be performed to get those results.
I also believe the tester then must have good networking and system/network administration knowledge to perform certain tasks or else the right results just won't appear.
[LinuxSecurity.Com] What are the tools that you find most useful for this testing?
[Pete Herzog] I must admit I get frustrated with certain security tools that only ALMOST do what is needed. I added in the manual a sort of tool wish list. At least with open-source tools, I can add the functionalities I need. I can tweak the code to run best on my system.
[LinuxSecurity.Com] How do you archive the information and create a summary that would be useful for presenting to management/sysadmins/security engineers?
[Pete Herzog] The summary is usually the biggest and most detailed part of the report because it tells the results, opinions, hopes, and fears of the tester in plain language. This has to be within the scope and business needs of the target organization. I have often suggested using two reports. One report could go to the management and the other to IT. And both reports should list the positive and the negative.
As far as archiving, I think it's important not to keep sensitive information like that if it doesn't need to be. Using a hashing mechanism and a public timestamp can later prove whether or not it is the same report as was delivered. Once it is delivered, the report needs to be treated as sensitive intelligence but that is really no longer in the tester's control.
[LinuxSecurity.Com] Does social engineering enter into your plan?
[Pete Herzog] SE is a topic that bounced around a bit and I wasn't sure if I should include it since I didn't know how it could exist as a quantitative tool. The arguments were good for it in the end and I learned that my concept of SE was far too narrow. Now it is in the manual-- a good indication that peer-review is indeed a benefit.
[LinuxSecurity.Com] How about coordinated attacks, or attacks that may require piecing together information from many sources?
[Pete Herzog] There is no reason why the more precision based attacks can't fit into some of the parameters which already exist such as Firewall and ACL testing, router and switch testing, port and protocol testing, etc. The other part of this is the danger of heavy volume and the possibility of bringing down routers between you and the target organization. In the end it was decided that there is no need to test against these kinds of flood attacks like DDoS because any analyst can tell you if you're vulnerable or not just by the system and network map.
[LinuxSecurity.Com] Would you recommend a proprietary or off-the-shelf solution to interrogate a corporate network?
[Pete Herzog] I haven't seen a single off-the-shelf solution that can do what a Red Team can do. I also haven't seen one that doesn't take some good security and networking knowledge to use properly. I think if used properly they can be of great help, especially in cases of internal auditing (which we won't cover until the Intranet supplement this Autumn). I still think it would be good if some of these automated scanners showed the flow methodology they follow so a security tester can best integrate it into his/her testing. So far, I have seen none which do.
[LinuxSecurity.Com] How much testing involves having a well-constructed security policy in advance of the actual testing?
[Pete Herzog] None. The security policy is just another parameter as far as the methodology is concerned. The security tester's sole job in this methodology is to collect information about the Internet presence regardless of the good security principles as set forth by chapter one of any O'Reilly security book or the good folks at SANS. If the policy exists, the tester can confirm its rules with appropriate testing but otherwise that can really be done in the analysis phase after testing.
[LinuxSecurity.Com] How would you recommend the process of adjusting the security policy, firewall, software, etc, given the results of the test?
[Pete Herzog] Regardless of the findings, all adjustments are a matter of mitigating risk. The rest that I could say here really is cliche: total security means an unusable system and the more complicated the security of a system or network the more likely mistakes will be made by the people in charge of it.
[LinuxSecurity.Com] Has there been much help in developing the standards so far?
[Pete Herzog] There has been a good deal of advice, some hardcore supporters, and so far no complainers. Those that help have done quite a bit. My wife has also done a lot of the formatting and design stuff as well as being pretty tolerant of my late hours on the PC after work.
[LinuxSecurity.Com] How do potential contributors help out?
[Pete Herzog] On the main page there is a paragraph on the submission process. Basically it's a way that I can keep in touch with the contributors and track the submissions. As it turns out, many people volunteer and never do submit. All I ask is that they send me an e-mail with their name and whether or not they are representing an organization. Then they should join the discussion and news mailing lists to keep up on changes. We don't mail much but when we do it's for decisions or the occassional update. Those on the discussion group always see the draft releases before they are released.
[LinuxSecurity.Com] What kind of help do you need from volunteers?
[Pete Herzog] I need people to comment on what they read. Anyone with any comments is appreciated. I need people who do something different as a methodology and tell me what works and what doesn't. I need to hear from people who are integrating the methodology and if it works for them.
There is a lot to do on the site-- from developing the manual itself to the three supplements: the secure programming supplement, the training supplement, and the tools development. Anyone who has the time and can contribute really should. There are so many parts of the meth that just need to be better.
