Our friend Rich pointed me to http://www.onlamp.com/pub/a/security/2004/09/16/open_source_security_myths.html.

Open Source Security: Still a Myth
by John Viega, coauthor of Secure Programming Cookbook for C and C++
09/16/2004

I have to agree with the main thesis, that many developers/users ≠ many security analysts/debuggers. I also get the impression he's just baiting the Slashdotters.

All in all, in some cases open source may have more eyeballs on it. Are those eyeballs looking for security problems, though? Are they doing it in a structured way? Do they have any compelling incentive? Do they have a reason to focus dozens or hundreds of hours on the problem to approach the level of effort generally given to a commercial audit? The answer to all of these questions is usually no. A good deal of software doesn't get examined for security at all, open source or not. When it does, commercial software tends to receive much more qualified attention.

Reading this, I thought about some specific projects: MS Windows, IIS, and Office (commercial non-open); Linux, FreeBSD, OpenBSD, and Apache httpd (open, generally free); Solaris (currently closed, discussing going open source); and Mac OS X (a commercial product layered on an open/free layer).

Microsoft can afford all the expensive tools they are willing to pay for, and I know they develop their own debugging tools in-house too. Something's horribly wrong, though. Having listened to Scott Charney, Chief Trustworthy Computing Strategist for MS, I don't think the real problem is incompetence -- although from a security perspective, that is a very real problem. The real problem is priorities.

Scott spent some time telling us why he's a credible authority on security, and poking holes in Dan Geer's argument. Mostly, though, he told us over and over that MS puts as much effort into security as customers will allow, and customers really don't want Microsoft to do any more than they already are. Surveys, meetings with managers, yadda yadda yadda -- they all point to people saying security is a top priority when asked, but when push comes to shove, what customers really demand is non-security features and lower cost. According to Scott, they're not willing to pay for security.

In my own work, I deal with the same conflicting demands for security versus unwillingness to give up flexibility, but as long as Bill Gates is saying security is job #1, and Scott is telling people that customers won't permit better security, they're full of it. I believe that MS has gone through a sea change, and security is now more important to them than it was 5 years ago, but they've been telling the world how great their security record is for much longer than that, and it still sucks rocks. As for the others, there really are lots of people reading Linux kernels, perhaps because they want to be cool, or are assigned homework, or want to fix a strange problem. Sun has some very serious engineers, and also can afford whatever auditing tools they want. If they open source Solaris, do they qualify as open under this article, or closed? Obviously open Solaris should get more review than closed Solaris, perhaps in those same CS classes.

The most hard-core security auditors I know of are the OpenBSD folks, and their fixes roll into FreeBSD, NetBSD, and OpenSSH. The TrustedBSD and SELinux projects also audit FreeBSD and Linux. I don't follow bugtraq &c., but most of the fixes I see are reported by third parties anyway, so I wonder how much this discussion of auditing is a red herring. I don't know how many security problems MS is sitting on -- either because they don't expect outsiders to find out, or because there's no public pressure to fix them. I do know that Apple's last security black eye, over application launching, wasn't dealt with quickly partially because it wasn't publicly a big deal; after it got more publicity, Apple moved much more aggressively.

I'm not sure it's a meaningful comparison, but I feel more comfortable knowing the Samba & Apache httpd projects don't sit on known bugs they know about (even only internally) indefinitely. I'm sure MS does, at least sometimes.

Developers usually haven't heard that; when they do, they tend to become defensive. Sometimes they'll insist that there's no problem until someone can actually demonstrate a working attack. That is, instead of fixing potential problems and moving on, they'll try to force security auditors to spend hours of precious time demonstrating exploitability. This actually tends to be more of a problem in the open source world than in the commercial world, because commercial projects typically are driven more by schedules. Managers often are already worried about sticking to their schedule and will try to railroad developers into taking the easy road, even if they start to question the audit results. In the open source world, developers tend to be quite independent, even when people are managing a project. With such projects, it's rare that anyone worries about a negative impact should a release take a bit longer to ship.

I've never managed developers (open or closed), but I don't buy his assertion that the best way to meet (commercial) deadlines is to fix tough bugs, and the easiest way to deal with (public) problems is to take as much time as is necessary to argue them into (unfixed) submission.

Certainly, commercial software organizations can fall into similar traps by assuming that they have security under control. However, commercial organizations are more likely to take security seriously, simply because they are more likely to have paying customers demanding some level of security assurance.

Bull. Again, MS is the poster-boy, and offering some level of security assurance ≠ fixing all security problems. Apple's not perfect either, although their model is reaping definite rewards from many eyeballs -- BSD and open source application fixes are constantly flowing into Mac OS X, and Apple's effort here is minimized. This might be a better argument for Sun and IBM (AIX & OS/400), but some of that goes back into Linux because they both have Linux systems as well...

Many market segments have not only identified security as a big concern but also realized that the root causes are better confronted by the development team, not network engineers. In particular, financial companies and government organizations are looking for assurance that the software they acquire is likely to jump some basic security bar. People who want to sell software to organizations in these markets have to answer tough questions about the security properties of their software. Many times, potential customers must fill out extensive documentation about their products and the processes and technologies used to build them. Sometimes, potential customers must even submit their software to independent third-party source code auditing before purchase. For instance, the U.S. Navy is currently working on the prototype of a process that all vendors will go through before they may sell software for use on the Navy's intranet.

They've been using Windows (at least since NT 4) on nuke subs for years. I'm pretty sure they didn't do an external code audit. I do like the CC certifications (recently for RHEL on IBM gear), but don't know how enough to judge their relevance personally.

These organizations have spent a lot of money improving their security process, knowing that if they address their security problems earlier in the life cycle, reliability goes up and, in the long term, cost goes down. Because open source projects aren't driven by traditional financial concerns, they're less likely meet the needs of the market. Open source projects tend to focus more on what the development community finds interesting rather than what potential users find interesting. The lack of financial incentive results in a relatively poor feedback mechanism.

Here he's just saying that open source is less relevant because they don't get paid. That argument has nothing to do with security, and everything to do with general perceptions of open source vs. commercial software. We're going to need to revisit nomenclature if and when Sun releases Solaris. At that point, Sun, Apple, and Red Hat will all be commercial open source companies in varying ways.

If the open source world wants to make sure that security is not an impediment to adoption, then it needs a strong answer. First, open source projects need to migrate to software engineering processes that resonate with the industry. Most projects are devoid of structured process. Programmers who use it tend to take the Extreme Programming approach, which may end up being too liberal for many buyers.

Pfui. I have no idea what programming methodologies MS and Sun use, and neither do the vast majority of their clients. This is not generally relevant, and much harder to judge (or even perceive) than code product quality, which is apparent in varying ways to each user.

Comparing all open source software with all commercial software is tough. Certainly, when it comes to security, there are good cases and disasters in each camp. I do believe that from a security point of view, Apache is probably better off than Microsoft's IIS and that djbdns is better off than almost anything competitive. While I do think the open source community has a long way to go in general, I don't think it is necessarily worse on the whole. I would evaluate it only on a case-by-case basis.

D'oh! "Apache is probably better off than IIS?" My head reels! This whole article has been evaluating open source as a lump against commercial software. If I were Groklaw's PJ, I'd be convinced this guy works for Microsoft. Myths about Open Source, hmm? Sounds a bit like MS' Linux Myths.

In the end it doesn't matter if open source systems tend to be more secure than proprietary systems, because on the whole they aren't yet coming close to being "secure enough."

Double d'oh! Earlier he said MS software was years from getting substantially better, but thinks that's an example of commercial software doing it right. Pfeh.

John Viega is chief scientist of Secure Software and the coauthor of "Building Secure Software" (Addison-Wesley) and "Network Security with OpenSSL" (O'Reilly).

Ah. He mentioned developers plugging in SSL libraries and thinking they were done. This may explain some of it, but the confidence in Microsoft is puzzling. I'm going to stick with my belief that someone was disappointed that open source isn't as perfect as some people expect it to be, and wrote a long rant based on that.

I agree many eyes aren't relevant to all problems, and security is one of the harder ones, but find much of this hard to credit. I'm largely in agreement with lozano, the first respondent.