Re: Code is truly awful, but sadly not unusual
My experience is not at all limited to megaprojects. I have worked on pretty much every size and type of software. My experience is a bit limited because I have almost always been on the development side rather than maintenance. I think that programmer productivity does fall with project size, but not entirely as much as you seem to think, nor for the same reasons. You cannot take a few exemplary and successful small projects and their developers and use that as a yardstick for even other small projects, let alone large ones.
Take a look at the actual projects and source code on sourceforge; count code age and lines. Does it exceed daily productivity by paid professionals from large consulting firms? I have seen a lot of code from both sources and I would say that it does not. Most big projects have been less cost-effective than I would have liked, but not all. I worked on a project costing tens of millions of dollars with Sybase consulting and they kicked ass. I also worked on a huge multi-billion dollar production system for one of the largest firms in the U.S. and it is the only non-trivial project where we managed to come in with quality code both ahead of time and below budget.
Re: "size it in that manner and expect - as you do - inevitable overruns."
That was a thumbnail quote based on a line count and inspection of a sampling of code. It may seem strange to some, but in my experience a survey of code quality and line count has been the best indicator of the time required. I expect that if I had that budget and was allowed to form my own team that I could come in below that budget. However, overruns are inevitable on most projects because they would not have been started at all if more realistic estimates were made. Writing software, even with an existing system as a spec (huge leg up), is still labor intensive and difficult. I have been a career consultant and worked on a lot of different sizes and types of projects including in software companies as such like Sybase and Microsoft. My experience, as far as I can tell, is about exactly in line with what generally happens.
BTW -- I have worked on large projects with both IBM and Accenture and if you think you are better than their top guys I suspect you have not met their top guys. They *do* pitch out some less than stellar programmers on huge projects, but they attract some really good people. I was out a bunch of times with IBM, Andersen and then Accenture and none of our projects failed to deliver. You could have done it cheaper, but when you are spending $250 million dollars on a mission critical system that has to be there on time, it is not prudent to wing it.
I am near certain that I could beat the average large consulting firm project in terms of time and budget. I am also near certain that I would not be able to beat them if they fielded their best people. In no case could I possibly offer the guarantees that they can. One of my Andersen projects came in on time because there was a series of something like $5 million penalties for coming in late against milestones and they had deep enough resources to insure they made it.
Whether we like it or not, cost and time over-runs are a fact of life in complex projects. Failing to plan for that is planning to fail because of that.
Re: One BSD developer beating thirty developers from a large consulting firm.
In my experience you would lose that bet in a spectacular way. The average team member would probably be a bit worse, but the kind of consulting dollars available on these projects attracts some very good people. Chances are the same guy would be offered $200 per hour to work on the project with the big team as well as being offered much greater resources and helpers to amplify his productivity. If he is really serious about his craft, he will join the big team. Heck, he might even learn something.
Career programmers with thousands of hours of hands-on experience and credentials like Master's degrees in Engineering and PhD's in math are not guaranteed to be the best software developers. Some are even bad, but in my experience they are pretty good.
Whatever you can say about the openSSL code quality, I would say that it reflects work below the median quality of most working professional programmers. That is not to say that the programmers who worked on it are less than mediocre. It is an enormous amount of intrinsically difficult code in a language that many programmers find challenging. My hacks may not include gotos or multiple points of exit or mutant stylistic habits, but they can still be pretty ugly. If you only have time to do the hack, even the best programmer will produce poor code. Is there an experienced working professional programmer anywhere that has never been boxed in like that? I highly doubt it.
Re: "I'd still be more likely to trust whatever the OpenBSD team ends up using themselves."
This could go either way, but I am pretty sure that, given the budget I mention I could do better than they will end up doing and so would Accenture or IBM.
Re: "Your team would likely still be at the PowerPoint stage phrasing vision and mission statements"
This might have been true in the past, but I doubt it is now. Given the budget I mention, a couple of people on the team would be formally interfacing with management and users. It is possible that the team might *also* produce management friendly documentation. In the past I have worked on ISO 9000 projects where various documents (a lot of them) were a formal requirement. You might want to have a bit of polish on the presentations and documentation too if you were spending tens of millions of dollars on a project.
Re: "and various sociopaths would scheme to get themselves into politically advisable positions."
I agree with you here. The climbers are everywhere and they muck up the works with their schemes. Since they are devoted to climbing and you are devoted to your actual job, they end up winning a lot even though they are visible to at least some. This may be over-represented in fat and happy environments, but no organization is immune to this, including open source.
Re: "Seeing that people have to use something right now and over the next year, what do you suggest they use instead?"
There are good arguments in favor of both the existing project and the forked one. I expect that the SSL code generally will benefit by this situation, though it may be painful to the openSSL maintainers. Either should do, but if I had to make one choice I would choose the original over the forked version.
SSL is important enough that it should support a budget in the tens of millions to seal it up. However, even if SSL achieves perfection with respect to its design and code, it will still suffer from the many architectural and philosophical difficulties the whole system has.
Hearbleed was about the worst security bug I have ever seen in terms of both breadth and depth. It potentially affected every net connected device in the world, including ones that did not even use openSSL. I am not making light of it, but if you think about it, the only thing that has changed before and after is that we found out about this specific misfeature and we fixed it. We are not in good shape, but we are in better shape than before.
My estimation of our security situation has not changed. If you look at prior postings of mine you will see that security and privacy are big issues for me and I have been pretty blunt that security is hopelessly broken. That has not changed -- up or down.
You have to ask yourself why IBM, Red Hat, HP, Apple, Microsoft and other large players have not put together the necessary budget and had this done. Even at a commitment of $20 million it is a drop in the bucket for them. The weakness of the current openSSL code base negatively affects everyone, including people who do not even use it. They all have people who would have, like I did, look at heartbleed and know that even by our shabby standards of security it had a very serious problem. Given a look at the code base I would say that it is almost certainly still broken.
Perhaps they know that the systemic problems render an SSL fix pointless. Maybe they are under orders from some government agency not to rock the boat. Maybe they really don't think the openSSL code is that bad. I can't think of a scenario where all the big players look good here.
Biting the hand that feeds IT
