I think that to the contrary, that it would allow people to determine whether the original results were due to flawed software by analyzing the code. Subsequent efforts would remain free to create their own implementations from scratch if desired.
Further, open code means that other researchers can come along and write unit tests. It amazes me how much code is still verified manually if at all in the sciences.
A lot of code in the sciences is used once or only a few times. You crunch the data, write the paper and move on to the next research project. You don't get tenure for writing unit tests.
And you don't make friends with salad, i.e. you certainly won't make tenure when your paper is demonstrated as flawed due to errors in crunching the data that would have turned up with testing.
I have tried to explain (With limited success) to my colleagues in bioinformatics that not unit-testing your code is like using an instrument that hasn't been calibrated, and blindly trusting the results to be right.
They may not unit test it, in the software development sense, but they probably hand-check a new bit of calculation code on a few data points before setting it loose on a whole data set.
Yes, or sometimes even less rigour than this: run some data and see if the results "look right".
This is the standard procedure for when you've developed a complex algorithm to do something novel, and actually working through it by hand would take hours.
I should know, this is how I used to do it before I knew better, and as a result one of my published papers has (very slight) numerical errors in.
True, and that's the definite advantage to publishing code. My point is just that if there's a lot of code and error is obscure, it will take a long time to identify the error and the mistake will be propagated in further research. Most scientists using the code will just skim it, see if it gets the results published in the original paper, and then move on. This has been my experience with my own research anyway.
I disagree, because your mental model of a large code base that only has one error is absurd. Scientific code is not magically easier to write than real code. Get two random code hacks to write two significantly-sized codebases, say, about the size of a decent web framework, and you won't get one perfect code base and one with a subtle bug that has ramifications down the line. You'll have two code bases so shot through with bugs that they will never reconcile, ever.
The quality issues aren't all that different than a web framework, either. Release one or a small number of code bases, and have everybody pound on and improve them, and you might get somewhere. Have everybody write their own code bases from scratch every time and you'll get yourself the scientific equivalent of http://osvdb.org/search?search[vuln_title]=php&search[te... .
To be honest at this point when I see a news article that says anything about a "computer model" I almost immediately tune out. The exception is that I read for some sign that the model has been verified against the real world; for instance, protein folding models don't bother me for that reason. But this is the exception, not the rule. When it became acceptable "science" to build immense computer models with no particular need to check them against reality before running off and announcing immense world-shattering results I'm not exactly sure, but it was a great loss to humanity.
