"The article is not about 'give us the code instead of a paper', but instead about 'give us the code as a supplement to the paper'."
is an appropriate interpretation of the paper. But I can't have much sympathy for that interpretation because the code can be so darned long and obscure.
Some 'pseudo-code' of, say, a dozen lines as a 'figure' is okay. In nearly all cases, any code very close to something that actually runs in practice would tend to be too darned long.
The 'knowledge' of the paper is supposed to be clear just from the paper. Then the code is supposed to be 'routine'.
If the main body of the paper does not make the knowledge clear, then including a few hundred lines of code has little promise of improving the exposition.
There's a good history supporting my point in books by Knuth, Ullman, Sedgewick, the paper
Ronald Fagin, Jurg Nievergelt, Nicholas Pippenger, H. Raymond Strong, 'Extendible hashing—a fast access method for dynamic files', "ACM Transactions on Database Systems", ISSN 0362-5915, Volume 4, Issue 3, September 1979, Pages: 315 - 344.
and more.
For your
"The basic argument is that sometimes text alone is not sufficient to explain - a point you concede to by mentioning that the formulae exist in textbooks along with the description."
I believe you are asking too much of the equations, not a lot too much but too much.
In simple terms, even pure math is, usually is, at best is, is supposed to be written in complete sentences in a natural language, say, English. E.g., in a physics book describing Newton's second law, we discuss force, mass, and acceleration, explain that force is a vector F, mass is a scalar m, and acceleration is a vector a, say that force is the product of mass and acceleration, e.g.,
F = ma
So, by the time get to the equation, almost don't need it. Of course, not ALL writers write this way!
A long sequence of algebraic derivations is supposed to have a start much like what I gave for F = ma and after that is just algebraic derivations, often with not much need for more text.
For
"Take for instance many protocol standards: they come with a reference implementation, because sometimes the text is harder to understand when you can't compare it to code which actually does what is being described."
yes, there is a practical challenge there. Finally the Unix world decided to 'go empirical' and have 'code runoffs' or whatever they were called where people just got together to see if the actual code, say, the TCP/IP stack with SSL, would actually appear to work. Of course, that empirical approach was always risky because it is a long way from a mathematical proof of correctness or even a solid engineering document. And we've seen problems, e.g., with security in DNS.
For more, the AI world decided to f'get about 'theory' or even usual scientific documentation and make the criterion just 'does the code appear to work'. This, too, has proven to be risky and, of course, is a big step back in documentation quality from good applied math, mathematical physics, and engineering of past decades. E.g., how the heck to know that the Manhattan Project was on track before the first test? Sure: There were a LOT of derivations and calculations; that is, the project didn't rely on just the empirical approach. Indeed, the uranium bomb was dropped on Japan as the first 'test'; the engineering was so solid there was no doubt it would explode.
Sure, there is a big role for testing, but, still, the 'knowledge' in a journal is supposed to be able to stand on its own without much help from detailed documentation on code, electronic circuits, mechanical mechanisms, or chemical reactions. In particular, in experimental science, the published paper is supposed to be clear enough to permit others to reproduce the experiment, and to do so without detailed photographs and drawings of apparatus.
When I was an undergrad, one summer I worked in the lab of a famous scientist. He explained to me how a scientist is supposed to keep a good notebook and really good records. Then if another scientist wants to check details or reproduce results, sending the notebooks and data is supposed to be an obligation, and the materials are expected to be handled with great care and then returned.
So, yes, at times, in practice, there can be more that is relevant than just the published paper.
Still, communicating the 'knowledge' of a research paper with a few KLOC of code is a bit much!
I really think you are fighting a strawman here. Your argument is all over the place too... Is the math supposed to be clearly explained in text or just the starting conditions from which the rest of the formulae just tell the story for you?
Moving on to DNS having security holes and other problems with protocols made with reference implementations -- how is this relevant? Lots of scientific papers get published that later turn out to be wrong, whole fields of study sometimes look really good and push our understanding of the world but ultimately go away as "the wrong approach". This is just progress, not a fundamental flaw in putting making code available.
When you say the "knowledge of a paper" should stand on its own, without the need for the circuits, code, etc to understand it -- this is true of many fields, but not true when the paper is about the circuit, the code, etc. Then that very thing should indeed be presented, otherwise no amount of text will properly explain it, other than a 1000 words where a picture would suffice.
As for your famous lab scientist -- why is it ok that he wants to make notebooks and other lab stuff available, but when we ask for code to be put in a public repo when reviewing papers we are suddenly breaking knowledge?
An finally, about your last sentence -- supplemental code is not an attempt to transmit the knowledge with the knowledge of a research paper. The word supplemental means "in addition to", the paper should still happen!
A last, slightly tangential thing: No matter what you say about text being the only way to transmit knowledge, there are plenty of occasions where a simple digram transmits much more meaning to me than reams of text, I am visual thinker and learner. Similarly there are times when a few lines of code can tell me as much as paragraphs describing them. There are times when I see an equation and get what the paper is trying to discuss. There are also times when the text is invaluable, and without it I would be lost, no matter how good the diagram or equations or code. The methods of knowledge transmission are many -- to be stuck on a single type as the "true way" is a bit absurd and/or pretentious.
The resolution appears to be there are two quite different worlds: First there is the world of peer-reviewed research papers in computer science where the goal is to present knowledge that is new, correct, and significant and that sometimes results in code. And second there is the world of commercial computing that is heavily about just code. But these two worlds are very different. In particular, the research papers are not trying to document or describe code. Moreover, in the knowledge presented in a research paper, that knowledge is supposed to be in the form of text, possibly with some math, but not in code. E.g., the original paper with heap sort didn't need code; Knuth's presentation of heap sort in TACP has only a little code, in his language MIX which few people bother to understand, that is not essential.
"Is the math supposed to be clearly explained in text or just the starting conditions from which the rest of the formulae just tell the story for you?"
Again, you are asking too much from the math. Again, 'math' is supposed to be written in complete sentences. In particular there is no 'new language'. I illustrated with F = ma: You are supposed to get nearly all "the story" from the text and not the equations. Well written math doesn't ask a reader to get "the story" "just" or even primarily from the equations. If you don't want to accept this description of good writing in math, then so be it, and our main difference will be right there and not "all over the place".
Again, yet again, once again, the knowledge is supposed to be in the text, in well written paragraphs, in complete sentences, and there is no 'new language'. For an equation such as F = ma, that is a part of speech, a noun. And again, yet again, the surrounding text is supposed to be so good that the equation is hardly needed. Again, yet again, one more time, please read it this time, you are not supposed to have to dig the meaning out of the equations in more than a peripheral sense.
"Moving on to DNS having security holes and other problems with protocols made with reference implementations -- how is this relevant?"
Because, again, the code, including in a 'reference implementation', doesn't mean anything. We shouldn't ask to determine correctness, or even the quality of the design or engineering, just from the code, not even code with mnemonic identifier names and many in-line comments. The 'relevance' is that we see that the DNS code had security problems, that is, was not high quality design or engineering, and the reason for those problems is that high quality design and engineering are in text and NOT in code. So, with just the code, we have no solid, rational support of the quality of the design or engineering. Instead, about all we have is the code. Then to know if the code has high quality, we just deploy it and wait for the bug reports. Then we study the code, see where the problem is, patch the code, deploy it again, and wait for more bug reports. NOT good. NOT high quality design or engineering.
We do NOT design or engineer airplanes like in a 'reference implementation' of DNS. Instead, before the plane carries passengers, we have a mountain of rock solid engineering, heavily in text, saying that the plane is safe. We do NOT primarily determine the safety of an airplane by putting one million passengers in it. In DNS, we determined the quality of the work primarily by deploying it, using it in real networks, getting the bug reports, and then fixing the bugs. So, the DNS engineering was shoddy work compared with nearly all of the rest of engineering.
Broadly we just cannot expect to have delivered high quality software when what is delivered is mostly just the software. Instead, the quality has to be in a document almost entirely in text. In high quality? Yes. In common commercial practice? No.
Fundamentally it is the text that really matters, NOT the code. Again, yet again, given the text, the code is supposed to be routine. This holds in 'research level' 'knowledge'. Sure, this doesn't hold in routine commercial practice.
"Lots of scientific papers get published that later turn out to be wrong, whole fields of study sometimes look really good and push our understanding of the world but ultimately go away as 'the wrong approach'. This is just progress,"
No, you are not describing "progress" but are describing just mistakes, that are sometimes costly, and are to be avoided strongly.
"When you say the 'knowledge of a paper' should stand on its own, without the need for the circuits, code, etc to understand it -- this is true of many fields, but not true when the paper is about the circuit, the code, etc."
No, not really: We're talking about peer-reviewed research publications of 'knowledge'. That 'knowledge' is not really supposed to be about the circuit or the code. Instead, the circuit or code are supposed to be routine implementations of the knowledge in the paper.
You want to elevate the circuit or code to the level of knowledge, and that is backwards and wrong. Your goal of elevating code to the level of knowledge and what is primary is not promising. In simple, blunt terms, so far without serious exceptions, 'knowledge' is communicated in text, possibly with some math, in a natural language. That's all we've got. There ain't no more.
"Then that very thing should indeed be presented, otherwise no amount of text will properly explain it, other than a 1000 words where a picture would suffice."
In the communication of knowledge, we don't depend primarily on pictures, figures, diagrams, schematics or code. Instead we communicate knowledge in text, possibly with some math. Again, good examples are in books by Knuth, Ullman, Sedgewick, etc.
Again, yet again, once again, please try actually to read it this time, there has been a long tradition in math that there should be no diagrams or pictures at all, not even in vector calculus. Much of the motivation was to have the discipline to make SURE that the content was in text, just text, with some math, and NOT in pictures. Again, we didn't want the content to be in pictures. This point is SERIOUS about keeping up the quality of the material.
For pedagogy, a picture can be terrific, but 100% of the content should also be in the text.
"As for your famous lab scientist -- why is it ok that he wants to make notebooks and other lab stuff available, but when we ask for code to be put in a public repo when reviewing papers we are suddenly breaking knowledge?"
We're not. Put all the code in public repositories you want. Put in 1000 KLOC. Fine with me. But the code in the repository is not going to be part of the paper, e.g., it will not be reviewed in the peer-review process. Moreover, the paper just MUST make sense with no reference at all to the code. Again, the code is NOT the knowledge or the subject of the research paper. Again, the paper is NOT about the code; it is not a paper describing the code; the code is not the 'contribution to knowledge'. Instead, the code is at most a hopefully routine implementation of the knowledge in the research paper.
"No matter what you say about text being the only way to transmit knowledge, there are plenty of occasions where a simple digram transmits much more meaning to me than reams of text, I am visual thinker and learner."
Fine. So are many people. But such ways of 'transmitting knowledge' are NOT what we are talking about. We're talking about peer-reviewed research papers. Tutorial presentations can be very different. Movies are very different. TV is very different, even some of the TV cooking shows that try to be instructional are very different.
Maybe the main point of misunderstanding is, practical computing is nearly all about code, just the code, and, thus, you have accepted that computer science research papers can be about describing code. No. Instead, for such papers, any code is supposed to be just an implementation of the knowledge in the research paper and routine. The knowledge in the research paper is NOT 'code'.
The knowledge in a peer-reviewed research paper is supposed to be rock solid, and, again, yet again, we don't communicate that with pictures or code.
Pictures and code? In tutorials, fine. In commercial work, sure. In research papers, only a few lines of 'pseudo-code' in a picture that is not essential to the paper. Understand now?
Your
"The article is not about 'give us the code instead of a paper', but instead about 'give us the code as a supplement to the paper'."
is an appropriate interpretation of the paper. But I can't have much sympathy for that interpretation because the code can be so darned long and obscure.
Some 'pseudo-code' of, say, a dozen lines as a 'figure' is okay. In nearly all cases, any code very close to something that actually runs in practice would tend to be too darned long.
The 'knowledge' of the paper is supposed to be clear just from the paper. Then the code is supposed to be 'routine'.
If the main body of the paper does not make the knowledge clear, then including a few hundred lines of code has little promise of improving the exposition.
There's a good history supporting my point in books by Knuth, Ullman, Sedgewick, the paper
Ronald Fagin, Jurg Nievergelt, Nicholas Pippenger, H. Raymond Strong, 'Extendible hashing—a fast access method for dynamic files', "ACM Transactions on Database Systems", ISSN 0362-5915, Volume 4, Issue 3, September 1979, Pages: 315 - 344.
and more.
For your
"The basic argument is that sometimes text alone is not sufficient to explain - a point you concede to by mentioning that the formulae exist in textbooks along with the description."
I believe you are asking too much of the equations, not a lot too much but too much.
In simple terms, even pure math is, usually is, at best is, is supposed to be written in complete sentences in a natural language, say, English. E.g., in a physics book describing Newton's second law, we discuss force, mass, and acceleration, explain that force is a vector F, mass is a scalar m, and acceleration is a vector a, say that force is the product of mass and acceleration, e.g.,
F = ma
So, by the time get to the equation, almost don't need it. Of course, not ALL writers write this way!
A long sequence of algebraic derivations is supposed to have a start much like what I gave for F = ma and after that is just algebraic derivations, often with not much need for more text.
For
"Take for instance many protocol standards: they come with a reference implementation, because sometimes the text is harder to understand when you can't compare it to code which actually does what is being described."
yes, there is a practical challenge there. Finally the Unix world decided to 'go empirical' and have 'code runoffs' or whatever they were called where people just got together to see if the actual code, say, the TCP/IP stack with SSL, would actually appear to work. Of course, that empirical approach was always risky because it is a long way from a mathematical proof of correctness or even a solid engineering document. And we've seen problems, e.g., with security in DNS.
For more, the AI world decided to f'get about 'theory' or even usual scientific documentation and make the criterion just 'does the code appear to work'. This, too, has proven to be risky and, of course, is a big step back in documentation quality from good applied math, mathematical physics, and engineering of past decades. E.g., how the heck to know that the Manhattan Project was on track before the first test? Sure: There were a LOT of derivations and calculations; that is, the project didn't rely on just the empirical approach. Indeed, the uranium bomb was dropped on Japan as the first 'test'; the engineering was so solid there was no doubt it would explode.
Sure, there is a big role for testing, but, still, the 'knowledge' in a journal is supposed to be able to stand on its own without much help from detailed documentation on code, electronic circuits, mechanical mechanisms, or chemical reactions. In particular, in experimental science, the published paper is supposed to be clear enough to permit others to reproduce the experiment, and to do so without detailed photographs and drawings of apparatus.
When I was an undergrad, one summer I worked in the lab of a famous scientist. He explained to me how a scientist is supposed to keep a good notebook and really good records. Then if another scientist wants to check details or reproduce results, sending the notebooks and data is supposed to be an obligation, and the materials are expected to be handled with great care and then returned.
So, yes, at times, in practice, there can be more that is relevant than just the published paper.
Still, communicating the 'knowledge' of a research paper with a few KLOC of code is a bit much!