Scope of CODECHECK Makefile

[mstimberg]

The CODECHECK guidelines state:

Write a Makefile to re-run the workflow based on provided documentation, i.e., recreate all files listed in the manifest by runnign the command make codecheck. This target should run the whole or most suitable subset of the workflow and create the report.

But looking at previous CODECHECK examples, this usually does not seem to be the case, the Makefile only recreates the report based on the stored output files (as a side note: make clean still cleans the output files). For example in the last two codechecks: https://github.com/codecheckers/LICD_article and https://github.com/codecheckers/driftage .

As a more general question: what is the advantage of actually using a Makefile instead of e.g. a simple shell script or just a line stating what to do in the README file? Not to be misunderstood: I'm definitely a fan of Makefiles in general, since they are great for keeping track of changed dependencies, etc., but is this actually a CODECHECK use case? This touches on questions of the reproduciblity of the code check report itself. For example, if I run make in the codecheck directory in one of the above mentioned repositories, I will get a new codecheck.pdf which is actually wrong in a sense, because it will dynamically include my R session info, even though this is not the environment that has been used to generate the output files included in the document. I understand that the PDF is the "frozen" document that states everything in a non-ambiguous way, but the whole Makefile + dynamic Rmarkdown setup to me gives the wrong impression that the codecheck itself could be reproduced. This would only be the case if either the Makefile runs the code separately from the Rmarkdown file (which seems to be the case in at least one repository: https://github.com/codecheckers/Hopfield-1982/), or if the Rmarkdown included the actual computations (which is not feasible in general, I guess).

[nuest]

Hi @mstimberg - good catch! I think we should strike the "write a Makefile" phrase, also related to #22. It's most important that the codechecker runs the whole workflow, second level goal should be that it is scripted/automated (if not already and not too much extra work). The codechecker may use any way to achieve that, and a Makefile is probably just the personal preference of @sje30 and me. Stephen, any thoughts?

Your concerns regarding the reproduction of the check and a clear documentation of the environment are very helpful!

Would it be better if we remove any environment documentation from the report that is related to the rendering of the report itself? The report should only document reproduction environment information, not authoring environment stuff.