An aspect of computational reproducibility which doesn’t get enough mention is the compatability of research performed by different scientists. A scientist claiming to have the next greatest algorithm needs to prove it by providing benchmark tests. However if another scientist claims to have a better algorithm, we now have to go about comparing them in a fair way. For this comparison to be apt without any additional work, their algorithm analysis must be performed in an identical way. This is not true in general however. This essentially means that claims from two scientists can’t really be compared in an apples-to-apples fashion.
My collaborators Vanessa Sochat, Victoria Stodden, and I are trying a new method to promote compatibility between scientific research. The goal of Containers FTW is to foster competitiveness between scientists while also offering a system to constrain their methods enough to allow apples-to-apples comparisons.
Containers FTW aims to provide scientists interested in a common problem to define a container which can be used to solve that problem. Inputs and Outputs can be specified, and containers can then be submitted to that problem and compete against other containers defined for that problem. Given a set of input data, the containers can then be compared based on how well they solve the problem given that input. Kaggle competitions offer something similar to this, however we wish to make those competitions more general.