Spack Packaging
One of the most serious barriers to sharing code which scientists publish is the difficulty of creating adequate documentation to enable another person to properly build and run the necessary code. Not only that, but even with the most effort or best intentions, small changes made to their code may not make it into this package they are preparing for their peer scientist. These two reasons underscore the necessity of a better system for sending source code to a peer scientist and having them successfully build and run that code.
Spack is currently the best candidate to solve this problem. It provides a method of building a package along with all dependencies while also specifying for instance, which compiler to use. It allows multiple versions of these packages to be installed simultaneously and manages the running environment for these codes. Spack is also portable, it consists of a single directory in which all of its source code resides. Other codes come close to this functionality, but miss some of it which make them non-ideal candidates. Honorable mentions are: Hashdist, Nix, and EasyBuild. Containers also fall short of this goal as they promote ‘black-box’ sharing whereby binary blobs are packaged into containers. This is better than nothing of course, however sharing the source code should be the end goal of any effort of sharing code.
Currently however, spack falls short of this goal. I am currently working on implementing the following features after which I believe it will be ready to fill this gap.
- Support for easy use of native system packages when they exist
- Ability to automatically detect native system packages
- Ability to create a ‘Spackfile’ similar to a Gemfile describing necessary packages needed in a runtime environment to run or develop a specific package
- Easy method to create a self-contained spack ‘package’ containing all current spack code along with the source code for all necessary dependencies and the work itself. This package should just be able to be unpacked, and then the user should be able to have the ported along spack install and run the code provided
After these features have been properly implemented, I’m planning on creating several demonstration packages and leading tutorials at conferences to spread awareness of this new technology.
