Reusable CLI integration tests

On my path of moving lab’s code to more human friendly program, I usually write some CLIs, to ease configuration and deployment. When developing the client, I want to test it, and see how complex it is to use it. The best language to express that is a shell as it is probably how the user will use it. But how to integrate it to the classic cargo/go test? What to do if we want to change language (especially during prototyping)? Let me explain how I’m writing theses tests now.

I start with having a root test directory where I write each test case in a separated file. Each imports a sh “library” containing some common helpers. It mostly contains the usual set -euo pipefail, some mktemp & trap cleanup EXIT QUIT to have a clean test environment. As such, the tests are self-contained so that you can easily run the one you want. You also want to build your project and append its output directory to PATH to find the most recently built binaries. Now, it’s easy to run the stack using find -executable -type f -execdir {} \;! If you find that a test fails, just run it directly. You can also see exactly what the user is doing using set -x.

Happy now? Ha no, right, you want to integrate it with your usual tools. That greatly depends on the language you’re using. You usually have to generate a testcase per executable file, modify the PATH and … and that’s it, nothing much to do here! I’ve already coded some project specific ones, take inspiration from the one for rust and for go. And if you want to change the project’s language, you can reuse all of your integration tests, a sympathetic added bonus.

In summary, you have a language-agnostic, reusable, CLI integrations stack of tests. Personally, I now write all of my integration tests using this principle and it saved me countless hours. It also reduce the friction of changing language, even going as far as having multiple implementation running all the same tests.