Common Lisp Dependency Vendoring with Submodules

Oh, I see you disabled JavaScript. Keep up the good work, my fellow cleanweb person!

Notice that there might be trace amounts of JS, used for e.g. runnable JS code blocks. JS is not required for use of the website though, it’s only enhancing the existing functionality.

A bright thumbnail with handwritten digital text on it. In the center, there’s a composition of four schematic boxes of differing sizes spread around. Some have lambdas (Greek letter, a symbol of functional programming and Lisp) drawn on them. A bright “SUBMODULE” is written over them. In the corners, attributions to “aartaka.me” and “Artyom Bologov” are written.

So there are half a dozen of Common Lisp package/system/project managers:

Quicklisp (with Ultralisp distribution)
CLPM
OCICL
Qlot
Vend
Qi

These mostly fall into two categories:

Package/project managers fetching things from the Internet right into the running Lisp image and not touching the project itself.
Project-local package managers that include all the dependencies into the project tree.

Project-local solutions bloat the checkout size and slow down repository cloning. Online ones require network connection. I’m not satisfied with either of the trade-offs, and I think I have a solution to that.

Okay, I’ll need a more involved problem statement than that. The features I’m looking for in my dependency management journey are:

Fully local build options
CLI/UNIX-friendliness, especially for running tests
Reproducibility, of a weaker kind—Lisp deps that don’t change underneath me
Not needing bloated Swiss knife binaries just to manage dependencies
Option to use both local deps and Quicklisp / Ultralisp / CLPM / whatever
Custom REPLs with dependencies pre-loaded
Not reinventing the wheel

Submodules #

Git submodules are this kind of solution.

They allow the library developer to pin dependencies precisely
The library user only needs to fetch the dependencies they require, when they need them
There’s no need to install any heavyweight tools like Roswell
Repository size doesn’t grow unboundedly for those that don’t want it
It’s CI-friendly and well-integrated with the UNIX tooling

So my setup is:

Make a project/library and write the code interactively using Quicklisp for dependencies.
Once the library is more or less stable dependency-wise, run a 100 LoC dependency-resolving script using Quicklisp metadata.
Run all the commands it suggests (it doesn’t run anything itself, by design!)
Add some leftover deps that Quicklisp doesn’t have or that are special in some other way. (I tend to use Nyxt’s .gitmodules to get dependency locations (seems like working on Nyxt left an imprint, huh?))
And then set CL_SOURCE_REGISTRY to the path with dependencies (usually repository root) when I need to run tests or a custom REPL.

Stripped-down project Makefile (yes, I use Make for Lisp projects) then looks like:

LISP ?= sbcl
LISP_FLAGS ?= ...

export CL_SOURCE_REGISTRY ?= $(PWD)//

executable:
	$(LISP) $(LISP_FLAGS) --eval '(require "asdf")' --eval '(asdf:make "system")' --eval '(uiop:quit)'

.PHONY: tests
tests:
	$(LISP) $(LISP_FLAGS) --eval '(require "asdf")' --eval '(asdf:test-system "system")' --eval '(uiop:quit)'

.PHONY: repl
repl:
	$(LISP) $(LISP_FLAGS) --eval '(require "asdf")' --eval '(asdf:load-system "system")'

Makefile for submodule vendoring approach

Things to note:

exported/shared CL_SOURCE_REGISTRY with double slash: That’s a vital hack mentioned by Alexander Artemenko on Reddit. This double slash means a given directory is an ASDF “tree” to be searched for systems recursively. Here’s the syntax explanation in ASDF manual.
--eval '(require "asdf")' --eval '(asdf:load-system "system")': All the bootstrapping is basically two expressions, all reliant on ASDF. Given that CL_SOURCE_REGISTRY is set up properly, all the dependencies are there. Build, test, and launch REPLs all you want. No Quicklisp needed.
repl: REPL is the same call to $LISP, but without the --eval '(quit)' It’s both rlwrap compatible and has all the necessary libraries accessible.
Swank/Slynk integration: Easy to support by adding two more ASDF subsystems and running them from the Makefile.

A thin strip of an image with lots of boxes, evidently written by an amateur. Some boxes have text on them: “deps”, “CI”, “REPL”, “resilience” (overflowing the box,) and “compat”.

This setup gives me:

Ability to clone the project without dependencies for use with Quicklisp—git clone project-url
Ability to clone the project with all the dependencies—git clone --recursive project-url
Ability to get only a subset of dependencies I need (say, for libraries not on Quicklisp yet)—git submodule update --init -- deps/something or git submodule update --init -- deps/ to update them all
Global reuse of dependencies with custom ASDF registries (as in .config/common-lisp/source-registry.conf.d/asdf.conf)
Option to run tests from CLI and on CI with all the dependencies accessible to them
Precisely pinned dependencies possibly going fresher than Quicklisp
Custom REPLs in the dev environment
Total Quicklisp / CLPM / OCICL independence, with the only hard project dependency being ASDF (and Quicklisp project repositories at vendoring time, but that’s minor and easy to have locally)
An option to use Quicklisp etc. nonetheless, just do make CL_SOURCE_REGISTRY=path/to/systems// and your dependencies are replacing the submodule ones

What not to like about it?

The Problem With ~/common-lisp #

Thanks to Konrad Hinsen for making me think about that! One scenario where this submodule-driven workflow might break is when you clone the repo into ~/common-lisp. And then there’s another version of one of the deps in there, provoking a version clash.

Cloning into ~/common-lisp is a bad practice that should be eradicated in favor of properly compartmentalized registries. But alas. Anyway, if you clone into ~/common-lisp or ~/.local/share/common-lisp/source/, then do a non-recursive clone instead. If you cloned the library there, you likely have all the dependencies available already. In the very same ~/common-lisp, no doubt. So no need for submodules.

But better not use ~/common-lisp at all, global state bad.

To see how the whole setup around submodules looks, you can check out my Quickproject template and its makefile and .asd file.

Common Lisp Dependency Vendoring with Submodules

Submodules #

The Problem With ~/common-lisp #

Leave feedback! (via email)