Lisp Logical Pathnames

A soft palette thumbnail with a blueberry yogurt colored background. Text reads “LOGICAL;”, “**;”, “PATNAMES;”, and ”*.*.*;”, imitating a logical pathname pattern. It’s written in soft black with asterisks in flowery purple. Oh, and it’s all handwritten!

Logical pathnames are a powerful feature of Common Lisp. Abstracting system-specific filenames and locations behind a special hostname. Making UNIX and Windows filenames easily representable with a common format. Making build-specific directories discoverable and reusable, regardless of packaging. Allowing weird shortcut names saving typing. Logical pathnames (LP (not Linkin Park) from now) are cool. Except...

LPs are under-specified and implementation-specific
LPs are syntactically restricted
There are no explanations of how to work with LPs on the Internet

So I (as any Lisper) decided to experiment with these obscure things. And figure out how they actually work. Including the gory details of how implementations do LPs and how to use them “portably”.

This post was prompted by Nicolas Martyanoff’s post on pathnames and the mention of logical pathnames there.

Logical Pathname Syntax #

Logical pathnames are host-prefixed semicolon-separated directory+file paths:

;; Implementation-reserved SYS: host
#p"sys:encodings;tools"
;; CCL-specific CCL and HOME
#p"ccl:"
#p"home:ccl-init"
;; Imaginary logical pathnames, might be defined later
#p"junk:test.lisp"
#p"l:"
#p"git:cl-minifloats;cl-minifloats.asd"
#p"site:lisp-design-patterns"

Examples of logical pathnames

The only mandatory part is the host. The rest should be directories (separated by semicolons) and files (regular name.type conventions.) One significant restrictions is that both directories and file names/extensions should only consist of letters, digits, and hyphens. Nothing else. Not even ASCII. Mere hyphenated alphanumerics. (Actually, SBCL and CCL allow arbitrary chars recognizing them as file name parts, while ECL and ABCL error out.)

Which is a significant restriction. But, given that we—people defining logical pathnames—are a programming minority and have a luxury of control over our file naming... We’re fine.

Match Patterns #

But Logical Pathname path syntax is not the only type of syntax one encounters with these. There are also match patterns that define what kind of LPs are valid for a host:

"" ;; Works everywhere except ECL
"*.*"
"**;*.*"
"L:**;*.*.*"
"MAIL;**;*.MAIL"
"CODE;DOCUMENTATION.*.*"

Examples of match patterns

Notice the abundance of asterisks there. These stars are :wild (*) and :wild-inferiors (**) components to be replaced later. Replaced with the content following the host in actual instantiated pathnames. So logical pathnames are actually a small pattern-matching DSL (one among the lots others) transforming paths. Now to actual use:

Translation APIs #

There are only three entry points defining how logical pathnames are processed:

logical-pathname-translations: to define (set) translations,
translate-logical-pathname: to resolve these,
and load-logical-pathname-translations: to do host-loading magic you’ll likely never need.

And that’s it. The rest is handled by Common Lisp internals, resolving logical pathnames when actually using them. (Maybe calling translate-logical-pathname too?) So the most interesting thing about LP API is logical-pathname-translations. Its setter (yes, it’s a setf-able accessor (?)) defines new translation from the setf-ed pattern-translation list:

(setf (logical-pathname-translations "l")
      `(("L:**;*.*" "/home/aartaka/junkyard/test.lisp")))

Simple logical pathname definition

OS portability

Note that I’m using a UNIX-like notation of forward slash separated dirs/files. Which makes my logical pathnames less portable than they might be. If I aimed for portability across Linux, Mac, and Windows, I’d rather do something like (merge-pathnames (make-pathname :directory '(:relative "junkyard") :name "test" :type "lisp") (user-homedir-pathname)) so that directory separators are OS-specific. But this will do for the sake of example and because I mainly use Linux.

Now, all this syntax looks off, so I’m going to define a macro:

(defmacro define-logical-pathname (host (pattern expansion) &rest patterns)
  `(setf (logical-pathname-translations ,host)
         (list (list ,pattern ,expansion)
               ,@(loop for (pattern expansion) in patterns
                       collect `(list ,pattern ,expansion)))))
;; Better now
(define-logical-pathname "l" ("L:**;*.*.*" "/home/aartaka/junkyard/test.lisp"))

Logical pathname definition macro

So what happens is that:

We take a host to add translation to;
We provide a match pattern (or several of them);
Plus the actual path to resolve to;
And setf it to be the (set of) patterns for the given host.

Simple Translation #

This LP I defined is supposed to be a shortcut to an individual file. So it will be optimal to use it as l:. While also minimizing the pattern. So how far can we get the match pattern to retain this constant property?

;; All OK
(define-logical-pathname "l" ("L:**;*.*.*" "/home/aartaka/junkyard/test.lisp")) ;; Version element
(define-logical-pathname "l" ("L:**;*.*" "/home/aartaka/junkyard/test.lisp")) ;; No version
(define-logical-pathname "l" ("**;*.*" "/home/aartaka/junkyard/test.lisp")) ;; Omitted obvious host
;; OK, but only for empty or file-only paths (l:file.type)
(define-logical-pathname "l" ("*.*" "/home/aartaka/junkyard/test.lisp"))
;; OK, but only for empty or extension-less file paths (l:file)
(define-logical-pathname "l" ("*" "/home/aartaka/junkyard/test.lisp"))
;; OK, but only for empty paths (l:) and not everywhere
(define-logical-pathname "l" ("" "/home/aartaka/junkyard/test.lisp"))

Shortening substitutions

Most of these are overkill for a single-file LP. But the range and the possibilities are clear: you can pattern-match paths of varying complexity with these.

Substituting Translation #

The fun stuff starts when you substitute path parts into the pattern:

(define-logical-pathname "a" ("**;*.*" "/home/aartaka/**/*.*"))
;; Same, a standard shortcut for file omission?
(define-logical-pathname "a" ("**;*.*" "/home/aartaka/**/"))

(translate-logical-pathname #p"a:web;logical-pathnames.htm")
;; => #P"/home/aartaka/web/logical-pathnames.htm"

Simple $HOME-resolving logical pathname

Notice the symmetry: we take :wild/* and :wild-inferiors/** from the pattern. And plug them into the translation. That’s the superpower: we can substitute nested paths. Well, as long as there’s not too many wild parts there.

The overall heuristic seems to be:

** is replaced with the arbitrary matching path in the provided logical pathname.

* in directories seem to be replaced in the order of appearance.

(define-logical-pathname "X" ("X:a;*;b;*;*.*" "/hello/*/hi/*/what/*.*"))

(translate-logical-pathname #p"x:a;bonjour;b;barev;greetings.me")
;; => #P"/hello/bonjour/hi/barev/what/greetings.me"

Multiple directory asterisks

Asterisks in file name, type, and version (I know, no one uses versions, which is a shame) are replaced with these from the provided logical pathname.
The rest is matched and replaced verbatim.

Oh, right! We can also provide multiple patterns to match paths against, to be matched in the order of appearance:

(define-logical-pathname "a"
    ("W;**;*.*" "/home/aartaka/web/**/*.*")
    ("G;**;*.*" "/home/aartaka/git/**/*.*")
    ("**;*.*" "/home/aartaka/**/"))

(translate-logical-pathname #p"a:w;logical-pathnames.htm")
;; => #P"/home/aartaka/web/logical-pathnames.htm"

Multi-pattern logical pathname (notice the dispatch prefix, it’s a frequent pattern!)

What if I tried doing something smarter, like repeating the same wildcard twice in replacement?

(define-logical-pathname "a"
    ("G;*.asd" "/home/aartaka/git/*/*.asd"))

(translate-logical-pathname #p"a:g;cl-minifloats.asd")
;; SBCL:
;;  :WILD-INFERIORS is not paired in from and to patterns:
;;  (:ABSOLUTE "G") (:ABSOLUTE "home" "aartaka" "git" :WILD)
;; ECL:
;;  #P"A:G;CL-MINIFLOATS.ASD" is not a specialization of path #P"A:G;*.ASD"
;; CCL:
;;  Error: No translation for #P"a:g;cl-minifloats.asd"
;; ABCL:
;;  Unsupported case in TRANSLATE-DIRECTORY-COMPONENTS.

Uh oh, no overly smart things

So we’re finally hitting the limitations of logical pathnames. They are powerful, but not in the Turing-complete way like other DSLs in CL. Use caution.

Logical pathnames are a nice idea with a lot of potential. But they are under-specified and implementations are inconsistent at times. To summarize the behavior:

Logical pathnames are a DSL for file path resolution.
These are logical host plus semicolon-separated hyphenated alphanumeric directories and files.
- Gotcha: some implementations allow other chars, but that’s not portable.
These get resolved according to the rules provided to logical-pathname-translations.
- Gotcha: rules are resolved in the order they are provided in.
Rules have a left hand side (pattern to match) and right hand side (resulting path.)
- Gotcha: Left and right hand side should (not must—that’s implementation-specific) have matching wild components.
These wild elements match the parts of the actual logical pathname and get inserted into the resulting path.
- One by one and strictly once per wildcard.
Translation happens in translate-logical-pathname, often implicit in using pathname APIs.
Logical pathnames are cool, albeit quite restricted.

So yeah, you can use logical pathnames for their many benefits, as long as you stick to portable patterns. I’m going to use some of these in my setup:

#-ccl
(define-logical-pathname "home"
    ("HOME:**;*.*.*" "/home/aartaka/**/*.*"))

(define-logical-pathname "cfg"
    ("CFG:**;*.*.*" "/home/aartaka/.config/common-lisp/**/*.*"))

(define-logical-pathname "git"
    ("GIT:**;*.*.*" "/home/aartaka/git/**/*.*"))

My potential logical pathnames