Logical pathnames are a powerful feature of Common Lisp. Abstracting system-specific filenames and locations behind a special hostname. Making UNIX and Windows filenames easily representable with a common format. Making build-specific directories discoverable and reusable, regardless of packaging. Allowing weird shortcut names saving typing. Logical pathnames (LP (not Linkin Park) from now) are cool. Except...

LPs are under-specified and implementation-specific
• LPs are syntactically restricted
• There are no explanations of how to work with LPs on the Internet

So I (as any Lisper) decided to experiment with these obscure things. And figure out how they actually work. Including the gory details of how implementations do LPs and how to use them “portably”.

This post was prompted by Nicolas Martyanoff’s post on pathnames and the mention of logical pathnames there.

Logical Pathname Syntax

Logical pathnames are host-prefixed semicolon-separated directory+file paths:

;; Implementation-reserved SYS: host
#p"sys:encodings;tools"
;; CCL-specific CCL and HOME
#p"ccl:"
#p"home:ccl-init"
;; Imaginary logical pathnames, might be defined later
#p"junk:test.lisp"
#p"l:"
#p"git:cl-minifloats;cl-minifloats.asd"
#p"site:lisp-design-patterns"

The only mandatory part is the host. The rest should be directories (separated by semicolons) and files (regular conventions.) One significant restrictions is that both directories and file names/extensions should only consist of letters, digits, and hyphens. Nothing else. Not even ASCII. Mere hyphenated alphanumerics. (Actually, SBCL and CCL allow arbitrary chars recognizing them as file name parts, while ECL and ABCL error out.) (Actually, letters are supposed to be uppercase, and all lowercase ones are converted to uppercase automatically. Quite some historic baggage...)

Which is a significant restriction. But, given that we—people defining logical pathnames—are a programming minority and have a luxury of control over our file naming... We’re fine.

Match Patterns

But Logical Pathname path syntax is not the only type of syntax one encounters with these. There are also match patterns that define what kind of LPs are valid for a host:

"" ;; Works everywhere except ECL
"*.*"
"**;*.*"
"L:**;*.*.*"
"MAIL;**;*.MAIL"
"CODE;DOCUMENTATION.*.*"

Notice the abundance of asterisks there. These stars are () and () components to be replaced later. Replaced with the content following the host in actual instantiated pathnames. So logical pathnames are actually a small pattern-matching DSL (one among the lots others) transforming paths. Now to actual use:

Translation APIs

There are only three entry points defining how logical pathnames are processed:

logical-pathname-translations

to define (set) translations,

translate-logical-pathname

to resolve these,

and load-logical-pathname-translations

to do host-loading magic you’ll likely never need.

And that’s it. The rest is handled by Common Lisp internals, resolving logical pathnames when actually using them. (Maybe calling too?) So the most interesting thing about LP API is . Its setter (yes, it’s a -able accessor (?)) defines new translation from the -ed pattern-translation list:

(setf (logical-pathname-translations "l")
      `(("L:**;*.*" "/home/aartaka/junkyard/test.lisp")))

Now, all this syntax looks off, so I’m going to define a macro:

(defmacro define-logical-pathname (host (pattern expansion) &rest patterns)
  `(setf (logical-pathname-translations ,host)
         (list (list ,pattern ,expansion)
               ,@(loop for (pattern expansion) in patterns
                       collect `(list ,pattern ,expansion)))))
;; Better now
(define-logical-pathname "l" ("L:**;*.*.*" "/home/aartaka/junkyard/test.lisp"))

So what happens is that:

We take a host to add translation to;
1. We provide a match pattern (or several of them);
2. Plus the actual path to resolve to;
3. And it to be the (set of) patterns for the given host.

Simple Translation

This LP I defined is supposed to be a shortcut to an individual file. So it will be optimal to use it as . While also minimizing the pattern. So how far can we get the match pattern to retain this constant property?

;; All OK
(define-logical-pathname "l" ("L:**;*.*.*" "/home/aartaka/junkyard/test.lisp")) ;; Version element
(define-logical-pathname "l" ("L:**;*.*" "/home/aartaka/junkyard/test.lisp")) ;; No version
(define-logical-pathname "l" ("**;*.*" "/home/aartaka/junkyard/test.lisp")) ;; Omitted obvious host
;; OK, but only for empty or file-only paths (l:file.type)
(define-logical-pathname "l" ("*.*" "/home/aartaka/junkyard/test.lisp"))
;; OK, but only for empty or extension-less file paths (l:file)
(define-logical-pathname "l" ("*" "/home/aartaka/junkyard/test.lisp"))
;; OK, but only for empty paths (l:) and not everywhere
(define-logical-pathname "l" ("" "/home/aartaka/junkyard/test.lisp"))

Most of these are overkill for a single-file LP. But the range and the possibilities are clear: you can pattern-match paths of varying complexity with these.

Substituting Translation

The fun stuff starts when you substitute path parts into the pattern:

(define-logical-pathname "a" ("**;*.*" "/home/aartaka/**/*.*"))
;; Same, a standard shortcut for file omission?
(define-logical-pathname "a" ("**;*.*" "/home/aartaka/**/"))

(translate-logical-pathname #p"a:web;logical-pathnames.htm")
;; => #P"/home/aartaka/web/logical-pathnames.htm"

Notice the symmetry: we take / and / from the pattern. And plug them into the translation. That’s the superpower: we can substitute nested paths. Well, as long as there’s not too many wild parts there.

The overall heuristic seems to be:

is replaced with the arbitrary matching path in the provided logical pathname.
• in directories seem to be replaced in the order of appearance.

  (define-logical-pathname "X" ("X:a;*;b;*;*.*" "/hello/*/hi/*/what/*.*"))
  
  (translate-logical-pathname #p"x:a;bonjour;b;barev;greetings.me")
  ;; => #P"/hello/bonjour/hi/barev/what/greetings.me"
  

• Asterisks in file name, type, and version (I know, no one uses versions, which is a shame) are replaced with these from the provided logical pathname.
• The rest is matched and replaced verbatim.

Oh, right! We can also provide multiple patterns to match paths against, to be matched in the order of appearance:

(define-logical-pathname "a"
    ("W;**;*.*" "/home/aartaka/web/**/*.*")
    ("G;**;*.*" "/home/aartaka/git/**/*.*")
    ("**;*.*" "/home/aartaka/**/"))

(translate-logical-pathname #p"a:w;logical-pathnames.htm")
;; => #P"/home/aartaka/web/logical-pathnames.htm"

What if I tried doing something smarter, like repeating the same wildcard twice in replacement?

(define-logical-pathname "a"
    ("G;*.asd" "/home/aartaka/git/*/*.asd"))

(translate-logical-pathname #p"a:g;cl-minifloats.asd")
;; SBCL:
;;  :WILD-INFERIORS is not paired in from and to patterns:
;;  (:ABSOLUTE "G") (:ABSOLUTE "home" "aartaka" "git" :WILD)
;; ECL:
;;  #P"A:G;CL-MINIFLOATS.ASD" is not a specialization of path #P"A:G;*.ASD"
;; CCL:
;;  Error: No translation for #P"a:g;cl-minifloats.asd"
;; ABCL:
;;  Unsupported case in TRANSLATE-DIRECTORY-COMPONENTS.

So we’re finally hitting the limitations of logical pathnames. They are powerful, but not in the Turing-complete way like other DSLs in CL. Use caution.

Logical pathnames are a nice idea with a lot of potential. But they are under-specified and implementations are inconsistent at times. To summarize the behavior:

Logical pathnames are a DSL for file path resolution.
• These are logical host plus semicolon-separated hyphenated alphanumeric directories and files.
  Gotcha: some implementations allow other chars, but that’s not portable.
• These get resolved according to the rules provided to .
  Gotcha: rules are resolved in the order they are provided in.
• Rules have a left hand side (pattern to match) and right hand side (resulting path.)
  Gotcha: Left and right hand side should (not must—that’s implementation-specific) have matching wild components.
• These wild elements match the parts of the actual logical pathname and get inserted into the resulting path.
  One by one and strictly once per wildcard.
• Translation happens in , often implicit in using pathname APIs.
• Logical pathnames are cool, albeit quite restricted.

So yeah, you can use logical pathnames for their many benefits, as long as you stick to portable patterns. I’m going to use some of these in my setup:

#-ccl
(define-logical-pathname "home"
    ("HOME:**;*.*.*" "/home/aartaka/**/*.*"))

(define-logical-pathname "cfg"
    ("CFG:**;*.*.*" "/home/aartaka/.config/common-lisp/**/*.*"))

(define-logical-pathname "git"
    ("GIT:**;*.*.*" "/home/aartaka/git/**/*.*"))