Common Lisp Is Not a Single Language, It Is Lots

The Language?

Probably the biggest thing that "proves" Common Lisp integrity is the book. "Common Lisp The Language". Funnily enough, Wikipedia says

But let's see whether there is The Language in the book. The book is 1000 pages long. And I'm not able to focus on anything longer than 30 pages. So I'll take a different direction: that of a programmer exploring the API. The API of trivial-cltl2, the CLtL portability library. Here's a listing of symbols that trivial-cltl2 exports in addition to standard ones:

(:export #:compiler-let
         #:variable-information
         #:function-information
         #:declaration-information
         #:augment-environment
         #:define-declaration
         #:parse-macro
         #:enclose)

And that's it. Not much. CLtL2 is the same as ANSI CL.

But notice what the list is lacking: the notion of objects. So CLtL2 (as both the book and the language), although including CLOS and other niceties discussed below... Is not object-oriented at its initial core. And all the mentions of CLOS etc. are only happening near the end of the book. An epilogue. Appendix. Erratum.

So we have our first language (out of many): CLtL. A language built out of variables, functions, and maybe structures. Like Scheme, but batteries included. Small and neat nonetheless. Here's how our parser could look in The Language:

;; We don't necessarily need structs if we use lists.
(defstruct element
  (type :paragraph :type keyword)
  (content "" :type string))
(defun parse-elements (stream)
  (if (not (peek-char nil stream nil nil))
      '()
      (cons
       (case (peek-char nil stream)
         (#\= (let ((line (read-line stream)))
                ;; Yes, I know links have text, but bear with me.
                (if (uiop:string-prefix-p "=> " line)
                    (make-element :type :link :content (subseq line 3))
                    (make-element :content line))))
         (#\> (make-element :type :quote :content (subseq (read-line stream) 2)))
         ;; Yes, headings have level. But oh well.
         (#\# (make-element :type :heading :content (subseq line 2)))
         (t (make-element :content (read-line stream))))
       (parse-elements stream))))

No fancy DSLs, no Turing-complete APIs, no OOP. Speaking of which, another Common Lisp language:

Common Lisp Object System

OOP is but a one approach to building Lisp software. There are CL codebases built entirely out of plain functions. Even the ones built entirely in a Scheme-y functional style (I worked with some.) There are others built out of objects and methods on them. There are ones that are built around generics and protocols. And it's quite hard to work with either if one's typical style is not the same as that of the codebase.

So here goes our first non-CLtL language: CLOS. Defining classes and methods on them:

(defclass base-element ()
  ((content :type string
            :initarg :content)))
(defclass paragraph-element (base-element) ())
(defclass quote-element (base-element) ())
(defclass link-element (base-element) ())
(defclass heading-element (base-element)
  ((level :type integer
          :initarg :level)))
(defmethod parse-elements-clos ((stream stream))
  (if (not (peek-char nil stream nil nil))
      '()
      (cons
       (case (peek-char nil stream)
         (#\= (let ((line (read-line stream)))
                (if (uiop:string-prefix-p "=> " line)
                    (make-instance 'link-element :content (subseq line 3))
                    (make-instance 'paragraph-element :content line))))
         (#\> (make-instance 'quote-element :content (subseq (read-line stream) 2)))
         (#\# (make-instance 'heading-element :level 1 :content (subseq (read-line stream) 2)))
         (t (make-instance 'paragraph-element :content (read-line stream))))
       (parse-elements-clos stream))))
(defmethod parse-elements-clos ((string string))
  (with-input-from-string (s string)
    (parse-elements-clos s)))

The code is more verbose than that in The Language. Yet it's also more reliable and introspectable. All these classes add type checking and convenient slot access with e.g. with-slots. The behavior is more overridable with :around methods. And the code can be extended with more classes.

I must admit that I cheated by reusing the code from above. So the parser stays kind of the same procedural piece of code. It's enhanced with more types and MOP introspection. But it still is procedural, because OOP (not the CLOS, but the general idea) is merely a facade over procedural programming, encapsulating the procedural behavior into classes. One can implement it with methods and character dispatch. But that's no longer a CLOS-y code, it's...

Generics And Protocols

Generics are part of CLOS, right? Why make them a separate "language" then? My reasoning is: one can program anything in them without touching classes. And if you can, then it's a separate language alright.

Generics shine the most when there's a library intended for extension. Like lots of libraries by Shinmera or Robert Strandh, where there's

• A general protocol.
• And implementation-specific extensions to it with the actual behavior.

Here's a weird version of the Gemtext parser with generics:

(defgeneric parse-elements-generic (stream state char)
  (:method ((stream stream) (state t) (char null))
    state)
  (:method ((stream stream) (state null) (char null))
    (nreverse
     (parse-elements-generic stream nil (peek-char nil stream nil nil)))))
(defmethod parse-elements-generic ((stream stream) (state list) (char (eql #\=)))
  (let ((line (read-line stream)))
    (parse-elements-generic
     stream
     (cons (if (uiop:string-prefix-p "=> " line)
               (list :link (subseq line 3))
               (list :paragraph line))
           state)
     (peek-char nil stream nil nil))))
(defmethod parse-elements-generic ((stream stream) (state list) (char (eql #\#)))
  (parse-elements-generic
   stream
   (cons (list :heading (subseq (read-line stream) 2))
         state)
   (peek-char nil stream nil nil)))
(defmethod parse-elements-generic ((stream stream) (state list) (char (eql #\>)))
  (parse-elements-generic
   stream
   (cons (list :quote (subseq (read-line stream) 2))
         state)
   (peek-char nil stream nil nil)))
(defmethod parse-elements-generic ((stream stream) (state list) (char character))
  (parse-elements-generic
   stream
   (cons (list :paragraph (read-line stream))
         state)
   (peek-char nil stream nil nil)))

That's where the example stops being useful. Generic style is nice, but this piece of code is by no means representative. I must've used classes for elements. I must've made a simpler protocol. I must'ven't used lists for state. I might have used state machine to drive the parsing.

But even this ugly code has the benefits of generics:

• It is easy to extend: just dispatch over the new character.
• It provides a strict protocol with type checking—if something goes wrong, a no-applicable-method signals and shows the problem.
• It makes the behaviors compose nicely and puts recursion termination into the method of its own.

Try generics for your new projects. They are nicer than I represented them here. Speaking of "nicer", how about some syntactic sugar?

Loop Macro

Using loop is always a risk: some Lispers love it, and some hate it. But one thing is undeniable: it's powerful and reads like English. Well, given that it's used right.

And no, it's not just "another iteration construct". loop has:

• Symbol binding.
• Conditionals.
• Iteration over (almost) all the standard data types.
• Data aggregation.

So loop is quite a free-standing entity. I've proof: I completed two weeks of Advent of Code in it.

So let's try implementing this simple Gemtext parser as one huge loop:

(defun parse-elements-loop (stream)
  (loop for char = (peek-char t stream nil nil)
        while char
        when (eql #\= char)
          collect (list :link (subseq (read-line stream) 3))
        else when (eql #\> char)
               collect (list :quote (subseq (read-line stream) 2))
        else when (eql #\# char)
               collect (list :heading (subseq (read-line stream) 2))
        else collect (list :paragraph (read-line stream))))

Beautiful, isn't it? Ugly, isn't it? Love it or hate it, it's a nice readable syntax. And you can advertise this small language to your C-family friends. "This is The Lisp, try it!" I did this once, and it kinda worked...

Clojure?!

The legend goes: Rich Hickey made Clojure after being irritated by ABCL. So Clojure feels (after programming some sufficiently big projects in it)... Lispy (sorry for the linguistic abuse here). Like an opinionated standard lib over CL. Some reader macros here and there. Square and curly braces. Some nicer functional programming constructs. More consistency and less burden of the past. A fancy OOP system based on interfaces.

We could've had that in CL. And there are projects enabling a "nicer CL", or even outright implementing Clojure in CL! Many CL programmers squint at them. These extensions are going "too far" making a different language out of CL. Yes, yes, yet another Common Lisp language.

Notable Mention: Format

format function is an enormous text formatting DSL with

• Conditionals.
• Iteration.
• Pretty-printing.
• Recursive function calls.
• And something I surely forgot to mention.

Here is how one of my format strings looks like:

"~s (~a bit~:p):
 #b~b, #o~o, #x~x~
~{~&Universal time: ~2,'0d:~2,'0d:~2,'0d ~
~[~;Jan~;Feb~;Mar~;Apr~;May~;Jun~;Jul~;Aug~;Sep~;Oct~;Nov~;Dec~] ~
~a~[th~;st~;nd~;rd~:;th~], year ~a.~} ~
~{~&Approximate UNIX time: ~2,'0d:~2,'0d:~2,'0d ~
~[~;Jan~;Feb~;Mar~;Apr~;May~;Jun~;Jul~;Aug~;Sep~;Oct~;Nov~;Dec~] ~
~a~[th~;st~;nd~;rd~:;th~], year ~a.~}"

Horrible, but it gets the job done. And it's much more effective than writing 100+ lines of printing code.

So while I'm hesitant about calling format another Common Lisp language... It is alien enough to be one.

See also: a case against format!

Conclusion

There's no Common Lisp The Language. There's a set of languages sloppily stitched together. Well, less sloppy than C++ anyway. Agree or disagree, these are quite orthogonal to each other:

• Core Scheme-y Common Lisp.
• CLOS.
• Generics.
• loop macro
• Clojure and Scheme?!
• format strings.

A lot of sub-languages for one (sorry) "language", right?

(Drafts of this post also included sections on namespaces and OS interfaces. I decided to drop them to save space and because they weren't adding much. Lots of languages!)

Leave feedback! (via email)