Don't think of it as modifying the language. Think of it as adding new ways of expressing ideas in the existing language.

These sorts of languages/language extensions pop up all the time in "ordinary" programming. For example, common DSLs we see often.

- regex is a language for string patterns

- JSON is a language for simple nested data

- SQL is a language for expressing DB queries

- OpenMP contains (essentially) a language for parallelizing loops

- (this might sound crazy because it's so common and basically built in standard in almost all programming languages:) infix notation for math

The problem is that many of these are absolute kludges in existing languages. Anybody who has had the displeasure of trying to interpolate DB table names into SQL SELECT strings knows this. Most languages try to make it a bit better by adding preprocessors, using builder-patterns, abusing reflection, or otherwise. Looking under the hood of some languages' implementations of these ideas will often lead you to see the most cursed of programmer things in existence.

Each of those languages (or close syntactic cousins of them) can be written directly in Lisp, as a library, without shoving them into strings, or using external tools, or making custom file formats. They call can exist as a part of the Lisp language as a syntax extension.

Making DSLs requires exercising that muscle a bit; it's non-trivial to design a new and useful language that fits orthogonally into an existing system.

It's like being introduced to OO programming where, at least for me, it made sense but it wasn't immediately obvious what a good object model for a problem might be. It took me lots and trial and error to understand how OO works in "the real world", where it's good, and where it falls short.

Once you get they hang of making DSLs—or even just small but powerful syntax extensions—it's an amazing tool to have at your disposal.

My second thought, which I excluded for brevity, was aren't functions and libraries language extensions? I mean technically you could write C without malloc, but it wouldn't be very useful. Regex is a great example of an embedded DSL, but I've never thought "I wish I could write a macro to change it's behavior on the fly in this special case". The last time I looked at this issue I saw that sure, if you don't have generics, it would be helpful to have a language that could write typed functions for you. That makes sense. Changing the parser and compiler behavior before any runtime code executes could be very interesting. There are certainly enough people waxing poetic about the expressiveness of it to make it worth learning.

Again, you don't want to change regex, you want to create it in the first place.

Maybe a simpler example, what if I wanted this in (pseudo-)C?

    with(s = fopen("/foo/bar", ...)) {
        ...;
        ...;
        println(readln(s));
        ...;
        ...;
     }

When the "with" clause ends, the file will automatically be closed. This is more or less bog standard RAII expressed as a new language construct.

In Lisp, this is a feature you could build and provide as a library feature. In fact, we can do it, straight here in an HN comment.

    (defgeneric destroy (x)
      (:documentation "A destructor for use with the WITH feature".))

    (defmacro with ((var = val) &body body)
      "Python-style RAII macro. VAR will be
       bound to the value of VAL, and upon
       exiting scope, will be destroyed with
       DESTROY."
      (assert (eq = ':=))
      (let ((orig (gensym)))
        `(let* ((,orig ,val)
                (,var ,orig))
           (unwind-protect
               (progn ,@body)
             (destroy ,orig))))

We can register a new destructor easily for open file streams:

    (defmethod destroy ((s stream))
      (close stream))

(We could add some more error handling, etc. to make it more robust.)

Now, we effectively have RAII in Lisp:

    (with (s := (open "/foo/bar" ...))
      ...
      ...
      (write-line (read-line s))
      ...
      ...)

This is now real Lisp code you can actually run, completely integrated into the language, and nicely orthogonal to the existing gamut of features.

This is a concept we can use almost immediately after learning about it, instead of waiting for Python's PEP 6363 to pass with consensus from people who may have different programming goals than you do.

I can see a lot of uses for macros, but in this case C# has a "using" keyword and the designers have put a lot of thought into it and how it is called in a wide variety of situations. Lisp was a language developed in a world when people worked on their own cars, like me. I would have loved Lisp if I had found it earlier, like about the time I rebuilt my VW engine in the 80's.

Many of the language simplifications on more recent C# versions, code generators, Rosylin analysers, expression trees are all features that in Lisp boil down to one thing, macros.

https://github.com/norvig/paip-lisp - Peter Norvig's Paradigm's of AI Programming

https://github.com/norvig/paip-lisp/search?l=Markdown&q=defm... - all references to defmacro in the markdown files

Chapter 3 shows a simple macro, just adding a while loop to the language.

Chapter 9 shows some more complex ones, including a with- macro and a grammar compiler macro.

Chapters 11 and 12 show the development of a Prolog implementation in CL using defmacro to aid in compilation again in Chapter 11.

Chapter 12 shows adding an OO system to the language. Technically not needed with CLOS, but a good demonstration of what can be done with macros.

There are other examples (why I included that search link). Macros let you change the language in ways large and small. Many uses could probably be replaced with functions, though you'd end up having to throw a bunch of quotes about or closures in order to delay processing things. You'd also be delaying that to runtime, which incurs its own penalties compared to macro expansion and compile time (if a compiled CL).

I like this but it seems like there would be many inconsistencies to building your own language without an actual language specification

So, as a designer of miniature languages and syntactic extensions, specify it!

A well written function must also abide by some promise or specification—contracts on the inputs and invariants on the outputs. I don't see why new syntactic structures are much different from that.

I think there's undue spookiness around macros among lots of programmers, especially those who haven't sought to solve some problem with them.

- "It's too powerful for the common person."

- "It's way too likely to make code confusing."

- "It's invariably too hard to reason about."

- "It won't work on a team because everybody is going to make their own weird incompatible language and it's just not scalable."

All of these sentiments are complete bunk, and I argue are against common wisdom around other programming abstractions, like functions or classes.

Extending a language with macros isn't all that different from extending a language with your own functions. At least in principle.

And a lot of what tends to be macros in Lisp turns into functions in eg Haskell. Eg you can define your own new branching and looping constructs (like 'if' or 'while') etc as functions in Haskell, but in Lisp these need to be macros.

Unlike call-by-need evaluation, macros provide the opportunity to compute things at compile time (think lookup tables) or analyze the arguments at compile time (think DB query syntax checking).

In Lisp, you can also define branching and looping as functions, but you need to concede and have the arguments themselves be (lambda) functions.

    (defun while (c body)
      (when (funcall c)
        (funcall body)
        (while c body)))

Yes, I know that laziness doesn't take care of all macros. That's why even Haskell has macros.

Yes, if you wrap your arguments in lambdas, you can write loops like that.

Though to be as performant as the built-in constructs, you need a rather clever compiler that 'removes' the lambdas again.

I don't know how this exactly fits, but you can do crazy stuff like query a SQL db in the compiler to check your Racket program for auto generated variable names. This extends the editor tooling to highlight names which don't match to columns in the DB tables.

I wrote a post about this here[1]

[1]: http://tech.perpetua.io/2022/01/generating-sqlite-bindings-w...