I think you're making very reasonable points in this thread :)
> This post persuaded me to not take much of a look at Janet.
I feel I should say that you shouldn't form too much of an opinion about Janet just because one person used it to prototype a weird macro system. This concept is not, like, part of Janet or anything -- Janet macros are basically Common Lisp's, but with an elegant solution to the function hygiene problem -- and I don't think that I'm even a very representative Janet user (as a big macro fan). Janet is a very nice Lua and Perl alternative even if you never use it to write a single macro! Its text parsing facilities alone are worth a look.
> The problem Lisp macros solve is "this code is more verbose/ugly/boilerplate-y/etc. than I want it to be", which just isn't the problem you're writing the program to solve.
I think this misses what I see as "the point" of macros, which is to be able to make a tiny language core. Consider "and:" I would be sad to program in a language without a short-circuiting "and." So most languages special-case that, right? But lisps don't. Macros mean that you don't have to make "and" a built-in part of the language. Or, I dunno, "defn." "for." Janet's only iteration primitive is "while," and then standard library uses macros to implement for, each, list comprehensions, etc.
And I feel like it's totally fair to not care about that all. After all, why does it matter to you, the programmer, whether "and" is special-cased in the language what is implemented as a macro in "user space?"
> Whenever you reach for a macro, there's another tool you could be reaching for to solve the actual problem at hand. At the very least, you can just write the code the macro would expand into. There's inherently never a case where the macro is the only way to solve the problem.
Of course! But I feel like this doesn't make a compelling point against macros to me, because you could say exactly the same thing about first-class functions, or generic types, or any other language feature. And people have!
> If you're good at writing macros, you won't always get burned by them, but nobody is ever perfect at writing macros, so everyone gets burned sometimes. If you're writing software that you actually need to work, the risk is rarely worth it.
This is, I think, a very good point. Which is why lots of languages (Racket, Scheme, Clojure) have macro systems that make it almost impossible to write macros that "burn you," if I understand your meaning correctly. But others don't! Janet is a bit weird in that it's a recent language that does not have a hygienic-by-default macro system.
> When I have written production code in a Lisp (mostly Clojure) I've rarely reached for macros, and often bugfixes have been removing a macro that was of the "so preoccupied with whether or not they could, that they didn't stop to think if they should" variety. And if you spend enough time avoiding and removing macros, you start to wonder why you're destroying your eyesight trying to match parentheses, when the entire reason for the parentheses is to enable something you have to avoid and remove.
I can certainly see how such an experience would sour you on macros. I am fortunate that I have never had to maintain buggy legacy macros -- sounds awful. I would again point out that you could have a similar experience with... inheritance, async/await, static type checking, I dunno. Anything applied poorly can seem like a terrible idea. You could write off entire languages if you only saw terribly written code in that language. But macros applied well can be great!
When I think of all the language features that have been added to, say, JavaScript over the last twenty years, and how many of them could have been written as macros, in a way that works for all browsers, without a need for something like Babel... it's a little silly that JavaScript developers had to wait for async/await to become an official language feature, when Clojure just implemented it as a library.
Counterpoint: it's reasonable to argue that it's bad to add language features to a language, because it makes your code harder to understand for the median developer. But of course the lack of macros doesn't stop language fragmentation, it just relegates that to a smaller group of programmers who have the time or inclination to write full parsers and compilers -- see JSX, Svelte... Clojure itself! Or Kotlin, or any other JVM language.
(Not that macros prevent fragmentation. If anything, the fact that macros make it so easy to implement a lisp is why we have so many different lisp implementations...)
> And don't get me wrong: macros are cool. "Because I like them" is a totally valid reason to write macros and Lisp.
Yeah :)
I wanted to respond to something else you said elsewhere:
> As a developer I don't have the resources to design and test my code to the extent that the language creators do.
I think that blurring the line between "code users can write" and "code language authors can write" is the point! To give programmers the resources to design and test code on the same level as language implementors. (Which, again: why should I care?)
> I feel I should say that you shouldn't form too much of an opinion about Janet just because one person used it to prototype a weird macro system. This concept is not, like, part of Janet or anything -- Janet macros are basically Common Lisp's, but with an elegant solution to the function hygiene problem -- and I don't think that I'm even a very representative Janet user (as a big macro fan). Janet is a very nice Lua and Perl alternative even if you never use it to write a single macro! Its text parsing facilities alone are worth a look.
Sure, the alternative to hygenic macros was what originally made me curious about that, but now I've learned enough about that to satisfy my interest. However, I'll retract my statement that I'm not going to look into it further, because I had forgotten I was also interested in how they get it to embed in C programs.
> Of course! But I feel like this doesn't make a compelling point against macros to me, because you could say exactly the same thing about first-class functions, or generic types, or any other language feature. And people have!
Right, but the complete argument isn't "there are other ways to do this", the argument is "there are other ways to do this, and nearly every one of them is less error-prone".
> When I think of all the language features that have been added to, say, JavaScript over the last twenty years, and how many of them could have been written as macros, in a way that works for all browsers, without a need for something like Babel... it's a little silly that JavaScript developers had to wait for async/await to become an official language feature, when Clojure just implemented it as a library.
I'll point out that JavaScript already had async callbacks and promises (implemented as a library) when async/await was added as a third way to do basically the same thing, resulting in JS codebases that now have half-baked glue to make the three ways work together. I'm not sure anybody was waiting for async/await to do anything that couldn't already be done, and the churn of reimplementing working code to use a new feature didn't do anyone much good. But that's sort of a tangent.
> Counterpoint: it's reasonable to argue that it's bad to add language features to a language, because it makes your code harder to understand for the median developer. But of course the lack of macros doesn't stop language fragmentation, it just relegates that to a smaller group of programmers who have the time or inclination to write full parsers and compilers -- see JSX, Svelte... Clojure itself! Or Kotlin, or any other JVM language.
Language fragmentation is a problem, but it's not the problem I'm talking about.
If I write code in a popular programming language such as Python, JavaScript, C++, Clojure (sans macros), etc., I create bugs, and I can be reasonably certain that those are my fault. I've been writing C longer than anything, and I've never found a bug in GCC or Clang in over 20 years (okay, there are a few that were retroactively declared features and forever-supported, but that's a separate issue). As The Pragmatic Programmer says, "Select isn't broken", and Coding Horror says, "It's always your fault"[1]. It's not necessarily rare for a popular programming language to have bugs, but it's extremely rare that you'll be the first one to find them.
The same is true for popular libraries and whatnot that ship with the language, which is why I'm not particularly concerned about "defn" or "for" are macros. Those are macros in most implementations, I'm aware, but they're really-well-tested macros because pretty much every Lisp developer to ever write a significant amount of Lisp has tested them. "defn" and "for" aren't broken.
If I write code in the half-baked DSL written by Bob two cubicles over using macros. That code definitely has bugs, and it's very likely I'll be the first to find them. Not to hate on Bob too much: if I wrote macros they'd have bugs too.
And sure, as you've said, Bob can write buggy functions too. The difference is, functions and I have really good boundaries. When I call a function it doesn't touch my code, and I don't touch it's code, and the expressions I pass into Bob's functions only get executed once, and the stack traces all have very understandable corresponding line numbers, and most of the time it's very easy to figure out if the bug is in Bob's function or my code calling Bob's function. And if it's in Bob's code I write a unit test and fix it, and if I'm feeling cheeky I send him a screenshot, and if it's in my code I fix it and git rebase my mistake out of existence to hide my shame.
Macros don't have those boundaries. The expression I pass as an argument to a macro might get called once, twice, ten times, or not at all, with any side effects of that occurring each time. Symbols might get leaked. If you pass (+ (* m x) b) into a macro it can do stuff like flatten a parent s-expression too far make that into (+ * m x b) and even that simple issue can be hard to debug because the line numbers get split up so you have to figure out what's going on. So you can't really tell whether the problem is in the macro or the code calling the macro. And you don't even know when you have to be careful about this, because it's not always obvious whether the code you're calling even is a macro.
Hygenic macros do help, but they don't eliminate all of these problems.
And half the time when I git blame, it wasn't even Bob who wrote the macro, it was me, five years ago. Ain't that embarrassing.
> I think that blurring the line between "code users can write" and "code language authors can write" is the point! To give programmers the resources to design and test code on the same level as language implementors. (Which, again: why should I care?)
And that's my point: macros don't give you thousands of programmers to test your language you made out of macros. So that's why I care whether it was me or Bob or the Common Lisp team who implemented the language: when the Common Lisp team implements the language it doesn't matter if they use macros or assembly because thousands of people will run the code before I even get a chance and they'll suss out the vast majority of the bugs and issues before I have to deal with them. When me or Bob implements the language, it's me, Bob, or the intern who has to suffer the consequences.
Codebases that use extensive macros to create DSLs eventually become write-only. The power and readability you see in toy examples and in the short run in your own code, rarely plays out in the long run, and when it does play out it's because of extensive testing and work--a lot more work than Bob and I have the bandwidth for.
> the expressions I pass into Bob's functions only get executed once
Sure not in functional programming languages, where one can pass functions or data structures which store functions. The code passed gets executed an arbitrary number of times and in different contexts.
> macros don't give you thousands of programmers to test your language you made out of macros
One does not need thousands of programmers. That's misguided. Macros can also be tested and used in any size of development context.
I'm not going to engage further with you if you continue to approach this as a religious zealot who will do anything to defend your precious macros, including:
1. Quoting me out of context (literally not even whole sentences) and ignoring anything I say that you don't have a convenient, shallow response to.
2. Assuming I have absolutely no knowledge of Lisp basics like higher-order functions. Yes, I'm aware of higher-order functions, and that's an exception which doesn't fundamentally change my point.
3. Posting irrelevant code snippets with no explanation.
It's quite possible for two people to have different experiences that lead them to have two different beliefs. My experience is that macros don't turn out to be a good tradeoff in the long run in most cases. I'm sure you have some experience that leads you to believe that macros are the second coming of Buddha or whatever, but that's not my experience. And given it's just a feature of a language which isn't that widely used, this isn't some life or death situation where either of our opinions are some sort of moral failing. The stakes are not that high and you can afford to be kind and thoughtful.
If you're willing to approach the discussion in that way I'll be happy to discuss this with you further. Otherwise this will be my last response to any of your posts.
> Right, but the complete argument isn't "there are other ways to do this", the argument is "there are other ways to do this, and nearly every one of them is less error-prone".
I feel like there's such a wide variety of things that you could be trying to do with macros that this definitely true a lot of the time.
> The same is true for popular libraries and whatnot that ship with the language, which is why I'm not particularly concerned about "defn" or "for" are macros. Those are macros in most implementations, I'm aware, but they're really-well-tested macros because pretty much every Lisp developer to ever write a significant amount of Lisp has tested them. "defn" and "for" aren't broken.
So I'd argue for this differently: defn and for and other pervasive macros aren't safe because they're well-tested, they're safe because they're trivial. You know that (defn ...) is short for (def (fn ...)). You know exactly the code that that macro expands to. And you can choose to type (def (fn ...)), or you can choose to type (defn ...). Same with short-circuiting "and", or "for", or "+=", or whatever.
It sounds like you're mostly talking about complex, hairy macros -- macros where you don't know exactly what code they expand to. But I dunno, the fact that macros can create monstrosities doesn't mean that you should have to type (def (fn ...)). It's a feature with an extremely broad scope, and can definitely be mis-used.
> I'll point out that JavaScript already had async callbacks and promises (implemented as a library) when async/await was added as a third way to do basically the same thing, resulting in JS codebases that now have half-baked glue to make the three ways work together. I'm not sure anybody was waiting for async/await to do anything that couldn't already be done, and the churn of reimplementing working code to use a new feature didn't do anyone much good. But that's sort of a tangent.
This is a very good point. Destructuring assignment, maybe? Arrow functions? I think we agree that some language features are good additions, even if I picked a bad example :)
> Codebases that use extensive macros to create DSLs eventually become write-only. The power and readability you see in toy examples and in the short run in your own code, rarely plays out in the long run, and when it does play out it's because of extensive testing and work--a lot more work than Bob and I have the bandwidth for.
I dunno! I can certainly see how an extensive macro-based DSL could degrade to something write-only.
I don't know when the long run starts, but my experience with macros is that they're just a huge productivity benefit that I wouldn't want to give up. I haven't seen a codebase degrade into an idiosyncratic mess -- perhaps, in part, because there's a lot more friction to writing macros in OCaml than in Clojure, so macros are only used where they provide a substantial and obvious benefit. Or perhaps it's that typechecking makes it much harder to write poorly-behaved macros. Or perhaps I just got lucky with my cubicle assignment :)
> So I'd argue for this differently: defn and for and other pervasive macros aren't safe because they're well-tested, they're safe because they're trivial. You know that (defn ...) is short for (def (fn ...)). You know exactly the code that that macro expands to. And you can choose to type (def (fn ...)), or you can choose to type (defn ...). Same with short-circuiting "and", or "for", or "+=", or whatever.
That's probably true of a lot of macros, but I would be surprised if the macros included in Lisp are all of that triviality.
And the biggest perceived payoff of macros developed in user-space is going to be the macros that aren't trivial.
> This is a very good point. Destructuring assignment, maybe? Arrow functions? I think we agree that some language features are good additions, even if I picked a bad example :)
Sure, I see your bigger point, and arrow functions are a good example of it. I mean, obviously you can do `function(foo) { ...; return bar; }` and then use bind() to fix the fact that that doesn't close around the right things at all, but arrow functions are just so much clearly less error prone that there's no real argument they're an improvement.
I'm actually not aware of destructuring assignment being available in JS, but it's great from Erlang, so I'll have to look into that.
This is something I think about a lot because I'm writing a compiler/interpreter for my own programming language: programming languages often introduce too many features before they're really thought out well enough--meanwhile lots of those features would be just fine as libraries. The result is an inconsistent language with lots of ways to do the same thing that don't play well together. C++ has this problem so badly that they've developed toolsets for subsetting--i.e. choosing the set of features of the language you use and returning errors if you use features outside that set. And then there's stuff like C which just solves that problem by rarely adding features.
The most powerful languages, I think, tend to be ones that didn't add a ton of features, and made good choices on the features they did add. For one example, I think Erlang got their threading features really, really right, and a lot of other programming languages are going to regret going with other threading models and bolting on an Erlang-style model later.
> I don't know when the long run starts, but my experience with macros is that they're just a huge productivity benefit that I wouldn't want to give up. I haven't seen a codebase degrade into an idiosyncratic mess -- perhaps, in part, because there's a lot more friction to writing macros in OCaml than in Clojure, so macros are only used where they provide a substantial and obvious benefit. Or perhaps it's that typechecking makes it much harder to write poorly-behaved macros. Or perhaps I just got lucky with my cubicle assignment :)
I dunno! I haven't written much OCaml, though I've written some F# which is supposed to be pretty similar. I do think making powerful-but-dangerous features harder to use can be a good strategy for dissuading their use except when they're really needed, so you might be on to something there.
> This post persuaded me to not take much of a look at Janet.
I feel I should say that you shouldn't form too much of an opinion about Janet just because one person used it to prototype a weird macro system. This concept is not, like, part of Janet or anything -- Janet macros are basically Common Lisp's, but with an elegant solution to the function hygiene problem -- and I don't think that I'm even a very representative Janet user (as a big macro fan). Janet is a very nice Lua and Perl alternative even if you never use it to write a single macro! Its text parsing facilities alone are worth a look.
> The problem Lisp macros solve is "this code is more verbose/ugly/boilerplate-y/etc. than I want it to be", which just isn't the problem you're writing the program to solve.
I think this misses what I see as "the point" of macros, which is to be able to make a tiny language core. Consider "and:" I would be sad to program in a language without a short-circuiting "and." So most languages special-case that, right? But lisps don't. Macros mean that you don't have to make "and" a built-in part of the language. Or, I dunno, "defn." "for." Janet's only iteration primitive is "while," and then standard library uses macros to implement for, each, list comprehensions, etc.
And I feel like it's totally fair to not care about that all. After all, why does it matter to you, the programmer, whether "and" is special-cased in the language what is implemented as a macro in "user space?"
> Whenever you reach for a macro, there's another tool you could be reaching for to solve the actual problem at hand. At the very least, you can just write the code the macro would expand into. There's inherently never a case where the macro is the only way to solve the problem.
Of course! But I feel like this doesn't make a compelling point against macros to me, because you could say exactly the same thing about first-class functions, or generic types, or any other language feature. And people have!
> If you're good at writing macros, you won't always get burned by them, but nobody is ever perfect at writing macros, so everyone gets burned sometimes. If you're writing software that you actually need to work, the risk is rarely worth it.
This is, I think, a very good point. Which is why lots of languages (Racket, Scheme, Clojure) have macro systems that make it almost impossible to write macros that "burn you," if I understand your meaning correctly. But others don't! Janet is a bit weird in that it's a recent language that does not have a hygienic-by-default macro system.
> When I have written production code in a Lisp (mostly Clojure) I've rarely reached for macros, and often bugfixes have been removing a macro that was of the "so preoccupied with whether or not they could, that they didn't stop to think if they should" variety. And if you spend enough time avoiding and removing macros, you start to wonder why you're destroying your eyesight trying to match parentheses, when the entire reason for the parentheses is to enable something you have to avoid and remove.
I can certainly see how such an experience would sour you on macros. I am fortunate that I have never had to maintain buggy legacy macros -- sounds awful. I would again point out that you could have a similar experience with... inheritance, async/await, static type checking, I dunno. Anything applied poorly can seem like a terrible idea. You could write off entire languages if you only saw terribly written code in that language. But macros applied well can be great!
When I think of all the language features that have been added to, say, JavaScript over the last twenty years, and how many of them could have been written as macros, in a way that works for all browsers, without a need for something like Babel... it's a little silly that JavaScript developers had to wait for async/await to become an official language feature, when Clojure just implemented it as a library.
Counterpoint: it's reasonable to argue that it's bad to add language features to a language, because it makes your code harder to understand for the median developer. But of course the lack of macros doesn't stop language fragmentation, it just relegates that to a smaller group of programmers who have the time or inclination to write full parsers and compilers -- see JSX, Svelte... Clojure itself! Or Kotlin, or any other JVM language.
(Not that macros prevent fragmentation. If anything, the fact that macros make it so easy to implement a lisp is why we have so many different lisp implementations...)
> And don't get me wrong: macros are cool. "Because I like them" is a totally valid reason to write macros and Lisp.
Yeah :)
I wanted to respond to something else you said elsewhere:
> As a developer I don't have the resources to design and test my code to the extent that the language creators do.
I think that blurring the line between "code users can write" and "code language authors can write" is the point! To give programmers the resources to design and test code on the same level as language implementors. (Which, again: why should I care?)