"what are the benefits of Guile Emacs in 2020 and after?"
Here is what Andy Wingo had to say on the subject[1]:
Guile can implement Emacs Lisp better than Emacs can.
We can compile Emacs Lisp to Guile's VM, and make C shims to present a suitably compatible C interface to the rest of Emacs' C primitives.
No one will notice! Except that after a switch, Emacs would be faster, more powerful, and have the ability to access all of Guile's facilities -- the Scheme language, other languages implemented for Guile (Javascript, Lua, ...), a proper ffi, dynamically loadable libraries, a module system, the numeric tower (rationals, bignums, etc), Guile's existing libraries, delimited continuations (!), fast bytevector access, native threads, etc.
On a language level, I am convinced that not only can Guile implement Emacs Lisp, but it can provide a coherent story for tight interaction between e.g. minor modes in Javascript and Scheme and Elisp, etc. More details will come later, though feel free to bring up specific points.
There have been some suggestions on the list that "Guile people want to rewrite Emacs in Scheme". Though Scheme does have some facilities to offer, personally I do not believe in rewriting software. I've failed enough times, thank-you-very-much -- and if we could retrofit a compiler into Guile in two years via correctness-preserving transformations, surely we can replace the implementation of Elisp without causing problems.
My goal is to make it the obvious decision for the Emacs maintainers to say, "yes, let's switch to Guile's implementation of elisp, because it will make Emacs better and have no drawbacks". We're not there yet -- and that's why I wanted to put off this mail -- but we'll get there.
I think it's safe to say at this point -more than 10 years later- that Andy Wingo was very optimistic.
Absolutely nothing of what he described in that email materialized. And it's no wonder it didn't, since a lot of it is simply wishful thinking. In addition, he didn't address the fact that there are political and not technical reasons behind the absence of certain features (e.g. FFI). They could be done today, without Guile, assuming a consensus on the political front existed.
On the other hand, Emacs Lisp has gotten a lot better and not only knocked out some of the features mentioned by Andy (dynamic modules, native threads) but is also making progress towards new directions (GCCEmacs).
And as I described in my other comment, I completely disagree with his premise that having Emacs be extended in JavaScript, Lua etc would be a plus. The last thing you do with a cohesive ecosystem like Emacs Lisp's is to introduce needless fragmentation that takes away its core strengths.
"I completely disagree with his premise that having Emacs be extended in JavaScript, Lua etc would be a plus"
I also disagree, but that boat sailed long ago... For years now Emacs has allowed dynamic module loading, which lets Emacs be extended by C or anything that compiles down to C (which is pretty much any language). So it can already be done.
What Guile will offer is in some cases skipping the module compilation step, and allow more direct access to languages which can be used through the Guile interpreter.
Furthermore, whether you or I like the idea of extending Emacs through Javascript or Lua, there are plenty of other Emacs users who do, and I say let them at it!
Emacs needs more new users, and if some can be attracted through compatibility with their favorite language, I welcome them.
I'd personally like to keep Emacs pure and Lispy, but that's already not true and it's very unlikely we're going to go back.
That's not to mention that significant parts of core Emacs has been written in C for decades now. It'd be much nicer if some of that C code could be rewritten in Scheme.
Going through C is not a counterargument, since it's doesn't compete with interactive development in Emacs Lisp. The technical drawbacks are many and for that very reason you see no fragmentation in the ecosystem today.
Being able to write a Lua or JavaScript function on the fly, inside Emacs, and having Guile execute it would make these languages direct competitors to Emacs Lisp. You would expect to see newcomers take the easy way out and never bother with Emacs Lisp. Fragmentation. Dilution. That is the core issue.
You seem to think that these sort of sacrifices should be made in the interest of popularity. I don't agree at all.
Emacs has no reason to make populist compromises. It has certain qualities that have stood the test of time and have kept it extremely powerful. To go on the populist road would ensure that Emacs becomes just like everything else out there. A mediocrity.
I would firmly oppose switching the entirety of the Emacs ecosystem to JS, Lua, Rust or any other non-Lispy language, but that's not what we're talking about here. If an occasionaly JS or Lua package becomes available through Guile, what's the harm?
Anyway, I don't see this as the primary benefit of Guile Emacs.. the rest benefits that Andy Wingo mentions are much more interesting and valuable to me, not JS/Lua compatibility.
I'd also much prefer to write my own Emacs code in Scheme than eLisp (though, again, Guile Emacs does not entail replacing eLisp packages with Scheme -- you can still use eLisp if you want, and existing eLisp packages will still work).
I don't think you'll be able to control occasional or not.
This is a Pandora's box [1] situation. Either you open it or you don't. And having alternative languages as direct competitors to Emacs Lisp, available for interactive development, would certainly be blowing the box apart.
I wouldn't mind Scheme as an alternative, the only alternative, except I don't think I'd use it myself since I would still see Emacs Lisp as domain-superior.
Agreed on the rest, except as I said, none of that materialized from the Guile camp whilst we had plenty of that goodness emerge in core Emacs.
The problem with your point of view is that if it came to pass, then anyone who wanted to work on Emacs packages would have to learn multiple programming languages when before they only needed one, Elisp.
Having packages in multiple languages hurts the ability of people to help contribute to each others' work (especially if they don't have time or space to learn multiple languages).
Agreed, and I was disappointed about xi-editor's approach of using an RPC so any language can be used for extension. For all its issues at least elisp is ubiquitous, and contributes to feature discoverability.
This smacks of the https://en.wikipedia.org/wiki/The_Innovator%27s_Dilemma. If we're afraid that people will want to write packages in Blub for some reason, and we refuse to accommodate Blub at all, we're just driving everyone to some other editor that embraces it.
I think the difference is that Emacs isn't a company, and it doesn't need to beat others in a competition, it just needs enough contributors to keep moving forward. Not everyone needs to use Emacs for Emacs to succeed in its goals.
True, I was thinking of http://www.paulgraham.com/avg.html (where Lisp was Viaweb's "secret weapon"). And I'm worried about Emacs ending up with too small an audience of package maintainers to cover mainstream workplace use cases.
It might be the case that being able to use Emacs Lisp outside of the Emacs editor in a (potentially) more robust environment designed for general purpose programming could potentially give Emacs Lisp more legitimacy as a programming language in its own right (not just "a scripting language for an editor").
However, it seems that some people have another goal in mind with the Guile-based Elisp: actually replacing Emacs Lisp, or phasing it out, or whatever. I think that is a terrible idea. I also think their goal of mixing languages inside of Emacs would have the negative consequences you describe.
FWIW I think you could say FFI is possible. I've written a Golang module for emacs and build using -buildmode=c-shared and loaded it into emacs. I think for generic FFI you could probably wrap some sort of libffi.so using elisp.
There exist multiple Emacs FFI implementations [1], [2].
None of them are included in Emacs though and that's the main issue. For something like FFI to be maximally useful, you do want it to be part of Emacs and not an additional -compiled- add-on.
The problem is not technical, but political. For years, RMS has opposed an Emacs FFI due to his belief that it could be used to subvert Emacs goals by making it possible to do proprietary extensions.
Here is what Andy Wingo had to say on the subject[1]:
Guile can implement Emacs Lisp better than Emacs can.
We can compile Emacs Lisp to Guile's VM, and make C shims to present a suitably compatible C interface to the rest of Emacs' C primitives.
No one will notice! Except that after a switch, Emacs would be faster, more powerful, and have the ability to access all of Guile's facilities -- the Scheme language, other languages implemented for Guile (Javascript, Lua, ...), a proper ffi, dynamically loadable libraries, a module system, the numeric tower (rationals, bignums, etc), Guile's existing libraries, delimited continuations (!), fast bytevector access, native threads, etc.
On a language level, I am convinced that not only can Guile implement Emacs Lisp, but it can provide a coherent story for tight interaction between e.g. minor modes in Javascript and Scheme and Elisp, etc. More details will come later, though feel free to bring up specific points.
There have been some suggestions on the list that "Guile people want to rewrite Emacs in Scheme". Though Scheme does have some facilities to offer, personally I do not believe in rewriting software. I've failed enough times, thank-you-very-much -- and if we could retrofit a compiler into Guile in two years via correctness-preserving transformations, surely we can replace the implementation of Elisp without causing problems.
My goal is to make it the obvious decision for the Emacs maintainers to say, "yes, let's switch to Guile's implementation of elisp, because it will make Emacs better and have no drawbacks". We're not there yet -- and that's why I wanted to put off this mail -- but we'll get there.
[1] - https://lists.gnu.org/archive/html/emacs-devel/2010-04/msg00...