But what’s the future of Dyalog? With the passing of John Scholes, then the CTO quitting to “work on his first love, compilers” while Dyalog talks of building compilers, then Gitte Christensen saying in a talk that one reason for changing the download terms is that there aren’t enough young people signing up, then after twenty or thirty years of development one person (Marshall) can go into the codebase and get enormous performance improvements..
I don’t at all mean this to be heckling from the peanut gallery, I have great respect for the people, the tools, the company; but from the distant outside there seems an air of slightly directionless faded grandeur, if you will, like looking at an old-timey cinema.
If Dyalog APL is the future of APL, how do you imagine that future?
Thanks for your interest in the future of Dyalog! You are absolutely right that 2019 marks a kind of watershed moment for Dyalog. The original two-man development team who built version 1.0 of the interpreter (released in 1983) recently celebrated their 70th birthdays. At the beginning of this year, both of them had reduced working hours to about half time. Geoff Streeter remains in good health, but sadly John Scholes passed away in February.
Before Dyalog was acquired in 2005 and Gitte and I came in as CEO and CTO, the team working on Dyalog APL consisted of 5 people. Today that number is heading towards 25, several of whom were hired over the past decade to be ready to take over the work of Scholes and Streeter.
Gitte's comment about "not enough" young people signing up needs some context. The problem was actually caused by significantly increased interest in APL, which made us realise that our "classical" registration process was putting potential new users off. It was often taking too long to get people a copy of Dyalog APL because of the manual processing that was required, people were confused by the licensing terms and afraid of using the software – and many young people are just allergic to providing any information about themselves online. Now, APL is available for Windows, Linux (including the Pi) and macOS with no questions asked – for non-commercial use.
Losing Jay as CTO was a bit of a blow, but I think we are on the way to recovery. Now that he has completed his PhD on the subject of a parallelizing compiler (research that Dyalog has funded for the last 5 years), Aaron Hsu will join young guns Marshall Lochbaum and Adam Brudzewsky at Dyalog, and we have one more C developer joining the team in the UK in January. We hired two fresh APL consultants in the USA in 2019, and will be looking to hire two more in 2020 to support both existing and new business. I've been in the APL business for 40 years now, and my opinion is that the "kids" at Dyalog are every bit as impressive as Scholes and Streeter were at the same age; the shoes will be filled. There is as much young talent at Dyalog as there has ever been working on any APL interpreter, throughout the 50+ year history of the language.
As the team grows, we are able to afford having one or two team members focusing most of their efforts on performance. These efforts have accelerated over time, as first Nic Delcros, then Roger Hui and now Marshall have spent significant time rewriting primitive functions to take advantage of vector instructions and new hashing, sorting and other algorithms that have been evolved over the last four decades. Hardware has evolved in ways which mean that many of the old algorithms are sub-optimal on modern chips where the relative speed of CPU cycles vs memory access is completely different from the Z8000. Yes, Marshall is very good indeed, but I can't see a problem there . Performance has improved consistently in most of the last ten versions of Dyalog APL, and I hope we may see even more spectacular improvements in the next decade, both in the interpreter and in GPU-based parallel compilers.
We have much work to do: In addition to language and performance work, we expect Aaron to work with Richard Park, another new recruit who was added to the team last year, to work on training materials and documentation – and with Marshall, Roger and myself on quote evangelism unquote; talks at Lambdaconf, Functionalconf and perhaps some new conferences in 2020 and beyond (invitations welcome!).
For decades, Dyalog was a company that made a very comfortable living providing a better APL interpreter to clients who converted to Dyalog APL after initially purchasing an APL system from one of the companies that did serious marketing of APL interpreters: companies like IBM, STSC and I.P.Sharp Associates. Thanks to our recent growth, and the fact that Dyalog APL now has decent tooling not only under Microsoft Windows, but Linux and macOS too, we are now starting to focus on developing new business and attracting the next generation of APL developers.
I'm sorry to hear that we appear "directionless" to you. I will admit that it is an interesting challenge to keep existing customers satisfied (which should always remain the first priority for any organisation – put on your oxygen mask before helping others) while also working on attracting completely new groups of users. The rate of development of new language features, development tools and libraries for Dyalog APL is as high as it has been for any APL interpreter in history, at the same time as we are making significant performance improvements. Is there anything in particular that you miss?
Aaron's work on using APL as a high-performance tool for manipulating tree structures demonstrates that APL is more relevant than ever before, as a programming language that has a natural mechanical sympathy with modern hardware, and allows relatively unskilled developers to write extremely efficient solutions without using complex tricks. I recently had the pleasure of sharing a podium with Aaron in Mumbai as part of the JioTalks series, where we covered some of the reasons why APL is a tool worth taking a look at in 2020, even though the core of the language is more than 50 years old: https://jiotalks.com/watch/204/home/Morten_Kromberg_&_Aaron_...
Hi Morten, I have watched your video of presenting with Aaron (and many more), taking an interest in APL for the past few months. It's been fun (if frustrating) to try and learn, so you mentioning training materials and documentation is particularly appealing.
> I'm sorry to hear that we appear "directionless" to you
I hoped for this to be quiet feedback in case you don't see it from inside, rather than criticism, I'm not a commercial user (yet?) and you have nothing to be apologetic about; I appreciate your detailed and comprehensive reply, am enthused about many of the things you've said, am glad Dyalog is thriving, and wish you all continued success!
> Is there anything in particular that you miss?
I miss the familiarity of an imperative/OOP language with intellisense completing the names of methods and ordering of parameters, but I'm sure that will pass with time. :)
Just a comment about the intellisense stuff. Dyalog's IDE(s) do provide name completion for methods, functions, variables, objects, and namespace bindings. You don't get any auto-complete for the ordering of parameters, but arguably, this is less of an issue with a 1 and 2-arity function requirement in which it is generally bad style to deviate too far from this for too long in too many places without good reason.
Thanks for taking the time to submit a lengthy reply Morten and I wish y'all the best of luck in 2020. I don't use APL/J/K professionally, but have played around with it for fun and found it to be quite enjoyable.
I keep up with Dyalog's twitter and conference videos, but wish there was a little more in the way of beginner material. The recent video on using the CSV/HTML/XML/JSON calls was great btw.
Two questions I have revolve around performance, numerical computing, and program distribution. To me, having to use Python + Numpy, Matlab, or low level languages for scientific and numerical computing is a shame as APL is basically math notation and should excel here. I hate wasting time writing loops (I parse lots and lots of text files) and how it makes my code harder to view and keep up with (APL fixes this). However, will Dyalog ever invest in writing some libraries or code snippets or primitives for things like sparse matrices and matrix factorizations that I probably don't want to implement myself and that make more sense to be done by the language implementers in C (I guess similar to how Dyalog has native support for JSON & CSV)? I'm guessing you can call out to BLAS/LAPACK (like J) or use Python or R's libraries via APL's bridge libraries, but that defeats most of the reason why I would want to use APL to begin with and I would end up with a franken-system and would be better off just using Python.
I guess what I'd like to see as a user is a way for me to write APL code in a tacit manner (I really like the organization of Aaron's compiler) and the compile it as a binary that could be shared with other users at work. It would be nice to have some built-in scientific primitives. Honestly though, the licensing is the real problem last I checked. It seems like running in production would be a non-starter, so I could only use as a single user tool (probably not worth the fight). I know this might be counter to how y'all currently make money, so I'm not exactly expecting a response on this note, but felt you should know what the few deal-breakers are from someone who has considered using it professionally but has shied away.
I'll comment a bit here. If you look into prior research, there are a number of papers on sparse representations in APL along with accompanying code in the literature. That's a good starting point.
Additionally, sparse matrices are on the roadmap for Co-dfns in the future, so there's hope for you there. If you have a specific problem that you wish would work, the best thing to do is to make it known to us so that we can actually prioritize it. I think myself and Dyalog often prioritize those things for which we know there are users. If you want to be able to do this stuff, it would be great to get some concrete programs that we can work off of.
Finally, licensing should, IMO, be a non-issue for most people in the vast majority of cases. Dyalog provides some of the best licensing terms that I've seen, and there are already models that scale well both to startups and to existing deployment in large-scale commercial enterprise situations. If something in the licensing terms doesn't seem to work, I would get in touch with Sales @ Dyalog and they will surely be able to work with you to figure out something that would be equitable.
Most people are surprised when they find out how easy it is to move forward with Dyalog on a commercial product, or how readily Dyalog is looking at making things work for customers or people who wish they could be customers.
I don’t at all mean this to be heckling from the peanut gallery, I have great respect for the people, the tools, the company; but from the distant outside there seems an air of slightly directionless faded grandeur, if you will, like looking at an old-timey cinema.
If Dyalog APL is the future of APL, how do you imagine that future?