This was amazingly common in the 2010s during the Big Data craze. I know, because I was the one slapping the bad queries together.
Most startups didn’t care (to a point) because at that point in their lifecycle, the information they needed to get from those queries (and actions they could take based on it, like which customers were likely to convert and worth spending sales time on, etc) was more important than the money spent on the insane redshift clusters.
The mantra was almost always some version of, “just do it now, as fast as possible, and if we’re still alive in a year we’ll optimize then.”
Unfortunately, having to mess around with a JVM is a tough sell for a lot of data analysis folks. I'm not saying it's rational or right, but a lot of people hear "JVM" and they go "no thank you". Personally I think it's a non-issue, but you have to meet people where they are.
I dunno, if you can slog through the Python ecosystem then the JVM is starting to look not so bad. Plus with Clojure you don't need to deal with the headache and heartache that is Maven.
I think that's true for only a limited subset of programs, though. The Clojure lib ecosystem is nowhere near the size of the broader Java ecosystem, so you frequently end up pulling Maven deps to plug holes anyway.
That is the goal of a polyglot runtime, and why Clojure was designed to be a hosted language that embraces the platform, unlike others that make their tiny island.
It's unfortunate, but people's associations with Java the lang bleed into their beliefs about the JVM, one of the most heavily-optimized VMs on the planet.
There's some historical cruft (especially the memory model), but picking the JVM as a target is a great decision (especially with Graal offering even more options).
Exactly, especially because there isn't THE JVM, rather a bunch of versions each with their own approaches to GC, JIT, JIT caches, ahead of time compilation.
Meanwhile, I find it very annoying to deal with the litany of Python versions and the distinction between global packages and user packages, and needing to manage virtual environments just to run scripts. That being said, I am not an expert but that's always been my experience when I need to do anything Python related.
idk, I don't think I've had to do anything beyond install the JVM to work with Clojure. I'm not really a fan of the clj commands flag choices though (-M, -X, etc. all make no sense)
It doesn't come off as unintuitive by my read. They had a bug that led to a massive performance regression. Rewriting the code didn't have that bug so it led to a performance improvement.
They found that they had fewer bugs in Python so they continued with it.
I think a lot of people (especially those who are only peripherally involved in development, like management) don't really consider performance regressions at all when thinking about how to get software to go faster.
Meanwhile my experience has been that whenever there has been a performance issue severe enough to actually matter, it's often been the result of some kind of performance bug, not so much language, runtime, or even algorithm choices for that matter.
Hence whenever the topic of how to improve performance comes up, I always, always insist that we profile first.
My experience has been that performance bugs show up in lots of places and I'm very lucky when it's just a bug. The far more painful performance issues are language and runtime limitations.
The leadership and product direction work are at least as hard as the code work. Astral/uv has absolutely proven this, otherwise Python wouldn't be a boneyard for build tools.
Projects - including forks - fail all the time because the leadership/product direction on a project goes missing despite the tech still being viable, which is why people are concerned about these people being locked up inside OpenAI. Successfully forking is much easier said than done.
I had a lot of trouble convincing people that a correct Python package manager was even possible. uv proved it was possible and won people over with speed.
I had a sketched out design for a correct package manager in 2018 but when I talked to people about it I couldn't get any interest in it. I think the brilliant idea that uv had that I missed was that it can't be written in Python because if is written in Python developers are going to corrupt its environment sooner or later and you lose your correctness.
I think that now that people are used to uv it won't be that hard to develop a competitor and get people to switch.
> The problem is that Python was meant for scripting not properly designed software system engineering.
What something was meant to do has never, ever stopped people. People find creative ways to use tools in unintended ways all the time. It's what we do.
We can call this dumb or get misanthropic about it, or we can try to understand why people all over the world choose to use Python in "weird" ways, and what this tells us about the way people relate to computing.
100%. The thing I'm currently working on has been a pain probably 80% because the work was underspecified and didn't take a bunch of legacy concerns into account and probably 20% because of nature of the code itself.
If it was better specified I'd be done already, but instead I've had to go back and forth with multiple people multiple times about what they actually wanted, and what legacy stuff is worth fixing and not, and how to coordinate some dependent changes.
Most of this work has been the oft-derided "soft skills" that I keep hearing software engineers don't need.
This is gonna rankle folks who like one or the other, but they're basically the same language. When it comes to languages that run on the same VM, Erlang and Elixir are very close together. They aren't nearly as far apart as say, Java and Clojure.
Elixir adds a few things (a lisp-style macro system, protocols, UTF-8 as the default string type, a builtin build tool, streams) but Elixir is not a huge departure from Erlang in the way that Clojure is a huge departure from Java.
By far the biggest things you're going to learn when you learn either one are going to be the BEAM runtime itself and the OTP libraries, which both Elixir and Erlang have in common.
You can (and should) always handle whatever errors you actually want to and are able handle, so if that means catching a lot of them and forwarding them to a model, that's not a problem.
The benefit comes mainly from what happens when you encounter unknown errors or errors that you can't handle or errors that would get you into an invalid state. It's normal in BEAM languages to handle the errors you want to/can handle and let the runtime deal with the other transient/unknown errors to restart the process into a known good state.
The big point really is preventing state corruption, so the types of patterns the BEAM encourages will go a long way toward preventing you from accidentally ending up in some kind of unknown zombie state with your model, like for example if your model or control plane think they are connected to each other but actually aren't. That kind of thing.
It doesn't sound strange, it actually sounds sane and what everyone should be doing.
At the same time, I can't imagine the last time I had a random exception I didn't think about in prod, but I guess that's the whole point of the BEAM, just don't think about it at all.
I might take a stab at Elixir, the concepts seem interesting and the syntax looks to be up my alley.
Most startups didn’t care (to a point) because at that point in their lifecycle, the information they needed to get from those queries (and actions they could take based on it, like which customers were likely to convert and worth spending sales time on, etc) was more important than the money spent on the insane redshift clusters.
The mantra was almost always some version of, “just do it now, as fast as possible, and if we’re still alive in a year we’ll optimize then.”
reply