Lots of comments here are stating that typing is half baked in Python, and that if you gotta use types, you should use another language.
But that's missing the point that Python is still not meant to be the best at anything, but good at most things.
And in this case, it's exactly what you get: optional typing, with decent safety if you need it.
You can quick script or design seriously, you can explore in a shell or commit a public API into a file, you can start with untyped code and add some later.
That makes it cumbersome sometimes, yes. mypy is slow, and the typing system + match / case have many rough edges.
But it also keeps Python incredibly versatile.
It's amazing what you can do with this (not so) little language, and the whole ecosystem is always getting better and better.
Every time I try another language, I keep getting back into Python, because the fields I need tools for are vast, and it's the only one that I'm pretty sure will handle a problem decently in all it's various forms, especially the one I didn't consider when I started.
There is immense value in this, because it keeps opportunities on the table no matter what you do. I often find my self during a project adding something I could only because Python let me do so easily.
On the one side, yes Python is incredibly versatile.
On the other side, I have worked on many Python projects, some of them fairly high profile, and I have seen exactly two kinds of Python codebases:
1. a few were written by extreme professionals, plugging at every single hole, with ~100% coverage, plus considerable maintenance because every dependency upgrade tends to break something;
2. many that feel cobbled together, with undocumented (and often inconsistent) invariants "because something will eventually throw a TypeError if I make a mistake", undocumented metaprogramming, and heroic levels of maintenance because this stuff works until it doesn't.
Option 1. feels like "writing Python like it's Rust", but of course without any of the benefits of Rust either on performance or on safety.
Option 2. feels like "experiments running out of control". That is the unfortunate price of this incredible versatility.
I don't have a clear conclusion to this, except perhaps that Python is really good for the initial experimentation (when versatility actually gets you to initial results much faster), but really bad for... well, let's call it industralization.
To be fair, a division between hell and heaven will happen with any language.
The question is: is this particular hell worth the result?
There is no generic answer to that of course, it just happens is has been the case for me during those 20 years.
First, you have to get to the industrialization phase. And of course, you have to get there, with the constraint of time, budgets, and talent.
Second, Python does have less benefits for that phase than rust or haskel, but it's not impotent. Industrialization is just hard, no matter the language.
Because we are an industry where a lot of self-taught people get a career, we tend to forget this last point. Creating a serious computing system is engineering, and this includes the project management part, and being rigorous about a lot of thing.
Granted, the rigor needs to be higher with Python once your reach that scale, and at a certain point (which I wish everybody would reach), you may want to ditch it. It's a good problem to have though.
Yet we have to remember not all industrialization attempts are equal. Most are really tame. I'd argue you could replace half the website code base out there with a bunch of bash scripts on a VPS and they would still make money. So even in this context, Python can deliver a lot.
Static type safety has interesting YAGNI characteristics. For the bits of the code that must work, and where defects and regressions due to type errors may be subtle and difficult to detect, it's indispensable.
But it can also be an impediment to iteration. Sometimes the code you're working on is still so experimental that you don't really know what the best structure and flow of data will be yet, and it's easier to just bodge it together with maps and heterogeneous lists for the first few iterations so you can let the code tell you how it wants to be structured. Having to start with static types subtly anchors you to your first idea. And the first is typically the worst.
Something I really like about Python for this sort of thing is that I have a lot more ability to delay this industrialization stuff to the last responsible moment, and be selective in where I spend my industrialization tokens. Here's a list of the languages where I've found selectively rewriting the important bits in Rust to work well: C, C++, Python. I know which of those I'd rather use for prototyping, scripting, and high level control.
Relevant Fred Brooks quote: "Plan to throw one away. You will, anyway."
> But it can also be an impediment to iteration. Sometimes the code you're working on is still so experimental that you don't really know what the best structure and flow of data will be yet, and it's easier to just bodge it together with maps and heterogeneous lists for the first few iterations so you can let the code tell you how it wants to be structured. Having to start with static types subtly anchors you to your first idea. And the first is typically the worst.
That's a very interesting observation.
Let me add a counterpoint, though. Indeed, you will refactor everything. But by having strong, static typing, you have a much clearer idea of what you're breaking along the way. Cue in hundreds of anecdotes by each of us, when we broke something during a refactoring because the dependency was not obvious and there were not enough tests to detect the breakage. I've seen two of these in Python code just this week.
That was what I originally thought, as someone who grew up on static typing and then migrated to Python.
But what I've discovered in practice is that, during those early iterations, I don't really need the compiler to help me predict what will break, because it's already in my head. The more common problem is that static typing results in more breaks than there would be in the code that just uses heterogeneous maps and lists, because I've got to set up special types, constructors, etc. for different states of the data such as "an ID has/has not been assigned yet". So it kind of ends up being the best solution to a problem largely of its own making.
I'm also working from the assumption here that one will go through and clean up code before putting it into production. That could be as simple as replacing dicts with dataclasses and adding type hints, but might also mean migrating modules to cython or Rust when it makes sense to do so. So you should still have good static type checking of code by the time it goes into production.
As someone who has primarily gone in the other direction, my anecdata supports the opposite conclusion: static typing tends to help me to prototype over more dynamic languages, even if it takes (slightly) longer to physically write things down. I think this is because of two things:
1. If I'm prototyping things, I find I spend a lot of time trying to figure out what sorts of shapes the data in my program will have - what sort of states are allowed, what sort of cases are present, what data will always exist vs what data will be optional, etc. If I'm doing that in my head, I may as well write it down at the same time, and voila, types. So I'm not usually doing much extra works by adding types.
2. If I change my code, which I often do when prototyping (some name turns out to be wrong, some switch needs extra cases, some function needs more data), then that is much easier in typed languages than untyped ones. Many times my IDE can do the refactoring for me, and if that isn't possible, I can start making the change somewhere (e.g. in a type declaration) and just follow the red lines until I've made the change everywhere. One of the big results of this is that statically typed prototypes are often immediately ready to be developed into a product, whereas in dynamic languages, the prototype already bears so many scars from refactoring and the natural back-and-forth that comes from prototyping, that it needs to be rewritten over more from scratch. (The corollary to that being that I have never once had a chance to rewrite a prototype before releasing it to production.)
I can imagine that some of this comes down to programming/architectural style. I tend to want to define my types up front, even in dynamic languages, because types and data are how I best understand the programs I work on. But if that's not how you work, the tradeoffs might not be the same. The other side is that the type systems I regularly use are almost exclusively modern ones, with decent support for things like product and sum types, so that I can use relatively simple types to model a lot of invariants.
> But what I've discovered in practice is that, during those early iterations, I don't really need the compiler to help me predict what will break, because it's already in my head.
Reading this, I have the feeling that you're talking mostly of single-person (or at least small team) projects. Am I wrong?
> The more common problem is that static typing results in more breaks than there would be in the code that just uses heterogeneous maps and lists, because I've got to set up special types, constructors, etc. for different states of the data such as "an ID has/has not been assigned yet". So it kind of ends up being the best solution to a problem largely of its own making.
There is definitely truth to this. I feel that this is a tax I'm absolutely willing to pay for most of my projects, but for single-person highly experimental projects, I agree that it sometimes feels unnecessary.
> I'm also working from the assumption here that one will go through and clean up code before putting it into production. That could be as simple as replacing dicts with dataclasses and adding type hints, but might also mean migrating modules to cython or Rust when it makes sense to do so. So you should still have good static type checking of code by the time it goes into production.
Just to be sure that we're talking of the same thing: do we agree that dataclasses and type hints are just the first step towards actually using types correctly? Just as putting things in `struct` or `enum` in Rust are just the first step.
> I have the feeling that you're talking mostly of single-person (or at least small team) projects. Am I wrong?
Small teams. And ones that work collaboratively, not ones that than carve the code up into bailiwicks so that they can mostly work in mini-silos.
I frankly don't like to work any other way. Conway's Law all but mandates that large teams produce excess complexity, because they're basically unable to develop efficient internal communication patterns. (Geometric scaling is a heck of a thing.) And then additive bias means that we tend to deal with that problem by pulling even more complexity into our tooling and development methodologies.
I used to believe that was just how it was, but now I'm getting to old for that crap. Better to push the whole "expensive communication" mess up to a macro scale where it belongs so that the day-to-day work can be easier.
> Small teams. And ones that work collaboratively, not ones that than carve the code up into bailiwicks so that they can mostly work in mini-silos.
Well, that may explain some of the difference between our points of view. Most of my experience is with medium (<20 developers) to pretty large (> 500 developers) applications. At some point, no matter how cooperative the team is, the amount of complexity that you can hold in your head is not sufficient to make sure that you're not breaking stuff during a simple-looking refactoring.
Sure but at that point we're probably not on the first iteration of code anyway. Even at a big tech company, I find it most effective to make a POC first-iteration that you prove out in a development or staging environment that uses the map-of-heterogeneous-types style development. Once you get the PMs and Designers onboard, you'll iterate through it until the POC is in an okay state, and then you turn that into a final product that goes through a larger architecture review and gets carved up into deliverables that medium and large-scale teams work on. This latter work is done in a language with better type systems that can better handle the complexity of coordinating across 10s or 100s of developers and can generally handle the potential scale of Big Tech.
There's something to be said that the demand for type systems is being driven by organizational bloat but it's also true that large organizations delivering complex software has been a constant for decades now.
Do you work in an organization that does this? Because most organizations I've seen who don't pick the approach of "write it like it's Rust" rather have the following workflow.
1. Iterate on early prototype.
2. Show prototype to stakeholders.
3. Stakeholders want more features. At best, one dev has a little time to tighten a few bolts here and there while working on second prototype.
4. Show second prototype to stakeholders.
5. Stakeholders want more features. At best, one dev has a little time to tighten a few bolts here and there while working on third prototype.
6. etc.
Of course, productivity decreases with each iteration because as things progress (and new developers join or historical developers leave), people lose sight of what every line of code means.
In the best case, at some point, a senior enough developer gets hired and has enough clout to warrant some time to tighten things a bit further. But that attempt never finishes, because stakeholders insist that new features are needed, and refactoring a codebase while everybody else is busy hacking through it is a burnout-inducing task.
Yup! I'm at a company that used to be a startup and ended up becoming Big Tech (over many years, I'm a dinosaur here.) Our initial phase involved building lots of quick-and-dirty services as we were iterating very quickly. These services were bad and unreliable but were quick to write and throwaway.
From there we had a "medium" phase where we built a lot of services in more strictly typed languages that we intended on living longer. The problem we encountered in this phase was that no matter the type safety or performance, we started hitting issues from the way our services were architected. We started putting too much load on our DBs, we didn't think through our service semantics properly and started encountering consistency issues/high network chatter, our caches started having hotspotting issues, our queues would block on too much shared state, etc, etc.
We decided to move to a model that's pretty common across Big Tech of having senior engineers/architects develop a PoC and using that PoC to shop around the service. For purely internal services with constrained problem domains and infrequent changes, we'd usually skip this step and move directly to a strictly typed, high performance language (for us that's Java or Go because we find them able to deal with < 15 ms P99 in-region latency guarantees (2 ms P50 latencies) just fine.) For services with more fluid requirements, the senior engineer usually creates a throwaway-ish service written in something like Node or Python and brings stakeholders together to iterate on the service. Iteration usually lasts a couple weeks to a couple months (big tech timelines can be slow), and then once requirements are agreed upon, we actually carve out the work necessary to stand up the real service into production. We specifically call out these two phases (pre-prod and GA) in each of our projects. Sometimes the mature service work occurs in parallel to the experimentation as a lot of the initial setup work is just boilerplate of plugging things in.
===
I have friends who work/have worked in places like you describe but a lot of them tell me that those shops end up in a morass of tech debt over time anyway and eventually find it very difficult to hire due to the huge amount of tech debt and end up mandating huge rewrites anyway.
The "aha!" moment that converted me from a Clojure guy to a Haskell guy was realizing that types aren't an impediment to iteration, they are an enabler of rapid design iterating. Once written, code has a way of not wanting to be changed. Types let me work "above the code" during that squishy beginning period when I'm not sure 'what the best structure and flow of data will be'. Emotionally, deleting types is a lot easier for me than deleting code.
This is in no way saying that types are the One True Way™ ^_^ Just that I've found them, given the way my brain is wired, to be a great tool for iteration.
Yeah, types are a type of (usually) easy to write, automatically enforced, documentation which I find endlessly useful during experimentation, if sometimes constraining. I'm sure that there are other means to achieve similar results (some variant of TDD, maybe?) but I haven't experienced them yet.
> Something I really like about Python for this sort of thing is that I have a lot more ability to delay this industrialization stuff to the last responsible moment
This is one of my favorite aspects of Python. I can start with every module in prototype form and industrialize each module as its design firms up. I can spend my early development getting an idea fleshed out with minimal overhead.
> I can start with every module in prototype form and industrialize each module as its design firms up.
That is definitely the theory. And this flexibility is indeed very precious for some types of work.
However... does it actually happen as you describe? I can count on half of the fingers of one hand the number of Python codebases that I've seen that actually feel like they've properly been reworked into something of industrial quality. All the other codebases I've seen are of the "I guess it works, maybe?" persuasion, quite possibly because there is always something higher priority than quality.
The thing is type hints in Python are less a code quality feature and more a quality of life feature for developers. As long as I've got descriptive argument names and docstrings I can just tell you how to use a method. Your IDE can at least tell you argument names.
Type hints help reduce cognitive load when someone else (or you in the future) is trying to use some code. If you have strict type requirements you're testing that inside a method or with a decorator or something (and verifying with tests).
Even a big project can hum along happily without type hints. They're also something you can add with relative ease at any point.
> The thing is type hints in Python are less a code quality feature and more a quality of life feature for developers.
They are absolutely a code quality feature.
> As long as I've got descriptive argument names and docstrings I can just tell you how to use a method.
Yes, you can, but that doesn’t seem to be germane to the argument, since “it is possible to communicate intended use without typing” doesn’t support your QoL vs. code quality argument.
With typing, the type-checker can statically find potential errors in your description, or in my attempt to follow it—that’s a code quality feature. (Of course, that it does provide a description, and a better chance that the description is correct, is also a QoL issue.)
> Yes, you can, but that doesn’t seem to be germane to the argument, since “it is possible to communicate intended use without typing” doesn’t support your QoL vs. code quality argument.
Jillions of lines of quality Python were written before type hints. They're not strictly necessary for writing quality code. If you find modern code that's high quality it probably uses type hints but type hints don't automatically make high quality code.
Wandering offtopic, perhaps, but I've noticed that this kind of behavior seems to strongly correlate with Scrum.
The work starts getting rushed toward the end of the sprint. Every two weeks, people start furiously cutting corners to meet a completely artificial due date. And then there's basically zero chance that you'll be able to get the PO to agree to cleaning it up in the next sprint, because they can't see the problem.
Scrum of course prescribes all sorts of adornments you can add to try and counteract this effect. But I'm a firm believer that an ounce of prevention is worth a pound of cure.
The two week rush + everything is always broken failure pattern is solvable.
Given dubious project management schemes, do the refactoring and cleanup first. Then the functional change is easier to review as it's against a sane background, and there's minimal planned cleanup after in case that phase gets dropped. Call writing the tests characterisation if that helps.
I think it's the application of Scrum to work that doesn't primarily fall inside the (Cynefin) "simple" domain-type work that Scrum was designed for.
It's fine if the work is straightforward and easy to estimate. But, if it isn't, things get problematic. There are three variables that interact with each other when working on a project: time, scope, and quality. If you pin down both your acceptance criteria and the time you have to implement (which is basically what happens in a sprint planning meeting), then quality is the only remaining variable you have to manipulate when things aren't going according to plan.
Something that just about seems to work is something that has often met the threshold for "solves a real problem" while also meeting the other requirements that a product needs to be successful.
It's easy to make something that works, is well designed, and either doesn't solve a problem someone has or nobody knows about it.
I realize that this is the common wisdom these days: write code that works just well enough that we have a chance to fix before it does too much damage whenever it breaks. I suspect that this approach is strongly fueled by the unlimited VC money available in tech, since it means that any company can employ an unlimited number of full-time developers (and PR) just to handle catastrophes.
We'll see how that wisdom holds if/when/as VC money dries up and/or moves to other sectors.
> I suspect that this approach is strongly fueled by the unlimited VC money available in tech,
Well, I suppose we could trade anecdotes and counter-examples, but my position largely comes from my own experience rather than received wisdom (though there's plenty of that).
Instead I'll just say that I disagree, largely because because a business is a complex and shifting arrangement of various factors competing for limited time and resources. Even in a software business, software is only one of those.
> Having to start with static types subtly anchors you to your first idea. And the first is typically the worst.
Not just that, but dynamic typing conveniently allows you to try multiple ideas in parallel. In static languages, you tend to refactor The One Representation for a thing and try multiple ideas sequentially, which may or may not be better.
Of course, none of this is really inherent to the type system -- plenty of Python folks try various shapes for their data sequentially, and you can have multiple representations of the same data in a static language too. But I feel like the languages encourage a particular kind of experimentation, and sometimes either is more helpful than the other.
However, the other side of the coin is that in many Python codebases I've seen, people keep using multiple antagonistic ideas/paradigms in the same code, way past the point where it should have become clear that these ideas are actually incompatible and no amount of heroic hackery will solve the issues.
One of the core values in the Python community is, "we're all adults here."
I think, though, that a lot of Python programmers - particularly less-experienced ones - fail to realize that that typically functions as more of an expectation, perhaps even an obligation, than a liberty.
All this discussion is kind of interesting to me since in the beginning Python seemed to be focused on being the anti-Perl.
Rejecting "There's More Than One Way To Do It" in favour of "There should be one - and preferably only one - obvious way to do it" (IMO they did not find the right balance there in terms of python3 strings)
Ultra minimalism on syntax to the point of introducing invisible errors...
I suppose this greater leniency on approaches now is simply due to broad adoption.
Wholeheartedly agree. I'm a self taught programmer who programs to get things done rather than just doing programming for fun (I started in ML/data science field). That way, my code often resembles the caricature of these hacky codebases alluded above. And I'm well aware of it.
I often have arguments about utility of Python with my more technically solid engineer friends who keep telling me "Python doesn't scale" etc. And I often come back to the same point you highlighted: is the particular hell worth the result? For 85-90% of the cases the answer is resounding yes.
For nimble teams, startup projects, internal tools : most of the code is often thrown away eventually. Python's beauty is that the language quickly gets out of the way. Everyone then is forced to focus on complexity of the business domain, "does the code solve the business problem". The architecture/scale/speed problems will eventually arrive with Python. But most purist engineers overestimate how soon it will come. At most of the failed startup attempts I was part of, the PMF problem was more pressing to solve than software constraints.
Early days of YouTube, Dropbox, Instagram (and maybe OpenAI too) are big testament to this. I've made my peace these days by not fighting to prove Python is the best language etc. If someone tells me "Rust beats Python" kind of argument: I say I agree, wish them luck and focus on shipping things.
tl:dr; Python is still the best choice to "quickly deliver value and test your business hypothesis".
I think you have to grade code in your project, i.e not every piece of code has the same value, high value => more strict, low value => less strict.
Less strict code is something that you can throw away and rewrite without too much effort.
This has the implication that you have bulkheads (like in a ship) to separate high value and low value code, thus you have to avoid code that is toxic (i.e magic) that spreads between them, e.g. ORM entities.
When using the type system correctly you don't need 100% coverage, now I'm not a Python programmer so I don't how feasible that is with that type system, but in PHP I use the type system extensible and I don't write that many tests anymore, except for legacy code or when I need to test something specific, like an algorithm. By using the type system you can catch most programmer errors, and combine that with routinely throwing exceptions for any unwanted state, you will detect most bugs.
The way unit testing has been interpreted and implemented it has become more of a nuisance than a help in fast moving projects. Usually what happens is that the team refactors some unit and then multiple test breaks and then time needs to be spent rewriting these test, but what that means is that the tests were actually useless after they where committed if you can change them as you like, when you change a test you are no longer testing the same thing.
Thus tests can also can be graded, from high value tests, i.e stable, and low value tests. Depending how you organize your code, high value tests will usually test a larger set of functionality in one run, like a system (or integration) test from call to finish, these should never change regardless of much you refactor. High level tests should only change when your business logic changes.
Low value test typically involves a lot of mocking and stubbing, those are pretty much pointless to commit after you implemented your unit. Too much mocking and stubbing is code smell.
This is my experience as well. There are very few code bases with 100% coverage and strict development practices. Moreover, if the average developers get their hands on one of them, they'll degrade soon.
The problem comes from the very "top", since CPython developers are in camp (2) and celebrate their development practices.
Other languages also have this divide of course but not to the extent that Python has.
As for maintenance: This is also true. I have rewritten a Python code base in C++. Even in C++ it is easier to maintain, because the C++ compiler does not break between versions and there are no new bugs. And modern C++ can look quite like Python.
The Firefox build system feels to me like kind 1. The Synapse Matrix server also feels pretty close (although I don't understand why it doesn't use Pydantic or something like Pydantic).
I think this is a form of selection bias. Certain other languages wouldn't even allow a project like your point 2. to get started. In Python, the bar for something to start kind of working is quite low, even if you're a beginner, a scientist of another discipline who needs a little script to manage some data, a teen who's learning to code, etc. So of course you see a lot of half-baked stuff. It's important we distinguish between 1. and 2., but 2s might sometimes be better than nothing.
Absolutely. In some domains, the choice is between 2. and nothing – and 2. is better. That's true for Python or JavaScript, for instance (in each their domain).
I believe that we agree that there are actually many domains in which you really want 1., and if you can't have it, then "nothing" is generally preferable to 2.
You are right, this is probably not specific to Python. It just happens that:
1. we're currently discussing Python;
2. anecdotally, the C++ projects on which I have worked were all case 1. ("code it like it's Rust") – I make no claim that this is representative, though.
> But that's missing the point that Python is still not meant to be the best at anything, but good at most things.
The most important thing about Python is readability and its part of its syntax and is one of the best languages out there for readability.
Zen of python:
>>> import this
The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
>>>
> Heck, the package manager having to execute random code in every package (setup.py).
Nope! Not since the mid-2010s. It took a long time, but with wheels[1], install-time actions are finally separate from packaging-time actions, and the former do not include any user-defined actions at all.
Just about every sufficiently general system allows for arbitrary code in the latter, be they in debian/rules or PKGBUILD or the buildPhase argument to mkDerivation or—indeed—in setup.py. (Most systems also try to sandbox those sooner or later, although e.g. the Arch Linux maintainers gave up on cutting off Go and Rust builds from the Internet AFAIU.)
Don’t forget to `python setup.py bdist_wheel` your stuff before you upload it!
> And speaking of beautiful and elegant, dunders everywhere, really?
It’s the one reserved namespace in the language, so its usage for __init__.py and __main__.py seems—perhaps not beautiful, but fairly reasonable?
> Just about every sufficiently general system allows for arbitrary code in the latter, be they in debian/rules or PKGBUILD or the buildPhase argument to mkDerivation or—indeed—in setup.py. (Most systems also try to sandbox those sooner or later, although e.g. the Arch Linux maintainers gave up on cutting off Go and Rust builds from the Internet AFAIU.)
Most other programming language package managers don't, see Maven for Java.
But then again, NONE of the scripting languages ever wanted to learn stuff from Java, especially regarding packaging (lack of packaging namespaces is another major blunder re-created by Python and several others, including Javascript).
> Most other programming language package managers don't, see Maven for Java.
So it’s not able to express native extensions or even codegen then? I suppose that’s OK for something that’s not the only available solution, but I don’t think I’d love it, either.
> But then again, NONE of the scripting languages ever wanted to learn stuff from Java, especially regarding packaging[.]
Both Java and its docs are just extremely tedious to read, to be honest. I say that with all due respect to its principal designers and acknowledging my ever-hopeless crush on Self where a lot of the tech originated. (And I don’t only mean 2000s Java—it took me literal days to trawl through the Android frame pacing library to find[1] the two lines constituting the actual predictor, commented “TODO: The exponential smoothing factor here is arbitrary”. The code of the Go standard library, nice as that is, gives me a similar feeling with how... vertical it is.)
That’s not an excuse for ignorance, but it’s a centuries-old observation that you need to avoid inflicting undue pain on your readers if you want your ideas to survive. The flip side is that there may be untapped ideas in obscure or unpleasant texts written by smart people.
So if you can point to things one could learn from Java, I’d very much be interested.
(And no, literature searches are not trivial—I’ve occasionally spent months on literature searches, sometimes to realize that the thing I wanted either isn’t in the literature or is in someone’s thesis that’s only been published a year ago.)
Well, they didn't need to read much about Java in this case, frankly. Just creating a simple Java project with Maven would have showed the groupId concept (namespaces preventing top level squatting), for example.
Lack of namespacing is a trap most language package managers have fallen into sadly: even Cargo sees it as an “advantage” not to have them, apparently.
I think that is extremely short sighted, and that all packages including standard ones should be namespaced.
and classes are most of the time not needed anyway, it's just Java programmers that aren't used to the idea that code can be perfectly correct and readable without a single class
Everything you just said is true for Typescript, as you can set it up as strict or as forgiving as you like, and writing one off scripts for node is just as easy as for Python. But unlike Python, it has a great type system.
First, the JS ecosystem is very web oriented, so if you want to dabble out of there, you often gonna fall short.
Secondly, the JS packaging has very poor support for compiled extensions, which mean everything that needs a perf boosts is unlikely to get good quality treatments.
Finally, the community makes it a constant moving target. After 20 years of writing both JS and Python, I can still install old django projects that use 2.7 (did it 2 months ago), but JS projects for even 5 years ago are a very hard to build.
Bottom line, I use JS for the Web because I have to, given it has monopoly on the browser, and now good GUI, but if I want to keep my options open, I would rather go rust or go than JS.
We gradually wanted to move our Java, C# and C++ into a more “generalised purpose” language because it would be easier to maintain and operate a small team with one language in what was becoming a non-tech enterprise. Python was our first go to, because well, it’s just a nice language that’s easy to learn, but we eventually ended up in Typescript and our story was basically the polar opposite to what you mention here.
We found the package support to be far superior in JavaScript. Even a lot of non-web things like working with solar inverters and various other green energy technologies (which at least here in Europe is very, very old school) we’re significantly easier with JavaScript than they were with Python. I guess FTP is web, but it’s the example I remember the best because I had to write my own Python library for it because the most used packages couldn’t handle a range of our needs. This may be because it’s FTP is not me shortening SFTP, no no, that’s just what solar plant engineers figured would be practical. Sure they recommend you don’t put the inverters directly on the internet in their manuals, but who reads those? Not solar plant installers at least. Anyway, I fully expected Python to be king for those things considering it’s what runs a lot of IoT, but JavaScript was just plain better at it.
Which is generally the story of our migration of basically everything to typescript. Some or our C++ code ended up as Rust, and I really, really, love Rust, but most of our code base is now Typescript. It might all have been Rust if Rust was a frontend web language, but it would never be Python. The reason for this is the environment. Not just the package and module systems, but also how you can set up a very “functional programming” (in quotes because it’s not really FP but JSFP) to cut down on the usage of OOP unless absolutely advantageous, type safety, specific coding systems and how easy it is to actually enforce all of those things automatically. Is just a much better experience with Typescript compared to Python in our experience. I think you could very likely achieve some of the same with JSDoc and JS and not Typescript, but we couldn’t get that to work (as well) in our internal tooling.
Somewhat ironically, the two areas JavaScript wasn’t better than Python were in what I’d consider web-related parts of your tech stack. There aren’t any good JS SQL ORMs, Prisma is ok and so is MikroOrm but none of them compare to what’s available in Python. The other is handling XML, which is soooo easy in Python. I mean theoretically it should always be easy to handle XML, but that would require the XML you need to handle to actually adhere to any form of standards, which I guess is utopia.
But I guess you can have very different experiences.
Java was never really an option because of how hard it is to hire for in my region of Denmark (and maybe the field of green energy). Java certainly has some presence at some of the larger tech focused enterprise orgs, but most developers we come in contact with aren’t interested in working with it. Not sure why considering C# is quite popular among them, but it is what it is.
Python has really poor support for compiled extensions. I know this sounds weird to say, given that they are used everywhere, but this is the number one pain point in Python. It’s really awkward to say, develop on Mac and deploy on Linux.
Might I ask what scripting-like language does have good support for compiled extensions? Such that you can easily develop on Mac and deploy on Linux?
Because it seems to me that once you compile something you are in the awkward world of ABI and CPU differences. And binary portability has been a paint point of programmers since before I was born (and I'm not that young).
So if there is a programming language that neatly gets around this problem, me and a lot of other folks would really like to know about it.
C# does. It's not exactly a scripting language and authoring a native dependency NuGet package isn't exactly an obvious task, but when you learn how, it's a straighforward solution:
- a NuGet package with native deps (win, linux, osx) cross join (x86, arm)
- a P/Invoke package that depends on the native one
- actual software that uses the dependency
When you publish the actual software, it pulls the deps and includes the correct native build.
I’m talking about the ecosystem. I was unable to get a small Python project with some mainstream native libraries to compile for Linux on a macOS host without Docker.
This works far better in Rust, for example.
Of course if Python wasn’t so dog slow we wouldn’t need so many native packages.
It’s mad how there isn’t a Django clone in the JS world. They just stitch together half finished, buggy ORMs, migration tools and web frameworks? After all this time? Something like Django requires focus and concerted effort over a period of many years, so I guess it makes sense.
I get the impression JS devs would rather have a new framework with bugs and cool emojis in the commits than something more stable and less buggy.
I'd like a Django in Go ! But so farI haven't found anything that productive. I guess the "traditional" web framework for multi-pages website is not trendy enough.
Typescript is way way better for one off scripts than Python - using Deno you can have single scripts that can actually import third party and first party dependencies reliably.
Neither of those work well in Python. To import third party libraries you need to use Pipenv or Poetry or one of the many other half baked options.
Importing first party dependencies (e.g. code from another file) is also a nightmare because the path searching strategy `import` uses is insane. It can even import different code depending on how you run Python!
Absolutely agree. I've been writing Typescript for years for work, but I constantly try and explore other languages. The only language I'm never 100% sure how to handle in terms of importing both local and third party code is Python. Virtualenvs are a joke, and version management is terrible. Local imports don't make sense too.
I’ve been a Python developer for about 15 years and it isn’t good at most things. Performance is bad, package management is bad, typing is bad, async is bad, etc. Mostly it shines these days because of early mindshare in an exploding niche (AI/ML/numeric computing).
I’ve seen a lot of outages in production which were very hard to debug because someone blocked the event loop with a sync call or some CPU-intensive thing. The failures weren’t in the route that was blocking the event loop, but all over the place, including health checks, which would cause instances to be bounced until the whole service fell over.
Go doesn’t have problems like this—you could theoretically block the event loop with something sufficiently CPU-intensive, but Go schedules work across all cores (and moreover, Go gets hundreds or thousands of times more work done per core than Python, so it’s far less likely to run into these problems) so this becomes highly unlikely.
Python can only be second-best at anything if you take advantage of its strengths and avoid its weaknesses. If you insist on taking advantage of its weaknesses, it will suck completely.
The typing system is a huge weakness. If you insist on focusing on it, it won't be a good language.
>Lots of comments here are stating that typing is half baked in Python, and that if you gotta use types, you should use another language.
Its weird that so many conflate "this would be nice to have" with "this is the most important factor when deciding on a programming language". I think, obviously, very few Python users have type safety as their most important factor when deciding on a programming language.
The main drawback of Python is that it ruins you for other languages.
Python is a scripting language, it's great for scripts and one-offs, etc. But once you get above, say, 100K LoC it starts to get out of its sweet spot and into more direct competition with languages like, e.g. OCaml.
In some cases I hate when I have to use Python for a task that really deserves full typings because Python’s type hints are so half baked. But all other times, I like just sprinkling them in where relevant. Python is great at being okay at everything. That’s a real power.
This comment inspired me because I was working on data-types at my company at the moment and I realized how static typing would have really benefited us.
If you eg interact with remote apis a lot, you can write the natural code: some api calls for setup, then your business logic, then more api calls... and then test this. In one linear function that is straightforward to understand. With great mock support, no need for code generation for mocking, etc.
Ruby is one of the few languages sharing this excellent support. All the other common languages require some combination of dependency injection, code generation for mocking, spewing unnecessary interfaces all over your code and turning it into a tangled mess of logic smeared across many functions, etc.
> Python is[] not meant to be the best at anything, but good at most things.
I haven’t yet had the time to look at dataclasses and pattern matching in recent versions, so serious question: do we finally have a standard solution for doing algebraic-datatype-style stuff? I do have a generic visitor implementation lying around somewhere, but that doesn’t change the fact that every time I see any writeup of the type “Let’s do <a thing involving syntax> in Python!”, I spend most of my reading time wishing they had used SML or OCaml—and I don’t even know OCaml.
> Lots of comments here are stating that typing is half baked in Python, and that if you gotta use types, you should use another language
... I thought this is how most of us use Python? I find it repulsive to use without types. type_hints enable me to use it for Software Engineering, and I get to keep the Data Science stack, which is solid gold.
> the fields I need tools for are vast, and it's the only one that I'm pretty sure will handle a problem decently in all it's various forms
Whilst I agree with this stance to some extent, we should be clear that Python is not in fact the panacea you've made it out to be here. It is very possible that a particular problem requires performance Python can't match, so that any Python solution will be too big/ slow/ clumsy and must be rewritten in a better language. It is also very possible that a particular problem requires safety Python can't match, so that any Python solution will be too dangerous and must be rewritten in an appropriate safe language to avoid unacceptable losses.
> It is very possible that a particular problem requires performance Python can't match, so that any Python solution will be too big/ slow/ clumsy and must be rewritten in a better language.
This has been discussed again and again on HN, and the answer to it still stands to this day. So now I'm not giving you the benefit of the doubt.
If you don't like the characterization as a panacea, what would you prefer - jack of all trades maybe? All-purpose language ?
> This has been discussed again and again on HN, and the answer to it still stands to this day. So now I'm not giving you the benefit of the doubt.
I'd guess you're thinking you'll measure and just rewrite the hot code paths. The problem is in too many cases when industrialising software it's basically all hot, the heatmap just all glows red. Google people did talks about this maybe a decade ago, it's why Go ended up getting internal support, because if you write software the first time in a language with better performance you don't need to do the rewrite.
I think Python's actual strength is as a language for people whose job isn't primarily to write software. Let me give an example of a choice Python made (admittedly not for years) that is exactly what you should do for that audience, and then the opposite:
Ordered Dictionaries make dict have reasonable performance and yet also behave how naive users who have only a limited understanding of how the machine works would expect, which means they produce less buggy software in practice
Multiple Inheritance is too complicated to teach to a class of say, Geographers, and yet it's not really crucial for the underpinnings of the language, so why go to such lengths to support this feature ?
> the whole ecosystem is always getting better and better.
That is debatable. In many ways the ecosystem is getting more enterprisy, more ceremonious, overall "heavier". I feel Python ecosystem is rapidly losing its winning characteristic lightness that was so defining for it 10ish years ago.
I don't think "modern" Python code like in the article would inspire comics like this https://xkcd.com/353/
But that's missing the point that Python is still not meant to be the best at anything, but good at most things.
And in this case, it's exactly what you get: optional typing, with decent safety if you need it.
You can quick script or design seriously, you can explore in a shell or commit a public API into a file, you can start with untyped code and add some later.
That makes it cumbersome sometimes, yes. mypy is slow, and the typing system + match / case have many rough edges.
But it also keeps Python incredibly versatile.
It's amazing what you can do with this (not so) little language, and the whole ecosystem is always getting better and better.
Every time I try another language, I keep getting back into Python, because the fields I need tools for are vast, and it's the only one that I'm pretty sure will handle a problem decently in all it's various forms, especially the one I didn't consider when I started.
There is immense value in this, because it keeps opportunities on the table no matter what you do. I often find my self during a project adding something I could only because Python let me do so easily.
Python is quirky, yet it still damn practical.