Optimized systems are fragile. Probably true of models.
The most profound class I had in college was thermodynamics, where all the complexity of physical systems is lumped into a few parameters. It seems crude but those parameters include all the energy and can be used to make very useful calculations.
Is this a fair restatement: “Models (and systems) that accurately reflect reality with a high fidelity due in part to their accurate assignment of values to a larger number of parameters are fragile”?
I agree that that's true, but my point (and I think part of what the article is saying) is that most “optimized” systems/models are not in fact doing that. In which case, the point I'd take (if I read you correctly) is that even once you _have_ managed to accurately set the parameters (which is harder to do and validate than it looks), you still have a fragile model/system.
Alternatively [edited to add]:
Define a model as a system that is attempting to match another system in behaviour. And so now you have two problems:
① a complicated model is itself fragile (by analogy to the system it's modelling)
② a complicated system has more ways to diverge from the system it's trying to model
Hmm, in which case it's kinda the same thing after all, except one is about making the model in the first place, and the other is maintaining it, maybe?
It occurs to me that there's a sort of “learned helplessness” trap you can fall into from the thermodynamic base case: you can fail to separate _anything_ from placebo if your model fails to include enough of the real levers in the modelled system vs the levers it has that don't exist in that system. You can turn any signal into noise if you slice it up too much; that's basically half of the replication crisis. Studies on anti-depressants are an interesting example, for instance.
The most profound class I had in college was thermodynamics, where all the complexity of physical systems is lumped into a few parameters. It seems crude but those parameters include all the energy and can be used to make very useful calculations.