> Finding a genuine security flaw in OpenSSL is extraordinarily difficult.
history suggests otherwise
> The fact that 12 previously unknown vulnerabilities could still be found there, including issues dating back to 1998, suggests that manual review faces significant limits, even in mature, heavily audited codebases.
no, the code is simply beyond horrible to read, not to mention diabolically bad
if you've never tried it, have a go, but bring plenty of eyebleach
If someone meant to engineer a codebase to hide subtle bugs which might be remotely exploitable, leak state, behave unexpectedly at runtime, or all of the above, the code would look like this.
> If someone meant to engineer a codebase to hide subtle bugs which might be remotely exploitable, leak state, behave unexpectedly at runtime, or all of the above, the code would look like this.
I wonder who could possibly be incentivized to make the cryptography package used by most of the worlds computers and communications networks full of subtly exploitable hard to find bugs. Surely everyone would want such a key piece of technology to be air tight and easy to debug
But also: surely a technology developed in a highly adversarial environment would be easy to maintain and keep understandable. You definitely would have no reason to play whackamole with random stuff as it arises
2. In practice, the risk of introducing a breakage probably makes upstream averse to refactoring for aesthetics alone; you’d need to prove that there’s a functional bug. But of course, you’re less likely to notice a functional bug if the aesthetic is so bad you can’t follow the code. And when people need a new feature, that will get shoehorned in while changing as little code as possible, because nobody fully understands why everything is there. Especially when execution speed is a potential attack vector.
So maybe shades of the trolley problem too - people would rather passively let multiple bugs exist, than be actively responsible for introducing one.
It reminds me of Google Dart, which was originally pitched as an alternate language that enabled web programming in the style Google likes (strong types etc.). There was a loud cry of scope creep from implementors and undo market influence in places like Hacker News. It was so poorly received that Google rescinded the proposal to make it a peer language to JavaScript.
Granted, the interests point in different directions for security software v.s. a mainstream platform. Still, audiences are quick to question the motives of companies that have the scale to invest in something like making a net-new security runtime.
Pointless nitpick, but you want "undue market influence." "Undo market influence" is what the FTC orders when they decide there's monopolistic practices going on.
> Surely everyone would want such a key piece of technology to be air tight and easy to debug
The incentives of different parties / actors are different. 'Everyone' necessarily comprises an extremely broad category, and we should only invoke that category with care.
I could claim "Everyone" wants banks to be secure - and you would be correct to reject that claim. Note that if the actual sense of the term in that sentence is really "almost everyone, but definitely not everyone", then threat landscape is entirely different.
I read that whole paragraph with a tinge of sarcasm. There's bad actors out there that want to exploit these security vulnerabilities for personal gain and then there's nation-state actors that just want to spy on everyone.
I expected much worse to be honest. Vim’s inline #ifdef hell is on a whole other level. Look at this nightmare to convince yourself: https://geoff.greer.fm/vim/#realwaitforchar
That's a lot of ifdefs, sure. But at least Vim doesn't have it's own malloc which never frees and can be dynamically replaced at runtime and occasionally logs sensitive information.
As long as you don't statically link you can easily replace malloc (LD_PRELOAD). Many debug libraries do. Why is this so special in openssl? (I don't know if there is some special reason, though openssl is a weird one to begin with)
Using OpenSSL's malloc may bypass protections of hardened libc mallocs like OpenBSD's.
If memory crosses the boundary between OpenSSL and your app, or some other library, freeing it with a different allocator than the one it was allocated with is undefined behavior.
OpenSSL's allocator doesn't free in in the same ways other mallocs do, which prevents memory sanitization tools like valgrind from finding memory bugs.
OpenSSL has a completely separate idea of a secure heap, with it's own additional malloc implementation, which can lead to state leakage or other issues if not used perfectly at the (non-existent because the entire library surface is exposed) security boundary and is accidentally intermingled with calls to the (insecure?) malloc.
It's just a big can of security worms which may have been useful on odd platforms like VMS, though that's questionable, and only serves to add additional layers of inscrutability and obfuscation to an already messy codebase today. It's not enough to know what malloc does, one must familiarize themselves with all the quirks of both(!) of OpenSSL's custom implementations, which are used precisely nowhere else, to judge the security or code correctness implications of virtually anything in the codebase. There's no good reason for it.
> Finally, taking an OpenSSL public API and attempting to trace the implementation to see how it is implemented has become an exercise in self-flagellation. Being able to read the source to understand how something works is important both as part of self-improvement in software engineering, but also because as sophisticated consumers there are inevitably things about how an implementation works that aren’t documented, and reading the source gives you ground truth. The number of indirect calls, optional paths, #ifdef, and other obstacles to comprehension is astounding. We cannot overstate the extent to which just reading the OpenSSL source code has become miserable — in a way that both wasn’t true previously, and isn’t true in LibreSSL, BoringSSL, or AWS-LC.
Also,
> OpenSSL’s CI is exceptionally flaky, and the OpenSSL project has grown to tolerate this flakiness, which masks serious bugs. OpenSSL 3.0.4 contained a critical buffer overflow in the RSA implementation on AVX-512-capable CPUs. This bug was actually caught by CI — but because the crash only occurred when the CI runner happened to have an AVX-512 CPU (not all did), the failures were apparently dismissed as flakiness. Three years later, the project still merges code with failing tests: the day we prepared our conference slides, five of ten recent commits had failing CI checks, and the day before we delivered the talk, every single commit had failing cross-compilation builds.
Even bugs caught by CI get ignored and end up in releases.
Wow, that is just crazy. You should investigate when developing software, but for something like OpenSSL... Makes me think this must be a heaven for state actors.
I'm surprised AI was even able to find bugs in that.
Given that it's been trained on "regular" code and that presentation points out that openssl might as well be written in brainfuck it shocks me that AI would be able to wrap its pretty digital head around it
> If someone meant to engineer a codebase to hide subtle bugs which might be remotely exploitable, leak state, behave unexpectedly at runtime, or all of the above, the code would look like this.
I'd wager if someone did that the codebase would look better than OpenSSLs
The codebase designed to hide bug would look just good enough that rewriting it doesn't seem worth it.
OpenSSL is so bad that looking at it there is just desire to rip parts straight out and replace them, and frankly only fear-mongering around writing security code kept people from doing just that and only after heartbleed the forks started to try. And that would also get rid of any hidden exploit.
It really is just a collection of several dozen research grade implementations for algorithms + a small handful of load bearing algorithms for the entire internet. Surprisingly, OpenSSL isn't the only critical piece of internet architecture like this.
maybe this is what blindsides most developers into disregarding the threat of AI to their jobs. We work off some idealised version of what the industry actually is which we presume AI will fail at, instead of the reality.
I remain surprised at how long people can flog horses I figured would be dead decades earlier in enterprise. Too scared to fix fundamental issues and still running off the fumes of vendor lock-in with exasperated end users.
I worry that software and the industry is more resistent then we might imagine. Consider the insanity of Elon Musk's arbitrary cuts to twitter and the resilience of that platform in the years that followed.
It might simply be the case that buying more tokens and kicking the code enough times might give a "good enough" result for the industry to continue. I don't want to believe this but the discussion of how awful the openssl code base is seems to suggest that might be the case. You just need to automate the process of caution we have around it. We should all be hoping that Gastown fails but I feel like it might succeed.
The insanity is how he enacted them. Like the idea that everyone should come to his office with print outs of the code they've written, or that everyone has to come into HQ to do some all-nighters. Just an absurd hunger-games attitude to his workforce, full of horrific coginative biases and discrimination against some of the workforce (e.g. against those with young children or those with disabilities who might be less able to commit to all-nighters).
There was an article on here 15ish years ago to the effect of "everything's broken all the time. Everyone who writes software knows it, yet we all tolerate it."
I'd love to find that sometime. Maybe it's time to ask Gemini once again to look for me.
Honestly, this is absurdly funny, but it makes me wonder whether we'll ever see Computer Science and Computer Engineering as seriously as other branches of STEM. I've been debating recently whether I should keep working in this field, after years of repeatedly seeing incompetence and complacency create disastrous effects in the real world.
Oftentimes, I wonder if the world wouldn't be a bit better without the last 10 or 15 years of computer technology.
This is really something that’s making me quite fed up with industry. I’m looking towards embedded and firmware in hopes that the lower in the stack I go the more people care about these type of things out of business necessity. But even then I’m unsure I’ll find the rigor I’m looking for
I’ve been thinking the same thing lately. It’s hard to tell if I’m just old and want everyone off my lawn, but I really feel like IT is a dead end lately. “Vintage” electronics are often nicer to use than modern equivalents. Like dials and buttons vs touch screens. Most of my electronics that have LCDs feel snappy and you sort of forget that you’re using them and just do what you were trying to do. I’m not necessarily a Luddite. I know tech _could_ be better theoretically but it’s distressing to know that it’s also not possible for things to be different for some other reasons. Economically, culturally? I don’t know.
Is it still a critical piece? I thought most everyone migrated to libressl or boringssl after the heartbleed fiasco and serious people took a look at OpenSSL and started to understand the horror show that is the codebase and also development practices that clearly have not gotten better, if not gotten even worse.
We don't know how to secure C codebases by manual review. It's been well known to security engineering people for decades. And has been wider industry and academic consensus for a long time. It's like "is man-made climate change real".
(We don't know how to secure other codebases either, but C is harder since its memory safety story is like a chainsaw juggling act so code has classes of vulnerabilities that other languages don't and this eats a lot of the attention).
Security certifications are one reason. OpenSSL maintains a module for FIPS compliance, which includes an entire boatload of weak and broken algorithms nobody else bothers with.
It is. There are other related issues like at some point RedHat patched back options removed/changed in openSSH 7.0 because
* they upgraded a major release (6.x to 7.x) in "stable" channel of their distro
* their customers ran some ancient stuff that required those options.
We've failed a security audit because our checks just compared OpenSSH version ("if version is above this it doesn't need any change in config") while Red Hat's OpenSSH version was downgraded to earlier version settings/security issues
A number of projects like Firefox and the Linux kernel uses them. It's boring at that point. The generated code is C and assembly can be used like any library, but it has been formally verified.
But, there is ring and rustls too. A number of projects are shifting to it
I can read C/C++ code about as well as I can read German. Bits and pieces make sense but I definitely don’t get the subtleties.
What’s eye bleachy about this beyond regular C/C++?
For context I’m fluent in C#/javascript/ruby and generally understand structs and pointers although not confident in writing performant code with them.
For one thing, "C/C++" is not a thing. If you see C-like C++, that is C.
Part of OpenSSL's incomprehensibility is that it is not C++ and therefore lacks automatic memory management. Because it doesn't have built-in allocation and initialization, it is filled with BLAH_grunk_new and QVQ_hurrr_init. "new" and "init" semantics vary between modules because it's all ad hoc. Sometimes callees deallocate their arguments.
The only reason it needs module prefixes like BLAH and QVQ and DERP is that again it is not C++ and lacks namespaces. To readers, this is just visual noise. Sometimes a function has the same name with a different module, and compatible function signature, so it's possible to accidentally call the wrong one.
history suggests otherwise
> The fact that 12 previously unknown vulnerabilities could still be found there, including issues dating back to 1998, suggests that manual review faces significant limits, even in mature, heavily audited codebases.
no, the code is simply beyond horrible to read, not to mention diabolically bad
if you've never tried it, have a go, but bring plenty of eyebleach