There seems to be some issue with unaccountable maintainers, and that’s a problem for innovation. If you can’t even get to the point of “technical debate” (we aren’t interested, sorry), then what hope innovation?
These are “people problems”, and there are no easy or good answers, but it doesn’t mean we should through our hands up and consider things “unfixable” either.
"... I come at this from the perspective of having worked on Linux since around December of 1991. I first met you and Tove in 1995 at the Free Software Conference at MIT that Stallman sponsored. ...
Probably the last technical contribution of my career is leading an initiative to provide the Linux community a generic security modeling architecture. Not to supplant or replace anything currently being done, but to provide a flexible alternative to the development of alternate and/or customized workload models, particularly in this era of machine learning and modeling.
Four patch series over two years, as of yesterday, not a single line of code ever reviewed. ...
We were meticulous in our submissions to avoid wasting maintainers time. We even waited two months without hearing a word before we sent an inquiry as to the status of one of the submissions. We were told, rather curtly, that anything we sent would likely be ignored if we ever inquired about them.
We tried to engage, perhaps to excess, in technical discussions attempting to explain why and how we chose to implement what we were proposing. ...
There were never any relevant technical exchanges. The discussion consisted of, we have decided to do things a certain way, no discussion, if you don't like that you should really consider doing something other than submitting to upstream Linux."
Okay, but how long is reasonable to take to tell someone that their patch is too big to be reviewed?
> We were meticulous in our submissions to avoid wasting maintainers
time. We even waited two months without hearing a word before we sent
an inquiry as to the status of one of the submissions. We were told,
rather curtly, that anything we sent would likely be ignored if we
ever inquired about them.
It's reasonable to ask that people make smaller-sized patches to get reviewed, and it's reasonable to have to rule out some things due to not having the bandwidth for it compared to other priorities, but it's pretty ridiculous to expect people to know that those are the reason their code isn't getting reviewed if they're not allowed to inquire once after two months of radio silence.
Not everything can be broken down into 5 line patches. Some of the bigger features need bigger code. You can look at the impact radius and see if the patch is behind a feature flag to make a decision. In this case, it seems like it is elective and does not have any changes beyond what is self-contained.
I don’t follow the Linux kernel development process closely, but on most projects—open source or proprietary—that I’ve been involved in, dropping a big diff without having proposed or discussed anything about it beforehand would have been fairly unwelcome, even if it’s something that would be infeasible to break down smaller. I’d also argue that even quite substantial changes can be made with incremental patches, if you choose to do so.
There had probably been a lot of discussion in the background
The "every change can be a small patch, can be split in such" is part now of the Linux folklore IMHO. As in, a belief that people kinda heard about but nobody has concrete proof
Also do you know what would make it much easier to review long changes? Not using email patches
GitHub issues with "load 450 hidden comments" on it? No threads, no comparisons between multiple submissions of the same patch, changes from rebasing impossible to separate from everything else? Having to drop anyway to the command line to merge a subset that has been reviewed? Encouraging squash merging makes it easier to work with long changes?
Email is complex and bare bones, but GitHub/GitLab reviews are a dumpster fire for anything that isn't trivial.
I've used Gerrit myself for nearly 10 years. I recognize it's strengths and like it for what it does. I also am comfortable with mailing lists and Git Hub and Git Lab. I've lived in "both worlds".
IIUC, the Kernel project doesn't want to use it because of the single-point-of-failure argument and other federation issues. Once the "main" Gerrit instance is down, suddenly it's become a massive liability. This[1] is good context from the kernel.org administrator, Konstantin Ryabitsev:
"From my perspective, there are several camps clashing when it comes to the kernel development model. One is people who are (rightfully) pointing out that using the mailing lists was fine 20 years ago, but the world of software development has vastly moved on to forges.
"The other camp is people who (also rightfully) point out that kernel development has always been decentralized and we should resist all attempts to get ourselves into a position where Linux is dependent on any single Benevolent Entity (Github, Gitlab, LF, kernel.org, etc), because this would give that entity too much political or commercial control or, at the very least, introduce SPoFs. [...]"
SPoF is not the only concern. Gerrit search is painful. You can't search the internet for a keyword that you wrote in v10 of some Gerrit patch, unlike on list-based workflows. Also, there's too much point-and-click for my taste, but I lived with it, as that's the tool chosen by that community. It's on me to adapt if I want to participate. (There are some ncurses-based tools, such as "gertty"[1]; but it was not reliable for me—the database crashed when I last tried it five years ago.)
In contrast, I use Mutt for dealing with high-volume email-based projects with thousands of emails. It's blazingly fast, and an absolute delight to use—particularly when you're dealing with lists that run like a fire-hose. It gives a good productivity boost.
I'm not saying, "don't drag me out of my email cave". I'm totally happy to adapt; I did live in "both worlds" :-)
That's a great post, and also the presentation linked there is very interesting and hints at some of the issues (though I disagree with some of the prescriptions there)
Yes, the model is not scaling. Yes the current model is turning people away from contributing
I don't think linux itself is risking extinction, but someday the last person in an IRC channel will turn the lights off and I wonder if then they will think about what could they have done better and why don't people care about finicky buildsystems with liberal use of M4. Probably not.
One of the first habits I learned at BigCo was to have a sense of the DAG in any work over 400 lines, so I could make patches arbitrarily large or small. Because whichever I chose, someone would complain eventually
Indeed, and that's exactly what the Rust for Linux guys are doing. They have Binder, multiple changes with various degrees of dependency on newer compilers, support for different kinds of drivers, and they are extracting them and sending them to Linux in small bits.
I'd get mildly angry if someone sent a giant patch without having discussed with me first. (Not working with any kernel things) But a quicker "no, don't have time, probably never will" response would have been nice I guess?
no feedback whatsoever for 2 months - just because it was 17k lines - and you are blindly defending it? if you don't know what you are posting & talking about, then maybe you shouldn't.
if someone is willing to put in the efforts to submit such a huge patch, how about just show them a little bit respect, I mean the minimum amount of respect, for such efforts and send them a one line reply asking them to break the patch into smaller ones.
surely that won't take more than 45 seconds. still too hard to get the point?
> if you don't know what you are posting & talking about, then maybe you shouldn't
I hate to appeal to authority, but I have been working with large cathedral-style open source projects (mostly three: GCC, QEMU, Linux itself) for about 20 years at this time and have been a maintainer for a Linux subsystem for 12. So I do know what I am posting and talking about.
The very fact that I had never learned about this subsystem from LWN, Linux Plumbers Conference, Linux Security Summit etc. means that the problem is not technical and is not with the maintainers not replying it.
A previous submission (7000 lines) did have some replies from the maintainers. Apparently instead of simplifying the submission it ballooned into something 150% larger. At some point I am not sure what the maintainers could do except ignoring it.
> At some point I am not sure what the maintainers could do except ignoring it.
Ignoring people is a childs approach to communication. If there is a problem, its your responsibility to communicate it. If you have communicated the issue and the other party is refusing to hear it, that is a different issue, but it is your responsibility to communicate.
Surely if there were insufficient established context for a reviewer to easily follow a large patch the correct response would be a swift rejection with a request that such context be established prior to submitting again.
It sounds like there was willingness to meet any requirements, but submitters end up in a position of not knowing what the requirements are and if they have met them or not.
It's very easy to develop tunnel vision when working remotely with people you've not even ever met. It's happened to me multiple times and there's a very high chance that this is happening here.
The short of his rant is that he wants a "Code Of Standards for maintainers" in addition to a CoC, in order to witch hunt people he feels "gets in the way."
Yikes, this is EXACTLY what Linus was talking about except an order of magnitude worse!
No. He just wants people to have some decency and standards in communication and not arbitrarily decide that their little kingdom can never be touched.
This does not read as a rant at all to me. Rather it seeks to highlight a problem using an example from his own work and proposes a possible solution (with a previous caveat of “I don’t know how to fix this”).
> "Code Of Standards for maintainers" in addition to a CoC
He wants maintainers to behave in some pre-determined understandable fashion. He wants some accountability, and that seems reasonable to me. This is not a “maintainers must do what I want”, this is “let’s set basic expectations” and ensure people follow them. Whatever those should be.
> in order to witch hunt people he feels "gets in the way."
B does not follow A. You are simply straw-manning here, so I have nothing to say to it other than it’s a fallacious point.
His complaint is, in essence, that people will block technical proposals for nontechnical reasons, and that attempts to resolve this result in people complaining that you're talking about nontechnical things at all.
Few people like dealing with interpersonal conflict. They want to think that technical meritocracy wins the arguments.
But in that discussion, a maintainer said "I am going to try to sabotage this because I hate it.", and there's no technical argument to resolve that. And there's not really any route other than escalation to some kind of authority if there's not a common negotiation baseline.
"You can't reason someone out of a position they weren't reasoned into."
>in order to witch hunt people he feels "gets in the way."
When has this ever happened with the Linux CoC? The only people who I've seen banging pots and pans together about the CoC are folks who like to sealion [1]
There are easy and good answers, just none that every single one would agree with, and that is the problem.
The current management model is based on consensus in some parts, and a dictatorship in others. Its not clear which parts are which because of sprawl, and non-responsiveness.
This is a core communication people issue. The problems arise because in consensus models, people are expected to adopt archetypical roles based somewhat on preference, but mostly on the needs of the group, towards productivity.
If no one else is doing that type of role, someone will be forced into doing it, and will adopt characteristics of that role, regardless of their personal opinions about it.
When they get punished for fulfilling the role, you lose people.
> doesn't mean we should throw our hands up and consider things "unfixable" either.
The given structure is unfixable, because it sets the stage for a trauma loop where actions can't take place without people who are expected to fulfill a role to meet consensus, but in doing so they get punished for volunteering.
There is no way around this structure so long as you use a consensus model in whole or part, in dysfunctional groups the people involved either fail to communicate or worse there are members that impose coercive cost by assuming the unproductive roles so no action happens while resources are wasted.
This really is basic stuff covered in your college level intro to communications coursework.
For those wanting to know more about this, you can read more about the roles at the link below.
I think "Code of Civility" is what Dr. G was describing. Conduct presupposes civility, but acceptable online discourse has changed amongst new internet natives raised in the era of moral totalitarianism, aggression as a service, and popular outrage incubators.
Looks like Dr Greg (not Greg KH) is just piling his own personal grievance on top of the drama. Saying "Jim are you listening?" makes me wonder who he's sniping at.
Speaking as an actual People Manager, his ideas sound pretty shallow.
> Jim Zemlin, who is the executive director of the Linux Foundation and has nothing to do with Linux.
That sounds... Not good. Seems like like this is exactly what's wrong with... Well, not just this, but much of the "FOSS" world. (Maybe about since "Free Software" became "Open Source"?)
There seems to be some issue with unaccountable maintainers, and that’s a problem for innovation. If you can’t even get to the point of “technical debate” (we aren’t interested, sorry), then what hope innovation?
These are “people problems”, and there are no easy or good answers, but it doesn’t mean we should through our hands up and consider things “unfixable” either.