That's why we built our Attribution Engine [0]. We help platforms to deal with CSAM, IBSA and any toxic content and copyright.
The way it works is that platform uses our SDK through which they "send" all uploaded content. The SDK generates a fingerprint that is sent to our service and a license is issued. The license is non/permissible, so they instruct platform and the uploader (creator) what to do. We also provide payment distribution, usage reporting and ADR (alternative dispute resolution).
If platform uses our service, we indemnify them from any liabilities under DMCA, EUCD and others up to $50M in damages (legal cost and/or court orders).
We charge % of revenue generated by the platform for our services. If platform generates no revenue, the service is free. We don't charge per lookup nor there is any scale limit (well, there are throughput limits, but they are quite far for most platforms [currently we can search around 1.1k hours of content every second]).
We cover video, sound recordings and compositions. Images are coming late this year and text sometimes next year.
Just a quick note on the moderation. The benefit of our structure is the moderation is "outsourced" back to law enforcement and gov agencies, like NCMEC and FBI in US, Bundespolizei in Germany, etc.). This means platforms don't have to hire people to moderate the covered content, because the liability is transferred back to the organizations that are create it in the first place.
We built our adminless Internet forum [1] for the same reason. It's splendid on the dev side because initially we built traditional moderation tools (they still exists in the Git), but then we deleted it all and it felt much simpler.
Still waiting to see if it will work from a moderation standpoint. The site's been live for almost a year with no issue yet.
Edit: It's a text only forum, which makes this a lot easier. I was thinking about allowing images but with a high per image fee.
It's complicated, because there are legal requirements across many different jurisdiction. The product was built to explicitly solve for the liability issues platforms are facing.
I like your solution. Up until it works for you, that's great.
First the pornography comes, then the child pornography. That can even be minors sexting which is not a criminal network but it can still get you in trouble.
Politics has a way of turning into violent threats, pictures of nooses, etc.
Then there is the spam, actually that comes before the pornography.
There's a reason there's been so much research into AI moderation, although so far it always seems to either be useless or throwing babies out with the bathwater. Not to mention that without human systems to review what the AI decides (or even with) it tends to appear to users as automatic censorship (and they're usually not wrong about that).
Once I had 10,000 NSFW images in a collection of 1,000,000. You might say that 99% of the images are good and for most classification problems 99% accuracy would be wonderful.
But it's not good enough.
You might find some simple tricks that eliminate 90% of those NSFW images and now you have 99.9% accuracy. That makes you look like a genius in the machine learning world.
But it's not good enough. Even 99.99% accuracy (100 bad images) is not good enough
Even if somebody found one NSFW image you could get kicked out of Adsense. One child porn image uploaded to your site won't put you in jail but it is still completely unacceptable.
We had pretty good success on https://paint.wtf (AI-judged Pictionary game) using CLIP for content moderation. Feel free to probe it a bit by drawing a dick or something on one of the prompts; it seems to do a pretty good job (if it doesn’t work please hit the “Report” button and let us know).
I think a large part of the issue could be solved through taking a more "common sense" approach to product design. Imagine you offer a platform like Facebook and I registered a few hours ago. Is it really smart to allow me to start uploading dozens of images or livestream to potentially thousands of existing users? Surely a better approach would be for my account to have restrictions on what I can do and the number of times/period for which I can do it. Once I become trusted in the eyes of the platform those restrictions can start to be loosened.
We've been studying this problem for a product we're working on, and it seems like the depths of this hole get dark and very deep. We expect to need to hire someone to moderate early on.
As mentioned, there is child porn, various other forms of illegal porn, illegal violence, hatred, propaganda to incite violence, etc... Then of course any kind of content you might not approve of due to any personal or company policies, if relevant.
All of our material will be forced to be public so the user should have no expectation of privacy, but for many people this won't matter. Just look at Facebook; that's a public-facing service, and people upload atrocities to it constantly.
The current discussion seems to fall under "Trust and Safety". I'm not aware of any specific guidance or organised thought, though I'd be really surprised if there weren't academic coursework or professional training beginning to appear.
You'd do well to look at established services. Craigslist's list of prohibited content, the T&C of Facebook, Twitter, Reddit, etc., are going to be useful.
Just off the top of my head:
Cyberstalking, pornography, child porn, piracy, malware, fraud, illicit goods (guns, drugs, black/grey market, stolen goods), intimidation, gangs, bullying, alcohol, tobacco, prescription medications, hoaxes, various ineefective / "alternative" products and remedies (which themselves run the gamut of legality, even defining this is at best difficult), advertising, advertising for protected or regulated sectors / goods (housing, employent, personal and professional services, beauty care, escorts, security services, licensed professional, ... As with the goods section, this rapidly gets complex), legal services / aid, political activities, fomenting revolutin / freedom fighters.
User-generate content is a massive concern.
One concept I'm seeing getting increased traction is a focus not on the quantity of posted content but the prevelance or level of access or views. Facebook and YouTube especially are increasingly discussing problematic content not in terms of posts or videos, but of views or presentations of those.
This ... starts making trade-offs in moderation much more viable, principally because there is an inverse logarithmic relationship between the number of items and the views: If n items gets n views, then 10*m items get n/10 views. Very roughly.
This means that you can set a goal in terms of the number of items viewed (and see what the maximum unmoderated prevalence will be), or target a specific prevalence and determine how many reviewers will be required.
For human moderators, the number of items reviewed per day seems to be in the 500--800 range. Note that 800 items/day in 8 hours is 100/hour, or 1.6 per minute, or 36 seconds per item. That's inclusive of breaks, overhead, and non-moderation tasks.
Moderation itself is a very psychologically loaded task. You'll either want to rotate people through it from other functions, or see a heck of a lot of staff turnover.
If anyone has greater insights from one of the current large UGC services (FB, Twitter, Instagram, Whatsapp, TikTok, Imgur, Reddit, etc.), I'd really like to know what current internal practices are.
Some of my previous work had some incidental exposure to this area (I was tasked with removing identified content, working on both our internal and external CDN provider to do so). After a couple of spot checks to see I was unlikely to be deleting content which wouldn't meet removal criteria, as in literally two, I decided I simply didn't want to take the risks of performing additional checks. My removal process turned out to be quite effective --- what the CDN provider's specs suggested might be a weeks-long process removed some millions of items over a weekend. That was on what is by current standards a very modest-sized social network.
