You could still do enforcement through legal rather than technical means, though.
Disallow installing apps from outside the App Store, provide no system UI to do so. Prohibit apps from being app stores themselves or running code that didn't pass app review, with exceptions for dev tools etc. Make apps able to escape the sandbox, at least in some ways.
Even if an app somehow sneaks past app review and gives users unfettered access to their devices, it can't ever get too many users. If it's unpopular, it's not a concern to Apple, if it becomes popular, Apple will know about it and can levy very heavy contractual fines on the dev.
It's worth mentioning none of these concerns affected Android / Google Play, even though it's fairly easy to sideload and even install custom app stores there.
The only phones that come with alternative stores are from Chinese manufacturers, which isn't going to be an issue for Apple as the operating system isn't open source.
The OPs main point was that the existence of sideloading or third-party app stores hasn't led to major security issues. And apps still have to comply with the sandboxing.
Your ideas are explicitly illegal in the EU and, if the law ever passes, would be illegal in the US too. Frankly I think we are past having to figure out ways for Apple to tax software usage in all circumstances, just wait for the law to catch up and it's all moot. In fact, just the Epic case seeks to rewrite what amounts to 70% of App Store spending: gacha games being able to link to their own billing options.
> The legislation aims to prevent Big Tech companies from "self-preferencing" their own products at the expense of competitors.[3] Under AICO, covered platforms would be forbidden from disadvantaging other companies' products or services.
Disallow installing apps from outside the App Store, provide no system UI to do so. Prohibit apps from being app stores themselves or running code that didn't pass app review, with exceptions for dev tools etc. Make apps able to escape the sandbox, at least in some ways.
Even if an app somehow sneaks past app review and gives users unfettered access to their devices, it can't ever get too many users. If it's unpopular, it's not a concern to Apple, if it becomes popular, Apple will know about it and can levy very heavy contractual fines on the dev.