The best solution would be at the OS level, preventing someone's entire contact list from reaching an app's server in the first place.

In iOS 14, Apple (re)implemented a photo picker so apps don't need access to your entire photo library for you to upload a single photo. I'm hoping they could come up with something similar for contacts as well.