And let me guess, they don't have your phone number memorized; they probably save your phone number in their phone's contacts list alongside your name, which is automatically backed up in the cloud in either Google Contacts or iCloud depending on what phone they use.
Expecting your friends not to use cloud services seems a bit unreasonable and unenforceable. Are you really going to tell all your friends they should write down or memorize your phone number instead of storing it in their phone?
While your name and phone number are public, the list of people who have your phone number saved is not public. These companies can effectively map who you associate with only by looking at other people's phones. They can profile you by association, and that is very wrong.
At some point privacy is no longer a choice, not a real choice anyway. You get to chose between participating in society or keeping your privacy. It shouldn't have to be this way, but it is.
Oh, uh, I thought you were talking about your name.
Microsoft collects your friends' names so it can spell their names correctly when you use speech-to-text or related features. I thought that's what you were objecting to. What kind of data are you talking about?
If you really are as charming in real life as you are online, people might indeed accede to your quaint insistence on anachronistic media, until they don't.
Personally, I require that all my so-called friends communicate with me in morse code over short wave radio. It works a treat.
Sucks to be you if your relationships, be they business or personal, are as fragile as a volatile technology.