You're conflating physical properties with other abilities/characteristics. Does a dwarf have a physical limitation preventing them from wielding magic? This is the argument.
I don't see anything innately wrong with a human who can breathe fire, or has wings, or a dwarf with four arms, so long as you're willing to RP it. It does seem silly to say that no, it's actually against the rules, your dwarf can't learn magic.
There are different versions of D&D. At least in some versions, you have some control over assigning some of the stats, and so you could assign some of them as you wish if you want better bonuses for magic.
It shouldn't limit you from using magic, and it doesn't, and that is good.
But, sometimes you will want to do things other than just casting spells (especially if you have run out of spells or if there is anti-magic preventing casting spells), so you can decide if you want to be good at one thing, or good at other things too but perhaps not as much.
That's actually a really good question. It's certainly one of the key debated questions at the heart of what WotC is doing here (and as others point out, somewhat lagging the rest of the TTRPG industry, many of which got bored of "races" in the D&D mechanical sense a long time, in one way or another).
Are they archetypes for builds? Why are they archetypes for builds? Is it problematic if they are archetypes for builds? Are they just flavor for settings and character backgrounds and other storytelling needs? Should they be? Why have "races" at all and not just "backgrounds"?
I don't see anything innately wrong with a human who can breathe fire, or has wings, or a dwarf with four arms, so long as you're willing to RP it. It does seem silly to say that no, it's actually against the rules, your dwarf can't learn magic.