I think both of these points are two sides of the same coin. There's an athletic advantage that most transgender women will have over most cisgender women, but there's also an athletic disadvantage that most trans women will have compared to cis men.
i.e., unless something fundamentally changes about how leagues are divided, there's going to be perceived unfairness in sports.
As long as things are unchanged, I think the real conversation boils down to who we prioritize: cis women or trans women.