That's like saying multiple programs compiled by different compilers from the same sources should have very similar binaries. You're looking in the wrong place! Similarities are to be expected in the structure of the latent space, not in model weights.