The usual transphobe argument is that chromosome abnormalities are too rare to worry about. Even if that were true statistically, which it isn't, it still leaves the question of what to do with those specific individuals who have them, and if you let them go where they obviously belong, you've made a mockery of any claims that only chromosomes are reality.