Because latex, even if your not allergic to it, can cause us to have a negative reaction, and leaves behind a smell, or bodily reactions.
It is fine for people who have sex and use a condom once in a while, but for people who need to use multiple condoms a day, latex can have adverse reactions...