It might. I spent years suffering repetitive yeast infections and realized after my allergy to latex was identified that condoms were probably causing them. Ask your doctor!
If you told your doctor when you starting seeing him that you're allergic to latex then it should be in your chart. The doctors and nurses who see you will use latex free gloves. So I doubt your pap was abnormal from the gloves. They probably used non-latex gloves.