Yes mildew does leave an ugly mess if you don't go on and get it off as soon as you can. I use bleach and a mild soap with a soft brush and it takes it off of your bath tub and things like that. Clothes it leaves a black stain on the piece.
I believe if it is on certain porous surfaces for very long it can actually discolor it. As with any residue, the sooner you clean it off the better. That way it is easier and has less time to stain.
