Well, it depends on how flirtatious he is being. There is always the possibility that you are misconstruing charm for flirtation. Then on the opposite end of the spectrum is sexual harassment. Myself, I would ask him how his wife would feel about his actions. Then depending on his answer I might tell him to f off or something.