Of course not. Attraction is an emotion with many many facets. Yes the physical part of it maybe the first area for many but two people need to connect mentally.
There are many different things that cause an attraction between a guy and a girl. The important thing to remember is that there is no set rule for any group of people. Guys are independently-thinking individuals.