I have never heard of fish smelling you. I know they feel your vibrations, but I dont see what it would hurt. I think you could give it a try and if it works better then go for it.
I go fishing very often and always use mineral oil soap. When using mineral oil soap the fish cannot smell you. I am very successful while fishing after using this oil.
I have been told many thinks in my life. I think that if you don't like the smell of fish on your hands you should use a pumice soap after fishing or ware rubber gloves when baiting your hook.