The same thing happened to me and a nurse gave me some good advice. Any mild hair conditioner works great at removing band aid adhesive. Just rub it in and let it sit for a few minutes, then rub gently with a washcloth and rinse. Conditioner will also loosen the edges of a band aid that sticks.
You can use nail polish remover (Cutex), soaking a cotton ball with baby oil, applying rubbing alcohol on a Q-tip followed by body lotion, and if all else fails there is always the tried and true use of soap and water.