Yes, you can skip the sugar pills provided you have a good memory. These pills do not contain medication, and are provided to keep you in the routine of taking a pill every day. It's important to start your next pack on time, so make note of the date!
You are supposed to take the sugar pills in your birth control compact so that you will be in the habit of taking pills every day. This way you won't forget the birth control pills or forget what day you are on. Take the pills to be safe.
It won't hurt you physically to skip them but the primary reason they are there is so that you do not forget to take your pills ever. Usually persons who have a strict routine of taking the same pill every day succeed at remembering whereas people who skip have a lot harder time remembering when they took what pill.