The most likely answers are poor insulation or drafty windows. Windows should seal well and, for energy efficiency, be thermal windows - i.e. double-pane glass.
Most likely the problem is poor sealing around your windows, doors, and other airways. Your air ducts themselves may need to be sealed and pressurized.
