When it cools down enough where you know nobody is going to swim anymore. Preferably before leaves start to fall off the trees to keep clean up to a minimum
You can close your pool whenever you feel that it is too cold for you to want to use it any more. If you live somewhere where it freezes, make sure the pool is empty by the time it starts freezing.
The general consensus is the first frost of the season. If you want a specific time however, generally by mid-September the temperature just isn't warm enough to swim anymore, so anytime after the 15th is usually pretty appropriate. You don't want to wait until the first snowfall.