Cooling and warming by a few degrees can happen in much less time, possibly just a few centuries - see Younger Dryas.

Which, from what I can tell, should make us more worried, since it shows how drastic and unpredictable the climate can respond to changing conditions, like greatly increasing the carbon in the atmosphere.