Why is the Summer Solstice only the beginning of warm weather for the season?

The same goes for the Winter Solstice as well.. I've always wondered. I mean if the sun is closest to the N. Hemisphere around June 21st, shouldn't that day be the peak of Summer's warm weather? How does it get progressively warmer in the weeks after the Solstice, if the N. Hemisphere is moving further and further away from the Sun? Is it the Ocean? Thanks.