Welcome to End Point’s blog

Ongoing observations by End Point people

Avoid 2:00 and 3:00 am cron jobs!

A word to the wise: Do not set any cron jobs for 2:00 am or 3:00 am on Sunday morning! Or to be safe, on other mornings besides Sunday as well, since jobs originally set to run on some particular day may eventually be changed to run on another day, or every day.

Most of the time such cron jobs will run fine, but if they run every Sunday morning, then twice per year they will run at the exact time daylight savings time (aka summer time) kicks in or ends, sometimes with very strange results.

On Linux with vixie-cron we saw two cron jobs run something like once per second between 3:00 and 3:01 when the most recent daylight savings time began. Thus they ran about 60 times, stepping all over each other and making a noisy mess in email. No serious harm was done, but that's only because they were not tasks capable of causing serious harm.

Feel free to wish for or agitate for or fund or write a better open source job scheduler that everyone will use, one that will ensure no overlapping runs, allow specifying time limits, etc. Better tools exist, but until one of them achieves cron's level of ubiquity, we have to live with cron at least some places and sometimes.

Alternatively, where possible set the server timezone to UTC so that no daylight savings changes will happen at all.

Or most preferable: Governments of the world, stop the twice-yearly dance of daylight saving time altogether.

But in the meantime this particular problem can be entirely avoided by just not scheduling any cron jobs to run on Sunday morning at 2:00 or 3:00 server time.


Anonymous said...

Set system clock to UTC, problem solved. Users can set their own TZ for cosmetic purposes.

3am is a great time for corn jobs since by that time the late night programmers have gone home to avoid stupid hour.

Brian Gadoury said...

Any time is a great time for corn jobs!

Jon Jensen said...

Anonymous: I think you missed the point. 2:59 is fine, and 3:01 is fine, just 3:00 is a problem. Those other times are roughly equivalent but avoid the trouble.

And yes, corn jobs are the way to go in any case. :)

Leigh said...

Was PS the guilty site? I think we have a 3am job and one that occurs every 20 minutes and one that occurs every 3 hours.

Jon Jensen said...

Hi, Leigh. It's possible but it actually happened two other places that I was aware of. :)

If the cron job runs quickly enough or doesn't produce any email output then it may not have been noticed that it ran many extra times during that one minute. But in these two other places it made some email noise.

Richard Templet said...

I double checked and actually we don't have any running at 2 am and I adjusted the 3 am one to run at 305 right after Jon posted this. It also happens to be a script that wouldn't generate any email output so it could have run into the same issue and we just didn't see it.

Jason Ellison said...

cron - daemon to execute scheduled commands (ISC Cron V4.1)

Daylight Saving Time and other time changes
Local time changes of less than three hours, such as those caused by the start or end of
Daylight Saving Time, are handled specially. This only applies to jobs that run at a spe-
cific time and jobs that are run with a granularity greater than one hour. Jobs that run
more frequently are scheduled normally.

If time has moved forward, those jobs that would have run in the interval that has been
skipped will be run immediately. Conversely, if time has moved backward, care is taken to
avoid running jobs twice.

Time changes of more than 3 hours are considered to be corrections to the clock or timezone,
and the new time is used immediately.

Jon Jensen said...

Jason, yes, it sure sounds like ISC Cron (aka Vixie cron) tries to handle DST. All I know is that it doesn't work right, at least not in several occasions I've personally encountered.