I've had this same issue over and over again in my career.

I've toyed with the idea of writing a daily "sanity checker" in crontab that verifies various concepts of system health.

Examples: Did the latest batch of data transfer to S3? Did we delete old customer accounts today? Did we get any signups (because if not, something may be broken, but not triggering an exception report etc)? Did we send out daily report emails?

But I could see this easily becoming a pointless exercise, and I doubt I'd have the time to keep the sanity checker updated with the latest requirements. In fact, the sanity checker would probably become insane pretty quickly.

Perhaps the platform itself should do this for you, in some way. Idea: while coding, indicate that this procedure should be running periodically, ie:

    Monitor.registerPeriodicTask('email-reports', 'daily')

and then the system would log every time it occurs, with a generic task that would run periodically and scan for things that should have occurred, but haven't in some while.

Monitor that the newest backup is less than N hours old.

Alternatively, if you bother to have proper exit codes for your backup scripts, you could verify exit != 0, and alert on that.

I have cron on all of my systems email STDERR to me, STDOUT is redirected to the normal logs.

One you clean up your crons, you really only get email once a month or so when something breaks.

Do you use something like http://habilis.net/cronic/ ?

Nope, just instead of this

    0 1 * * * backup >/var/log/foo 2>&1

I do this

    0 1 * * * backup >/var/log/foo

My errors are emailed to me via standard cron conventions. Normal logging is available for me if I care to look at it.