These days it should be possible to roll back a few steps (à 15 min / 1 hour) with a copy on write filesystem like zfs? Full scale restore from backup should only be necessary if the storage hardware fails (IMHO).

You still need to apologize for a some data loss, though. So make sure that everything you do has one or two safety nets before it hits the customer.