Systemd status mail on unit failure

So while setting up a Systemd timer and service in my previous article about systemd I ran into a bit of a problem with failure handling.

What I wanted was to receive a mail if my service (started by a timer) failed, sound quite simple and the default behavior of cron (more or less). Looking through the manual for systemd.unit I found the attribute “OnFailure”, sound like the one you want to use right? However from what I understood this argument only takes a reference to another Unit, not a command…

My solution for this was to create a reusable service that will send a status mail about any service, and let OnFailure trigger the service.

Step 1

In the service defined during the previous article (“/etc/systemd/system/sitename-drupal-cron.service”) the following was added:

[Unit]
...
OnFailure=unit-status-mail@%n.service

What is important to understand here is the “%n” after “@”, “%n” will be substituted for “Full unit name” in this case “sitename-drupal-cron.service” (it will be escaped, but that is not a problem here), and “@” which means that what is after “@” but before “.service” will be considered the “Instance name”. The instance name will be used as the argument of which service we are going to send the status mail for.

Step 2

Next we create the unit-status-mail service, “/etc/systemd/system/unit-status-mail@.service” (yes the file name contains “@.” with nothing in between):

[Unit]
Description=Unit Status Mailer Service
After=network.target

[Service]
Type=simple
ExecStart=/bin/unit-status-mail.sh %I "Hostname: %H" "Machine ID: %m" "Boot ID: %b"

As you can see, here we call a script that will send the actual mail, and we also send along some (possibly) useful Systemd paramters.

Step 3

Last step is the script which will send the actual mail. Just a disclaimer, I am not an experienced Bash scripter, so no code convention or so has been used ;)

“/bin/unit-status-mail.sh”

#!/bin/bash
MAILTO="root"
MAILFROM="unit-status-mailer"
UNIT=$1

EXTRA=""
for e in "${@:2}"; do
  EXTRA+="$e"$'\n'
done

UNITSTATUS=$(systemctl status $UNIT)

sendmail $MAILTO <<EOF
From:$MAILFROM
To:$MAILTO
Subject:Status mail for unit: $UNIT

Status report for unit: $UNIT
$EXTRA

$UNITSTATUS
EOF

echo -e "Status mail sent to: $MAILTO for unit: $UNIT"

Remember to make the script executable, “chmod +x /bin/unit-status-mail.sh”.

Test

You can try the unit-status-mail script out by running:

/bin/unit-status-mail.sh sitename-drupal-cron

Remember, this script and service should work for any service that you want status mail for ;)