I received an email from the moderation team. At the end it stated:
This email was sent from an unmonitored account. Do not reply directly to this email; if you feel this message is incorrect, please respond on Stack Overflow here: https://stackoverflow.com/users/message/XXXXXX.
The link did not work until I was somehow able to log in under the right account. After logging in it works. Maybe worth putting a note about that or something in the email, because when it fails all you get is a "page not found" message and no indication of why the failure happened or what to do about it.