Users have long been asking for and discussing the pros and cons of a feature to proactively inform the question asker when their question has been closed. Although the initial response many years ago was that after consideration we would not implement this feature (for a variety of reasons), following the release earlier this year of the follow post feature (which greatly expanded inbox messaging relating to post updates, and has been positively received by the community), we decided to revisit this feature.
During our announcement about the first phase of Question Close updates, we included a notice about an experiment that we would be running related to Email and Inbox notifications for closed questions:
This is launching on Monday (April 20) on Stack Overflow as an A/B test. This update sends question authors an inbox notification when their question is closed. The idea is that the inbox notification will guide users to their closed question and the post notice displayed there will encourage them to edit the question to improve it.
If there are no edits, the question remains closed, and hasn't been deleted within an hour of closure, we'll also send a follow-up email with guidance about what to do next.
It’s running as an A/B test so that we can measure how effective these notifications are in guiding users to edit their questions.
The experiment ran for approximately two months on Stack Overflow only. The goals of the experiment were to see how adding inbox notifications to the post author would affect:
- The percentage of closed posts edited by the author after it was closed. We were hoping to see a positive change here.
- Site satisfaction/user reaction. We were hoping that users would not have a negative reaction to the new notifications and emails.
- Determine if the custom email notifications were more effective than just the inbox notifications were for leading to more post owner edits after posts are closed.
Results
| Total | Edited | Diff | Reopened after edit | Diff | |
|---|---|---|---|---|---|
| No notification | 43619 | 5464 (12.53%) | 590 (1.35%) | ||
| Inbox only | 296 | 41 (13.85%) | 10.6% | 9 (3.04%) | 124.8% |
| Inbox and email | 39428 | 5437 (13.79%) | 10.1% | 616 (1.56%) | 15.5% |
| Inbox total | 39724 | 5478 (13.79%) | 10.1% | 625 (1.67%) | 16.3% |
Relating back to the goals of the experiment:
- Percent of closed posts edited:
- As shown by the data, in the control group (no notification) 12.53% of posts were edited by the post owner.
- This can be compared to 13.79% of posts edited after inbox notifications were sent, an increase of over 10% in the percentage of posts edited (corresponding to our first goal). This indicates that the notifications and/or email are having a positive effect in getting post owners to reengage after their posts are closed.
- This also led to an increase in the number of posts reopened after edits (1.35% to 1.57%). While this was not itself a goal for the experiment (there is nothing in this feature that is designed to lead to higher quality edits that are more likely to lead to posts being reopened), it is nice to see and is worth noting.
- Community reaction: We did not discover any significant negative feedback about the new notifications, nor was there any unexplained drop in site satisfaction numbers during this time period which might be correlated with a negative reception to this experiment.
- Inbox only vs email:
- The original experiment did not test this question well enough. The logic put into place was to send the email out one hour after the inbox notification went out (as long as no edits were made, the question was still closed, and had not been deleted).
- Additionally, the system default interval to receive an email notification for unread inbox messages is 3 hours. Since 99% of participants in the experiment received this custom email, there is no way for us to tell if the 3 hour interval email (which sends the standard inbox notifications, without any special formatting relating to post closure) would have been more or less effective.
- Because of these factors, no determination can be made as to the effectiveness of the emails (on their own).
- It is worth noting that the Click Through Rate on the emails was 12.4%, which is a relatively high click through rate.
Experiment, take 2
Based on what we saw, we decided to graduate the inbox notifications. The following is now live, network wide:
- Post owner inbox notifications when their question is closed
- Post follower inbox notifications when a question that they follow is closed
- No notifications are sent to:
- Anyone when the question is closed due to being migrated to another site (this has its own notification)
- The OP or any follower who cast the last vote for closing the question (including when a question author self-dupes their own question)
As far as whether or not to graduate the email, we opened a new experiment in mid-November (that ran for approximately two months), in which we had three groups:
- Control: Inbox notification only, no email
- A: Inbox notification sent upon closure and email sent after one hour
- B: Inbox notification sent upon closure and email sent after one day
Final Result: Closed emails after one day
The second round of the experiment included over 87,000 closed posts.
Findings:
- The email sent one hour after the post
CloseDatehad no significant effect on edit rates for those posts by post owner (8.50% edited with inbox only versus 8.64% with email). - The email sent after one day did have a significant effect on the edit rates by post owner: 3.86% (versus 2.08% of posts edited after one day in the inbox-only control group).
Based on these results, we have graduated the closed email experiment, sending emails to users one day after post close. This excludes posts where the post owner has already edited the post, where the post is deleted or reopened, or where the post owner has indicated that they do not want to receive emails of this nature.
from scipy.stats import fisher_exact; fisher_exact([[5464,43619-5464],[5478,39724-5478]])→ 7·10⁻⁸). The more interesting question is whether the effect size was worth it.