89

Users have long been asking for and discussing the pros and cons of a feature to proactively inform the question asker when their question has been closed. Although the initial response many years ago was that after consideration we would not implement this feature (for a variety of reasons), following the release earlier this year of the follow post feature (which greatly expanded inbox messaging relating to post updates, and has been positively received by the community), we decided to revisit this feature. 

During our announcement about the first phase of Question Close updates, we included a notice about an experiment that we would be running related to Email and Inbox notifications for closed questions

This is launching on Monday (April 20) on Stack Overflow as an A/B test. This update sends question authors an inbox notification when their question is closed. The idea is that the inbox notification will guide users to their closed question and the post notice displayed there will encourage them to edit the question to improve it.

If there are no edits, the question remains closed, and hasn't been deleted within an hour of closure, we'll also send a follow-up email with guidance about what to do next.

It’s running as an A/B test so that we can measure how effective these notifications are in guiding users to edit their questions.

The experiment ran for approximately two months on Stack Overflow only. The goals of the experiment were to see how adding inbox notifications to the post author would affect:

  1. The percentage of closed posts edited by the author after it was closed. We were hoping to see a positive change here.
  2. Site satisfaction/user reaction. We were hoping that users would not have a negative reaction to the new notifications and emails.
  3. Determine if the custom email notifications were more effective than just the inbox notifications were for leading to more post owner edits after posts are closed.

Results

Result data from the experiment. No notification group: 43,619 total posts, 5,464 edited (12.53%), with 590 posts being reopened after edit (1.35%). Inbox only group: 296 total posts, 41 edited the post (13.65%), with 9 posts being reopened after edit (3.04%). Inbox and email group: 39,428 total posts, with 5,437 edited (13.79%), with 616 posts being reopened after edit (1.56%). Aggregate total for inbox (with or without email): 39,724 total posts,5,478 edited (13.79%) and 625 being reopened (1.57%).

Relating back to the goals of the experiment:

  1. Percent of closed posts edited:
    1. As shown by the data, in the control group (no notification) 12.53% of posts were edited by the post owner.
    2. This can be compared to 13.79% of posts edited after inbox notifications were sent, an increase of over 10% in the percentage of posts edited (corresponding to our first goal). This indicates that the notifications and/or email are having a positive effect in getting post owners to reengage after their posts are closed.
    3. This also led to an increase in the number of posts reopened after edits (1.35% to 1.57%). While this was not itself a goal for the experiment (there is nothing in this feature that is designed to lead to higher quality edits that are more likely to lead to posts being reopened), it is nice to see and is worth noting.
  2. Community reaction: We did not discover any significant negative feedback about the new notifications, nor was there any unexplained drop in site satisfaction numbers during this time period which might be correlated with a negative reception to this experiment. 
  3. Inbox only vs email:
    1. The original experiment did not test this question well enough. The logic put into place was to send the email out one hour after the inbox notification went out (as long as no edits were made, the question was still closed, and had not been deleted).
    2. Additionally, the system default interval to receive an email notification for unread inbox messages is 3 hours. Since 99% of participants in the experiment received this custom email, there is no way for us to tell if the 3 hour interval email (which sends the standard inbox notifications, without any special formatting relating to post closure) would have been more or less effective.
    3. Because of these factors, no determination can be made as to the effectiveness of the emails (on their own).
    4. It is worth noting that the Click Through Rate on the emails was 12.4%, which is a relatively high click through rate.

So, what now?

Based on what we saw, we have decided to graduate the inbox notifications. The following is now live, network wide:

  • Post owner inbox notifications when their question is closed
  • Post follower inbox notifications when a question that they follow is closed
  • No notifications are sent to:
    • Anyone when the question is closed due to being migrated to another site (this has its own notification)
    • The OP or any follower who cast the last vote for closing the question (including when a question author self-dupes their own question)

As far as whether or not to graduate the email, we are going to open a new experiment (that will run for approximately one month), in which we will have three groups:

  1. Control: Inbox notification only, no email
  2. A: Inbox notification sent upon closure and email sent after one hour
  3. B: Inbox notification sent upon closure and email sent after one day

We will make a final decision about the email based on the results of this experiment. We welcome your feedback below and questions on the experiment and new features.

  • 3
    “This can be compared to 13.79% of posts edited after inbox notifications were sent, an increase of over 10% in the percentage of posts edited (corresponding to our first goal).” - How exactly do you figure that it’s an increase of 10%? The difference between 12.53% and 13.79% is only a little over 1% increase. – Ramhound Nov 17 at 10:23
  • 33
    @Ramhound 1 percentage point increase, but (13.79 - 12.53)/12.53 ~ 10.05% increase in the rate. – muru Nov 17 at 10:24
  • 14
    I wish to add that while the effect on SO may be rather small, I expect this to have a much bigger impact on some other sites. – Wrzlprmft Nov 17 at 10:26
  • 2
    @Wrzlprmft why would that be? – rene Nov 17 at 10:28
  • 20
    @rene: Several reasons: A lower ratio of questions that are unsalvageable, fewer help vampires, people being more invested in their questions, more timely closure. – Wrzlprmft Nov 17 at 10:40
  • 14
    @Wrzlprmft interesting theory (and I agree with your premise). I will try to dig into the data after the experiment to see how this plays out on SO vs non-SO sites. – Yaakov Ellis Nov 17 at 10:42
  • 12
    @Ramhound: The probability of this difference occurring by chance is astronomically low (from scipy.stats import fisher_exact; fisher_exact([[5464,43619-5464],[5478,39724-5478]]) → 7·10⁻⁸). The more interesting question is whether the effect size was worth it. – Wrzlprmft Nov 17 at 10:52
  • 2
    10% more edits result in 15% higher chance to reopen. Is there any possible explanation of how these extra edits seem to be more potent than normal edits? – Trilarion Nov 17 at 11:36
  • 6
    @YaakovEllis, Not directly, no. But if I've had a post closed and deleted, unless I have high enough rep, it disappears. I assume these notifications stay your inbox for atleast 6 months and include a link to the question? Wouldn't an indirect result of this be the ability to view the post past that 90 day cut-off? – ouflak Nov 17 at 12:28
  • 4
    @Trilarion possible explanation: the email that goes out contains different instructions based on the close reason (similar to what is shown in the post notice). Perhaps the instructions given here lead to higher quality edits. But that is just conjecture. – Yaakov Ellis Nov 17 at 12:30
  • 2
    Will the OP also be able to view close votes on their question(s)? – ouflak Nov 17 at 12:31
  • 2
    @ouflak this is not changing anything about the ability to view close votes. Just adding a notification when the post is closed. – Yaakov Ellis Nov 17 at 12:33
  • 2
    Can a 10k user screenshot Jeff Atwood's now-deleted original answer to the original question? I'd like to read about the thinking back then. This seems like such an essential feature. – pkamb Nov 17 at 17:30
  • 5
    One of the other answers actually has nearly all of what Jeff says. i.stack.imgur.com/hycsI.png is what Jeff said, as he said it – Journeyman Geek Nov 17 at 17:50
  • 4
    Are four significant digits (e.g., 12.53%) warranted? The sample size alone would suggest a relative accuracy of about 0.5% (not percentage points) at best. – P.Mort. - forgot Clay Shirky_q Nov 18 at 2:16
46

Update from Yaakov Ellis's comment:

These emails are only sent to users who have their email settings for Tips & Reminders set to On. Users who have this set to Off should not receive the email, even if they are in the experiment group.

This kind of solves my concern. I still think the one-hour waiting time is too "nagging", but this is still being experimented on.


Original answer:

The email should be an opt-in feature. There are users that do not want to receive almost any SE-related notifications outside of the SE inbox, and this should not be an exception.

The advice that comes with the email could be shown on the SE website after the user clicks a link (only visible to them) in the close notice.

If this is not an option, I suggest to at least make the one-hour wait time between closure and email considerably longer. A closed question on SE is in no way a matter so urgent that it needs to be attended to in one hour.

| improve this answer | |
  • 27
    "The email should be an opt-in feature" This!!! – leonheess Nov 17 at 16:42
  • 7
    This new feature needs to be attached to email settings that are already available in preferences, or maybe a new unique one. It doesn't make sense to not make it toggleable like all other SE emails. – zcoop98 Nov 17 at 17:51
  • 2
    What does this have to do with the GDPR? – user2357112 supports Monica Nov 17 at 23:25
  • I don't think the GDPR says anything that requires this feature to be opt-in. The closest thing is that there needs to be a legal basis for processing personal data used to send emails, but contrary to popular misconception, that legal basis does not have to be explicit consent in advance. – user2357112 supports Monica Nov 17 at 23:36
  • 1
    There should be an opt-out option, but I'm pretty sure the GDPR doesn't require this to be opt-in. – user2357112 supports Monica Nov 17 at 23:43
  • 4
    GDPR has absolutely nothing to do with this. – Asteroids With Wings Nov 18 at 1:32
  • 1
    @AsteroidsWithWings Maybe not, but the GDPR-concerned user has something to do with it. :) – Conrado Nov 18 at 3:44
  • 8
    An opt-in would guarantee that virtually no-one gets the emails. Users who are engaged enough to go opt into this would almost certainly be the sort of users who don't need it. These emails obviously shouldn't be sent to people who've opted out of the relevant categories in their email settings, but making it opt-in would defeat the user benefit of sending the email. – Ryan M Nov 18 at 7:24
  • I just removed the "GDPR" part, as it might be wrong and is not really necessary anyway. – wimi Nov 18 at 7:37
  • It should respect the overall email notification settings, of course. – Trilarion Nov 18 at 8:26
  • 1
    Well here in the Netherlands you are not allowed to send newsletter-type of emails without consent. That probably boils down from EU legislation. That consent NEEDS to be opt-in. It can't be the old fashion remove this checkmark or check this box to NOT receive the newsletter. It has to be click here to subscribe. However these emails are of a different type, so it might not be needed in this case. – Luuklag Nov 18 at 8:43
  • 3
    These messages are considered Transactional Email under the terms of GDPR and therefore exempt from necessary opt in/out. It might be nice to allow the user to opt in/out but there is no legal requirement to do so – roaima Nov 19 at 9:41
  • 9
    These emails are only sent to users who have their email settings for Tips & Reminders set to On. Users who have this set to Off should not receive the email, even if they are in the experiment group. – Yaakov Ellis Nov 19 at 11:01
  • @roaima "transactional email" are not a concept from GDPR. What you are probably referring to is the processing of user data on the basis of fulfilling a contract. This would typically include things like "please pick up your order" emails and the general processing of an address to deliver their order. This would not apply to the processing of data in this case as far as I can tell. Most likely this would fall under the processing of PIIs based on a legitimate interest, which is something the user can legally object to which is typically implemented as an opt-out. – David Mulder yesterday
  • @DavidMulder Law refers to this very phrase in Where is the line between transactional vs marketing emails with GDPR? – roaima yesterday
8

"... Additionally, the system default interval to receive an email notification for unread inbox messages is 3 hours. Since 99% of participants in the experiment received this custom email, there is no way for us to tell if the 3 hour interval email (which sends the standard inbox notifications, without any special formatting relating to post closure) would have been more or less effective."

and

[...]

  1. A: Inbox notification sent upon closure and email sent after one hour
  2. B: ″ ″ one day.

I think if people are fairly active they may or may not react before one hour, and likely will before one day; whereas people who check in every few days or longer (the majority?) are basically in the same category (receive email (group 2 and 3), or not (group 1)).

My point: For the majority, an hour or a day are too similar an interval (too close together).

You could guess which group they might fall into (assuming a correct email address) by checking how often they visit - so perhaps group 3 would benefit from being sent after 3-5 days.

| improve this answer | |
  • 2
    In the last experiment, users received the special email before there was a chance at getting the inbox notification. So no way of knowing if maybe the inbox notification by itself could do the trick or not, since the quickest that that can come is within 3 hours. Extending the experiment to a day gives some differentiation there to see if the inbox email might do the trick. That said, considering changing this to a longer email to make it more obvious. – Yaakov Ellis Nov 19 at 10:59
3
  • No notifications are sent to:
    • [...]
    • The user who cast the last vote for closing the question (including when a question author self-dupes their own question)

The wording of this is a bit confusing to me. I understand not sending a notification to OP when they vote to close their own question. However, the main, non-parenthetical part of the sentence sounds like you are sending a notification to the first and second (and 3rd/4th for non-SO sites) close voters when a question they voted on is closed, but surely this is not the case.

Assuming I'm correct there, I'm wondering why you don't just say no notifications are sent to:

the post owner when they cast the vote that closes their question/vote to close their own question

or something similar?

| improve this answer | |
  • 11
    If a user who followed the question cast a close vote on the question, and that vote is not the last vote, they will receive a notification. If they cast the final vote, then will not receive a notification. – Yaakov Ellis Nov 17 at 14:43
  • 4
    @YaakovEllis So a user who casts a close vote, but does not follow the question, will not get notified? – TylerH Nov 17 at 15:00
  • 4
    That is how I interpreted things from the start @TylerH. You need to be either OP or a follower of the Q to be notified. – Luuklag Nov 17 at 15:21
  • 4
    @Luuklag correct – Yaakov Ellis Nov 17 at 15:24
  • 2
    @Luuklag It's how I'm hoping it was meant to be interpreted but since it's unclear based on the wording I wanted some explicit confirmation, because the alternative would be, IMO, pretty bad. – TylerH Nov 17 at 16:27
  • It would be very weird to send a notification about the closure that you were the last nail of. – Braiam Nov 17 at 17:41
  • @Braiam I agree. That's not what this answer is about. – TylerH Nov 17 at 22:05
2

Great to have this new feature! I've had countless of questions being closed silently, which is extremely annoying when waiting for answers to questions.

Giving a chance to the OPs to improve/edit/update/etc. their questions before the Stack Exchange system makes it impossible to other users to answer the questions is very important. In that spirit, I believe one should also receive a notification when one's question gets deleted.

| improve this answer | |
  • I'm in doubt on notifications upon question deletion. Yes I'm all about (increased) transparancy, but on the other hand it will lead to a lot of re-posting questions, which I see as a major drawback. But perhaps this hypothesis is best put to the test. – Luuklag 9 hours ago
-10

"... Additionally, the system default interval to receive an email notification for unread inbox messages is 3 hours. —copied from https://meta.stackexchange.com/a/356767/163204 & Question closed notifications experiment results and graduation where I originally commented, thank you https://meta.stackexchange.com/users/282094/rob for the encouragement

Why "3 hours"? Is Stackcorp's bulk/mass emailing really the issue (server stress and spam complaints basically?)? I want instant and every notifcation, personally*. Can I please pay $5/mo. or buy credits for instant and you pay for longer timing if you prefer? Picking "three" hours for everyone is a lack of thought.

Programmatically a customizable form to allow the user to set the setting is optimal, and would lead the way as a role model for other websites suffering with the same systemic/formulaic poverty of thought.

*I can not trust the current Notification emails to be regular, I manually refresh SE.com constantly just to stay informed (clicking the menubar's uninformative red light sequare, again and again), the Android app is the most reliable notification I get yet it's ethereal (restart phone and poof, missed several mods made in under 30 minutes, again), versus a copy of email apparently timed currently for dealing with server stress instead of user stress.

| improve this answer | |
  • 4
    For instant notifications there is the inbox on the site. – Luuklag Nov 18 at 15:45
  • @Luuklag I mentioned that being a problem in the astericks (*), did you read that line? I added the word "menubar" for that line/part to be clearer. – prosody-Gab Vereable Context Nov 18 at 15:46
  • 4
    I don't get what is unreliable about it. It works perfectly fine for me. – Luuklag Nov 18 at 15:49
  • @Luuklag If you can stay loaded 100% of the time, have a 100% working internet connection, always have your browser working, then you are privileged to get all your notifications with that unfriendly barebone feature. You're privileged then, I am not able to trust that works like you. Privilege. – prosody-Gab Vereable Context Nov 18 at 15:50
  • 14
    I'm sorry you're struggling with the on site notifications but you're asking for something that'd be a huge change to our defaults for notifications. That's not a bad thing but would need to be a separate discussion since this question relates to one specific notification only. – Catija Nov 18 at 17:28
  • 1
    I reorganized as another q at meta.stackexchange.com/questions/356817/…. Thank you @Catija for the encouragement, it's definitely not personal. I agree and realize the work is a burden of complications (why nobody wants to muddle O:], and I myself can not help code ASP.NET the current SE lang last I checked) as well as you. :) I value your time. – prosody-Gab Vereable Context Nov 18 at 18:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .