81

I've been looking at what happens when new users post their first questions on Stack Overflow... Not really done yet, but I figured I'd toss what I have up here in case anyone else has useful suggestions.

For window of time between 30 and 90 days ago, I looked at users who posted their first question within this window and broke them into two groups: those who returned and posted at least one more question, and those who never posted anything else again (or at least, haven't yet). I then collected some data on what sorts of things happened to their first post...

New users who came back after their first question was...

Total Upvoted Downvoted Answered Closed Deleted Commented Edited Nothing
----- ------- --------- -------- ------ ------- --------- ------ -------
20227 4520    5080      13979    3883   2899    13651     8962   891    

Percentage of this group's total:
      23%     25%       69%      19%    14%     67%       44%    4%

New users who never came back after their first question was...

Total Upvoted Downvoted Answered Closed Deleted Commented Edited Nothing
----- ------- --------- -------- ------ ------- --------- ------ -------
52720 10644   12143     28636    9836   10000   31677     19822  5111   

Percentage of this group's total:
      20%     23%       54%      18%    18%     60%       37%    9%
  • 17
    How many of those "never came back"s do you think are people just trying to get a quick fix on their homework? – Dennis Meng Jan 18 '14 at 18:44
  • 4
    Lots, @Dennis. Lots. But that's just my subjective impression. – Shog9 Jan 18 '14 at 18:45
  • 10
    At a glance, of the "never came back"s, the close/delete/nothing rate is really high. That doesn't surprise me; I think a lot of us feel like there's a strong correlation between one and done users and low quality questions. – Dennis Meng Jan 18 '14 at 18:48
  • 15
    Do they really never come back, or do they just reappear later under another user123 account (after their cookies get cleared or from another machine in the computer lab or whatever)? – The Community Jan 18 '14 at 18:51
  • 1
    See above comment, @TheCommunity – Shog9 Jan 18 '14 at 19:00
  • 2
    Funny enough, the closed% actually decreases slightly for the "did not return" folks, @Dennis. The close % for new users is pretty high period. The most significant spikes appear to be deleted and "nothing". – Shog9 Jan 18 '14 at 19:03
  • 10
    Is this only for new users asking questions? There are some epic new users who come in and just start answering questions as well. Can you show any analysis or statistics for new users answering questions? – Travis J Jan 18 '14 at 19:05
  • 1
    That would seem like a reasonable definition @shog9. Something that might be interesting to see is if there is some form of "first timer difficulty". I.e. are they generally having a hard time compared to those who have been around longer? How significant is this? How do the percentage differences between your two groups compare with this baseline? Things like that. We've all seen the "SO seems hard for beginners. Is that just me?" questions on Meta. I think it might be nice to be able to state "Actually, overall new users perform similarly to those who've been around for a while" or not. – Bart Jan 18 '14 at 19:06
  • 5
    The Answered / Deleted ratio for returning users is at 3.6:1, while non returning ones are at 2.86:1. In fact, most of these statistics are very similar. Considering that the first set is roughly 2.5 times less than the non returners and adjusting the only real difference is the "nothing" field which is twice as frequent for the non returners. – Travis J Jan 18 '14 at 19:12
  • 3
    Point out the inaccurate assumptions / predictions in this post, @Alien. Or suggest an alternate set of data to look at, or a better way of analyzing it. You know, like the rest of the good folks in this thread have been doing. I posted this because I'm looking for suggestions and input - if you have some of those, don't hold back. – Shog9 Jan 18 '14 at 19:54
  • 2
    @Shog9 I'd be very curious of what the long term trends are like, specifically... how did new users fair a year ago, 2 years ago etc. Are we seeing an increase in downvoted, closed and/or deleted questions. What about users who ended up achieving a "decent" amount of rep. I guess my question is are we really suffering from a problem in terms of declining quality or are the precentages rather stable and it is just the numbers that are increasing? – psubsee2003 Jan 18 '14 at 20:04
  • 1
    Thanks (maybe "PositiveScore"/"NegativeScore" would be more intuitive?). And thanks for sharing in the first place. Interesting data. – Aaron Bertrand Jan 18 '14 at 20:31
  • 3
    Do the "never came back" statistics include accounts that were deleted by moderators after their first post? If so, that would tend to skew some of those numbers a little high due to the inclusion of spammers and trolls. Those aren't legitimate users, so we probably shouldn't count them. – Brad Larson Jan 18 '14 at 21:38
  • 1
    Deleted accounts are hard-deleted, @brad - so they are excluded completely. – Shog9 Jan 18 '14 at 22:52
  • 3
    Originally when I joined the site it was just to get help on one thing in a game I was making for a class. I registered with no intention of really coming back. If you look these stats during the time that I joined it would look like I also never came back, but I did. After getting more into what I was programming and having more challenges I was having to Google a lot of questions. From google I would usually end up on StackOverflow. I became used to googling and visiting only the stack links because that is where I got the answer. I ended up coming back after finding it so useful. – ComputerLocus Jan 22 '14 at 20:11
49

Yes, we do need more pictures, as the raw data in the question is not so easy to see clearly.

Below are some views comparing percentages of the Came back vs. Never came back.

enter image description here

Initial thoughts...

Nothing, Deleted
When no activity occurs, does this indicate a poor question or an
obscure question? Is the question good enough to leave open,
but too niche for anyone to care to delve into?

Edited, Commented, Answered
More participation equals greater chances of return; likely
a two-way street.

Closed
The lack of difference is interesting.

Downvoted/Upvoted
A slight edge for the "Come back", perhaps related somewhat
to the Edited, Commented, Answered,
where greater participation equals more chances of return.

  • In the red/green chart, which is which? – Sergio Jan 19 '14 at 7:56
  • @Sergio Its "Came back - Never". Answered came back: 69%, Answered but never came back: 54%. 69 - 54 = 14. That's the green "14.79%" in the chart. – user213963 Jan 19 '14 at 8:41
  • @mg1075 you should switch the order of "Came Back" and "Never Came Back" on your second chart because its confusing. It seems like "Came Back" should be red because it is on the red side of the line, and "Never Came Back" should be green for the same reason. – WolfLink Jan 20 '14 at 9:04
  • I think that it's totally logical that 'Nothing' is the reaction most likely to make people never come back: (in the same vein as 'hating is better than ignoring'...) – Benjol Jan 24 '14 at 6:46
16

Pictures! We need pictures and pie charts!

Raw nmbers

Percentages

Please ignore the blue guide lines for sizing the image. And the lack of pie. Thank you.

  • 6
    Looking at the percent, the biggest points of interest for me is that the ones that got an answer were more likely to come back, and the ones that had nothing (whatever that entails) were less likely to come back. The overall seems to be the amount of engagement we have with the person (answering, closing, commenting, editing), the more likely they are to return... which makes some sense. – user213963 Jan 18 '14 at 21:12
  • 11
    Just a minor tip, you usually don't want to connect those points, since there is no relation between the individual items on the x-axis. Connecting them makes it appear like there's a progression from one item to the other, while in fact they're unrelated and categorical. – slhck Jan 19 '14 at 10:47
14

It's encouraging to see that answering someone's question is the best thing we can do to get them to come back to the site again. This shouldn't come as a surprise to anyone, it's just nice that this is working. As mg1075 mentioned, this is a two-way street. People will likely come back if we answer their bad/off-topic questions too.

It's also not surprising that doing nothing and deleting someone's first question are the most effective ways to influence people to not come back. I'm concerned about the "do nothing" category, as many of those are likely decent questions that are just hard to answer. I'm not as worried about the "deleted first question" group.

The stat that puzzles me the most is that closing a first question seems to have negligible impact on whether someone comes back or not (19% vs. 18%). Are you only looking at the current open/closed state of first questions? If so, we might be mis-categorizing some questions that get closed, edited by the OP, then reopened. I'd like to see how many times that happened in the timeframe of the data you collected to see if closing questions is having a bigger effect.

One other thing I'd like to see is what effect each community action on first questions has on people coming back and contributing more than just new questions. Can we get the same kind of data for how many users came back and gave at least one upvoted answer after their first question was upvoted, downvoted, answered, closed, etc.?

  • 3
    I'm currently looking at the history of the post to determine if it has ever been closed; Bill Woodger had the idea of limiting this further to events occurring prior to their last visit, which I might try. Including re-open events is a good idea as well. I think the best course here would be to break out first posts with a close event with all else being equal: all answered (or unanswered), none deleted, etc. – Shog9 Jan 19 '14 at 16:25
  • It's uncommon for closed posts to be reopened. – Raedwald May 1 '18 at 6:57
7

With such raw data, graphical representations are premature. Some result has to be pretty substantial, and be an expected result, to be likely to stand up through further analysis. The 69% for answers given may be such a thing. It sounds right. However, 54% of those marked as "never came back" (NCB) also got answers, and so far did not come back.

At least two questions might be looked at: what happens to new users (or, more pointedly, why do some never come back)?; how do we, as a community, treat new users?

The second question can be considered because it likely has an effect on the outcomes for new users. Once bitten, twice shy. Perhaps we seem too biting? How to turn "Perhaps" into some meaningful knowledge?

Useless questions are useless. However, a user asking a useless question first up, isn't necessarily going to ask a useless question in the future. If their experience of asking their first question, which was important to them, but not the sort of question that fits on SO, is so bad that they never want to hear of SO again, then perhaps something is going wrong. I know that a lot has been put into the site to avoid being overtly "you are rubbish", but analysis of the data may reveal further areas where things need to be considered in that light.

Back to the data for now. There is more than one type of new user. Not all users just come to ask a question. Some are keen to answer. Some are keen to learn, and contribute where they can through edits, comments, and voting, before they feel confident enough to answer something. Some just come to learn, look, or something, without further contribution. Some of any of these groups may later ask questions.

For those who ask first up, there are two simple situations: after their first question, they ask another at some point; after their first question they don't ask a question - within the time-period being monitored.

That "time-period being monitored" is an obvious problem. Going back a year, or two years, may show a different sort of profile for the NCBs, although care has to be shown to consider whether SO was sufficiently the same a year, or two years ago, to attempt to understand any differences/similarities (big changes to the Closing last summer, for instance).

Then the simple situations can be a little more complex. Some of those "never came backs" may still be visiting the site, even though they have not asked another question. Even developers with experience sometimes ask questions, but less likely twice in a two-month period. Are some of the NCBs still visiting the site, and how many? What are they doing? Answering, commenting, editing, researching or just otherwise here from time-to-time?

A definitive definition of NCB will not be easy, but maybe someone who has visited the site more than seven days after the last activity on their question which was the last thing when the question was active (some formula to decide when community activity on a question stopped, which is not undone by someone providing a new answer or comment or vote three weeks later).

So, an NCB would be someone who has not asked a subsequent question and has at least not visited the site after activity on their question died down.

New users end up in one of three categories, CB, NCB and Others (who make the totals balance, even if too small in number to be worth analysing further - assuming the numbers are relatively small).

Then the "smack in the face" VS "encouragement" have to be broken out from the data.

The first are Downvotes, non-positive comments (yes, this is subjective, but it is the way the new user understands the comments which will affect how they feel about it), Closing and Deleting.

The second are Answered, Upvotes, neutral/positive comments, Editing.

"Nothing" doesn't really fit either group, until we know what "Nothing" is - which is going to be difficult without an initial manual reviewing of a sample of them. Results of analysis dictate when/if/priority of this being done.

So, CBs and NCBs broken-down by (net) numbers of Downvotes; another, higher priority, sampling for Comments; analysis of speed of Closing/Delete; number with net zero and actual zero question votes; number of net Upvotes; answered, no of answers per question, voting on answers (count, broad indicator of quality answer if at least n votes), number of acceptances, correlation between acceptance and highest-voted answer; number and type of edit (tags, title, body).

We can expect that Voting has an impact. We can expect that Comments have an impact, but not easy to evaluate up front. Closing/Deleting seems to be a candidate, and is open to some automated analysis. Edits may not affect things much, as an expectation, but would tend to be positive.

From the user, their own edits, Accepts and comments may indicate that they have an engagement. If many CBs take some of this action, it may be expected that some proportion of NCBs who act in a similar way on their initial question may still ask a second question, but it just hasn't come up yet.

OK, knock the above into shape, drop out the rubbish, include anything else rational, prioritise, estimate, and I want something substantial on my desk by this time next Monday.

Actually, you might want to explain a model for new users first. If I had mailed a leaflet to these 73,000 people, then 20/73 is a fantastic conversion rate. If I had made one sale to 73,000 people, I'd be very disappointed if I made a second sale to only 20/73 of them. So, the model is neither of those, but is there one?

Ask a good question, reply to comments, comment/vote on answers, accept what helped with the problem, continue using the site, asking, answering, editing, commenting... it is rarely going to happen with a first-time question from someone with little experience. Someone with little experience will have other questions to ask. They should be coming back. Next time their question should be better. Most of those 53,000 should be a source of future questions.

Just lastly, a breakdown by tags may be interesting. If Downvotes are a big influence, I expect that in some tags there is a greater tendency for over-the-top Downvotes than in others.

  • 1
    Your point about time is a good one. There was 8 months between me asking my first question and doing anything else on the site at all and almost 11 months before I posted a second time. – ben is uǝq backwards Jan 20 '14 at 12:42
6

Here it goes one suggestion for data analysis:

Introduction

Let's look for statistic percentage of answers, which had the highest difference between groups.

In the below chart we can see that despite newcomers who came back got more answers (proportionally) than who never came back, the ratio upvoted_questions/answered_threads was smaller for the former.

enter image description here

That means regardless the amount of answers newcomers have got by each group, the quality of questions probably was very similar for both of them.

This is reasonable to assume because (generally) they:

  • do not have StackExchange experience,
  • do not master the subject which is being asked.

By the action of coming back, they are more providing details to make the question answerable rather than improving substantially its content.

Thesis Statement

Starting from this premise that newcomers questions do not differ so much in quality among groups why not:

Searching for quality embedded on answers.

Proposal for data analysis

What are the average and variance of top answer's score in those newcomers threads?

Hypothesis

I would expect higher or similar average score on answers from newcomers who come back than from the ones who never come back. I would also expect lower score variance for answers in threads from people who come back.

If this is true, then it is good that newcomers come back!!

  • "came back" doesn't mean edited the question, it means asked another question later – Kate Gregory Jan 19 '14 at 0:46
  • 2
    @Kate, I know..Just having an active profile could be enough for people answering and relying on receiving OP's feedback. But in meantime for first/second question (or even after them) they can edit and comment too (who never came back don't). – Andre Silva Jan 19 '14 at 0:52
  • "Starting from this premise that newcomers questions do not differ so much in quality among groups..." I don't think this is a reasonable premise. I think the data is telling us that we're creating the two groups by answering the good first questions and ignoring/deleting the bad ones. – Bill the Lizard Jan 19 '14 at 13:49
  • @Bill, 1- Deleted and Closed (%) statistics are the same for both groups, such as Upvoted threads; 2- The group who comes back differentiates from the other (I guess), but my hypothesis is that answers will show this better. – Andre Silva Jan 19 '14 at 14:46
5

Some people commented on the surprising lack of impact of the question closure. I conjecture this is because the table does not separate duplicates from other closure reasons.

Having one's question closed as a duplicate is (or should be) nearly equivalent to having it answered - the answer is just given in another place. This ought to encourage the user to come back; at any rate, being pointed to a duplicate is better than getting nothing. Other closure reasons probably have a discouraging effect, and the two effects cancel each other in the "closed" category.

3

The main outcome from these statistics is that it is relatively irrelevant what we do to the question as to whether users "come back".

There are the expected flows towards any feedback increasing the likelihood of return use and no feedback decreasing it, but they are barely or not significant.

In general: need more data and alternate ways to break it down.

Firstly, I'd like the above statistics broken down into 4 groups, where the extra distinction is whether they came back to at least mark an answer to their first question as accepted.

3

I would caution against reading too much into the statistics presented (so far). Among other things, reading even correlative value (or lack thereof) into the percentages as posted would be incorrect; given the directions of potential effect is " -> ", the percentages should be reversed: % of users who come back whose questions were closed out of all users whose first questions were closed, etc. It is the same information, and in some ways isn't really different, but it's the right way to look at it, and it makes it easier to see relative relationships. (FYI, this is like MPG for cars - that's the wrong way to see it, GPM or some version thereof is far superior, since that's what you care about - how many gallons does it take you to get [somewhere].)

For example, 28% of people whose first question was closed came back to the site. Comparatively, 22% of people whose first question was deleted came back, and 33% whose first question was answered came back and 30% whose question was upvoted - and 29% whose question was downvoted(!).

Answering certainly seems to have some small benefit, and deletion has a pretty big detriment.

Nothing, however, is the biggest, at only 15% return rate. Basically, as long as something happens to someone's question, they come back at a normal rate - which tells you that the user needs to feel involved in some way, which isn't surprising to students of retail or brand management.

In terms of future analysis, it would be interesting to see a survival analysis performed. This is the kind of analysis done to see which cancer treatment works best; but it also works well for customer retention.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .