What happens to new users?

Question

I've been looking at what happens when new users post their first questions on Stack Overflow... Not really done yet, but I figured I'd toss what I have up here in case anyone else has useful suggestions.

For window of time between 30 and 90 days ago, I looked at users who posted their first question within this window and broke them into two groups: those who returned and posted at least one more question, and those who never posted anything else again (or at least, haven't yet). I then collected some data on what sorts of things happened to their first post...

New users who came back after their first question was...

Total Upvoted Downvoted Answered Closed Deleted Commented Edited Nothing
----- ------- --------- -------- ------ ------- --------- ------ -------
20227 4520    5080      13979    3883   2899    13651     8962   891    

Percentage of this group's total:
      23%     25%       69%      19%    14%     67%       44%    4%

New users who never came back after their first question was...

Total Upvoted Downvoted Answered Closed Deleted Commented Edited Nothing
----- ------- --------- -------- ------ ------- --------- ------ -------
52720 10644   12143     28636    9836   10000   31677     19822  5111   

Percentage of this group's total:
      20%     23%       54%      18%    18%     60%       37%    9%

Answer 1

Yes, we do need more pictures, as the raw data in the question is not so easy to see clearly.

Below are some views comparing percentages of the Came back vs. Never came back.

Initial thoughts...

Nothing, Deleted
When no activity occurs, does this indicate a poor question or an
obscure question? Is the question good enough to leave open,
but too niche for anyone to care to delve into?

Edited, Commented, Answered
More participation equals greater chances of return; likely
a two-way street.

Closed
The lack of difference is interesting.

Downvoted/Upvoted
A slight edge for the "Come back", perhaps related somewhat
to the Edited, Commented, Answered,
where greater participation equals more chances of return.

Answer 2

Pictures! We need pictures and pie charts!

_{Please ignore the blue guide lines for sizing the image. And the lack of pie. Thank you.}

Answer 3

It's encouraging to see that answering someone's question is the best thing we can do to get them to come back to the site again. This shouldn't come as a surprise to anyone, it's just nice that this is working. As mg1075 mentioned, this is a two-way street. People will likely come back if we answer their bad/off-topic questions too.

It's also not surprising that doing nothing and deleting someone's first question are the most effective ways to influence people to not come back. I'm concerned about the "do nothing" category, as many of those are likely decent questions that are just hard to answer. I'm not as worried about the "deleted first question" group.

The stat that puzzles me the most is that closing a first question seems to have negligible impact on whether someone comes back or not (19% vs. 18%). Are you only looking at the current open/closed state of first questions? If so, we might be mis-categorizing some questions that get closed, edited by the OP, then reopened. I'd like to see how many times that happened in the timeframe of the data you collected to see if closing questions is having a bigger effect.

One other thing I'd like to see is what effect each community action on first questions has on people coming back and contributing more than just new questions. Can we get the same kind of data for how many users came back and gave at least one upvoted answer after their first question was upvoted, downvoted, answered, closed, etc.?

Answer 4

With such raw data, graphical representations are premature. Some result has to be pretty substantial, and be an expected result, to be likely to stand up through further analysis. The 69% for answers given may be such a thing. It sounds right. However, 54% of those marked as "never came back" (NCB) also got answers, and so far did not come back.

At least two questions might be looked at: what happens to new users (or, more pointedly, why do some never come back)?; how do we, as a community, treat new users?

The second question can be considered because it likely has an effect on the outcomes for new users. Once bitten, twice shy. Perhaps we seem too biting? How to turn "Perhaps" into some meaningful knowledge?

Useless questions are useless. However, a user asking a useless question first up, isn't necessarily going to ask a useless question in the future. If their experience of asking their first question, which was important to them, but not the sort of question that fits on SO, is so bad that they never want to hear of SO again, then perhaps something is going wrong. I know that a lot has been put into the site to avoid being overtly "you are rubbish", but analysis of the data may reveal further areas where things need to be considered in that light.

Back to the data for now. There is more than one type of new user. Not all users just come to ask a question. Some are keen to answer. Some are keen to learn, and contribute where they can through edits, comments, and voting, before they feel confident enough to answer something. Some just come to learn, look, or something, without further contribution. Some of any of these groups may later ask questions.

For those who ask first up, there are two simple situations: after their first question, they ask another at some point; after their first question they don't ask a question - within the time-period being monitored.

That "time-period being monitored" is an obvious problem. Going back a year, or two years, may show a different sort of profile for the NCBs, although care has to be shown to consider whether SO was sufficiently the same a year, or two years ago, to attempt to understand any differences/similarities (big changes to the Closing last summer, for instance).

Then the simple situations can be a little more complex. Some of those "never came backs" may still be visiting the site, even though they have not asked another question. Even developers with experience sometimes ask questions, but less likely twice in a two-month period. Are some of the NCBs still visiting the site, and how many? What are they doing? Answering, commenting, editing, researching or just otherwise here from time-to-time?

A definitive definition of NCB will not be easy, but maybe someone who has visited the site more than seven days after the last activity on their question which was the last thing when the question was active (some formula to decide when community activity on a question stopped, which is not undone by someone providing a new answer or comment or vote three weeks later).

So, an NCB would be someone who has not asked a subsequent question and has at least not visited the site after activity on their question died down.

New users end up in one of three categories, CB, NCB and Others (who make the totals balance, even if too small in number to be worth analysing further - assuming the numbers are relatively small).

Then the "smack in the face" VS "encouragement" have to be broken out from the data.

The first are Downvotes, non-positive comments (yes, this is subjective, but it is the way the new user understands the comments which will affect how they feel about it), Closing and Deleting.

The second are Answered, Upvotes, neutral/positive comments, Editing.

"Nothing" doesn't really fit either group, until we know what "Nothing" is - which is going to be difficult without an initial manual reviewing of a sample of them. Results of analysis dictate when/if/priority of this being done.

So, CBs and NCBs broken-down by (net) numbers of Downvotes; another, higher priority, sampling for Comments; analysis of speed of Closing/Delete; number with net zero and actual zero question votes; number of net Upvotes; answered, no of answers per question, voting on answers (count, broad indicator of quality answer if at least n votes), number of acceptances, correlation between acceptance and highest-voted answer; number and type of edit (tags, title, body).

We can expect that Voting has an impact. We can expect that Comments have an impact, but not easy to evaluate up front. Closing/Deleting seems to be a candidate, and is open to some automated analysis. Edits may not affect things much, as an expectation, but would tend to be positive.

From the user, their own edits, Accepts and comments may indicate that they have an engagement. If many CBs take some of this action, it may be expected that some proportion of NCBs who act in a similar way on their initial question may still ask a second question, but it just hasn't come up yet.

OK, knock the above into shape, drop out the rubbish, include anything else rational, prioritise, estimate, and I want something substantial on my desk by this time next Monday.

Actually, you might want to explain a model for new users first. If I had mailed a leaflet to these 73,000 people, then 20/73 is a fantastic conversion rate. If I had made one sale to 73,000 people, I'd be very disappointed if I made a second sale to only 20/73 of them. So, the model is neither of those, but is there one?

Ask a good question, reply to comments, comment/vote on answers, accept what helped with the problem, continue using the site, asking, answering, editing, commenting... it is rarely going to happen with a first-time question from someone with little experience. Someone with little experience will have other questions to ask. They should be coming back. Next time their question should be better. Most of those 53,000 should be a source of future questions.

Just lastly, a breakdown by tags may be interesting. If Downvotes are a big influence, I expect that in some tags there is a greater tendency for over-the-top Downvotes than in others.

Answer 5

_{Here it goes one suggestion for data analysis:}

Introduction

Let's look for statistic percentage of answers, which had the highest difference between groups.

In the below chart we can see that despite newcomers who came back got more answers (proportionally) than who never came back, the ratio upvoted_questions/answered_threads was smaller for the former.

That means regardless the amount of answers newcomers have got by each group, the quality of questions probably was very similar for both of them.

This is reasonable to assume because (generally) they:

• do not have StackExchange experience,
• do not master the subject which is being asked.

By the action of coming back, they are more providing details to make the question answerable rather than improving substantially its content.

Thesis Statement

Starting from this premise that newcomers questions do not differ so much in quality among groups why not:

Searching for quality embedded on answers.

Proposal for data analysis

What are the average and variance of top answer's score in those newcomers threads?

Hypothesis

I would expect higher or similar average score on answers from newcomers who come back than from the ones who never come back. I would also expect lower score variance for answers in threads from people who come back.

If this is true, then it is good that newcomers come back!!

Answer 6

Some people commented on the surprising lack of impact of the question closure. I conjecture this is because the table does not separate duplicates from other closure reasons.

Having one's question closed as a duplicate is (or should be) nearly equivalent to having it answered - the answer is just given in another place. This ought to encourage the user to come back; at any rate, being pointed to a duplicate is better than getting nothing. Other closure reasons probably have a discouraging effect, and the two effects cancel each other in the "closed" category.

Answer 7

The main outcome from these statistics is that it is relatively irrelevant what we do to the question as to whether users "come back".

There are the expected flows towards any feedback increasing the likelihood of return use and no feedback decreasing it, but they are barely or not significant.

In general: need more data and alternate ways to break it down.

Firstly, I'd like the above statistics broken down into 4 groups, where the extra distinction is whether they came back to at least mark an answer to their first question as accepted.

Answer 8

I would caution against reading too much into the statistics presented (so far). Among other things, reading even correlative value (or lack thereof) into the percentages as posted would be incorrect; given the directions of potential effect is " -> ", the percentages should be reversed: % of users who come back whose questions were closed out of all users whose first questions were closed, etc. It is the same information, and in some ways isn't really different, but it's the right way to look at it, and it makes it easier to see relative relationships. (FYI, this is like MPG for cars - that's the wrong way to see it, GPM or some version thereof is far superior, since that's what you care about - how many gallons does it take you to get [somewhere].)

For example, 28% of people whose first question was closed came back to the site. Comparatively, 22% of people whose first question was deleted came back, and 33% whose first question was answered came back and 30% whose question was upvoted - and 29% whose question was downvoted(!).

Answering certainly seems to have some small benefit, and deletion has a pretty big detriment.

Nothing, however, is the biggest, at only 15% return rate. Basically, as long as something happens to someone's question, they come back at a normal rate - which tells you that the user needs to feel involved in some way, which isn't surprising to students of retail or brand management.

In terms of future analysis, it would be interesting to see a survival analysis performed. This is the kind of analysis done to see which cancer treatment works best; but it also works well for customer retention.