What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 134 Stack Exchange communities.

A few days ago I had a post that provided data scraped from the profile pages of over 23,000 StackOverflow users with the intention of having the community mine the database and produce interesting graphs and statistics.

Unfortunately, due to privacy concerns, the database and scraping script are no longer available. Instead, I will attempt to make these graphs myself. Ideas for future graphs are welcome - the database contains the following schema:

sqlite> .schema
CREATE TABLE profile ( 
    uid INTEGER PRIMARY KEY, name, rep INTEGER, registered, joined INTEGER, lastseen INTEGER, website,
    location, age INTEGER, bio, questions INTEGER, answers INTEGER, votes INTEGER, tags INTEGER,
    badges INTEGER);

joined and lastseen are UNIX timestamps. lastseen is accurate to the minute, but joined is only accurate to the day. registered can have one of three values:

  • Registered User
  • Unregistered User
  • Moderator

Some Data:

sqlite> select age, count(age), sum(rep), avg(rep) from profile group by age;
-7992|2|808|404.0
8|15|7193|479.533333333333
9|3|153|51.0
11|1|1|1.0
12|1|405|405.0
14|1|1|1.0
15|4|4|1.0
16|17|4092|240.705882352941
17|16|4766|297.875
18|36|8131|225.861111111111
19|74|11224|151.675675675676
20|98|19933|203.397959183673
21|141|39768|282.042553191489
22|230|44023|191.404347826087
23|295|70772|239.905084745763
24|403|102279|253.794044665012
25|404|90655|224.393564356436
26|457|119718|261.964989059081
27|443|122917|277.465011286682
28|441|99973|226.696145124717
29|372|79790|214.489247311828
30|385|94075|244.350649350649
31|360|83887|233.019444444444
32|296|64477|217.827702702703
33|291|55692|191.381443298969
34|261|59625|228.448275862069
35|200|41846|209.23
36|199|45489|228.587939698492
37|194|52054|268.319587628866
38|182|57922|318.252747252747
39|139|27038|194.517985611511
40|124|22128|178.451612903226
41|112|26616|237.642857142857
42|84|9947|118.416666666667
43|56|8309|148.375
44|50|6957|139.14
45|43|10363|241.0
46|42|12846|305.857142857143
47|34|5579|164.088235294118
48|23|1980|86.0869565217391
49|23|2335|101.521739130435
50|16|861|53.8125
51|12|2890|240.833333333333
52|15|2745|183.0
53|4|675|168.75
54|12|2230|185.833333333333
55|6|880|146.666666666667
56|4|409|102.25
57|1|9|9.0
58|5|159|31.8
59|2|44|22.0
60|4|24|6.0
61|4|195|48.75
63|1|581|581.0
65|1|11|11.0
66|1|4315|4315.0
68|1|25|25.0
88|20|6810|340.5
108|1|231|231.0
255|1|843|843.0
|16346|516447|31.5947020677842

Data:

Registered Users: 86.9% (19998/23001)
Unregistered Users: 13.1% (3007/23001)
Moderators: 0.017% (4/23001)

Graphs:

Scatter plot of all ages and all rep scores: [note: yes, I do know I misspelled "stack". I'll see if I can update this soon]

alt text

'Hockey Stick' graph of reputation vs reputation rank:

alt text

Same 'hockey stick' graph, focused on the top 1000 users:

alt text

Any further requests for data or graphs are welcome, and feel free to use the sample data above to create your own!

share

migration rejected from stackoverflow.com Dec 16 at 12:31

This question came from our site for professional and enthusiast programmers. Votes, comments, and answers are locked due to the question being closed here, but it may be eligible for editing and reopening on the site where it originated.

closed as off-topic by Martijn Pieters, Shadow Wizard, Aziz Shaikh, nicael, ProgramFOX Dec 16 at 12:31

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question pertains only to a specific site in the Stack Exchange Network. Questions on Meta Stack Exchange should pertain to our network or software that drives it as a whole, within the guidelines defined in the help center. You should ask this question on the meta site where your concern originated." – Martijn Pieters, Shadow Wizard, Aziz Shaikh, nicael, ProgramFOX
If this question can be reworded to fit the rules in the help center, please edit the question.

4 Answers 4

Seriously, privacy concerns? I'm assuming it wasn't you who decided this was a problem. How can there be privacy concerns with data that is fully publicly available on the web site? It's not like you were publishing data that you obtained through some means other than, you know, browsing.

share
    
Even though the data is available, the fact that it is aggregated and easily searcheable raises the risk. Unfortunatley it often only takes one complainer to get rid of a useful utility for everyone else, and chances are the complainer had 'what if' scenarios, instead of a real, good reason. –  Adam Davis Oct 2 '08 at 18:29
    
My thoughts exactly. You'd have to ask Joel - if he (or Jeff) say it's OK I'll post my database and scraper again; otherwise, I'll just take requests. –  Kyle Cronin Oct 2 '08 at 18:31
    
People who have real concerns should be hiding their true identity anyways. There's no background check with polygraph to get an OpenID. Use Thomas Pain as your pseudonym if that's what you're going to be anyways. –  Mark Brady Oct 3 '08 at 17:16
    
humans aren't like computers, and the 'what is possible' is a lot less relevant than the 'what is'. –  Dustin Getz Nov 29 '08 at 21:12

This graph (below) you did would have been more useful without the 1 who has a too high value compared to other. The graph is almost blank because of 1 person. Scale of the graph needed to be changed or remove few "noise user". But well, good job. alt text

share
    
I'll look into it, but as you can see with the top 1000 graph, it's more than just one individual. In fact, that exponential spike is the hallmark of the hockey stick graph. –  Kyle Cronin Oct 2 '08 at 18:32
    
Indeed, that's kind of the point. As expected with phenomena having to do with ranks, reputation, popularity and the like, they follow power-law, expoenential, or pareto-like distributions. –  Gregg Lind Oct 10 '08 at 21:16
    
Logarithmic y axis? –  MarkJ Mar 28 '09 at 21:25

replacing the *.bmp with *.png/gif/jpeg would be great :-)

share
    
As would updating the script and rerunning it. I plan to do something with this this weekend, but I don't have an easy way of making graphs. I'll work something out though. –  Kyle Cronin Dec 5 '08 at 22:15

I've computed some new statistics on my site. We need a "meta" corner for things like that, which aren't really questions.

share

Not the answer you're looking for? Browse other questions tagged .