2

Is it possible to obtain the individual user reputations dataset (users and reputation at a given time) from Stack Overflow for example? Aim: modeling purposes.

Improve this question
11
  • 1
    If you want to look at all users, you'll need to download the Data Dump (45GB of XML), or perform your query on the Data Explorer (a tool for running SQL queries against the data set online, though with a timeout that may be prohibitive depending on what you want to do). The API also allows you to gather information about users (also subject to rate limits).
    – Jeremy
    Mar 28, 2017 at 23:39
  • 2
    Can you clarify "dataset"? Do you just mean a list of users and their reputation? Or do you mean the full details as in this? The former can be obtained through SEDE or the API like Jeremy mentioned. The latter is not available to us common folk, you can reconstruct some of the info from SEDE but not all. If you're looking for rep changes over time for a specific user I suppose you could scrape stackexchange.com/users/USER_ID?tab=reputation
    – Jason C
    Mar 29, 2017 at 0:09
  • I mean the info containing just the users and the reputation at a given time.
    – Edgar Santos
    Mar 29, 2017 at 0:47
  • @JasonC The data SE gives back at most 50000 rows for a query.
    – peterh
    Mar 29, 2017 at 2:37
  • @user54652 Then all of the methods Jeremy listed will work. Note that there are 6892675 users on SO, but SEDE queries are limited to 50000 results, so you'd want to either use OFFSET/FETCH to page the results (tedious) or come up with some limiting criteria to keep the result set small. You might even be able to do a lot of the work of your analysis itself in a query. For the API it's easier to automate, subject to rate and quota limits. For the downloadable dumps you can do whatever, you'll have to parse XML.
    – Jason C
    Mar 29, 2017 at 2:40
  • @peterh Yes that is correct. You can still use SEDE to get it all. Just not in one shot.
    – Jason C
    Mar 29, 2017 at 2:40
  • @JasonC Right :-)
    – peterh
    Mar 29, 2017 at 2:43
  • you can't get reputation at a given time, only current rep from a single datasource (be it SEDE or the datadump) so you have to get all datadumps and then buildup the timeseries from that. If you only need a few columns from a table but want to fetch more then 50,000 rows have a look at this trick
    – rene
    Mar 29, 2017 at 7:10
  • I think that 25k reputation is required to access that sort of data.
    – John Militer
    Mar 29, 2017 at 18:16
  • @John The 25k "site analytics" privilege you're referring to contains some aggregate statistics that are not available anywhere else, but doesn't provide any large data sets like are being requested here.
    – Jeremy
    Mar 29, 2017 at 19:29
  • I forgot the other option for querying the data without downloading it: the Data Dump data set is also available in Google BigQuery public datasets. However, this requires a Google Cloud account and may const money if you're not in the free tier.
    – Jeremy
    Mar 29, 2017 at 19:31

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .