4

I have a set of 141 user IDs on the math.stackexchange site, and I would like to pull information about these users' reputation histories through the Stack Exchange API. I have set the parameters for pagesize as 100, which I believe is the maximum, and have partitioned the set of user IDs several different groups (6) to be certain that the text does not exceed the page limit.

However, the program only returns reputation histories for a set of 63 unique user IDs. Briefly going through the user profiles, I have reason to believe that the program is missing some reputation events. Also, with one particular account that has around 200 reputation, I have only noticed two reputation events returned when there should be more. I don't think these reputation events would be private, so why am I encountering this problem and how would I go about fixing it?

1
  • Why would one need this? Nov 8 '20 at 8:22
5

The API uses Paging to prevent that to much data needs to be sent. When you make a call to any of the API end-points you start at page 1. So for my reputation history the first call (for page 1, the default) is this:

/2.2/users/578411/reputation-history?site=stackoverflow

which returns this:

{
    "items" : [{
            "reputation_history_type" : "post_upvoted",
            "reputation_change" : 10,
            "post_id" : 20923631,
            "creation_date" : 1469610299,
            "user_id" : 578411
        }, /*many more */{
            "reputation_history_type" : "answer_unaccepted",
            "reputation_change" : -15,
            "post_id" : 24241649,
            "creation_date" : 1468394533,
            "user_id" : 578411
        }
    ],
    "has_more" : true,
    "quota_max" : 300,
    "quota_remaining" : 291
}

As you can see has_more has a value of true which means I can make another call for the next page. So we add the page parameter to our call

/2.2/users/578411/reputation-history?site=stackoverflow&page=2

This will return:

{
    "items" : [{
            "reputation_history_type" : "answer_accepted",
            "reputation_change" : 15,
            "post_id" : 24241649,
            "creation_date" : 1468394518,
            "user_id" : 578411
        }, /* many more */{
            "reputation_history_type" : "post_upvoted",
            "reputation_change" : 10,
            "post_id" : 38208136,
            "creation_date" : 1467735361,
            "user_id" : 578411
        }
    ],
    "has_more" : true,
    "quota_max" : 300,
    "quota_remaining" : 290
}

As it turns out as of today I can continue till page 88 until has_morereturns false.

It is a bit of an hassle to do the above by hand but it is not hard to script. Don't forget to check for the backoff parameter and honor the number of seconds returned there.

Here is a snippet that will apply above mentioned method:

(function () {
  var start,
      result,
      userid = 578411; // which userid to use 
  
  // this is called when you click the start button 
  function reputationHistory() {
    
    // this is called for each page
    function getApi(page) {
      // ajax calls
      var xhr = new XMLHttpRequest();
      
      // this is called when the API returns its data
      function loadComplete() {
        // make a javascript object out of the response
        var data = JSON.parse(xhr.responseText),
            backoff,
            i,
            event,
            item;
        // process results, by going over each elemet in items
        for(i = 0; i < data.items.length; i = i + 1) {
          item = data.items[i];
          // to show something, create a new div in the html
          event = document.createElement('div');
          // merge some of the fields in a string
          event.textContent = item.post_id + '; ' + item.reputation_change;
          // show it
          result.appendChild(event);
        }
        
        // honor backoff if present, if not you'll get a 503 error
        backoff =  data.backoff || 0;
        // if has_more is true there more results
        if (data.has_more) {
          // fetch the next page by calling getApi 
          // again with an higher pagenumber
          // use setTimeout to wait a bit
          // notice we call getApi here again 
          setTimeout(function () { getApi(page + 1); }, backoff * 1000);
        } else {
          // let the user know we are done
          event = document.createElement('div');
          event.textContent  = 'done';
          result.appendChild(event);
        }
      }
      
      // make sure something will happen when data is returned
      xhr.addEventListener("load", loadComplete);
      // notice the page parameter here
      // here the actual api call is created
      xhr.open(
        'GET', 
        'https://api.stackexchange.com/2.2/users/' + 
        userid + 
        '/reputation-history?site=stackoverflow&page=' + 
        page);
      // put this API call in motion
      xhr.send();
    }
    
    // this is our first call
    // which starts at page 1, hence the 1 here
    getApi(1);
  }
  
  //  this is called when the user is about to see something
  // and might click on a  button
  function init() {
    console.log('init')
    start = document.getElementById('start');
    result = document.getElementById('result');
    // click the button will put everything in motion
    start.addEventListener('click', reputationHistory);
  }
  // make sure to call init() when loading is done.
  window.onload = init();
})()
<button id="start">Start</button>
<div id="result">
</div>

2
  • Forgive me for not entirely understanding your meticulously written post. So, is it okay to still use the api.stackexchange to obtain the information on reputation history? In other words, you are saying that I should be partition the set of user IDs such that 'has_more' returns false?
    – Sarah
    Jul 26 '16 at 20:05
  • When you call any of the api endpoints and in items there are more elements available then will fit based on the pagesize the has_more property will be true. This means that you can make another call for the next page (page +1) with the same parameters to get a new result with items. You keep changing the page until has_more returns false.
    – rene
    Jul 26 '16 at 20:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .