6

I'm a 10k user on Math.SE. As a human using my browser, I can access deleted questions and view them, just like non-deleted questions. I'd like to use this access to collect data for a personal project of predicting some kind of "question quality" based on my voting record. I have retrieved my voting record using a Python script, but accessing the content of questions I've voted on that were later deleted is proving tricky.

My 10k privileges obviously don't extend to Data Explorer access, and using the requests module in Python is (for some reason) locking me out after a couple dozen page requests.

Is there a good way to access the content and metadata of deleted questions on a site where I'm a 10k user?

9
  • 3
    "using the requests module is... locking me out" - how frequent are your requests? Human-level frequent, or bot-level frequent where you're requesting a new page every couple milliseconds? The latter will get you locked out quickly; the former should not.
    – ArtOfCode
    Oct 5, 2016 at 19:33
  • @ArtOfCode I'm going at human-level speeds (probably like 1 request a second), but the problem is that my login cookie stops working after a few dozen requests. If I log in again, I can get a new cookie and that works, but since I have several thousand posts I'd like to request, I don't really want to keep manually logging in to get the cookie. (Doing Google login to StackExchange and getting the resulting cookie is a little too advanced for me, so I end up logging in on my browser and copying the cookie manually into my program.)
    – apnorton
    Oct 5, 2016 at 20:55
  • 1
    1 request/second is not human speed. Humans don't make that many sustained requests. They might send a few that close together, but for the most part there are tens of seconds if not minutes between requests.
    – ArtOfCode
    Oct 5, 2016 at 20:57
  • 1
    You can download a Stack Exchange Data Dump. It has the same data as Data Explorer, but you can process it on your own machine, without any limits. Oct 5, 2016 at 21:39
  • @ArtOfCode Ok, I'll try lowering the rate at which I request pages and see if that fixes anything
    – apnorton
    Oct 6, 2016 at 4:26
  • @Gothdo That's true, but I can't access deleted question content in the dump. I'll probably end up querying the data dump for questions that aren't yet deleted, but as it stands I want info that's not in the dump.
    – apnorton
    Oct 6, 2016 at 4:27
  • 1
    In my tests here I learned that you need more then just the auth cookie, specially when you move cookies between browsers. I didn't figure out yet what more that might be but I expect one of the other long standing cookies to be involved
    – rene
    Oct 6, 2016 at 19:03
  • @rene Hmm... ok. I might have to figure out a different way to approach this, then.
    – apnorton
    Oct 7, 2016 at 1:56

0

You must log in to answer this question.

Browse other questions tagged .