4

I'm a 10k user on Math.SE. As a human using my browser, I can access deleted questions and view them, just like non-deleted questions. I'd like to use this access to collect data for a personal project of predicting some kind of "question quality" based on my voting record. I have retrieved my voting record using a Python script, but accessing the content of questions I've voted on that were later deleted is proving tricky.

My 10k privileges obviously don't extend to Data Explorer access, and using the requests module in Python is (for some reason) locking me out after a couple dozen page requests.

Is there a good way to access the content and metadata of deleted questions on a site where I'm a 10k user?

  • 3
    "using the requests module is... locking me out" - how frequent are your requests? Human-level frequent, or bot-level frequent where you're requesting a new page every couple milliseconds? The latter will get you locked out quickly; the former should not. – ArtOfCode Oct 5 '16 at 19:33
  • @ArtOfCode I'm going at human-level speeds (probably like 1 request a second), but the problem is that my login cookie stops working after a few dozen requests. If I log in again, I can get a new cookie and that works, but since I have several thousand posts I'd like to request, I don't really want to keep manually logging in to get the cookie. (Doing Google login to StackExchange and getting the resulting cookie is a little too advanced for me, so I end up logging in on my browser and copying the cookie manually into my program.) – apnorton Oct 5 '16 at 20:55
  • 1 request/second is not human speed. Humans don't make that many sustained requests. They might send a few that close together, but for the most part there are tens of seconds if not minutes between requests. – ArtOfCode Oct 5 '16 at 20:57
  • You can download a Stack Exchange Data Dump. It has the same data as Data Explorer, but you can process it on your own machine, without any limits. – Michał Perłakowski Oct 5 '16 at 21:39
  • @ArtOfCode Ok, I'll try lowering the rate at which I request pages and see if that fixes anything – apnorton Oct 6 '16 at 4:26
  • @Gothdo That's true, but I can't access deleted question content in the dump. I'll probably end up querying the data dump for questions that aren't yet deleted, but as it stands I want info that's not in the dump. – apnorton Oct 6 '16 at 4:27
  • In my tests here I learned that you need more then just the auth cookie, specially when you move cookies between browsers. I didn't figure out yet what more that might be but I expect one of the other long standing cookies to be involved – rene Oct 6 '16 at 19:03
  • @rene Hmm... ok. I might have to figure out a different way to approach this, then. – apnorton Oct 7 '16 at 1:56

You must log in to answer this question.

Browse other questions tagged .