7

I would like to have a working offline version of all questions I worked on, including questions I have commented on, edited, voted up,...

I know, you can export your data from each Stack Exchange site, using http://data.stackexchange.com

But how can you mirror all pages on all stacks where you were active?

I tried using wget -m on my Stack Exchange profile site in the "activity" view, but that didn't work correctly.

It would be perfect, if the export were still working html sites that are linked between each other whenever possible. Also the side bar with "related questions" should be working.

  • Looks like you got a mess of questions here. Please focus on one request and elaborate what exactly you want to be done. – Shadow The Curly Braced Wizard Sep 7 '15 at 6:33
  • I added a summary – rubo77 Sep 7 '15 at 6:37
  • 1
    An offline mess of questions is still a mess of questions – user289879 Sep 7 '15 at 6:44
  • It shouldn't be a mess. If i have a local folder I could search through it for keywords. and I would have a backup of all the work I have done, which is a lot of important documentation I accumulated here already – rubo77 Sep 7 '15 at 9:52
  • Also deleted questions would be an interesting point. – Qwertiy Mar 22 '17 at 17:25
4

You can get the posts you have asked, answered or commented on through queries in the data explorer, no need for wget. That would make retrieving the side bar not possible, which isn't useful in my opinion.

Some queries:

Of course you can still get the HTML through the post IDs.

  • Interesting, but it returns only 494 results for me, and I have.... way many more. – Shadow The Curly Braced Wizard Sep 7 '15 at 6:50
  • On which of the query? In fact, the queries are just samples of the possibilities of SEDE. OP can change the queries according to his wishes. – Patrick Hofman Sep 7 '15 at 6:51
  • 1
    Oops, first one - thought it's for all activity in one place. So all is good. :) – Shadow The Curly Braced Wizard Sep 7 '15 at 6:53
  • This is not enough. I would need the whole question with all answers, when I added comment there or just did anything in that question – rubo77 Sep 7 '15 at 9:53
  • So you can combine all together in one right? – Patrick Hofman Sep 7 '15 at 9:58
  • but then I wouldn't have the other questions and answers on questions where I provided just one answer. I want to have backupped the whole HTML page like it is visible in Firefox for all activities I ever did. This might get large, but it is worth it – rubo77 Sep 7 '15 at 14:40
  • I think a solution to this is still needed. also see meta.stackexchange.com/questions/158546/… – rubo77 Sep 28 '15 at 19:50
  • I would love to supply you with a better solution by I haven't one. I am sorry. @rubo77 – Patrick Hofman Sep 28 '15 at 19:57
  • What about deleted items? – Qwertiy Mar 22 '17 at 17:31
2

A solution to get at least all first 10 pages of questions and answers as browseable html pages:

#/bin/bash

# your User and Stackexchange ID from your "Network Profile" page
SE_ID=1047481
U=rubo77

mkdir -p stackexchange
cd stackexchange

# get all user ids across SE sites
users=$(curl https://stackexchange.com/users/$SE_ID/$U?tab=accounts|grep account-container -A1|grep http|cut -d"/" -f3,5|xargs)
for i in $users; do
  site=$(echo $i|sed "s~/.*$~~g")
  id=$(echo $i|sed "s~^.*/~~g")
  for a in questions answers; do
    for i in $(seq 10|xargs); do
      URL="http://$site/users/$id/$U?tab=$a&sort=newest&page=$i"
      echo "################### $URL #######################"
      wget -c -r -l 1 --wait 0.1 --random-wait -e robots=off -p -k -H "$URL"
      # sleep timer has to be added here to prevent a rate limit trigger of SE
      sleep 15
    done
  done
done

explain wget options

This solution works quite well, but it triggers the rate limiter on stackoverflow after about 80MB of sites downloaded:

Too many requests

So we have to add a sleep timer between downloads. What would be a good value, that is accepted by SE?

  • In general when scraping pages on SE I go with 5 second intervals and it seems to treat me well, most of the time. – Jason C Mar 27 '17 at 3:50
  • a sleep timer after each wget call wasn't enough, I added ` --wait 0.1 --random-wait` to the wget call so it waits randomly 0.05 to 0.15 seconds between each download call in the recursive scraping. (This makes it quite slow, because also the page requisites are downloaded with that delay) – rubo77 Mar 27 '17 at 13:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .