6

I am trying to download the Stack Exchange data dump from archive.org through the command line using the internetarchive python package developed by archive.org, as follows:

pip install internetarchive
ia download stackexchange

but I get the errors error message. I tried a few variants and they have the same error message:

user@server:~$ ia download stackexchange
stackexchange: eedeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeddddeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee - errors
user@server:~$
user@server:~$ ia download  stackexchange --glob="*.7z"
stackexchange: eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee - errors
user@server:~$
user@server:~$ ia download stackexchange 3dprinting.meta.stackexchange.com.7z
stackexchange: e - errors

Using wget didn't work either:

user@server1:~$ wget https://archive.org/download/stackexchange/3dprinting.stackexchange.com.7z
--2018-02-03 17:36:05--  https://archive.org/download/stackexchange/3dprinting.stackexchange.com.7z
Resolving archive.org (archive.org)... 207.241.224.2
Connecting to archive.org (archive.org)|207.241.224.2|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://ia800801.us.archive.org/23/items/stackexchange/3dprinting.stackexchange.com.7z [following]
--2018-02-03 17:36:06--  https://ia800801.us.archive.org/23/items/stackexchange/3dprinting.stackexchange.com.7z
Resolving ia800801.us.archive.org (ia800801.us.archive.org)... 207.241.230.81
Connecting to ia800801.us.archive.org (ia800801.us.archive.org)|207.241.230.81|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2018-02-03 17:36:06 ERROR 403: Forbidden.

How can I download the Stack Exchange data dump from archive.org through the command line?

I don't want to download through torrent.

  • 1
    It is probably something to do with the recent change of login being required in order to download the SE dumps. Did you configure that package with a username and password? – animuson Feb 3 '18 at 22:49
  • @animuson that worked, thanks! – Franck Dernoncourt Feb 3 '18 at 22:55
  • 1
    @animuson Why was this changed? – Sonic the Anonymous Hedgehog Feb 3 '18 at 22:57
  • 4
    @Sonic Internet Archive classified it as high bandwidth sometime at the beginning of the year, and subsequently threw a login requirement on it at the same time. Don't believe we had any say in the decision. – animuson Feb 3 '18 at 23:01
6

As animuson mentioned in the comment, one has to login to be able to download the Stack Exchange dump. This can be done using ia configure:

user@server:~$ ia configure
Enter your Archive.org credentials below to configure 'ia'.

Email address: franck@gmail.com
Password:

Config saved to: /home/ubuntu/.config/ia.ini

I have opened an issue on the internetarchive python package: Improve error message when user needs to login #234

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .