7

I am trying to download the Stack Exchange data dump from archive.org through the command line using the internetarchive python package developed by archive.org, as follows:

pip install internetarchive
ia download stackexchange

but I get the errors error message. I tried a few variants and they have the same error message:

user@server:~$ ia download stackexchange
stackexchange: eedeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeddddeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee - errors
user@server:~$
user@server:~$ ia download  stackexchange --glob="*.7z"
stackexchange: eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee - errors
user@server:~$
user@server:~$ ia download stackexchange 3dprinting.meta.stackexchange.com.7z
stackexchange: e - errors

Using wget didn't work either:

user@server1:~$ wget https://archive.org/download/stackexchange/3dprinting.stackexchange.com.7z
--2018-02-03 17:36:05--  https://archive.org/download/stackexchange/3dprinting.stackexchange.com.7z
Resolving archive.org (archive.org)... 207.241.224.2
Connecting to archive.org (archive.org)|207.241.224.2|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://ia800801.us.archive.org/23/items/stackexchange/3dprinting.stackexchange.com.7z [following]
--2018-02-03 17:36:06--  https://ia800801.us.archive.org/23/items/stackexchange/3dprinting.stackexchange.com.7z
Resolving ia800801.us.archive.org (ia800801.us.archive.org)... 207.241.230.81
Connecting to ia800801.us.archive.org (ia800801.us.archive.org)|207.241.230.81|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2018-02-03 17:36:06 ERROR 403: Forbidden.

How can I download the Stack Exchange data dump from archive.org through the command line?

I don't want to download through torrent.

4
  • 1
    It is probably something to do with the recent change of login being required in order to download the SE dumps. Did you configure that package with a username and password?
    – animuson StaffMod
    Feb 3, 2018 at 22:49
  • @animuson that worked, thanks! Feb 3, 2018 at 22:55
  • 1
    @animuson Why was this changed? Feb 3, 2018 at 22:57
  • 4
    @Sonic Internet Archive classified it as high bandwidth sometime at the beginning of the year, and subsequently threw a login requirement on it at the same time. Don't believe we had any say in the decision.
    – animuson StaffMod
    Feb 3, 2018 at 23:01

1 Answer 1

9

As animuson mentioned in the comment, one has to login to be able to download the Stack Exchange dump. This can be done using ia configure:

user@server:~$ ia configure
Enter your Archive.org credentials below to configure 'ia'.

Email address: [email protected]
Password:

Config saved to: /home/ubuntu/.config/ia.ini

I have opened an issue on the internetarchive python package: Improve error message when user needs to login #234.


Note that downloading sometimes fails, e.g.:

[...]
 downloading sharepoint.meta.stackexchange.com.7z: 100%|█████████████████████████| 1.35M/1.35M [00:00<00:00, 3.21MiB/s]
 downloading sharepoint.stackexchange.com.7z:  71%|██████████████████████▊         | 111M/156M [01:32<00:37, 1.26MiB/s]
 error downloading file stackexchange\sharepoint.stackexchange.com.7z, exception raised: HTTPSConnectionPool(host='ia904700.us.archive.org', port=443): Read timed out.
 downloading sitecore.meta.stackexchange.com.7z: 100%|█████████████████████████████| 457k/457k [00:00<00:00, 2.97MiB/s]
 downloading sitecore.stackexchange.com.7z: 100%|████████████████████████████████| 26.0M/26.0M [00:06<00:00, 3.93MiB/s]
 downloading skeptics.meta.stackexchange.com.7z: 100%|███████████████████████████| 6.30M/6.30M [00:01<00:00, 5.13MiB/s]
[...]

To mitigate that, use the --retries parameters:

ia download stackexchange --retries==100

and run the command several times until you'll see that it doesn't download anything anymore (it'll skip any already download files. Partially downloaded files will be redownloaded, so no worries about that).

For more info on the ia download options, enter:

ia help download

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .