16

I downloaded Stack Overflow posts dump ('stackoverflow.com-Posts.7z') on 13 Sept 2019 which was lastly modified on 4 Sept 2019 according to https://archive.org. However, after I extracted the data I found that the most recent post on the dump was created on March 2018. Initially, I thought there is something wrong with my script. So, I viewed the data using one of the software where I got the same results as my script.

I am confused now what to do next! Does anyone know from where I can get the dump of the most recent Stack Overflow posts? I tried the uploaded dump in the Big Query but I found a mistake in the dump where the tags are not located between <>.

Looking forward to your helping in this issue.

Thanks all

Improve this question
4
  • 5
    We had some minor issues with the upload of the new archive, let me take a look at and see if something got messed up.
    – Taryn StaffMod
    Commented Sep 30, 2019 at 16:06
  • 1
    I downloaded the stackoverflow.com-Posts.7z file from archive and imported it into SQL Server. I'm seeing that the most recent date on a post is 2019-09-01 05:24:14.550, so I'm not reproducing the issue that you're seeing. Can you try to pull the file again?
    – Taryn StaffMod
    Commented Oct 1, 2019 at 22:04
  • Thanks @Taryn, I am downloading the file again, and will get back to you once I finished extracting it.
    – Maghaj
    Commented Oct 2, 2019 at 17:51
  • 2
    I checked the data quickly and you are right the recent post in the dump is 2019-09-01. I do not know why this happened, although, I downloaded the data again. Anyway, thanks for checking the issue with me.
    – Maghaj
    Commented Oct 3, 2019 at 17:48

2 Answers 2

Reset to default
5

I'm going to blame maybe caching on this.

I downloaded the file from our archive and loaded it into SQL Server. I'm seeing the max date on the Posts as 2019-09-01 05:24:14.550. It looks like maybe you got an old copy, but seems to be resolved now.

Improve this answer
2

I just want to confirm that the latest dump appears to be complete at a first look.

I'm talking about the torrent with info_hash fd11cc265a7964b42ac80f21c216420372c9cada, whose 7z files were created in December 2019.

It contains almost 50M posts, the last of which is from 2019-12-01:

$ 7z e -so stackoverflow.com-Posts.7z | wc -l
46947635
$ 7z e -so stackoverflow.com-Posts.7z | tail -n 1 | grep -Eo "<row .+Body"
<row Id="59123008" PostTypeId="1" CreationDate="2019-12-01T06:13:36.580" Score="0" ViewCount="1" Body

The total uncompressed size for all XML files is over 330 GiB:

$ for i in $(ls stackoverflow*7z); do 7z l $i | grep xml ; done
2019-12-02 03:57:59 ....A   3970886018    254479804  Badges.xml
2019-12-02 04:08:16 ....A  20024216942   4489154698  Comments.xml
2019-12-02 05:07:35 ....A 131615538659  26884195792  PostHistory.xml
2019-12-02 05:41:30 ....A    772322307     88782172  PostLinks.xml
2019-12-02 05:41:04 ....A  76537966910  15332719142  Posts.xml
2019-12-02 05:41:30 ....A      5088121       816900  Tags.xml
2019-12-02 05:44:10 ....A   3693845282    529296484  Users.xml
2019-12-02 05:56:01 ....A  17944120471   1148494387  Votes.xml
$ for i in $(ls *7z); do 7z l $i | grep xml ; done | sed --regexp-extended 's,.+\.A +([0-9]+).+$,\1,g' | paste -s -d+|bc
357018286071
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .