10

As I was working with the last SE data dump, I noticed that there are PostLinks that reference posts that do not exist.

As an example, in opensource.meta PostLinks.xml there is a row:

<row Id="330" CreationDate="2015-06-25T20:35:59.847" PostId="130" RelatedPostId="93" LinkTypeId="1" />

However, when I look in the Posts.xml file, there are no search results for Id="93".

A quick query in SEDE shows that it is pointing to a deleted question.

SELECT *
FROM PostsWithDeleted
WHERE Id = 93

The same situation occurs with PostId in Votes.xml and AcceptedAnswerId in Posts.xml. Here is a quick SEDE query to demonstrate the latter: https://data.stackexchange.com/serverfault/query/1853165

I believe either deleted posts should be included in the data dump, or the PostLinks should not include links to deleted posts.

Edit: I also noticed a similar situation for Badges in Mathematics. They are pointing to Users that do not exist. https://data.stackexchange.com/math/query/1853349

Edit2: Also in Mathematics: a post owned by a user id that does not exist: https://data.stackexchange.com/math/query/1853355/

Edit3: There is a user that is in Mathematics SEDE but not in the dump. User Id 1308569 is available in SEDE, but I can't find it in the dump. cat Users.xml | grep 'Id="1308569"' returns nothing.

Improve this question
9
  • 3
    @starball PostsWithDeleted is in SEDE, but not published in the dump. It’s subtly documented here.
    – user1502910
    Commented Jul 25 at 11:24
  • The existing dump is three months old... it's not going to have any user (or anything else, for that matter) created on or after April 1st.
    – user1502910
    Commented Jul 26 at 16:36
  • 2
    remember that both data-dump and SEDE refresh aren't based of a static source. The views / queries run against a live database (IIRC against the secondary in the DAG), one by one and are not guarded by a transaction of any kind, let alone a read isolated one. On the edge of producing a dump / refresh you wil find data inconsistencies, caused by either pure "luck" or because a batch job was running at the same time. That type of data mishaps should resolve itself with the next dump / refresh. Leaves "regular" issues which exist as well and are by-design at this stage.
    – rene Mod
    Commented Jul 26 at 17:02
  • 1
    Votes.xml is by-design having the vote records for deleted posts as well: meta.stackexchange.com/questions/354977/…
    – rene Mod
    Commented Jul 26 at 17:06
  • bug report on PostLinks: meta.stackexchange.com/questions/260676/…
    – rene Mod
    Commented Jul 26 at 17:09
  • These issues seem to have been addressed in the latest Data Explorer refresh, but obviously nothing can be fixed in dumps produced in the past. As rene explained, though, there are cases when objects might point to posts that have been deleted. And since deleted posts aren’t included in the dump…
    – user1502910
    Commented Jul 28 at 14:26
  • @rene Is there any existing report about the Comments file entirely missing the ContentLicense column? I can't seem to find it by searching.
    – Maxwell175
    Commented Jul 31 at 2:46
  • @Maxwell175 not that I know of.
    – rene Mod
    Commented Jul 31 at 4:26
  • 1
    I think it's nice to have this data for archival value. Even if it's doesn't reflect a coherent database state, one may potential reference these ID references against each other, or data from other sources (such as previous releases), so they're still carrying information. They're an annoying case to handle when importing this into a real database, but I'd rather that annoyance than losing the information.
    – Jeremy
    Commented Aug 6 at 14:59

1 Answer 1

Reset to default
2

This change has been made. Please see this post for more details on other fixes and changes related to data dumps.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .