What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 149 Stack Exchange communities.

I like to use Data.SE to view usage stats for some of the sites, however deleted posts are not included on Data.SE and I think this skews the numbers quite a bit, particularly on sites with a lot of deletions.

Would it be possible to include some limited data about deleted posts in Data.SE to make statistical queries more accurate?

You could clear the non-public data fields of deleted posts, such as Posts.Title, Posts.Body, and Posts.OwnerUserId to address privacy concerns explained here. You could even leave the PostId set to 0 since I know SE doesn't want to provide users with a way of searching deleted questions.

The data I would mostly be interested in is CreationDate, LastActivityDate, PostHistory (date closed, date reopened, date deleted, etc), and the fact there was a deleted question in the first place. Up/Down votes, Score, Tags, and View/Answer/Favorite count would also be preferred.

This would make the statistical queries accurate, while still removing the information not meant to be visible to the public.

share|improve this question
I've been meaning to post this for quite a while. I would absolutely love it if this were implemented. –  ben is uǝq backwards Dec 3 '12 at 16:25
"now that Data.SE is automated"...automation or not was never really a motivator for hiding deleted questions, evident by the fact they're consistently hidden everywhere. We simply don't publish a list of deleted content (which you'd have by Id, with this request) anywhere. –  Nick Craver Dec 27 '12 at 11:52
@NickCraver Hrrmm I thought I saw a question like this a while back, and someone declined it because doing a data dump to Data.SE was a manual process and stripping non-public data would be too much of a bother. –  Rachel Dec 27 '12 at 12:32
Hmm, not directly related, but did you ever make a request for keeping successful close/reopen votes in the votes table? I was thinking the other day that the current behaviour is rather useless. –  Tim Stone Dec 31 '12 at 17:02
@TimStone I have now :) –  Rachel Dec 31 '12 at 17:10
@NickCraver - I understand the value of not exposing deleted content as it's generally crud with little to no redeeming value. However, not having access to that information makes it difficult to identify behavioral patterns regarding the genesis of that content. It's hard to suggest quality improvement when I can't definitively say where some of the crud is coming from. –  GlenH7 Apr 17 '13 at 15:10
Can you give some examples of the kind of statistics you're hoping to garner from this? I suspect I know what you're going for, but it's not entirely clear from what you're requesting. –  Shog9 Jun 3 '13 at 19:54
@Shog9 I like to run queries on things like post activity and site usage over a time frame, and new user activity and retention rates. These can be very skewed without deleted posts, especially if the user had a bad start on the site. I've seen other users run statistical queries on Data.SE too, and keep having to remind them that deleted posts are not included. It wasn't a big deal in the past because deleted posts were such a small percentage, but with so many more closed posts now getting deleted with the tweaked auto-delete script, the numbers have shifted quite a bit. –  Rachel Jun 3 '13 at 21:54
@Rachel: actually, the tweaks aren't live yet - they're waiting on a huge pile of other changes that should be going out Real Soon Now. Of course, they will tend to skew things a tad when they do go live, so you may want to investigate whatever oddities you're observing before then. –  Shog9 Jun 5 '13 at 23:25
@Shog9 Hrrmmm I thought the changes were live due to the number of posts deleted by Community on Programmers.SE recently –  Rachel Jun 6 '13 at 3:49
Well, there are two other auto-delete scripts that've been running weekly / monthly for years now... –  Shog9 Jun 6 '13 at 4:26
@Shog9 I was re-running the queries displayed here and noticed a pretty dramatic shift in some of the numbers. Those were posted about a year ago. Around the same time I also noticed the Community user deleting a lot of older posts on Programmers, so thought the script had been updated. It may have just been coincidence though. –  Rachel Jun 10 '13 at 13:38

You must log in to answer this question.

Browse other questions tagged .