# Useless low entropy answers on SO

Using the data dump I came up with a list of "answers" containing no useful text. Those answers are all just noise and should be deleted by a moderator.

And instead of spending the next days with flagging them for moderator attention one by one, here they are:

Answers posted in duplicate:

-
(for info, I think we've got most of them...) – Marc Gravell Jan 2 '10 at 11:19

To get the list I searched for posts containing only a small amount of different characters, based on the assumption that random typing like "hgfghhgffhgfgh" usually contains the same characters over and over again. To measure this redundancy I basically calculated the entropy of all posts:

``````double calc_entropy(const std::string &txt) {
static const double log_2 = log(2.0);
const int sz = txt.size();
int chars[256] = {0};

for (int i = 0; i < sz; i++) {
if (txt[i] == '>' || txt[i] == '<' || txt[i] == ' ')
continue;
chars[(unsigned int)(unsigned char)txt[i]]++;
}

double H = 0.0;
for (int i = 0; i < 256; i++) {
if (chars[i]) {
double p = (double)chars[i] / sz;
H += p * log(p) / log_2;
}
}
return -H;
}
``````

When sorting the posts by the score calculated by this function (or minor variants of it), the problematic answers came to the top. But of course there are also false positives, like this or many variations of "Yes" and "No".

-
length seems to be a reasonably good indicator as well. – Jeff Atwood Jan 2 '10 at 11:51
+1 for doin' it in c++ – Shog9 Jan 2 '10 at 16:58

Hmm... in some cases there is no edit history, but in some cases it looks like the answer has been removed (usually by the OP, perhaps lacking a "delete" button). I wonder which is the greater evil - rollback (against poster) or delete (of perhaps useful content).

-
there's no restriction on deleting answers, though -- only some restrictions on deleting questions because of the implied auto-delete of all answers. – Jeff Atwood Jan 2 '10 at 10:14