What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 127 Stack Exchange communities.

I downloaded the Stack Overflow dataset. I want to use the stopword list to further process the dump file. But I don't have it. Could you tell me how I can obtain the stopword list for the Stack Overflow dump file?

share|improve this question
Is there some reason you think that you should hve the same stopword list that Stack Excahnge uses? I mean the list exists in order to condition the input for some puprpose and the kind of conditioning needed depends on the processing you expect to do. So why would you expect Stack Exchange to have a better list for your purposes than you can come up with? –  dmckee Jan 21 '13 at 22:16
I want to extract keywords from each question and its answers. So I need to remove stopwords using stopword list. Usually, I only downloaded stopword list from online. but since stack overflow is designed for programming only, the stopword is also different. –  Qiongjie Jan 22 '13 at 17:18
My point is that there is nothing that stops you from doing your own stopword list. Indeed, if you are looking for keywords you might want to consider whitelisting the words you want instead of blacklisting the words you don't want. Or a little of both. Start by scanning a subset of the corpus to learn the overall frequency of words and symbols in the text, then figure out what you want from there. –  dmckee Jan 22 '13 at 17:35
@dmckee Is there a reason that he shouldn't have the same stopword list that Stack Exchange uses? Wanting to interpret posts the same way Stack Exchange does isn't outlandish. –  casperOne Jan 22 '13 at 19:06
@casperOne None that I know of, but I will still content that unless his use of the data is the same then it does not make sense to simple assume he needs the same list. My point is not "you can't have it" since I have no control over that but "are you sure that is what you want?" because "further process" is vague and the use cases I am imaging have different needs that live searching. –  dmckee Jan 22 '13 at 19:10
@dmckee It doesn't matter what "further process" is. I'm pretty sure that no matter what we want to do with the raw data, we all want to start in the same place that Stack Exchange does. The stopword list gets you there, moreso than not having it. It's not like you need to show what you're going to use it for or what he's tried for a request on meta tagged support. –  casperOne Jan 22 '13 at 19:15
add comment

You must log in to answer this question.

Browse other questions tagged .