What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 131 Stack Exchange communities.

I downloaded the Stack Overflow dataset. I want to use the stopword list to further process the dump file. But I don't have it. Could you tell me how I can obtain the stopword list for the Stack Overflow dump file?

share|improve this question
    
Is there some reason you think that you should hve the same stopword list that Stack Excahnge uses? I mean the list exists in order to condition the input for some puprpose and the kind of conditioning needed depends on the processing you expect to do. So why would you expect Stack Exchange to have a better list for your purposes than you can come up with? –  dmckee Jan 21 '13 at 22:16
1  
I want to extract keywords from each question and its answers. So I need to remove stopwords using stopword list. Usually, I only downloaded stopword list from online. but since stack overflow is designed for programming only, the stopword is also different. –  Qiongjie Jan 22 '13 at 17:18
    
My point is that there is nothing that stops you from doing your own stopword list. Indeed, if you are looking for keywords you might want to consider whitelisting the words you want instead of blacklisting the words you don't want. Or a little of both. Start by scanning a subset of the corpus to learn the overall frequency of words and symbols in the text, then figure out what you want from there. –  dmckee Jan 22 '13 at 17:35
2  
@dmckee Is there a reason that he shouldn't have the same stopword list that Stack Exchange uses? Wanting to interpret posts the same way Stack Exchange does isn't outlandish. –  casperOne Jan 22 '13 at 19:06
1  
@casperOne None that I know of, but I will still content that unless his use of the data is the same then it does not make sense to simple assume he needs the same list. My point is not "you can't have it" since I have no control over that but "are you sure that is what you want?" because "further process" is vague and the use cases I am imaging have different needs that live searching. –  dmckee Jan 22 '13 at 19:10
1  
@dmckee It doesn't matter what "further process" is. I'm pretty sure that no matter what we want to do with the raw data, we all want to start in the same place that Stack Exchange does. The stopword list gets you there, moreso than not having it. It's not like you need to show what you're going to use it for or what he's tried for a request on meta tagged support. –  casperOne Jan 22 '13 at 19:15
1  
What? Off-topic, seriously? I'm wondering if I need to think about moving the bar for close votes here significantly higher than it currently is, because apparently, you can get 3k+ rep and still not know how to read. –  Tim Post Oct 9 at 18:06
    
@TimPost Well, the subreason half-way makes sense... Not that I think closing this question makes sense in any way, but there's more to it than just "off-topic". –  Anna Lear Oct 9 at 18:16
    
@TimPost No, the reputation system stops working after around 4k. After that reputation doesn't reflect participation or value or understanding well. –  bjb568 Oct 9 at 18:54
    
@Tim sorry, it tricked even me. Title and wording of question suggests it's specific to Stack Overflow and it appeared in the middle of retagging requests in the Close queue. –  Shadow Wizard Oct 9 at 22:01
    
@ShadowWizard Yeah, I think we might need a tweak here and there on the close reasons for child meta sites, and this particular site (even though it's not technically a child meta site). It's the "I know what this means, why doesn't everyone else?" syndrome, me thinks, covering up the need to change the language a bit. –  Tim Post Oct 10 at 5:42

You must log in to answer this question.

Browse other questions tagged .