Stop words removal

[geraldstanje]

Hi,

I currently test it with golang 1.3.
does go-porterstemmer also remove stop words or can you suggest a lib?

Gerald

[reiver]

Hello Gerald,

The "go-porterstemmer" algorithm does NOT remove stop words.

(It just implements the "porter stemmer" stemming algorithm (in Go).)

But you could code it yourself pretty easily.

First, you want a list of all the English stop words.

There are various lists of them on the Internet. Here are some lists:

• http://norm.al/2009/04/14/list-of-english-stop-words/
• http://dev.mysql.com/doc/refman/5.5/en/fulltext-stopwords.html
• http://www.textfixer.com/resources/common-english-words.txt

Then just extract that list of stop words and get it into your Go code.

And before you send something to the "go-porterstemmer" check if it in your list of stop words.

[geraldstanje]

Hell Charles,

thanks for your reply. Already implemented :)

does porter handle unicode? e.g. don\u2019t

does porter stemmer handle negation markers like: don’t, doesn’t, won’t, can’t,
mustn’t, isn’t, aren’t, wasn’t, weren’t, couldn’t, shouldn’t, wouldn’t ?
They should be transformed in do not, will not, cannot, must not, is not,
etc...

Thanks,
Gerald

2014-08-02 7:02 GMT+02:00 Charles [email protected]:

Hello Gerald,

The "go-porterstemmer" algorithm does NOT remove stop words.

(It just implements the "porter stemmer" stemming algorithm (in Go).)

But you could code it yourself pretty easily.

First, you want a list of all the English stop words.

There are various lists of them on the Internet. Here are some lists:

• http://norm.al/2009/04/14/list-of-english-stop-words/
• http://dev.mysql.com/doc/refman/5.5/en/fulltext-stopwords.html
• http://www.textfixer.com/resources/common-english-words.txt

Then just extract that list of stop words and get it into your Go code.

And before you send something to the "go-porterstemmer" check if it in
your list of stop words.

—
Reply to this email directly or view it on GitHub
#5 (comment)
.