WikipediaTokenizer incorrectly splits certain syntax into multiple tokens [LUCENE-1141]

[asfimport]

WikipediaTokenizer incorrectly splits tokens that have italics/bold inside the token, for instance '''F'''oo is a bold Foo, not F, oo

---

Migrated from LUCENE-1141 by Grant Ingersoll (@gsingers), updated May 16 2011
Attachments: LUCENE-1141-test.patch

[asfimport]

Grant Ingersoll (@gsingers) (migrated from JIRA)

Here's a test case for the problem

[asfimport]

Jens Muecke (@ryd) (migrated from JIRA)

Patch doesn't apply any more.

common.compile-test:
[javac] Compiling 1 source file to /home/jens/projects/java/lucene-git/build/contrib/wikipedia/classes/test
[javac] /home/jens/projects/java/lucene-git/contrib/wikipedia/src/test/org/apache/lucene/wikipedia/analysis/WikipediaTokenizerTest.java:232: cannot find symbol
[javac] Token token = new Token();
[javac] ^
[javac] symbol: class Token
[javac] location: class WikipediaTokenizerTest
[javac] /home/jens/projects/java/lucene-git/contrib/wikipedia/src/test/org/apache/lucene/wikipedia/analysis/WikipediaTokenizerTest.java:232: cannot find symbol
[javac] Token token = new Token();
[javac] ^
[javac] symbol: class Token
[javac] location: class WikipediaTokenizerTest
[javac] Note: /home/jens/projects/java/lucene-git/contrib/wikipedia/src/test/org/apache/lucene/wikipedia/analysis/WikipediaTokenizerTest.java uses unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 2 errors

BUILD FAILED

It's not by adding the import for the Token class.