TF/IDF and Okapi BM25 word frequency retrieval

[pkrakesh]

It would be great if there is some word frequency retrieval utility using TF/IDF and BM25 with Wink js. Even if there are many features like tokenizers and Wink BM25 search with the ecosystem, this is really missing. Since TF/IDF or BM25 leaves only very low footprint and doesn't requires any heavy models to relay upon, this will be very useful with fronted implementation with modern front-end libraries like React js.

[sanjayaksaxena]

While we had plans to add BM25 search capabilities to winkNLP, this is indeed a good idea. We can partition the task into two tasks — computing word weights using BM25 and later add search capabilities.

How would you like to see the output — a sparse matrix of weights or a bag of words like structure e.g. { word: weight, ... } format — any thoughts?

[pkrakesh]

Of course the { word: weight, ... } output will be more expressive and useful. A threshold variable to control the output (the number of words) also will be nice.

[sanjayaksaxena]

We have initiated work on a BM25 based utility and would be releasing it soon. It will follow wink-nlp styled API i.e. it will have an .out() API. You will be able to get multiple outputs/formats such as its.bow, its.docTermMatrix, its.idf, its.terms, or its.modelJSON (TF + IDF). Similarly at document level you will have a doc( index ).out() API, where you can get its.bow, its.vector, etc. And finally, you will be able to obtain the vector of a new text using .vectorOf() API.

Stay tuned.

[rachnachakraborty]

Hello @pkrakesh

We have released the new version(1.2.0) of wink-nlp on npm which has the BM25 Vectorizer. While the documentation is in progress, we are sharing a sample self explanatory code to get you started.

It is hosted on runkit

Best,
Rachna

[pkrakesh]

@sanjayaksaxena @rachnachakraborty @prtksxna Very excited to hear the new release. This is a great addition to the Wink ecosystem. Thanks & Regards.

[pkrakesh]

I was using the new TFDF implementation in my react project and I encountered a problem. In react, components are often re-rendered and in effect, the learn() function is calling again and when it happens the below error occurs.

`Error: wink-nlp: learn can not be used after a call to out() API in BM25 Vectorizer
bm25Vectorizer/methods.learn
node_modules/wink-nlp/utilities/bm25-vectorizer.js:185
AnalysePageWrapper/</<
src/AnalysePageWrapper.js:22

19 |
20 | useEffect(() => {
21 |

22 | corpus.forEach( ( doc ) => bm25.learn( nlp.readDoc( doc ).tokens().out( its.normal ) ) );
| ^ 23 | // learned = true;
24 | setVector(bm25);
25 | }, []);

AnalysePageWrapper/<
http://localhost:3000/static/js/main.chunk.js:2090:12

2087 | const corpus = postArr;
2088 | console.log(bm25 == null);
2089 | Object(react__WEBPACK_IMPORTED_MODULE_0__["useEffect"])(() => {

2090 | corpus.forEach(doc => bm25.learn(nlp.readDoc(doc).tokens().out(its.normal))); // learned = true;
| ^ 2091 |
2092 | setVector(bm25);
2093 | }, []); // console.log(vector);`

It is like something persisting in the memory after calling out(). Is there any option to nullify the learn() call so that we could be able to relearn with the new data (or any other option)? Can I get any help in getting around this problem?

[sanjayaksaxena]

Simple way out is to create a new instance every time.

[pkrakesh]

Got it. Made it working. Thanks a lot for your valuable reply.