Human Suggestions: Semantic Similarity Classification

[navvye]

Classified data on the basis of various semantic similarity tools.

[rkrishnasanka]

Here's my feedback @navvye

• ignore all the telugu / non-english words (manually remove it from the excel file
• In all your approaches you are loading the entire set of unique values, group them per unique column name value because you only need to sort between them. The options will only be one of them.
• Use a combo of approaches
• You can use the Levensthein distance to group the ones that are just going to be spelling errors / variations to create some preliminary sets
• You can use the other word-2-vec versions after that keep adding and remove things from the sets
• I'm curious to see how the k-means will work when you group similar words together

Just to clarify an example:

Tailoring
Driver
Ammavadi
Lari Driver
Travels
Contract worker
Bullero (auto)
Dairy
Millets&bullero
Chakali ( battalu wash)
Tractor
Lari
Tailor
Auto
Cloths Iron
Track tar
Lari driver
RMP Docter
Tracter
NREGS
Chepalu pattadam
24000

Will split into the following groups:

(Tailoring, Tailor)
Driver
Ammavadi
Travels
Contract worker
(Bullero (auto), Millets&bullero)
Dairy
(Chakali ( battalu wash), Cloths Iron)
(Lari, Lari driver, Lari Driver)
Auto
(Track tar, Tracter, Tractor)
RMP Docter
NREGS
Chepalu pattadam
24000