Implement hero.describe(s)

[mk2510]

In pandas, there is a s.describe() function that provides some information / statistics about a series or dataframe. Example:

>>> s = pd.read_csv("https://raw.githubusercontent.com/jbesomi/texthero/master/dataset/bbcsport.csv")["text"]
>>> s.describe()
count                                                   737
unique                                                  727
top       India's top six secure - Ganguly\n\nCaptain So...
freq                                                      2
Name: text, dtype: object

We can see that it's not all that useful for text data.

TODO

Implement a function hero.describe(s) where s is a TextSeries (so a Series where every cell is a string, like s in the example above. Output might include

• number of documents (= count from above = len(s))
• number of unique documents (= unique from above = len(s.unique())
• number of empty / missing (NaN) documents (= (~hero.has_content(s)).values.sum() )
• most common single words (= s.pipe(hero.tokenize).pipe(hero.top_words)[:10] for the 10 most common words)
• average length of documents (= s.pipe(hero.tokenize).map(lambda x: len(x)).mean()
• shortest document (= s.pipe(hero.tokenize).map(lambda x: len(x)).min()
• longest document (= s.pipe(hero.tokenize).map(lambda x: len(x)).max()

and should be a nice-looking Series like above.

We believe this is a great way for our users to get first insights into their text datasets.

[henrifroese]

Will start work on this now 🍺

[mk2510]

Great 🍻

[k0pernicus]

Solved on #168 if I don't make any mistake - we can close this issue I think :)

[jbesomi]

Thanks, Antoin! Keep up the good work!! 👍