Investigate the possibility of removing dataframe field from Document

[anakin87]

I've been thinking about dropping the Dataframe field in Haystack Document dataclass for a few reasons:

• Users are already using text representations (CSV, Markdown) that LLMs handle great - even for originally tabular data
• Pandas DataFrame creates serialization headaches. (e.g. in Hayhooks)
• Pandas is a heavy dependency that complicates things, especially in serverless environments like Lambda. We could make it optional.
• Supporting dataframes across different Document Stores requires complex workarounds.

I will reach out to internal and external users to validate my assumptions.
We should also investigate how impactful this change would be.

[anakin87]

@EdAbati, the author of dataframes-haystack confirmed that this idea makes sense to him.
@sjrl too.