Skip to content

Commit

Permalink
Update dataframe.py (#28871)
Browse files Browse the repository at this point in the history
community: optimize DataFrame document loader

**Description:**
Simplify the `lazy_load` method in the DataFrame document loader by
combining text extraction and metadata cleanup into a single operation.
This makes the code more concise while maintaining the same
functionality.

**Issue:** N/A

**Dependencies:** None

**Twitter handle:** N/A
  • Loading branch information
darien-schettler authored Dec 23, 2024
1 parent cb4e6ac commit 32917a0
Showing 1 changed file with 1 addition and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,8 @@ def lazy_load(self) -> Iterator[Document]:
"""Lazy load records from dataframe."""

for _, row in self.data_frame.iterrows():
text = row[self.page_content_column]
metadata = row.to_dict()
metadata.pop(self.page_content_column)
text = metadata.pop(self.page_content_column)
yield Document(page_content=text, metadata=metadata)


Expand Down

0 comments on commit 32917a0

Please sign in to comment.