Generic file generator

[tazarov]

Use LC mime router and python-magic to route files automatically:

from langchain_community.document_loaders.mime_router import MimeRouter
from langchain_community.document_loaders.text import TextLoader
from langchain_community.document_loaders.pdf import PDFLoader

# Create an instance of the MimeRouter
router = MimeRouter()

# Register loaders for specific MIME types
router.register_loader('text/plain', TextLoader)
router.register_loader('application/pdf', PDFLoader)

# Specify the path to the file
file_path = '/path/to/file.pdf'

# Determine the MIME type of the file
mime_type = 'application/pdf'  # You can use a library like python-magic to detect the MIME type

# Get the appropriate loader for the MIME type
loader_cls = router.get_loader(mime_type)

# Create an instance of the loader and load the file
loader = loader_cls(file_path)
document = loader.load()

# Process the loaded document as needed
content = document.content
# ...

[tazarov]

the use case for this is to specify a target dir and a glob pattern or just a series of file types to process.

Examples:

cdp file /target/dir --glob *.pdf --recursive
# or file types flags
cdp file /target/dir --pdf --word --recursive