langchain[minor]: Firecrawl Document Loader #5180

nickscamara · 2024-04-22T23:57:32Z

Added the FireCrawl document loader. Firecrawl crawls and convert any website into LLM-ready data. It crawls all accessible subpages and give you clean markdown for each.

Similar to langchain-ai/langchain#20364 but JS/TS :)

Thank you!

vercel · 2024-04-22T23:57:36Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchainjs-api-refs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 24, 2024 9:26pm
langchainjs-docs	✅ Ready (Inspect)	Visit Preview		Apr 24, 2024 9:26pm

dosubot · 2024-04-22T23:58:05Z

examples/src/document_loaders/firecrawl.ts

@@ -0,0 +1,13 @@
+import { FireCrawlLoader } from "langchain/document_loaders/web/firecrawl";


Hey there! 👋 I've reviewed the code and noticed that the added lines explicitly access an environment variable using process.env. I've flagged this for your review to ensure it aligns with our best practices for handling environment variables. Let me know if you need further assistance with this.

dosubot · 2024-04-22T23:58:07Z

langchain/package.json

@@ -1314,6 +1315,7 @@
    "@gomomento/sdk-web": "^1.51.1",


Hey there! I noticed that this PR adds a new hard dependency with the addition of "@mendable/firecrawl-js". I've flagged this for your review as it's important to ensure the impact of this change on the project's dependencies. Keep up the great work!

dosubot · 2024-04-22T23:58:08Z

langchain/src/document_loaders/tests/firecrawl.int.test.ts

@@ -0,0 +1,34 @@
+/* eslint-disable no-process-env */


Hey there! I noticed that the recent changes in the FireCrawlLoader tests are accessing environment variables directly. I've flagged this for your review to ensure proper handling of sensitive information. Let me know if you need any further assistance with this.

dosubot · 2024-04-22T23:58:09Z

langchain/src/document_loaders/web/firecrawl.ts

@@ -0,0 +1,96 @@
+import FirecrawlApp from "@mendable/firecrawl-js";


Hey there! I noticed that the recent code changes explicitly access an environment variable using getEnvironmentVariable. I've flagged this for your review to ensure it aligns with the intended functionality. Keep up the great work!

bracesproul

A couple nits, besides that it looks pretty good!

bracesproul · 2024-04-23T00:04:10Z

docs/core_docs/docs/integrations/document_loaders/web_loaders/firecrawl.mdx

+
+import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx";
+
+<IntegrationInstallTooltip></IntegrationInstallTooltip>


No need for an IntegrationInstallTooltip for non LC deps. You only this for langchain package installs

bracesproul · 2024-04-23T00:13:26Z

docs/core_docs/docs/integrations/document_loaders/web_loaders/firecrawl.mdx

+
+## Setup
+
+You'll need to sign up and retrieve your [FireCrawl API key](https://firecrawl.dev).


Suggested change

You'll need to sign up and retrieve your [FireCrawl API key](https://firecrawl.dev).

You'll need to sign up and retrieve your [FireCrawl API key](https://firecrawl.dev).

I would add some mention of the 100 free credits. I could easily see a user reading up until the API key requirement part and getting discouraged before they're even able to try it out

examples/src/document_loaders/firecrawl.ts

bracesproul · 2024-04-23T00:17:58Z

langchain/src/document_loaders/web/firecrawl.ts

+  apiKey?: string;
+  mode?: "crawl" | "scrape";


Let's add JSDoc here documenting what the default values are for both of these

langchain/src/document_loaders/web/firecrawl.ts

bracesproul · 2024-04-23T00:21:39Z

langchain/src/document_loaders/web/firecrawl.ts

+   * @returns An array of Documents representing the retrieved data.
+   * @throws An error if the data could not be loaded.
+   */
+  public async load(): Promise<Document[]> {


Prefer DocumentInterface

Suggested change

public async load(): Promise<Document[]> {

public async load(): Promise<DocumentInterface[]> {

Co-authored-by: Brace Sproul <[email protected]>

nickscamara · 2024-04-23T00:34:06Z

@bracesproul Thank you. Just fixed all the suggestions. Let me know if you spot any other ones.

bracesproul

lgtm, thanks for adding this!!

bracesproul

~~hmm, the amount of changed lines in the yarn.lock is concerning. I'm going to push up a PR in a second resolving~~

Resolved

…nsc/firecrawl-loader

jacoblee93 · 2024-04-24T21:16:54Z

Thank you!

Note that we'll move this to @langchain/community in 0.2 in a few weeks.

* Nick: init * Update firecrawl.ts * Nick: * Nick: * Update package.json * Nick: fixes docs * Update yarn.lock * Update examples/src/document_loaders/firecrawl.ts Co-authored-by: Brace Sproul <[email protected]> * Update langchain/src/document_loaders/web/firecrawl.ts Co-authored-by: Brace Sproul <[email protected]> * Nick: fixes * Update yarn.lock * Fix yarn.lock * lint & format * Update firecrawl.ts * Add entrypoint --------- Co-authored-by: Brace Sproul <[email protected]> Co-authored-by: Jacob Lee <[email protected]>

nickscamara added 6 commits April 22, 2024 15:44

Nick: init

602ef9a

Update firecrawl.ts

6aa8895

Nick:

da7be5b

Nick:

8782cc1

Update package.json

192a563

Nick: fixes docs

9d30f99

dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Apr 22, 2024

dosubot bot added the auto:enhancement A large net-new component, integration, or chain. Use sparingly. The largest features label Apr 22, 2024

dosubot bot reviewed Apr 22, 2024

View reviewed changes

vercel bot had a problem deploying to Preview – langchainjs-api-refs April 22, 2024 23:58 Failure

vercel bot had a problem deploying to Preview – langchainjs-docs April 22, 2024 23:58 Failure

Update yarn.lock

3980dc5

vercel bot deployed to Preview – langchainjs-api-refs April 23, 2024 00:14 View deployment

vercel bot deployed to Preview – langchainjs-docs April 23, 2024 00:15 View deployment

bracesproul reviewed Apr 23, 2024

View reviewed changes

nickscamara and others added 3 commits April 22, 2024 17:24

Update examples/src/document_loaders/firecrawl.ts

6a7bc20

Co-authored-by: Brace Sproul <[email protected]>

Update langchain/src/document_loaders/web/firecrawl.ts

29108ca

Co-authored-by: Brace Sproul <[email protected]>

Nick: fixes

3b5c75b

vercel bot deployed to Preview – langchainjs-docs April 23, 2024 00:43 View deployment

vercel bot deployed to Preview – langchainjs-api-refs April 23, 2024 00:43 View deployment

bracesproul approved these changes Apr 23, 2024

View reviewed changes

dosubot bot added the lgtm PRs that are ready to be merged as-is label Apr 23, 2024

nickscamara added 2 commits April 23, 2024 10:31

Update yarn.lock

4327d2a

Merge remote-tracking branch 'upstream/main' into nsc/firecrawl-loader

f9a09ec

vercel bot had a problem deploying to Preview – langchainjs-docs April 23, 2024 17:35 Failure

vercel bot deployed to Preview – langchainjs-api-refs April 23, 2024 17:43 View deployment

vercel bot deployed to Preview – langchainjs-docs April 23, 2024 17:43 View deployment

jacoblee93 changed the title ~~langchain: Firecrawl Document Loader~~ langchain[minor]: Firecrawl Document Loader Apr 23, 2024

bracesproul reviewed Apr 24, 2024

View reviewed changes

bracesproul added hold On hold and removed lgtm PRs that are ready to be merged as-is labels Apr 24, 2024

bracesproul added 2 commits April 24, 2024 13:28

Fix yarn.lock

bed72a3

lint & format

5744672

bracesproul added lgtm PRs that are ready to be merged as-is and removed hold On hold labels Apr 24, 2024

Merge branch 'main' into nsc/firecrawl-loader

68edbd0

vercel bot deployed to Preview – langchainjs-docs April 24, 2024 20:42 View deployment

vercel bot deployed to Preview – langchainjs-api-refs April 24, 2024 20:42 View deployment

Update firecrawl.ts

e28ef50

vercel bot deployed to Preview – langchainjs-docs April 24, 2024 21:08 View deployment

vercel bot deployed to Preview – langchainjs-api-refs April 24, 2024 21:13 View deployment

jacoblee93 added 2 commits April 24, 2024 14:13

Add entrypoint

ac50398

Merge branch 'main' of https://github.com/hwchase17/langchainjs into …

94bad49

…nsc/firecrawl-loader

vercel bot deployed to Preview – langchainjs-api-refs April 24, 2024 21:25 View deployment

vercel bot deployed to Preview – langchainjs-docs April 24, 2024 21:26 View deployment

jacoblee93 merged commit e0da231 into langchain-ai:main Apr 24, 2024
17 checks passed

WilliamEspegren mentioned this pull request May 16, 2024

community[minor]: Spider Document Loader #5415

Merged

rafaelsideguide mentioned this pull request May 16, 2024

langchain[minor]: Added Search Functionality to FireCrawl Document Loader #5418

Closed

naresh4dev mentioned this pull request May 30, 2024

[Snyk] Upgrade langchain from 0.0.153 to 0.1.36 naresh4dev/askDev-slackChatBot#1

Open

aechegoyan17 mentioned this pull request Jun 6, 2024

[Snyk] Upgrade langchain from 0.0.214 to 0.1.37 aechegoyan17/LibreChat#1

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

langchain[minor]: Firecrawl Document Loader #5180

langchain[minor]: Firecrawl Document Loader #5180

nickscamara commented Apr 22, 2024

vercel bot commented Apr 22, 2024 •

edited

Loading

dosubot bot Apr 22, 2024

dosubot bot Apr 22, 2024

dosubot bot Apr 22, 2024

dosubot bot Apr 22, 2024

bracesproul left a comment

bracesproul Apr 23, 2024

bracesproul Apr 23, 2024

nickscamara Apr 23, 2024

bracesproul Apr 23, 2024

bracesproul Apr 23, 2024

nickscamara commented Apr 23, 2024

bracesproul left a comment

bracesproul left a comment •

edited

Loading

jacoblee93 commented Apr 24, 2024

		@@ -0,0 +1,13 @@
		import { FireCrawlLoader } from "langchain/document_loaders/web/firecrawl";

		@@ -0,0 +1,96 @@
		import FirecrawlApp from "@mendable/firecrawl-js";


		import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx";

		<IntegrationInstallTooltip></IntegrationInstallTooltip>


		## Setup

		You'll need to sign up and retrieve your [FireCrawl API key](https://firecrawl.dev).

	public async load(): Promise<Document[]> {
	public async load(): Promise<DocumentInterface[]> {

langchain[minor]: Firecrawl Document Loader #5180

langchain[minor]: Firecrawl Document Loader #5180

Conversation

nickscamara commented Apr 22, 2024

vercel bot commented Apr 22, 2024 • edited Loading

dosubot bot Apr 22, 2024

Choose a reason for hiding this comment

dosubot bot Apr 22, 2024

Choose a reason for hiding this comment

dosubot bot Apr 22, 2024

Choose a reason for hiding this comment

dosubot bot Apr 22, 2024

Choose a reason for hiding this comment

bracesproul left a comment

Choose a reason for hiding this comment

bracesproul Apr 23, 2024

Choose a reason for hiding this comment

bracesproul Apr 23, 2024

Choose a reason for hiding this comment

nickscamara Apr 23, 2024

Choose a reason for hiding this comment

bracesproul Apr 23, 2024

Choose a reason for hiding this comment

bracesproul Apr 23, 2024

Choose a reason for hiding this comment

nickscamara commented Apr 23, 2024

bracesproul left a comment

Choose a reason for hiding this comment

bracesproul left a comment • edited Loading

Choose a reason for hiding this comment

jacoblee93 commented Apr 24, 2024

vercel bot commented Apr 22, 2024 •

edited

Loading

bracesproul left a comment •

edited

Loading