Firecrawl + Convex: Scrape the Web Straight Into Your Database  🔗 **Source:** [Firecrawl + Convex: Scrape the Web Straight Into Your Database](https://www.youtube.com/watch?v=5XyirpOIO5M) > [!note]+ Document Notes > tags: [convex, firecrawl, web-scraping, serverless-architecture, webhooks, developer-tools] > author: [MISSING] > source: YouTube > reference: https://www.youtube.com/watch?v=5XyirpOIO5M > date created: 2026-08-14 > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > # 🏷️ Title: Firecrawl + Convex: Scrape the Web Straight Into Your Database <!-- deep-read:sections start --> ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Firecrawl](https://www.firecrawl.dev) — the scraping/crawling API the video wires into Convex. - [Convex](https://www.convex.dev) — the backend-as-a-service platform (queries, mutations, actions, components) the video builds on. - Firecrawl Convex component docs (referenced on-screen, not linked in the transcript). - Source video: https://www.youtube.com/watch?v=5XyirpOIO5M ## 📝 Summary - **What happened.** A screen-recorded tutorial walks through installing and wiring the official Firecrawl Convex component into a small React + Convex app, then uses it to add web search, page scraping, and full-site crawling that writes results directly into a Convex database. - **Why it matters.** The video is a compact, concrete demonstration of a recurring serverless pattern: functions that must call third-party APIs (Firecrawl) and receive long-running, asynchronous results (a multi-page crawl) can't be implemented as ordinary sandboxed backend functions — they need an explicit "action" boundary plus a webhook callback, not polling. - **Key stakeholders.** Individual developers/indie hackers building small tools (the presenter's own "documentation hub" search feature is the running example); Convex and Firecrawl as vendors with a referral incentive baked into the video (500 extra Firecrawl credits via the presenter's code). - **Risks and tradeoffs.** The video flags one real security tradeoff explicitly — the webhook endpoint that receives scraped pages is publicly reachable, so skipping Firecrawl's webhook-signature verification would let anyone POST arbitrary data into the app's database. It also implicitly trades "no infrastructure to run" (no polling workers, no queues) for a dependency on both vendors' uptime and API pricing. - **Bottom line.** For a Convex app, the presenter's claim is that full web-search + scrape + durable, webhook-driven crawl support fits in "under 100 lines of code" once the component, environment variables, and one action/query file are set up. ## 📖 Main Content - The video opens with a demo: the presenter's own documentation-hub UI lets a user search across coding docs (a JavaScript framework is named on screen but transcribed inconsistently — see the Accessibility Assessment note on caption quality below); a Firecrawl search call returns the five closest pages, and clicking "crawl" scrapes markdown plus a screenshot for a chosen page and persists it to the Convex database. - **The core architectural problem it solves:** Convex's ordinary functions — queries and mutations — execute inside a deterministic, network-isolated sandbox. They cannot make outbound HTTP calls to a third-party API. Convex's third function type, **actions**, is the one type allowed to talk to the outside world, so any Firecrawl call has to originate from an action. - **The component pattern:** rather than hand-writing HTTP calls and a database schema, the video installs the official Firecrawl *component* for Convex, which ships with its own tables. An action calls into the component; the component talks to the Firecrawl API; results flow back and get written to the app's tables through a mutation. - **Setup steps demonstrated on screen, in order:** 1. Start from a minimal React + Convex app that already reads/writes test data. 2. Use Convex's **cloud** deployment, not local dev — local Convex has no publicly routable URL, and Firecrawl's webhook can't reach it. 3. `npm install` the Firecrawl component, then register it in `convex.config.ts` (created if it doesn't already exist). 4. Create a Firecrawl account, copy the API key from the Firecrawl dashboard, and set it as a Convex environment variable via the CLI. 5. Optionally (but "definitely" recommended) copy a webhook secret from Firecrawl's Settings → Advanced screen and set that as an environment variable too, so the app can verify that inbound webhook POSTs really came from Firecrawl. 6. Create a `convex/web.ts` file for the actions; instantiate the Firecrawl component; define a `search` action (mirrors the Firecrawl Node SDK's search call, `limit: 5` in the demo) and return normalized results. 7. Wire a React `useAction` hook to a search form; submitting the form calls the action and results render immediately (the demo narrows a query from a general framework name to a more specific phrase and shows the ranking shift). - **Search vs. crawl — the key distinction the video draws:** search works fine even on local Convex because it's a single fast round trip with no webhook involved. **Crawling does not**, because Convex functions run in a serverless environment unsuited to long-running work (e.g., scraping hundreds of pages). Instead of holding a function open, the action calls Firecrawl's `startCrawl`-equivalent, which immediately returns a crawl ID while Firecrawl keeps working server-side. - **The "durable crawl" mechanism:** as each page finishes scraping, Firecrawl POSTs it to an HTTP endpoint inside the Convex deployment; that endpoint verifies the webhook signature and calls a mutation to write the page. This repeats per page and survives the client app being closed entirely — there's no polling loop and no worker process to keep alive. - **Crawl code shown:** a `startCrawl` action (limit 25 pages, markdown format, `onlyMainContent: true` to skip headers/footers), a `crawlProgress` query for a progress indicator, and a `crawlPages` query returning paginated results. The demo crawl needs the Convex dashboard's "Firecrawl component" table view (not the app's own tables) to see the two component-owned tables, `crawls` and `pages`, fill in live. - **Payoff:** because the component's scrape options mirror Firecrawl's own scrape options directly, the same setup can pull markdown *and* screenshots — which is what powers the presenter's documentation-hub screenshots shown at the start. - **Close:** a link to the finished sample code, a Firecrawl referral code for 500 extra credits, and a teaser for a follow-up video about using Firecrawl to have an agent read 200 PDFs in under 3 seconds. ### 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=5XyirpOIO5M&t=0) What the component does + live demo - Web search, single-URL scrape, and full-site crawl, all writable into Convex; crawls are "durable" — pages persist as they're found even if the client app is closed. - Demo: docs-hub search → Firecrawl search returns top 5 pages → crawl a chosen page → markdown + screenshot saved to Convex, "under 100 lines of code." #### ⏱️ [01:00](https://www.youtube.com/watch?v=5XyirpOIO5M&t=60) Why actions, not queries/mutations > [!quote] > how does this all work? Well, the UI for our app was built in React and Convex is what handles the data using functions like queries and mutations. But these functions run inside a sandbox. So, how can they access the internet and do web searches? - Queries/mutations are sandboxed and cannot reach third-party APIs; actions are the es <!-- deep-read:sections end --> 📄 **Full text:** [[Full Document Contents/Articles/Firecrawl + Convex Scrape the Web Straight Into Your Database.md|Firecrawl + Convex: Scrape the Web Straight Into Your Database by YouTube]] ## Highlights - how does this all work? Well, the UI for our app was built in React and Convex is what handles the data using functions like queries and mutations. But these functions run inside a sandbox. So, how can they access the internet and do web searches? Well, this is where actions come in because actions can communicate with the outside world using third party APIs like the firecraw. And in the case of firecru it works by using a component which has its own tables. So any action ([View Highlight](https://read.readwise.io/read/01m0164c6fhrbanwpxwyr96ttq)) - Note: Core architecture claim: Convex queries/mutations are sandboxed; actions are the only way out to third-party APIs like Firecrawl. - Tags: atom/literature/h1, xref - crawl. And so when each page is scraped, firecrawl posts to a HTTP ends point in that deployment that verifies the signature and calls a mutation to update the database. And this process happens each time a new page is scraped, meaning you could completely close your app. So turn it off. And this will still keep happening inside the deployment, showing you new scraped pages when you next open the app. So what does that look like in code? So back in the web.ts ts file beneath our search action. We've now ([View Highlight](https://read.readwise.io/read/01m0164hgjqgpdye90rgh4j7fd)) - Note: Durable-crawl mechanism: per-page webhook + signature verification + mutation, working even after the client app closes. - Tags: atom/literature/h2, xref