Firecrawl + Convex Scrape the Web Straight Into Your Database

Firecrawl + Convex: Scrape the Web Straight Into Your Database ![rw-book-cover](https://img.youtube.com/vi/5XyirpOIO5M/maxresdefault.jpg) 🔗 **Source:** [Firecrawl + Convex: Scrape the Web Straight Into Your Database](https://www.youtube.com/watch?v=5XyirpOIO5M) > [!note]+ Document Notes > tags: [convex, firecrawl, web-scraping, serverless-architecture, webhooks, developer-tools] > author: [MISSING] > source: YouTube > reference: https://www.youtube.com/watch?v=5XyirpOIO5M > date created: 2026-08-14 > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > > # 🏷️ Title: Firecrawl + Convex: Scrape the Web Straight Into Your Database >  ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Firecrawl](https://www.firecrawl.dev) — the scraping/crawling API the video wires into Convex. - [Convex](https://www.convex.dev) — the backend-as-a-service platform (queries, mutations, actions, components) the video builds on. - Firecrawl Convex component docs (referenced on-screen, not linked in the transcript). - Source video: https://www.youtube.com/watch?v=5XyirpOIO5M ## 📝 Summary - **What happened.** A screen-recorded tutorial walks through installing and wiring the official Firecrawl Convex component into a small React + Convex app, then uses it to add web search, page scraping, and full-site crawling that writes results directly into a Convex database. - **Why it matters.** The video is a compact, concrete demonstration of a recurring serverless pattern: functions that must call third-party APIs (Firecrawl) and receive long-running, asynchronous results (a multi-page crawl) can't be implemented as ordinary sandboxed backend functions — they need an explicit "action" boundary plus a webhook callback, not polling. - **Key stakeholders.** Individual developers/indie hackers building small tools (the presenter's own "documentation hub" search feature is the running example); Convex and Firecrawl as vendors with a referral incentive baked into the video (500 extra Firecrawl credits via the presenter's code). - **Risks and tradeoffs.** The video flags one real security tradeoff explicitly — the webhook endpoint that receives scraped pages is publicly reachable, so skipping Firecrawl's webhook-signature verification would let anyone POST arbitrary data into the app's database. It also implicitly trades "no infrastructure to run" (no polling workers, no queues) for a dependency on both vendors' uptime and API pricing. - **Bottom line.** For a Convex app, the presenter's claim is that full web-search + scrape + durable, webhook-driven crawl support fits in "under 100 lines of code" once the component, environment variables, and one action/query file are set up. ## 📖 Main Content - The video opens with a demo: the presenter's own documentation-hub UI lets a user search across coding docs (a JavaScript framework is named on screen but transcribed inconsistently — see the Accessibility Assessment note on caption quality below); a Firecrawl search call returns the five closest pages, and clicking "crawl" scrapes markdown plus a screenshot for a chosen page and persists it to the Convex database. - **The core architectural problem it solves:** Convex's ordinary functions — queries and mutations — execute inside a deterministic, network-isolated sandbox. They cannot make outbound HTTP calls to a third-party API. Convex's third function type, **actions**, is the one type allowed to talk to the outside world, so any Firecrawl call has to originate from an action. - **The component pattern:** rather than hand-writing HTTP calls and a database schema, the video installs the official Firecrawl *component* for Convex, which ships with its own tables. An action calls into the component; the component talks to the Firecrawl API; results flow back and get written to the app's tables through a mutation. - **Setup steps demonstrated on screen, in order:** 1. Start from a minimal React + Convex app that already reads/writes test data. 2. Use Convex's **cloud** deployment, not local dev — local Convex has no publicly routable URL, and Firecrawl's webhook can't reach it. 3. `npm install` the Firecrawl component, then register it in `convex.config.ts` (created if it doesn't already exist). 4. Create a Firecrawl account, copy the API key from the Firecrawl dashboard, and set it as a Convex environment variable via the CLI. 5. Optionally (but "definitely" recommended) copy a webhook secret from Firecrawl's Settings → Advanced screen and set that as an environment variable too, so the app can verify that inbound webhook POSTs really came from Firecrawl. 6. Create a `convex/web.ts` file for the actions; instantiate the Firecrawl component; define a `search` action (mirrors the Firecrawl Node SDK's search call, `limit: 5` in the demo) and return normalized results. 7. Wire a React `useAction` hook to a search form; submitting the form calls the action and results render immediately (the demo narrows a query from a general framework name to a more specific phrase and shows the ranking shift). - **Search vs. crawl — the key distinction the video draws:** search works fine even on local Convex because it's a single fast round trip with no webhook involved. **Crawling does not**, because Convex functions run in a serverless environment unsuited to long-running work (e.g., scraping hundreds of pages). Instead of holding a function open, the action calls Firecrawl's `startCrawl`-equivalent, which immediately returns a crawl ID while Firecrawl keeps working server-side. - **The "durable crawl" mechanism:** as each page finishes scraping, Firecrawl POSTs it to an HTTP endpoint inside the Convex deployment; that endpoint verifies the webhook signature and calls a mutation to write the page. This repeats per page and survives the client app being closed entirely — there's no polling loop and no worker process to keep alive. - **Crawl code shown:** a `startCrawl` action (limit 25 pages, markdown format, `onlyMainContent: true` to skip headers/footers), a `crawlProgress` query for a progress indicator, and a `crawlPages` query returning paginated results. The demo crawl needs the Convex dashboard's "Firecrawl component" table view (not the app's own tables) to see the two component-owned tables, `crawls` and `pages`, fill in live. - **Payoff:** because the component's scrape options mirror Firecrawl's own scrape options directly, the same setup can pull markdown *and* screenshots — which is what powers the presenter's documentation-hub screenshots shown at the start. - **Close:** a link to the finished sample code, a Firecrawl referral code for 500 extra credits, and a teaser for a follow-up video about using Firecrawl to have an agent read 200 PDFs in under 3 seconds. ### 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=5XyirpOIO5M&t=0) What the component does + live demo - Web search, single-URL scrape, and full-site crawl, all writable into Convex; crawls are "durable" — pages persist as they're found even if the client app is closed. - Demo: docs-hub search → Firecrawl search returns top 5 pages → crawl a chosen page → markdown + screenshot saved to Convex, "under 100 lines of code." #### ⏱️ [01:00](https://www.youtube.com/watch?v=5XyirpOIO5M&t=60) Why actions, not queries/mutations > [!quote] > how does this all work? Well, the UI for our app was built in React and Convex is what handles the data using functions like queries and mutations. But these functions run inside a sandbox. So, how can they access the internet and do web searches? - Queries/mutations are sandboxed and cannot reach third-party APIs; actions are the escape hatch. The Firecrawl integration ships as a Convex *component* with its own tables, called from an action. #### ⏱️ [02:15](https://www.youtube.com/watch?v=5XyirpOIO5M&t=135) Installing the component and wiring credentials - `npm install` the component, register it in `convex.config.ts`. - Must use Convex **cloud**, not local — local has no public URL for Firecrawl's webhook to hit. - Get a Firecrawl API key from the dashboard; set it via the Convex CLI. - Get an (optional but recommended) webhook secret from Firecrawl Settings → Advanced; used to verify inbound webhook signatures, because the webhook URL is public and unverified POSTs could otherwise write arbitrary data to the database. #### ⏱️ [03:20](https://www.youtube.com/watch?v=5XyirpOIO5M&t=200) The search action + UI - `convex/web.ts`: instantiate the component, define a `search` action (`limit: 5`, matches the Firecrawl Node SDK shape), return normalized results. - React `useAction` hook + form `onSubmit` → results render immediately; narrowing the query text reorders the results sensibly in the demo. #### ⏱️ [05:00](https://www.youtube.com/watch?v=5XyirpOIO5M&t=300) Why crawling needs webhooks and search doesn't > [!quote] > crawl. And so when each page is scraped, firecrawl posts to a HTTP ends point in that deployment that verifies the signature and calls a mutation to update the database. - Convex functions run in a serverless environment, a poor fit for a long crawl (e.g., 500 pages). The action starts the crawl on Firecrawl's servers and returns immediately with a crawl ID; Firecrawl streams results back per-page via the verified webhook, independent of whether the client app is still open. #### ⏱️ [05:40](https://www.youtube.com/watch?v=5XyirpOIO5M&t=340) Crawl code + live crawl demo - `startCrawl` action: limit 25 pages, markdown format, `onlyMainContent: true`. - `crawlProgress` query drives a progress indicator; `crawlPages` query returns paginated page data. - Demo crawl of a docs site shows a live "scraping" indicator and a running scraped-page count; new data only appears once the dashboard view is switched to the Firecrawl component's own tables (`crawls`, `pages`). #### ⏱️ [06:50](https://www.youtube.com/watch?v=5XyirpOIO5M&t=410) Wrap-up - Scrape options match Firecrawl's own API options directly (markdown, screenshots, etc.) — this is what feeds the documentation-hub screenshots shown in the opening demo. - Sample code link, 500-credit referral code, and a teaser for a follow-up video on using Firecrawl to have an agent read 200 PDFs in under 3 seconds. ## 📚 References - [Firecrawl](https://www.firecrawl.dev) — primary source for the API/component being demonstrated. - [Convex](https://www.convex.dev) — primary source for the platform (queries, mutations, actions, components) underpinning the whole video. - No third-party citations, studies, or external sources appear in the transcript — this is a first-person product walkthrough. ## 🗣️ Paraphrase - Opening demo: the presenter shows a working documentation-search tool built on the Firecrawl Convex component, which can search, scrape, and crawl sites and store results in Convex with, in their words, "under 100 lines of code." - Architecture explainer: Convex's ordinary functions (queries/mutations) are sandboxed and can't call external APIs; only actions can, which is why any Firecrawl call has to run through an action, talking to a Firecrawl-provided component with its own database tables. - Setup walkthrough: install the npm package, register it in `convex.config.ts`, create a Firecrawl account for an API key, optionally grab a webhook secret to verify inbound signatures, then write an action file that instantiates the component and exposes a `search` action to the React UI via `useAction`. - Search vs. crawl: search is a fast synchronous round trip and works even on local Convex; crawling is long-running, so it has to be asynchronous — the action kicks off a Firecrawl crawl and returns a crawl ID immediately, while Firecrawl calls back to a webhook endpoint in the Convex deployment as each page finishes, independent of the client app staying open. - Crawl implementation: a `startCrawl` action with page-limit/format options, a `crawlProgress` query for UI feedback, and a `crawlPages` query for paginated results, backed by two tables the component manages itself. - Close: code link, referral credits, and a teaser for a related video on Firecrawl + PDF reading at scale. ## 📰 Newsletter Not applicable to this source — this video is a general web-development (Convex/Firecrawl) product tutorial with no WordPress connection, so a WordPress-focused newsletter blurb would misrepresent it. A general-audience blurb: *Building a "search my docs" feature usually means standing up a scraper, a job queue, and a webhook receiver. This walkthrough shows the shortcut: Firecrawl's official Convex component handles search/scrape/crawl and writes straight into your Convex tables, with the crawl step running as a durable, webhook-driven job so it keeps working even after you close the tab.* ## ✅ Action Items This source is a general developer-tooling tutorial (Firecrawl + Convex), not WordPress or open-source-community content. Most subteams in the standard checklist below don't apply to it; only the generic DevRel/OSPO/Community items are grounded in the actual video. Everything else is marked not applicable rather than padded with filler. #### DevRel - Developer experience: the video itself is a strong DevRel artifact pattern worth reusing — a single worked example (search → scrape → durable crawl) that gets a developer from zero to a working integration in under 8 minutes; consider this shape for any "quickstart video" brief. - Developer advocacy: the built-in referral-credit CTA (500 extra Firecrawl credits) is a concrete, low-effort acquisition mechanic worth noting for any partner-integration content plan. - Education: the "sandbox vs. actions" explanation is a reusable teaching moment for any serverless/BaaS platform with a similar function-privilege split — could be excerpted as a standalone concept clip. #### OSPO - Not applicable — no open-source governance, licensing, or compliance topic appears in this source. #### Community - Not applicable — no community, maintainer, or contributor dynamic appears in this source; it is a solo product walkthrough. #### WordPress - Not applicable to this source — no WordPress Core, Core Performance, Core AI, Core Program Team, Mobile, Playground, Community, Training, Marketing, Support, Design, Documentation, Photos, TV, Polyglots, Accessibility, Meta, CLI, Hosting, Tide, Openverse, Themes, or Plugins topic is discussed. (The Firecrawl-powered "documentation hub" search shown in the demo is a *generic pattern* a WordPress Documentation or DevHub team could borrow, but the video itself makes no WordPress reference.) #### The WPOCC - Not applicable — the video has no connection to The WPOCC (The WP Open Community Collective) or its programs. ### FAIR - Not applicable — no FAIR governance, funding, or technical-independence topic appears in this source. ## 🧪 Hypotheses - **H1.** Convex queries and mutations cannot make outbound HTTP requests to third-party APIs; only actions can, because queries/mutations run in a deterministic, network-isolated sandbox. - **H2.** For crawls beyond a handful of pages, a webhook-driven ("durable") architecture completes more reliably than holding a single serverless function open, because Convex actions are subject to execution-time limits unsuited to long-running scrapes. - **H3.** Skipping Firecrawl webhook-signature verification on the Convex endpoint would allow an attacker to POST forged "scraped page" payloads and write arbitrary data into the app's database, since the endpoint URL is public. - **H4.** Using the official Firecrawl Convex component (with its own `crawls`/`pages` tables) requires substantially less custom code than hand-rolling a webhook receiver and schema, consistent with the presenter's "under 100 lines" claim. - **H5.** A local (non-cloud) Convex dev deployment cannot receive Firecrawl webhook callbacks because it lacks a publicly routable URL, so crawling (though not searching) requires the cloud deployment. ### 📊 Methodology - **H1:** attempt an outbound `fetch()` inside a Convex query and a Convex mutation (expect a sandbox error), then repeat inside an action (expect success) — a minimal controlled code test, not present in the video itself. - **H2:** benchmark two implementations crawling the same 500-page site — one holding an action open synchronously, one using the webhook pattern shown — and measure timeout/failure rates. - **H3:** temporarily disable signature verification in a test deployment, POST a forged payload to the public webhook endpoint, and confirm whether it is accepted and written to the database. - **H4:** compare lines of code and setup time between the component-based approach shown and a hand-rolled webhook handler + schema built from scratch for the same feature set. - **H5:** attempt to register a local Convex dev URL as a Firecrawl webhook target and confirm the failure mode (unreachable/refused) described in the video. ### ❓ Questions - If webhook signature verification fails on a delivery, or the Convex deployment is briefly unreachable, does Firecrawl retry that page, or is it silently dropped from the crawl? - How does per-page mutation cost scale on Convex's pricing/quota model for crawls in the thousands-of-pages range, well beyond the 25-page demo? - Does the component expose crawl cancellation, or is `startCrawl` strictly fire-and-forget once started? - How is the Firecrawl API key scoped and rotated, and can one Convex deployment use multiple Firecrawl accounts/keys? ### 🔮 Future Research - Compare this webhook-push ingestion pattern against alternatives (client polling, Convex subscriptions/WebSockets) for latency and infrastructure cost at crawl volumes beyond the demo's 25 pages. - Independently verify the "durable crawl" reliability claim under adverse conditions — deployment redeploys mid-crawl, webhook timeouts, Firecrawl-side retries. - Follow up on the teased next video (Firecrawl + 200-PDF ingestion) to see whether the same action/webhook pattern generalizes to non-HTML content. ### 💡 Implications - **Practice.** Lowers the barrier for small teams/indie developers to add "search across my own content" or web-ingestion features without standing up separate scraper/queue infrastructure — the component absorbs both concerns. - **Security.** The video's own caveat is the load-bearing one: a webhook endpoint that writes to a database is a public write surface, and the presenter is explicit that signature verification is what stands between "convenient integration" and "anyone can POST fake data into your app." That generalizes to any Convex+webhook integration, not just this one. - **Education.** The queries/mutations-vs-actions explanation is a clean, transferable teaching example for the broader "sandboxed compute vs. network-capable compute" split found in many serverless/BaaS platforms. - **Funding/cost.** The referral-credit mechanic (500 extra Firecrawl credits via the presenter's code) is a growth lever viewers should recognize as marketing, not a platform default, when estimating what a similar integration would cost them. ### 🧠 Learning Styles Assessment - **Reading level estimate:** the transcript is conversational spoken tutorial narration, not prose; approximating with a Flesch–Kincaid-style read on the transcript text lands around grade 8–9 — short clauses, informal connectors ("So," "Well," "But"), minimal subordinate-clause density. #### 👀 Visual learners - Present: screen-recorded live demo, on-screen code (`convex.config.ts`, `web.ts`), a live crawl-progress UI, and the Convex dashboard table view. - Improve: an architecture diagram (React UI → Convex action → Firecrawl component/API → webhook → mutation → tables) would make the sandbox/action distinction land faster than narration alone. #### 🔊 Aural learners - Present: full narrated walkthrough is the primary format; no separate audio-only cut needed since the video *is* audio-led. - Improve: a 60–90 second audio-only recap of just the "why actions, why webhooks" architecture logic would let this travel as a podcast-style clip. #### 🗣️ Verbal learners - Present: the "sandbox" and "durable crawl" framing are effectively working analogies/definitions used consistently through the video. - Improve: a one-line glossary (action, component, webhook, durable crawl) up front would help viewers who join partway through. #### 🧩 Kinesthetic learners - Present: the video is itself a reproducible step-by-step (install → config → key → webhook secret → action file → UI hook). - Improve: an explicit numbered checklist (as reconstructed in this note's Main Content) would let a viewer follow along without repeatedly scrubbing the video. #### 🧠 Logical learners - Present: a clear causal chain — sandboxed functions can't reach the network → actions can → long-running work can't block an action → webhooks close the loop. - Improve: a small sequence diagram of the crawl request/webhook/mutation cycle would formalize what's currently explained only verbally. #### 🧑‍🤝‍🧑 Social learners - Present: none — this is a solo screen-recorded tutorial with no discussion, Q&A, or community prompt. - Improve: a companion Discord/community thread for "what did you build with this" would add a social layer the video itself doesn't have. #### 🧘‍♀️ Solitary learners - Present: self-paced video format, code-along friendly, referral link for independent follow-up. - Improve: a linked starter repo (mentioned but not shown/linked in the transcript itself) would let a solitary learner build without transcribing code from the screen. #### 🌿 Naturalistic learners - Present: none — no sustainability, environment, or lifecycle framing appears in this technical content. - Improve: not a natural fit for this source; skip rather than force an analogy. ### 📊 Accessibility Assessment #### Screen reader semantics - Present: not assessable from the transcript alone — this note doesn't have access to the video's own captions/description markup. - Improve: if publishing captions, ensure product/technical terms (Firecrawl, Convex, webhook) are hand-corrected rather than left to auto-caption inference (see Media and captions below). #### Keyboard navigation - Not applicable — the source is a video, not an interactive UI; no keyboard-navigation surface to assess. #### Contrast and readability - Not assessable from the transcript — on-screen code/UI contrast would need a visual review of the actual video frames, which weren't available to this analysis. #### Text resizing and reflow - Not applicable — video content; no reflow surface. #### Media and captions - **Present:** the video has machine-generated captions (this is the transcript basis for this whole note). - **Gap found directly in this source:** the auto-captions repeatedly mangle the two product names central to the video — "Firecrawl" appears transcribed as "Firecrol," "firecru," "fire call," "firecraw," "fireall," "firecore," and "firewall" across different segments; "webhook" sometimes splits into "web hook"; "endpoint" becomes "ends point"; "paginated" becomes "pageentated"; "onSubmit" becomes "onsert." A viewer relying on captions alone (not spoken audio) would likely misread the product's actual name. - **Improve:** publish a human-corrected transcript/caption pass, at minimum for proper nouns and technical terms — this is a low-effort, high-value fix for a tutorial whose entire value is "go install this exact tool." #### Forms and interactions - Not applicable — no interactive form exists in the video content itself (the React search form shown is inside the demo app, not the video player). #### Standards reference - WCAG 2.2 AA 1.2.2 (Captions, pre-recorded) is technically met — captions exist — but the accuracy gap above undermines the *intent* of the criterion for a technical audience that needs exact product/API names. ### 🌍 DEIB Assessment #### Cultural breadth - Not applicable — no regional, cultural, or multilingual content appears; this is a single-presenter technical screen recording. #### Inclusive language - Present: neutral, direct instructional language throughout ("you," "your app"); no gendered or exclusionary phrasing observed. - Improve: none needed based on the transcript. #### Neurodiversity support - Present: consistent step-by-step structure (demo → why → how → demo again) is easy to chunk. - Improve: the video relies entirely on spoken narration plus on-screen code with no text-overlay summaries between steps; brief on-screen step labels (as reconstructed in this note's Notes section) would reduce reliance on continuous audio tracking. #### Representation - Not applicable — no people, roles, or identities are depicted or discussed beyond the single presenter. #### Psychological safety - Not applicable — no sensitive, contentious, or interpersonal content; purely technical instruction. ### ✅ Logical Validity #### Strong reasoning - The video's core argument is a clean deductive chain, explicitly stated: Convex queries/mutations run sandboxed → sandboxed code cannot reach third-party APIs → therefore any Firecrawl call must go through an action → long crawls can't block an action → therefore crawling needs an asynchronous, webhook-driven design. Each step is stated and immediately demonstrated in code, which is a strong pattern for technical argumentation (claim → live proof). #### Fallacies - No fallacies identified. The video makes narrow, falsifiable technical claims (e.g., "under 100 lines of code," "local Convex has no public URL") that are demonstrated on screen rather than asserted rhetorically. The like/subscribe request and referral-code plug are ordinary promotional asks, not disguised as technical argument, so they aren't scored as fallacious reasoning here. #### Tone of voice - Casual, first-person, instructional — frequent verbal fillers and self-corrections typical of unscripted screen-recorded narration ("So," "Well," "But," "Now don't worry about the style..."), consistent throughout. ### 🛡️ DARVO Analysis Not applicable to this source — DARVO (Deny, Attack, Reverse Victim/Offender) analysis applies to interpersonal or institutional accusation dynamics. This is a solo technical tutorial with no accusation, denial, or victim/offender framing present anywhere in the transcript. ## 🎯 Conclusion - **Bottom line:** the Firecrawl Convex component collapses "add search/scrape/crawl to a Convex app" into an install, an env-var setup, and one actions file — with the crawl path specifically designed around Convex's serverless execution limits via a signed webhook callback rather than polling. - **Practical next steps for readers:** (1) if evaluating this for a real project, explicitly test the webhook-signature-verification path before shipping, since that's the one place the video flags real risk; (2) budget for the cloud Convex requirement if currently developing locally; (3) treat the "under 100 lines" and referral-credit claims as presenter-stated, unverified by this analysis, and confirm against the linked sample code before relying on them. ## 🧱 Refactor Appendix — Firecrawl + Convex: Scrape the Web Straight Into Your Database > Density: 2–5 Literature atoms (H3), 0–3 Permanent candidates (H4). > Defaults: 3 Literature, 1 Permanent. Resonance filter: if you can't > articulate in one line why a claim matters, drop it. > > Konik claim-header rules: present tense, declarative, ≤10 words when > possible, searchable, unique, mirrors annotation focus if present. > > Atomicity: one declarative sentence per atom. "and"/"also" → split. > > Tag pool: #bmf · #addendum · #xref · #articleseed · #storystem · #share · #aphorisms ### 📚 Literature Atoms (H3) #### [H1] Convex actions are the only sandbox exit > [!quote] ID:rw_atom1 > these functions run inside a sandbox. So, how can they access the internet and do web searches? **Context & use**: - ~[01:10](https://www.youtube.com/watch?v=5XyirpOIO5M&t=70) - Names the exact reason the whole integration needs an "action" rather than a query/mutation — the load-bearing architectural fact of the video. **Tags**: #xref **Highlight**: [view highlight](readwise://)  #### [H2] Durable crawls write to Convex via signed webhook > [!quote] ID:rw_atom2 > when each page is scraped, firecrawl posts to a HTTP ends point in that deployment that verifies the signature and calls a mutation to update the database **Context & use**: - ~[05:00](https://www.youtube.com/watch?v=5XyirpOIO5M&t=300) - Explains the "durable crawl" claim mechanically: per-page webhook + signature check + mutation, not a held-open connection or polling loop. **Tags**: #xref **Highlight**: [view highlight](readwise://)  #### [H3] Local Convex can't host Firecrawl's webhook > [!quote] ID:rw_atom3 > I'm using the cloud version of Convex, not the local one since they don't have public URLs **Context & use**: - ~[02:15](https://www.youtube.com/watch?v=5XyirpOIO5M&t=135) - A concrete, easy-to-miss setup requirement — anyone following along on local Convex will hit a silent wall on the crawl feature specifically (search still works locally). **Tags**: #xref **Highlight**: [view highlight](readwise://)  ### 🧠 Permanent Candidates (H4) #### [P1] Serverless function limits force webhook-driven design for long external jobs **Claim** (one sentence, Courtney's voice): When a serverless function can't stay open long enough to finish a job, the job has to hand off to the external service and let that service call back in, rather than the function waiting on it. **Evidence pointer**: [[#H2]] **Scope & conditions**: Holds for any BaaS/serverless action that triggers a long-running third-party process (crawls, batch jobs, video transcodes); breaks down if the third-party service has no webhook/callback mechanism, forcing a polling fallback instead. **Links** (≥1 with annotation): - [[Serverless execution time limits]] — because extends - [[Webhook signature verification]] — because applies-in the Firecrawl/Convex case - [[Async job patterns]] — because contrasts on push (webhook) vs. pull (polling)  ### ✏️ Fleeting Queue - [F1] Check whether the Firecrawl Convex component exposes crawl cancellation, or only fire-and-forget `startCrawl` — https://www.youtube.com/watch?v=5XyirpOIO5M #fleeting #inbox - [F2] Watch the teased follow-up video (Firecrawl + agent reading 200 PDFs in under 3 seconds) and compare its ingestion pattern to this one — https://www.youtube.com/watch?v=5XyirpOIO5M #fleeting #inbox ### → Things 3 (max 2 actions) - [ ] Try the Firecrawl Convex component on a small side project (doc search) and use the 500-credit referral code from the video before it expires — dev-tools eval #things  ## Highlights - these functions run inside a sandbox. So, how can they access the internet and do web searches? - Note: [atom/literature/h1] Convex actions are the only sandbox exit -- names why any Firecrawl call must originate from an action, not a query/mutation. #atom/literature/h1 #xref - when each page is scraped, firecrawl posts to a HTTP ends point in that deployment that verifies the signature and calls a mutation to update the database - Note: [atom/literature/h2] Durable crawls write to Convex via signed webhook -- the mechanical core of the 'durable crawl' claim. #atom/literature/h2 #xref - I'm using the cloud version of Convex, not the local one since they don't have public URLs - Note: [atom/literature/h3] Local Convex can't host Firecrawl's webhook -- easy-to-miss setup requirement. #atom/literature/h3 #xref - When a serverless function can't stay open long enough to finish a job, the job has to hand off to the external service and let that service call back in, rather than the function waiting on it. - Note: [atom/zettel/z1] Permanent candidate: serverless execution limits force webhook-driven design for long external jobs. #atom/zettel/z1 - A screen-recorded tutorial walks through installing and wiring the official Firecrawl Convex component into a small React + Convex app, then uses it to add web search, page scraping, and full-site crawling that writes results directly into a Convex database. - Note: [summary] Top-level claim from the Summary section. #atom/literature/h1