fix(firecrawl): count depth for scheme-less start URLs; doc prerequisites and CI wording

Author: waleedlatif1

.github/CI.md

@@ -4,7 +4,7 @@ The `changes` job runs Turborepo 2.9.14's `affectedTasks` query before installin
 
 A PR compares the checked-out GitHub merge commit with its first parent: the actual target revision tested by that merge. The PR event's base SHA can become stale when the target advances. Scoping requires a matching GitHub merge ref and SHA, exactly two merge parents, and a usable target parent; otherwise every check runs. This works with the existing depth-two checkout without fetching older ancestry. Main/staging pushes execute all checks; the parent `ci` workflow skips checks on `dev` pushes, while PRs targeting `dev` still run checks. A manual `checks` run executes all checks unless supplied a full base commit SHA to reproduce a scoped plan. Missing history, an empty diff, unknown files outside the workspace graph, root script/CI changes, or a planner error run every check. Missing selection outputs fail the aggregate gate. The gate also rejects an omitted dependency, any failed/cancelled job, and any skip that its explicit selection flag did not authorize.
 
-Dedicated docs verification uses the same planner, comparing pushes with their previous commit and PRs with the tested merge's first parent. Site acceptance follows `docs#build` and its transitive package dependencies. The app's docs-consumer task depends on server runtime and that docs build: backend-only changes run the platform consumer against the docs app, while site changes also run corpus and browser checks. Independent SDK and unit-only edits skip those production jobs. The required `docs HTTP` gate verifies explicit site/platform selections before accepting any skip.
+Dedicated docs verification uses the same planner, comparing pushes with their previous commit and PRs with the tested merge's first parent. Site acceptance follows `docs#build` and its transitive package dependencies. The app's docs-consumer task depends on server runtime and that docs build: backend-only changes run the platform consumer against the docs app, while site changes also run corpus and browser checks. Standalone package and unit-only edits skip those production jobs. The required `docs HTTP` gate verifies explicit site/platform selections before accepting any skip.
 
 Docs production builds run once and publish an artifact to isolated HTTP, corpus, Chromium, and WebKit jobs. Source checks keep their own checkout because generator verification mutates inputs. Content caches use separate lint/source/build namespaces; generation validates every restore, repairs missing declarations, and removes obsolete modules. The Next build cache keeps its supported incremental behavior. Content and Next caches have size limits, and Next cache files expire after seven days; successful content eviction happens after all consumers finish.
 

.github/CONTRIBUTING.md

@@ -152,7 +152,8 @@ It provisions the database, generates secrets, writes `.env`, and offers the sou
 
 #### Requirements
 
-- Docker must be installed and running on your machine
+- [Bun](https://bun.sh/) for `bun run sim-setup` inside the clone, or Node.js 20+ for `npx sim-setup`
+- Docker with Docker Compose, installed and running
 
 ### Option 2: Using Docker Compose
 

apps/sim/tools/firecrawl/crawl.test.ts

@@ -82,6 +82,12 @@ describe('Firecrawl legacy crawl depth', () => {
     expect(accepts('/guides/auth/tokens/rotate')).toBe(false)
   })
 
+  it('measures depth from a scheme-less start URL the same way', () => {
+    const body = crawlBody({ url: 'docs.example.com/guides/auth', maxDepth: 0 })
+    const patterns = (body.includePaths as string[]).map((pattern) => new RegExp(pattern))
+    expect(patterns.some((pattern) => pattern.test('/guides/auth'))).toBe(true)
+  })
+
   it('keeps a deep start URL when the caller also sets includePaths', () => {
     const body = crawlBody({
       url: 'https://example.com/docs/guides/auth/index.html',

apps/sim/tools/firecrawl/crawl.ts

@@ -22,9 +22,12 @@ const INDEX_FILE_PATTERN = /^index\.(html|php)$/
  * start URL, and Firecrawl drops the start URL itself when it fails `includePaths`.
  */
 function startPathDepth(url: string): number {
+  const trimmed = url.trim()
   let pathname: string
   try {
-    pathname = new URL(url).pathname
+    // Firecrawl accepts a bare host like `docs.example.com/guides`, so parse it as https.
+    pathname = new URL(/^[a-z][a-z\d+.-]*:\/\//i.test(trimmed) ? trimmed : `https://${trimmed}`)
+      .pathname
   } catch {
     return 0
   }