GitHub - agrawal-2005/Cortex: Extract how your company actually works. Turn it into workflows AI agents can run.

cortex

The AI that knows how your company actually works. Cortex turns your team's knowledge into workflows your agents can run.

Cortex ingests a company's raw operational exhaust (Slack, Jira, Confluence, GitHub, Discord, file uploads), clusters it by topic, extracts each recurring topic into a structured Skill — a typed, machine-readable description of a process with real tool bindings, approval gates, and success checks — and exposes those skills through an API any AI agent can consume. A safety-enforcing execution guard sits between a skill and any real-world action, refusing to run anything incomplete and blocking anything gated until a human approves it.

Cortex is deliberately not a search tool or a chatbot over documents — a structured skill is returned only when the system is genuinely confident it applies; otherwise it says so and falls back to citing relevant source material.

"If we want every company to run on AI automation, we need a new primitive: a company brain." — Tom Blomfield, YC Partner (RFS Summer 2026)

The Problem

Every company's real operating knowledge — how refunds get handled, how pricing exceptions are decided, how engineers respond to incidents — lives scattered across Slack threads, Jira tickets, Confluence pages, and people's heads. Humans navigate this by vaguely remembering where things are. AI agents can't.

The blocker to AI automation is no longer model quality — it's the absence of a structured "company brain" that pulls knowledge out of fragmented sources, structures it, keeps it current, and turns it into an executable skills file an AI agent can actually run.

Existing tools help humans find information. Cortex makes company knowledge executable by machines.

	Glean / Notion AI	Cortex
Output	Natural language paragraphs	Structured, executable JSON with typed tool bindings
Consumer	Humans reading answers	AI agents executing workflows
Knowledge source	Retrieves existing docs	Synthesizes from fragments across tools
Safety	None — text answers have no guardrails	Execution guard with approval gates, completeness checks, and grounding validation
Honesty	Returns the best match, right or wrong	Returns nothing rather than a wrong answer; falls back to citing source material

Proof Points

These are verified in the codebase, not claims in a pitch deck.

Claim	Evidence
Extraction is accurate	Golden-set evaluation: 34/39 checks passed (87%) across 5 hand-written golden skills scored against real LLM extractions
Extraction is honest about its own gaps	`automation_readiness.level` is code-enforced, not just LLM self-reported — a step lacking a real tool binding or success check is force-downgraded from `"executable"` to `"assisted"`
The system doesn't hallucinate confidently	Regression-tested: querying "GDPR deletion request" against a dataset where GDPR content got absorbed into an unrelated cluster correctly returns no skill plus relevant citations, instead of a wrong answer
Skills are actually executable, not just structured JSON	A working execution guard runs a real extracted skill step by step — blocking at a real approval gate, resuming after approval, completing with typed success verification
Reliability	514 passing automated tests; 38 of 38 architecture-hardening items closed with code-level verification

Honest Gaps

Single-tenant by design — safe for one pilot at a time; multi-tenancy is a known, unimplemented upgrade path, deliberately deferred until a second customer exists.
No live/incremental sync yet — ingestion is export-upload plus two live connectors (GitHub, Discord); Slack/Jira/Confluence require re-upload to refresh.
Clustering granularity — HDBSCAN can split one real workflow across two topic clusters. Mitigated at query time via a skill-level relevance gate so a clustering miss degrades to an honest "I don't know," not a wrong answer.
Fernet key rotation infrastructure — single static encryption key today; rotation isn't built.

How It Works

┌──────────────┐     ┌───────────────┐     ┌─────────────┐
│  Company's   │     │               │     │  AI Agents  │
│  scattered   │ ──→ │    Cortex     │ ──→ │  execute    │
│  knowledge   │     │               │     │  reliably   │
│              │     │  Ingests      │     │             │
│  Slack       │     │  Embeds       │     │  Support    │
│  Jira        │     │  Clusters     │     │  Ops        │
│  Confluence  │     │  Extracts     │     │  Sales      │
│  GitHub      │     │  Validates    │     │  Eng        │
│  Discord     │     │  Guards       │     │  Finance    │
│  Files       │     │  Learns       │     │  HR         │
└──────────────┘     └───────────────┘     └─────────────┘

Input: Fragmented knowledge across company tools.

Output: Structured executable skills like:

{
  "skill": "handle_refund_request",
  "confidence": 0.71,
  "automation_readiness": {
    "level": "assisted",
    "safe_to_automate": false,
    "missing_for_automation": ["approval gate required for amounts > $5,000"],
    "requires_human_review": ["Account manager approval"]
  },
  "steps": [
    {
      "action": "Verify refund eligibility",
      "tool_call": {
        "type": "api",
        "name": "Stripe",
        "endpoint": "GET /v1/charges/{{input.charge_id}}",
        "auth_ref": "STRIPE_SECRET_KEY"
      },
      "parameters": { "charge_id": "{{input.charge_id}}" },
      "outputs": { "charge_status": "whether the charge succeeded" },
      "success_check": { "type": "output_equals", "field": "charge_status", "expected": "succeeded" },
      "on_failure": { "action": "abort" },
      "approval_gate": null,
      "sources": [
        { "type": "confluence", "author": "Mike Torres", "link": "confluence://..." },
        { "type": "slack", "author": "Sarah Chen", "link": "slack://..." }
      ]
    },
    {
      "action": "Obtain account manager approval",
      "tool_call": { "type": "manual", "name": "Account manager approval" },
      "approval_gate": { "if": "amount > 5000", "require": "account_manager" }
    }
  ]
}

Every step must cite its sources — uncited steps are rejected at extraction time. AI agents consume the JSON. Humans read a plain-English version on the dashboard.

Architecture

flowchart TD
    subgraph Sources["Data Sources"]
        A1[Slack export]
        A2[Jira export]
        A3[Confluence export]
        A4[GitHub export + live API]
        A5[Discord export + live bot]
        A6[CSV / JSON upload]
    end
    Sources --> ING["Ingestion Layer\n(dedup, noise filter, streaming size cap)"]
    ING --> DOC[("Document table\n(Postgres)")]
    DOC --> EMB["Embedding Service\n(MiniLM, unit-normalized)"]
    EMB --> VEC[("ChromaDB\nvector store")]
    DOC --> CLU["HDBSCAN Clustering\n(cosine metric)"]
    CLU --> PEND[("Pending clusters")]
    CLU --> TOPN["Top-N clusters\n(pre-extraction)"]
    TOPN --> LLM["LLM Extraction\n(Groq, two-tier models)"]
    PEND -. "on-demand at query time" .-> LLM
    LLM --> VAL["Validation & Safety Layer\n- schema validator\n- command grounding\n- safety validator\n- automation-level enforcement"]
    VAL --> SKILL[("Skill table\n(Postgres, v2 schema)")]
    Q["User / Agent Question"] --> QSYS["Query System\n(hybrid search + relevance gate)"]
    VEC --> QSYS
    DOC --> QSYS
    SKILL --> QSYS
    QSYS -->|"confident skill match"| RESP1["Structured skill response"]
    QSYS -->|"no confident match"| RESP2["Composed answer + citations"]
    SKILL --> GUARD["Execution Guard\n(validate → resolve → approval gate → invoke → verify)"]
    GUARD --> AGENT["Consuming AI Agent /\nreal-world tool"]

Layer	Responsibility
Ingestion	Normalize 6 source types into a common `Document` row; dedup, filter noise, cap upload size
Embedding	Turn document text into a 384-dim meaning vector (unit-normalized, cosine-comparable)
Clustering	Group documents by topic using HDBSCAN; separate "extract now" (top-N) from "pending" (extract on demand)
LLM Extraction	Turn a document cluster into a structured Skill via a two-tier Groq model strategy
Validation/Safety	Enforce that a skill's self-reported automation-readiness is actually backed by real fields — never trust the LLM's word alone
Query System	Hybrid (embedding + keyword) search, with a skill-level relevance gate that prevents a marginal match from being presented as a confident answer
Execution Guard	The runtime contract between a skill's JSON and anything that actually executes it

Tech Stack

Component	Technology
Backend	FastAPI (Python 3.12), async throughout
LLM	Groq (llama-3.3-70b-versatile + llama-3.1-8b-instant) — HuggingFace and Ollama as alternatives
Embeddings	sentence-transformers/all-MiniLM-L6-v2 (local, 384-dim)
Vector DB	ChromaDB (local)
Database	PostgreSQL 16
Search	Postgres full-text search (tsvector/GIN) + ChromaDB embeddings
Frontend	React + Tailwind CSS
LLM Framework	LangChain
Clustering	HDBSCAN (cosine metric) with boilerplate stripping
Security	API keys (SHA-256), Fernet encryption, Redis sliding-window rate limiting
CI/CD	GitHub Actions (514 tests + frontend & website builds)

Data Ingestion

What becomes a "Document"

Each source has one natural unit — never word-by-word or line-by-line:

Source	One row =
Slack	One standalone message, or one whole thread (replies grouped)
Jira	One whole issue, with all comments concatenated
Confluence	One whole page
GitHub	One issue, PR, discussion, or doc file (comments appended)
Discord	One message
CSV / JSON upload	One row / one object

Every row carries: id, content, source_type, source_id, source_link, source_label, channel_or_project, author_name, created_at, embedding_id, low_signal.

Ingestion protections

Deduplication: UniqueConstraint("source_type", "source_id") at the database level, paired with a begin_nested() savepoint + IntegrityError catch that falls back to row-by-row insert — re-uploading the same export skips existing records instead of duplicating or crashing.
Noise filtering (is_low_signal): short acknowledgment-pattern messages ("ok", "thanks", "+1", emoji-only) are still inserted (so document counts stay honest) but excluded from embedding and clustering.
Streaming upload cap (_read_upload_capped): reads uploads in 64KB chunks and rejects with 413 as soon as the configured limit is exceeded — never buffers the whole file into memory.
Zip Slip protection: Slack export ZIP extraction validates every archive member's path before extracting.
Attachment extraction: PDFs (pdfplumber), DOCX (python-docx), CSVs (pandas), and images (pytesseract OCR) — all wrapped in asyncio.to_thread so CPU/IO-bound work doesn't block the event loop.
Async, non-blocking: every ingest route responds immediately with a task ID via asyncio.create_task.

Database Design

Cortex uses two coordinated stores: Postgres for structured, relational data, and ChromaDB for vector similarity search.

Entity-relationship diagram

erDiagram
    DOCUMENT ||--o{ SKILL_DOCUMENT : "cited by"
    SKILL ||--o{ SKILL_DOCUMENT : "cites"
    SKILL ||--o{ SKILL_STEP : "has"
    SKILL_STEP ||--o{ STEP_SOURCE : "cites"
    DOCUMENT ||--o{ STEP_SOURCE : "is cited in"
    SKILL ||--o{ FEEDBACK : "receives"
    CONNECTED_SOURCE ||--o{ DOCUMENT : "produces"
    SOURCE_TRUST }o--|| CONNECTED_SOURCE : "scores"
    CONVERSATION_TURN }o--o| SKILL : "matched (nullable)"
    DOCUMENT {
        uuid id PK
        text content
        string source_type
        string source_id
        string source_link
        string source_label
        string channel_or_project
        string author_name
        timestamp created_at
        string embedding_id FK
        bool low_signal
    }
    SKILL {
        uuid id PK
        string name
        text description
        string department
        float confidence
        json automation_readiness
        bool is_repeatable
        int version
        json embedding
        string status
    }
    SKILL_STEP {
        uuid id PK
        uuid skill_id FK
        int step_order
        text action
        json details
        float confidence
        json depends_on
        string status
    }
    SKILL_DOCUMENT {
        uuid skill_id FK
        uuid document_id FK
        string cluster_provenance
    }
    STEP_SOURCE {
        uuid step_id FK
        uuid document_id FK
        text quoted_snippet
    }
    FEEDBACK {
        uuid id PK
        uuid skill_id FK
        string action
        timestamp created_at
    }
    SOURCE_TRUST {
        string source_type PK
        float trust_score
    }
    PENDING_CLUSTER {
        uuid id PK
        string cluster_label
        json document_ids
        string status
    }
    CONVERSATION_TURN {
        uuid id PK
        uuid conversation_id
        text question
        text rewritten_question
        uuid matched_skill_id
        timestamp created_at
    }
    API_KEY {
        uuid id PK
        string key_hash
        timestamp created_at
    }
    CONNECTED_SOURCE {
        uuid id PK
        string source_type
        string encrypted_token
        timestamp last_synced_at
    }

Indexing strategy

Index	Table / Column	Purpose
`UNIQUE(source_type, source_id)`	`documents`	Prevents duplicate ingestion, race-condition-safe
GIN index on `tsvector`	`documents`, `skills`	Powers Postgres full-text keyword search
B-tree on `embedding_id`	`documents`	Avoids full table scan when finding not-yet-embedded documents
B-tree on `document_id`	`skill_documents`	Fast join for "which skill(s) cite this document"
Composite `(conversation_id, created_at DESC)`	`conversation_turns`	Fast retrieval of a conversation's recent turns

How Postgres and ChromaDB stay connected

Chroma and Postgres writes are coordinated explicitly: after upserting a vector into ChromaDB, the application flushes the corresponding Postgres row — if Postgres fails, the just-upserted ChromaDB vectors are deleted to prevent orphans. A startup reconciliation sweep cross-checks ChromaDB vector IDs against live Postgres document IDs, removing any drift from a hard crash mid-write.

Skill embeddings are cached directly as a JSON column on the Skill row (computed once at extraction, reused on every query) rather than stored in a separate Chroma collection — avoiding redundant re-embedding of skill name/description on every single query.

Embedding & Clustering

Embedding

Every document (and skill name/description) is converted to a 384-dimension vector using sentence-transformers/all-MiniLM-L6-v2, explicitly unit-normalized (normalize_embeddings=True), with a startup assertion verifying ‖v‖ ≈ 1.0 — so a future model swap that doesn't normalize internally fails loudly at boot instead of silently miscalibrating every relevance score. Unit normalization matters because the downstream relevance-scoring formula (relevance = 1 − distance/2) is only mathematically valid for unit vectors.

Clustering

HDBSCAN groups documents by topic:

Metric: cosine (not euclidean — the self-documenting, correct choice for semantic similarity).
min_cluster_size = 3 — fewer than 3 related documents cannot form a cluster; they become "noise" and are reassigned to the nearest existing cluster by cosine similarity if above a threshold, or left unclustered otherwise.
Known limitation: a genuinely rare topic (e.g., only 2 GDPR-related documents) can never form its own cluster. Mitigated at query time (see Query System), not by lowering min_cluster_size.

Lazy extraction

Not every cluster is extracted into a skill immediately — that would be expensive and would burn through LLM rate limits on topics nobody ever asks about.

The top N clusters (ranked by size) are extracted immediately after ingestion.
Every other cluster becomes a cheap, LLM-free pending cluster — just metadata.
A pending cluster is only extracted on demand, the first time a user's question is relevant enough.
Extraction across clusters is parallelized (bounded by asyncio.Semaphore(3)), with each parallel task using its own isolated database session — never sharing one AsyncSession across concurrent coroutines.

LLM Extraction Pipeline

Two-tier model strategy

Tier	Model	Used for	Why
Bulk	Fast/cheap Groq model	Pre-extraction of top-N clusters at ingestion time	Cost-sensitive, runs automatically, high volume
Live	Larger Groq model (128k context)	On-demand extraction when a query needs a pending cluster	Quality matters more — only fires on a cache miss, so cost is naturally bounded

Prompt construction

Documents feeding a single extraction prompt are scored by _doc_quality_score (recency, evidence-type authority — Jira/GitHub outranking casual Slack — source trust, author seniority, content substance) and greedily packed by descending quality into a fixed token budget, leaving headroom under Groq's per-request rate cap. The LLM call uses response_format={"type": "json_object"} (native structured-output enforcement).

Validation and safety layers (applied after every extraction)

flowchart LR
    LLM["LLM raw output"] --> V1["Schema validation\n(steps ≥3, every step cited)"]
    V1 --> V2["Command-level grounding\n(does the claimed tool/endpoint\nactually appear in cited source?)"]
    V2 --> V3["Safety validator\n(risky-pattern tools force\nsafe_to_automate=false\nunless approval_gate exists)"]
    V3 --> V4["Automation-level enforcement\n(level forced down if any step\nlacks real tool_call/success_check)"]
    V4 --> V5["Repeatability filter\n(rejects one-time projects)"]
    V5 --> V6["depends_on wiring\n(linear chain default,\nbranch-goto override,\nself-loop guard)"]
    V6 --> PERSIST[("Persisted Skill")]

Schema validation: rejects extractions with fewer than 3 steps, or any step lacking a cited source document.
Command-level grounding: checks whether the LLM's claimed tool name/endpoint actually appears in the step's cited source content — if not, the step is downgraded.
Safety validator: independent of the LLM's judgment — any step whose action matches a risky pattern (refund, payment, delete, deploy, rollback, etc.) forces safe_to_automate: false unless a real approval_gate exists. The validator's decision overrides the LLM's self-report.
Automation-level enforcement: a skill can't claim "level": "executable" while any step lacks a genuine tool_call or success_check — the level is force-downgraded to "assisted".
Repeatability filter: one-time projects get rejected, not mislabeled as skills.
depends_on wiring: each step's predecessor UUID is populated by default (linear chain); a branch.then: "goto step N" overrides this — with a guard preventing self-referential loops.

Confidence scoring

Signal	Max weight
Base (successfully extracted)	0.40
Recency of cited source	0.15
Author authority (seniority)	0.15
Source trust (historical reliability)	0.15
Evidence type (Jira/GitHub outrank Slack)	0.10
Corroboration (multiple sources agreeing)	0.15

This measures sourcing quality, separate from automation_readiness (which measures executability) — a step can be well-sourced but still not executable, and vice versa.

Skill Schema (v2)

Full annotated example

{
  "schema_version": "2.0",
  "skill_id": "uuid",
  "name": "Handle a refund request",
  "description": "Process a customer refund when requested via support ticket.",
  "department": "support",
  "confidence": 0.71,
  "trigger": { "type": "manual", "condition": "Customer requests a refund" },
  "inputs_schema": { "charge_id": "Stripe charge ID", "amount": "refund amount in USD" },
  "automation_readiness": {
    "level": "assisted",
    "safe_to_automate": false,
    "missing_for_automation": ["approval gate required for amounts > $5,000"],
    "requires_human_review": ["Account manager approval"]
  },
  "is_repeatable": true,
  "execution_plan": [
    {
      "step_id": "uuid",
      "order": 1,
      "action": "Verify refund eligibility",
      "depends_on": [],
      "tool_call": {
        "type": "api",
        "name": "Stripe",
        "endpoint": "GET /v1/charges/{{input.charge_id}}",
        "auth_ref": "STRIPE_SECRET_KEY"
      },
      "parameters": { "charge_id": "{{input.charge_id}}" },
      "outputs": { "charge_status": "whether the charge succeeded" },
      "success_criteria": "Charge status is 'succeeded' and within 30 days",
      "success_check": { "type": "output_equals", "field": "charge_status", "expected": "succeeded" },
      "on_failure": { "action": "abort", "then": null },
      "approval_gate": null
    },
    {
      "step_id": "uuid",
      "order": 3,
      "action": "Obtain account manager approval",
      "depends_on": ["<step-1-uuid>"],
      "tool_call": { "type": "manual", "name": "Account manager approval" },
      "approval_gate": { "if": "amount > 5000", "require": "account_manager" }
    }
  ]
}

Field reference

Field	Type	Purpose
`tool_call`	`{type, name, endpoint, auth_ref}`	A bound, structured invocation target — not prose. `auth_ref` is resolved to a real credential at execution time
`parameters`	dict of template strings	`{{input.x}}` (run-time inputs) or `{{steps.<id>.output.key}}` (prior step's result)
`outputs`	dict	What this step produces for later steps to reference
`success_check`	`{type, field, expected}`	A typed, programmatically-evaluable success condition
`on_failure`	`{action, then}`	A bounded action (`abort`/`retry`/`escalate`), not free text
`approval_gate`	`{if, require}`	A condition and a named approver role — enforced by the execution guard
`depends_on`	list of step UUIDs	Real prerequisite steps — enables building an execution graph
`automation_readiness`	see above	Code-enforced, not merely LLM self-reported

Backward compatibility

A schema_compat.py normalization layer (normalize_step_details) dual-reads older-shape data (tool + command) and converts it to the v2 shape (tool_call) on read — no database backfill required.

Execution Guard

The runtime contract between a skill's JSON and any real-world action. Lives in backend/execution/guard.py.

flowchart TD
    STEP["Skill step (v2 schema)"] --> V["1. validate_step\nReject if tool_call or\nsuccess_criteria/success_check missing"]
    V -->|invalid| REJECTED["REJECTED\n(never invoked)"]
    V -->|valid| R["2. resolve_templates\n{{input.x}} and\n{{steps.ID.output.key}}"]
    R --> RCHECK{"Unresolved template\nremains?"}
    RCHECK -->|yes| REJECTED2["REJECTED\n(garbage input never invoked)"]
    RCHECK -->|no| G["3. check_approval_gate"]
    G -->|"gate fires,\nno callback or denied"| BLOCKED["BLOCKED\n(safe by default)"]
    G -->|"no gate, or approved"| INV["4. invoke_tool\n(caller-supplied callback)"]
    INV --> HRC["5. requires_human_review check"]
    HRC --> SC["6. check_success\nmanual → passes\noutput_equals → typed comparison"]
    SC -->|unresolved template| WARN["Passes, marked UNVERIFIED"]
    SC -->|fails| FAILED["FAILED → on_failure handling"]
    SC -->|passes| BIND["7. bind_outputs"]
    BIND --> SUCCESS["SUCCESS"]

Design principles:

invoke_tool is always a caller-supplied callback — the guard itself never calls a real external system.
No approval callback = blocked by default — safe-by-construction.
Pre-invocation vs. post-invocation unresolved templates are handled differently: an unresolved template in parameters/endpoint means garbage input — rejected before it runs. An unresolved template in success_check.expected means the tool ran successfully but can't be programmatically verified — passed through but visibly marked unverified.
requires_human_review is enforced, not just displayed — steps matching this field are blocked until a human approves.

Query System

Hybrid retrieval

Natural language queries use hybrid retrieval — embedding search (ChromaDB) + Postgres full-text search (tsvector/GIN indexes) — with scores merged (max/union).

Why hybrid, not embeddings alone

Pure semantic search can miss exact terms — acronyms, ticket numbers, rare proper nouns — that a keyword match catches immediately. Postgres FTS runs in parallel with embedding search so literal rare terms (e.g., "GDPR") are correctly weighted even when embedding similarity alone under-ranked them.

The relevance gate

Three independently-tunable thresholds:

Threshold	Gates
`SKILL_MATCH_THRESHOLD`	Whether a matched document's skill gets returned as a structured answer
`DOCUMENT_CITATION_THRESHOLD`	Whether a document is relevant enough to include in a citation fallback
`EXTRACTION_WORTHY_THRESHOLD`	Whether a pending cluster is relevant enough to trigger on-demand extraction

On top of document-level matching, a skill-level re-rank independently compares the query's embedding directly against the candidate skill's own name+description embedding. If a document matched well but its skill doesn't relate to the query (e.g., a clustering-misplaced document), the skill is rejected and the system falls back to citing the document honestly.

Conversation memory

A conversation_id persists across turns. Follow-up questions are rewritten into standalone form using the last two turns before re-entering retrieval — e.g., "what about over $10k?" becomes "how do we handle refunds over $10,000?"

API Reference

All endpoints (except /health) require an X-API-Key header and are rate-limited (Redis sliding-window, in-memory fallback).

Ingestion

Method	Path	Body	Response
`POST`	`/api/ingest/slack`	Slack export ZIP (streamed)	`202 {task_id}`
`POST`	`/api/ingest/file`	CSV/JSON file	`202 {task_id}`
`POST`	`/api/ingest/jira`	Jira export JSON	`202 {task_id}`
`POST`	`/api/ingest/confluence`	Confluence export JSON	`202 {task_id}`
`POST`	`/api/ingest/github`	GitHub export JSON	`202 {task_id}`
`POST`	`/api/ingest/github/live`	`{repo, github_token_ref}`	`202 {task_id}` — live API pull
`POST`	`/api/ingest/discord`	Discord export JSON	`202 {task_id}`
`GET`	`/api/ingest/status/{task_id}`	—	`{status, progress, result}`

Query

Method	Path	Body	Response
`POST`	`/api/query`	`{question, conversation_id?}`	`{skill

Skills

Method	Path	Response
`GET`	`/api/skills`	List, filterable by status/department/confidence
`GET`	`/api/skills/{id}`	Full skill detail
`GET`	`/api/skills/{id}/executable`	Machine-readable v2 schema — the artifact an agent consumes
`GET`	`/api/skills/search?q=`	Keyword search across skills
`DELETE`	`/api/skills/{id}`	Removes skill + Chroma vectors; flags cluster to prevent regeneration

Feedback, Sources, Data, Workspace

Method	Path	Purpose
`POST`	`/api/feedback`	Record approve/edit/reject on a skill; updates source trust
`GET`	`/api/sources`	List connected/available data source connectors
`GET`	`/api/data-overview`	Full account of ingested data (paginated counts, sample content)
`DELETE`	`/api/workspace/data`	Hard-delete everything (documents, skills, embeddings, tokens); post-deletion verified
`GET`	`/health`	Unauthenticated liveness probe

Processing

Method	Path	Purpose
`POST`	`/api/v1/processing/cluster`	Run document clustering
`POST`	`/api/v1/processing/lazy-extract`	Cluster all docs, extract top clusters, store rest as pending (409 if already in progress)

Security & Data Trust

Encryption: connected-source tokens encrypted at rest (Fernet).
Secret redaction: error messages and logs are scrubbed of known key patterns before being surfaced.
Rate limiting: Redis sliding-window limiter (sorted sets), per-key, with graceful in-memory fallback.
Upload validation: extension allow-list, streaming size cap, Zip Slip path-traversal protection.
Search input sanitization: %/_ wildcard characters escaped in ILIKE queries.
Data deletion: workspace-level hard delete removes Postgres rows and Chroma vectors together, with post-deletion verification — not a soft flag.
Single-tenant isolation: one deployment per pilot customer; no cross-tenant data path exists.
API key authentication on every route (SHA-256 hashed, shown once at creation).
CORS restriction (localhost in dev, explicit origins in production).

Testing & Quality

514 automated tests passing, covering:

Category	Tests	What's Covered
Execution guard	98	Step completeness, approval gates, template resolution, human-review enforcement, schema compat
LLM failure modes	50	All 10 failure types: retries, JSON repair, graceful stop
Skill extraction	34	Core pipeline, quality gates, safety validator, JSON parsing
Conversation rewriting	24	Follow-up rewrite, standalone pass-through, LLM fallback, turn persistence
Discord ingester	22	Export parsing, reply chains, live bot ingestion
Security	22	Auth, rate limits, encryption, CORS, validation, token hygiene
Slack ingester	20	Export parsing, threading, idempotent re-ingest
Lazy extraction	21	Pre-extract, pending topics, on-demand, concurrency guard, parallel session isolation
Schemas	18	Request/response validation
JSON export ingesters	17	Jira, Confluence, GitHub export parsing + dedup
Confidence scoring	15	Recency, authority, trust weighting
API integration	15	All endpoints, error responses
Query matching	12	Cluster provenance, semantic ranking, skill relevance gate, FTS, re-rank
Skills API	8	CRUD, executable JSON
Edge-case audit	7	Empty files, duplicate exports, tiny clusters

Golden-set evaluation

5 hand-written "correct" skills scored against real LLM extractions — 34/39 checks passed (87%):

Golden skill	Score	Notes
Refund Processing	8/8 (100%)	Cross-source citations from Confluence, Slack, Jira
Incident Response	8/10 (80%)	Step coverage strong; one phrasing mismatch
Production Deployment	pass	Deploy-to-prod triggers safety validator correctly
New Hire Onboarding	partial	HDBSCAN split onboarding docs across clusters
Code Review	pass after re-run	Initial cluster match picked wrong cluster; correct cluster passes

Performance benchmarks

Operation	Time	Threshold
Ingest 100 docs	2.0s (20ms/doc)	< 500ms/doc
Embed 100 docs	1.2s (12ms/doc)	< 200ms/doc
Embed batch speedup	7.3x vs single	—
Cluster 400 docs	0.14s	< 30s
Extract 1 skill (LLM)	8.4s	< 60s
Query response	32ms (p95: 60ms)	< 2000ms

# Run all tests
.venv/bin/python -m pytest tests/ -q --ignore=tests/stress_test.py --ignore=tests/benchmark.py

# Run stress test (requires running server)
PYTHONPATH=. .venv/bin/python tests/stress_test.py

# Run performance benchmark
PYTHONPATH=. .venv/bin/python tests/benchmark.py

Quick Start

Prerequisites

Docker and Docker Compose
Python 3.12+
Node.js 20+
A Groq API key (free tier works) or HuggingFace account or Ollama (local, free)

Setup

# Clone the repository
git clone https://github.com/agrawal-2005/Cortex.git
cd Cortex

# Copy environment template and configure
cp .env.example .env
# Edit .env — set at minimum:
#   LLM_PROVIDER=groq
#   GROQ_API_KEY=gsk_...          (free at console.groq.com)
#   TOKEN_ENCRYPTION_KEY          (generate with command below)

# Generate encryption key
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

# Start infrastructure (PostgreSQL + Redis)
docker-compose up -d

# Install backend dependencies
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Run database migrations
alembic upgrade head

# Create your first API key
PYTHONPATH=. .venv/bin/python scripts/create_api_key.py "dev-key"
# Save the printed key — it's shown only once

# Start the backend
uvicorn backend.main:app --reload --port 8000

# In another terminal — start the frontend
cd frontend
npm install
npm run dev

Open http://localhost:3000 — you'll see the Cortex dashboard.

LLM Backends

Groq (default): fast, generous free tier. Cortex uses two models:

GROQ_MODEL=llama-3.1-8b-instant — bulk extraction after ingestion
GROQ_LIVE_MODEL=llama-3.3-70b-versatile — on-demand extraction at query time

HuggingFace: set LLM_PROVIDER=huggingface and HUGGINGFACE_API_TOKEN.

Ollama (local, unlimited, free):

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1

# In .env
LLM_PROVIDER=ollama
OLLAMA_MODEL=llama3.1
OLLAMA_BASE_URL=http://localhost:11434

Ingest Your First Data

Option A — GitHub repo (easiest):

Go to Data Sources → GitHub → "Connect"
Enter any public repo (e.g., fastapi/fastapi)
Click "Ingest" — documents appear within minutes

Option B — Slack export:

Export your Slack workspace (Settings → Import/Export)
Go to Data Sources → Slack → "Upload Export"
Upload the zip file

Option C — Jira / Confluence / GitHub / Discord JSON export:

Go to Data Sources → pick the source → "Upload Export"
Upload the JSON export file (see data/synthetic/ for expected shapes)

Option D — File upload:

Go to Data Sources → File Upload
Drag and drop any CSV or JSON file

Skill extraction runs automatically after every ingestion (top clusters up front, the rest on demand at query time).

Project Structure

Cortex/
├── .github/workflows/ci.yml     # GitHub Actions CI
├── backend/
│   ├── main.py                   # FastAPI app with auth + rate limiting
│   ├── config.py                 # Environment settings (Groq/HF/Ollama)
│   ├── database.py               # Async SQLAlchemy setup
│   ├── security/                 # Security layer
│   │   ├── auth.py               # API key authentication (SHA-256)
│   │   ├── crypto.py             # Fernet token encryption
│   │   ├── ratelimit.py          # Redis sliding-window rate limits
│   │   └── validation.py         # Input validation + secret hygiene
│   ├── ingestion/                # Source connectors
│   │   ├── slack_ingester.py
│   │   ├── github_ingester.py    # Live GitHub API
│   │   ├── github_json_ingester.py
│   │   ├── jira_json_ingester.py
│   │   ├── confluence_json_ingester.py
│   │   ├── discord_ingester.py
│   │   └── file_upload_ingester.py
│   ├── processing/               # Knowledge extraction
│   │   ├── embedder.py           # Chroma/PG coordination
│   │   ├── embeddings.py         # MiniLM embedding service
│   │   ├── clustering.py         # HDBSCAN (cosine)
│   │   ├── skill_extractor.py    # LLM extraction + quality gates
│   │   ├── lazy_extraction.py    # Parallel lazy extraction
│   │   ├── query_rewriter.py     # Multi-turn conversation rewriting
│   │   ├── renderer.py           # Skill rendering
│   │   └── prompts/
│   │       └── extraction.py     # Extraction prompt templates
│   ├── execution/                # Agent execution layer
│   │   ├── guard.py              # Execution guard (7-stage pipeline)
│   │   └── schema_compat.py      # V1→V2 step normalization
│   ├── knowledge/
│   │   └── models.py             # 11 SQLAlchemy models
│   └── api/
│       ├── routes_ingest.py      # Async ingestion + JSON exports
│       ├── routes_skills.py      # Skill CRUD + /executable
│       ├── routes_query.py       # Hybrid retrieval, relevance gates
│       ├── routes_feedback.py    # Feedback with trust updates
│       ├── routes_sources.py     # Encrypted source management
│       ├── routes_workspace.py   # Verified cascading hard delete
│       └── routes_data_overview.py
├── frontend/                     # React + Tailwind dashboard
├── website/                      # Marketing site (Vite + React + Framer Motion)
├── data/synthetic/               # AcmeTech synthetic dataset
├── scripts/
│   ├── create_api_key.py         # Mint API keys
│   ├── acmetech_pipeline.py      # End-to-end synthetic dataset pipeline
│   └── golden_set_eval.py        # Golden-set quality evaluation
├── tests/                        # 514 automated tests
├── alembic/                      # Database migrations
├── LICENSE                       # MIT
└── docs/
    └── assets/                   # Brand mark + wordmark SVGs

Known Limitations & Roadmap

Limitation	Status	Path forward
Single-tenant only	Deliberate	Add `organization_id` scoping once a second pilot customer exists
No live/incremental sync for Slack/Jira/Confluence	Deliberate	Webhook-based ingestion + `last_synced_timestamp`, deferred until extraction is proven on a real pilot
Clustering can split or misplace a workflow	Diagnosed, mitigated	Improves with more real data; query-time relevance gate prevents wrong answers
No document-chunk-level citation	Not built	Would improve precision on very long documents; current whole-document citation is adequate at typical message/ticket size
Fernet key rotation infrastructure	Not built	Requires versioned-key + re-encryption tooling
Time-range skill versioning	Not built	Handled implicitly via recency-weighted confidence; explicit versioning would make history queryable
Live tool execution	Guard built, callback mocked	Wiring one real integration (e.g., Stripe test-mode) is the natural next proof point

Done

Origin Story

Built by Prashant Agrawal — inspired by manually extracting a 4+ hour client onboarding process from scattered Slack threads and tribal knowledge at Locus.sh, then automating it into a single API call.

Every company has hundreds of processes like this. Cortex automates the extraction.

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 31 Commits
.github/workflows		.github/workflows
alembic		alembic
backend		backend
data/synthetic		data/synthetic
docker		docker
docs/assets		docs/assets
frontend		frontend
graphify-out		graphify-out
scripts		scripts
tests		tests
website		website
.env.example		.env.example
.gitignore		.gitignore
ARCHITECTURE_FIXES.md		ARCHITECTURE_FIXES.md
CLAUDE.md		CLAUDE.md
LICENSE		LICENSE
Makefile		Makefile
PITCH_READINESS.md		PITCH_READINESS.md
README.md		README.md
alembic.ini		alembic.ini
docker-compose.yml		docker-compose.yml
requirements.txt		requirements.txt

Folders and files

Latest commit

History

Repository files navigation

Table of Contents