GML-2022 Release 1.4.0 for schema-aware knowledge graph ingestion and query

[chengbiao-jin]

PR Type

Enhancement, Tests

---

Description

• Add schema-aware ingest foundation

  • Parse, diff, preview, and apply schemas
  • Persist type metadata and schema text
  • Install retrievers from live schema

• Expand LLM prompt customization

  • Add built-in fallback prompt templates
  • Support shared top-level prompt_path
  • Add schema extraction and retriever prompts

• Improve embedding store lifecycle

  • Add async background initialization
  • Cache per-graph embedding stores
  • Rebuild stores on config reload

• Strengthen observability and validation

  • Collect per-request LLM token usage
  • Skip redundant GDS installs
  • Add broad ingest and prompt tests

---

Diagram Walkthrough

flowchart LR
  a["Sample docs or pasted GSQL"] -- "extract / parse" --> b["SchemaProposal"]
  b -- "diff and apply" --> c["TigerGraph schema"]
  b -- "upsert descriptions" --> d["Type metadata"]
  c -- "render live schema" --> e["AllowedSchema"]
  e -- "install" --> f["Domain retrievers"]
  g["Prompt path + fallback prompts"] -- "guide" --> h["LLM tools"]
  i["Embedding store cache"] -- "serve per graph" --> h
  h -- "record usage" --> j["Trace logs / token tracking"]

Loading

File Walkthrough

	Relevant files
Enhancement	19 files schema_utils.py Add full schema proposal and apply pipeline                            +1872/-0 base_llm.py Add usage collection and fallback prompt templates              +454/-103 config.py Propagate prompt paths and cache embedding stores                +160/-4  tigergraph_embedding_store.py Avoid redundant GDS installs and quieter logging                  +50/-6    schema_extraction.py Add schema extraction support from sample documents            +345/-0  retriever_render.py Render and install retrievers from domain schema                  +236/-0  prompt_validation.py Add prompt validation utilities                                                    +142/-0  graphrag_stream_entity_community_pairs.gsql Add query to stream entity community pairs                              +21/-0    graphrag_delete_all_communities.gsql Add query to clear community data                                                +21/-0    graphrag_stream_all_ids.gsql Add query to stream graph identifiers                                        +17/-0    method_selector.py Add LLM-based retrieval method selector                                    +300/-0  ui.py Add UI endpoints for schema and prompts                                    +1194/-126 LLMEntityRelationshipExtractor.py Wire extractor to domain schema controls                                  +266/-141 workers.py Update ECC workers for schema-aware ingest                              +374/-66 util.py Support schema-aware ECC utility flow                                        +318/-16 graph_rag.py Integrate schema-aware graph ingestion behavior                    +165/-34 text_extractors.py Expand document extraction for CSV and Excel                          +194/-47 agent_graph.py Integrate retriever selection and trace data                          +233/-11 agent.py Capture node usage and tracing details                                      +124/-8
Tests	1 files test_schema_utils.py Add unit coverage for schema utilities                                      +1399/-0
Additional files	101 files CHANGELOG.md +88/-0    README.md +8/-0      VERSION +1/-1      graphrag_louvain_communities.gsql +2/-1      graphrag_louvain_init.gsql +2/-7      modularity.gsql +2/-1      stream_community.gsql +2/-1      SupportAI_Schema.gsql +2/-3      create_entity_type_relationships.gsql +0/-20    Content_Similarity_Search.gsql +3/-3      Content_Similarity_Vector_Search.gsql +8/-8      GraphRAG_Hybrid_Search.gsql +9/-3      GraphRAG_Hybrid_Search_Display.gsql +5/-3      GraphRAG_Hybrid_Vector_Search.gsql +5/-3      aws_bedrock_service.py +79/-1    prometheus_metrics.py +5/-0      chatbot_response.txt +0/-17    community_summarization.txt +0/-11    entity_relationship_extraction.txt +0/-24    generate_cypher.txt +0/-85    generate_function.txt +0/-27    graphrag_scoring.txt +0/-7      map_question_to_schema.txt +0/-14    generate_function.txt +0/-14    map_question_to_schema.txt +0/-19    entity_relationship_extraction.txt +0/-24    generate_function.txt +0/-29    map_question_to_schema.txt +0/-17    chatbot_response.txt +0/-29    community_summarization.txt +0/-11    entity_relationship_extraction.txt +0/-24    generate_cypher.txt +0/-85    generate_function.txt +0/-27    graphrag_scoring.txt +0/-7      map_question_to_schema.txt +0/-14    community_summarization.txt +0/-11    entity_relationship_extraction.txt +0/-24    generate_function.txt +0/-33    map_question_to_schema.txt +0/-14    chatbot_response.txt +0/-17    community_summarization.txt +0/-11    entity_relationship_extraction.txt +0/-24    generate_cypher.txt +0/-84    generate_function.txt +0/-24    graphrag_scoring.txt +0/-7      map_question_to_schema.txt +0/-15    question_expansion.txt +0/-6      generate_function.txt +0/-24    map_question_to_schema.txt +0/-15    chatbot_response.txt +0/-17    community_summarization.txt +0/-11    entity_relationship_extraction.txt +0/-24    generate_cypher.txt +0/-85    generate_function.txt +0/-27    graphrag_scoring.txt +0/-7      map_question_to_schema.txt +0/-15    question_expansion.txt +0/-6      requirements.txt +2/-0      image_data_extractor.py +44/-5    docker-compose.yml +5/-3      nginx.conf +8/-0      Dockerfile +4/-1      ecc_util.py +2/-2      eventual_consistency_checker.py +4/-18    community_summarizer.py +1/-1      main.py +59/-8    supportai_init.py +7/-5      util.py +5/-4      workers.py +16/-38  .npmrc +2/-0      Dockerfile +9/-1      package.json +10/-3    ActionProvider.tsx +28/-3    Bot.tsx +16/-6    ConfigScopeToggle.tsx +1/-1      CustomChatMessage.tsx +93/-1    Interact.tsx +58/-15  ModeToggle.tsx +32/-9    SideMenu.tsx +33/-7    alert-dialog.tsx +48/-0    tag-input.tsx +186/-0  useAlert.tsx +34/-0    useIdleTimeout.ts +13/-0    index.css +13/-1    main.tsx +6/-0      Chat.tsx +103/-4  Setup.tsx +34/-8    TraceLogs.tsx +1018/-0 CustomizePrompts.tsx +47/-41  GraphRAGConfig.tsx +308/-51 IngestGraph.tsx +122/-28 KGAdmin.tsx +1868/-55 LLMConfig.tsx +2/-2      SetupLayout.tsx +25/-3    safeJson.ts +18/-0    Dockerfile +8/-2      inquiryai.py +28/-25  supportai.py +21/-16  BaseRetriever.py +9/-3      supportai_ingest.py +11/-52  Additional files not shown

---

[tg-pr-agent]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 5 🔵🔵🔵🔵🔵

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Cache Race Per-graph embedding stores are built outside the cache lock, so concurrent first-time requests for the same graph can create multiple stores and duplicate expensive initialization or installation work before one instance is discarded. if graphname: with _embedding_stores_lock: cached = _embedding_stores.get(graphname) if cached is not None: return cached # Build outside the lock so first-time setup for one graph # doesn't serialize first-time setup for another. store = _build_embedding_store(graphname) with _embedding_stores_lock: existing = _embedding_stores.get(graphname) if existing is not None: return existing # racing thread won _embedding_stores[graphname] = store return store Atomicity Gap The schema apply flow now runs ADD and ALTER statements in separate schema-change jobs. If the first phase succeeds and the second fails, the graph is left partially updated even though the API describes the operation as atomic. # Split into two phases so TG's job-validator never sees an ALTER # referencing a vertex/edge type created elsewhere in the same # job. The ADD phase runs first; the ALTER phase (e.g. ADD PAIR # on existing edges) runs only after the ADD phase commits. add_stmts = [s for s in statements if s.lstrip().upper().startswith("ADD ")] alter_stmts = [s for s in statements if s.lstrip().upper().startswith("ALTER ")] def _run_phase(phase_stmts: List[str], phase_job: Optional[str]) -> Tuple[str, str]: block, name = build_schema_change_job(graphname, phase_stmts, phase_job) try: out = conn.gsql(block) except Exception: try: conn.gsql(f"USE GRAPH {graphname}\nDROP JOB {name}") except Exception: pass raise return out, name # The two-phase split (ADD then ALTER) is internal mechanics; the # user just sees a single "Applying domain schema" message that # spans both phases plus the brief metadata upsert. The wording # matches both schema-source paths (sample extraction and pasted # GSQL) — "extracted" would be misleading for the paste mode. _report("Applying domain schema") phase_outputs: List[str] = [] phase_jobs: List[str] = [] first_job_name: Optional[str] = None for phase_stmts in (add_stmts, alter_stmts): if not phase_stmts: continue # Only honor the caller-supplied job_name on the first phase that # actually runs; subsequent phases get auto-generated names so # they don't collide. phase_job = job_name if first_job_name is None else None output, ran_name = _run_phase(phase_stmts, phase_job) phase_outputs.append(output) phase_jobs.append(ran_name) if first_job_name is None: first_job_name = ran_name err = gsql_output_error(output) if err: try: conn.gsql(f"USE GRAPH {graphname}\nDROP JOB {ran_name}") except Exception: pass return { "status": "error", "statements": statements, "job_name": first_job_name, "job_names": phase_jobs, "gsql_output": "\n".join(phase_outputs), "error": err, "summary": summary, "metadata": {"entity_types": [], "relationship_types": []}, "retrievers": {"status": "skipped", "reason": "schema apply failed"}, } Node Typing Relationship endpoints are constructed with node type equal to the endpoint id rather than the extracted entity type, which can break downstream logic that expects relationship endpoint nodes to preserve semantic types. relationships.append(Relationship( source=Node(id=rel["source"], type=rel["source"], properties=src_props), target=Node(id=rel["target"], type=rel["target"], properties=tgt_props), type=rel["type"], properties=edge_props,