Video Analysis

A video semantic search application built on Azure Content Understanding and Azure AI Search.
The goal is to map natural-language questions directly to video timestamps and support one-click playback jump.

Workflow

The current end-to-end processing flow is:

1. Upload a video from the frontend to POST /pipeline/upload-and-process.
2. Backend stores the file in Blob Storage (default container: raw-videos).
3. Backend generates a short-lived SAS URL and calls Content Understanding for analysis.
4. Parse CU output segments and text (prefer transcriptPhrases, fallback to markdown).
5. Build normalized chunk documents and write them to Blob (default path: search-docs/<video_id>/latest.jsonl).
6. Trigger Azure AI Search Indexer to ingest JSONL and generate vectors.
7. At query time, run Hybrid + Semantic retrieval and return start_ms/end_ms/jump_start_ms/video_url for direct player seek.

Features

• End-to-end upload, analysis, and indexing workflow.
• Indexed video listing and switching (GET /videos).
• Semantic retrieval (keyword + vector + semantic reranking).
• Intra-chunk jump refinement (jump_start_ms, configurable via ENV).
• Video summary (GET /video-summary, cache-enabled).
• Video chat (POST /chat) and streaming chat (POST /chat/stream).
• Startup auto-provisioning for required resources (Index/DataSource/Skillset/Indexer/Blob containers).

Quick Start

Docker

1）Generate a strong random token (optional, used as LOCAL_AUTH_PASSWORD)

openssl rand -hex 32

2）Start the container

docker run -itd -p 8000:8000 --name VideoSemanticSearch \
  --restart unless-stopped \
  -e FOUNDRY_CU_BASE_URL=https://<foundry-resource>.cognitiveservices.azure.com \
  -e FOUNDRY_CU_API_KEY=your-content-understanding-api-key \
  -e FOUNDRY_AOAI_BASE_URL=https://<foundry-resource>.openai.azure.com/openai/v1 \
  -e FOUNDRY_AOAI_API_KEY=your-foundry-api-key \
  -e BLOB_CONNECTION_STRING=your-storage-connection-string \
  -e AZURE_AI_SEARCH_ENDPOINT=https://<ai-search-service>.search.windows.net \
  -e AZURE_AI_SEARCH_ADMIN_KEY=your-ai-search-admin-key \
  -e LOCAL_AUTH_USERNAME=demo \
  -e LOCAL_AUTH_PASSWORD=your-strong-random-token \
  ghcr.io/heyjiqingcode/videosemanticsearch:1.0.0

See Configuration for every available variable.

Local

1）Set up

# Clone code and install requirements
git clone https://github.com/HeyJiqingCode/VideoSemanticSearch.git
cd VideoSemanticSearch
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Copy .env.example and fill in your Azure OpenAI / Foundry settings
cp .env.example .env

2）Run the server from source

# run the web app
uvicorn app.main:app --reload --port 8000

# Then open http://127.0.0.1:8000

See Configuration for every available variable.

Basic Usage

Web UI

1. Open /login and sign in with your configured local credentials.
2. In Upload & Index, select a video and click Upload & Process.
3. After processing, select a target video in Video Library (or keep All videos).
4. Enter a natural-language question in the top search box and click Search.
5. Click any item in Matched Segments; the player jumps to the returned timestamp.
6. View Video Summary on the right; open OpenAI Chat Bot on the left for grounded Q&A.
7. Use Logout in the sidebar to clear session and return to login page.

API Endpoints

• GET /: frontend page
• GET /login: local login page
• GET /auth/session: current auth state
• POST /auth/login: local login
• POST /auth/logout: local logout
• GET /favicon.ico: site icon
• GET /health: service status + startup provision snapshot
• GET /ui-config: UI runtime config
• GET /videos: indexed video list
• POST /pipeline/upload-and-process: upload and process video
• POST /search: semantic search
• POST /admin/provision: manual provision
• POST /admin/rebuild: rebuild search resources
• GET /video-summary: video summary
• POST /chat: non-streaming chat
• POST /chat/stream: streaming chat (NDJSON)

Common Operations

• Incremental ingestion for new videos: upload and process directly, no reset required.
• Refresh video list in UI: click Reload Videos.
• Recreate index structure: click Rebuild Index (or call POST /admin/rebuild).

Configuration

Variable	Purpose	Default
LOCAL_AUTH_ENABLE	Enable the local login gate	true
LOCAL_AUTH_USERNAME	Local login username	admin
LOCAL_AUTH_PASSWORD	Local login password	admin123
LOCAL_AUTH_SESSION_HOURS	Session lifetime in hours	24
LOCAL_AUTH_COOKIE_NAME	Session cookie name	video_auth_session
BLOB_CONNECTION_STRING	Azure Blob Storage connection string	— (required)
BLOB_CONTAINER_RAW_VIDEO	Container for uploaded raw videos	raw-videos
BLOB_CONTAINER_SEARCH_DOCS	Container for chunked JSONL search docs	search-docs
AZURE_AI_SEARCH_ENDPOINT	Search service endpoint, e.g. https://<ai-search-service>.search.windows.net	— (required)
AZURE_AI_SEARCH_ADMIN_KEY	Search admin key	— (required)
AZURE_AI_SEARCH_INDEX_NAME	Index name	idrx-video-semantic-search
AZURE_AI_SEARCH_DATASOURCE_NAME	Data source name	ds-video-semantic-search
AZURE_AI_SEARCH_SKILLSET_NAME	Skillset name	ss-video-semantic-search
AZURE_AI_SEARCH_INDEXER_NAME	Indexer name	idx-video-semantic-search
FOUNDRY_CU_BASE_URL	Foundry CU endpoint, e.g. https://<foundry-resource>.cognitiveservices.azure.com	— (required)
FOUNDRY_CU_API_KEY	Content Understanding API key	— (required)
FOUNDRY_CU_ANALYZER_ID	Analyzer used for video analysis	prebuilt-videoSearch
FOUNDRY_AOAI_BASE_URL	Azure OpenAI v1 endpoint, must end with /openai/v1, e.g. https://<foundry-resource>.openai.azure.com/openai/v1	— (required)
FOUNDRY_AOAI_API_KEY	Azure OpenAI API key	— (required)
FOUNDRY_AOAI_EMBEDDING_MODEL_NAME	Embedding model; the deployment name must equal the model name (vector dimensions are fixed at 3072 in src/provision.py)	text-embedding-3-large
FOUNDRY_AOAI_CHAT_MODEL_DEPLOYMENT	Chat/summary model deployment	gpt-5.6-terra
TOP_K	Number of returned segments	5
CANDIDATE_K	Vector candidate pool size	50
ENABLE_INTRA_CHUNK_JUMP	Enable intra-chunk jump refinement	true
SEARCH_JUMP_PREROLL_SECONDS	Playback pre-roll before jump target (read by frontend)	0
AUTO_PROVISION_ON_STARTUP	Run idempotent provisioning at startup	true
AUTO_PROVISION_FAIL_FAST	Fail service startup when provisioning fails	false