Repository navigation
Wechuli/update to genkit - #24
Merged
Merged
Conversation
- Refactored AI settings form to support OpenAI, Anthropic, and DeepSeek providers. - Removed support for custom endpoints in the database and updated migration scripts. - Implemented new Genkit integration for AI workflows, including provider-specific configurations. - Added tests for AI review and conversion flows to ensure proper functionality and usage tracking. - Introduced a new module for managing AI provider options and validation.
- Deleted the extraction worker's package.json, which contained dependencies for document extraction. - Removed the extraction.go file, which implemented the extraction logic for various document types. - Eliminated the CLI script (extract-import-source-cli.mjs) that handled input and output for the extraction process. - Removed the main extraction script (extract-import-source.mjs) that processed document extraction logic. - Deleted the extraction worker daemon (extraction-worker-daemon.mjs) that managed socket connections for the extraction service. - Removed the mock import provider used for testing the extraction process. - Deleted tests related to the extraction worker protocol, ensuring no remnants of the extraction worker remain in the codebase.
…nctionality - Add extractSourceFromBytes function to handle various file types including text, PDF, XLSX, and DOCX. - Introduce a new SourceBuilder class to manage extracted content and warnings. - Implement extraction logic for text files (TXT, CSV, TSV, JSON) and office documents (XLSX, DOCX) with error handling. - Create a dedicated Web Worker for asynchronous extraction processes, ensuring document passwords and file sizes are validated. - Add unit tests for the extraction process, covering various scenarios including worker responses, timeouts, and error handling. - Implement ZIP file reading and validation for office documents, rejecting unsupported formats and encrypted files. - Ensure compliance with size limits for extracted content and provide meaningful error messages for invalid inputs.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request removes all server-side and containerized document extraction functionality (PDF/XLSX/DOCX) and migrates all import file extraction to run entirely in the browser. The changes simplify deployment by eliminating the Node.js extraction worker, associated Docker targets, and related configuration, and update documentation and specifications to reflect the new browser-only extraction approach and the removal of custom AI endpoints.
Major changes include:
Removal of Extraction Worker and Server-Side Extraction
extraction-worker, including dependencies, test targets, and socket directory setup, so the application image no longer contains or references any extraction worker or Node.js runtime. (Dockerfile) [1] [2]AI_EXTRACTION_SOCKET,AI_EXTRACTION_SCRIPT,AI_ALLOWED_ENDPOINTS) from.env.exampleand updated the documentation to reflect their removal. (.env.example,README.md) [1] [2]Makefileand CI configs. (.github/workflows/publish-container.yml,Makefile,.github/workflows/validation.yml) [1] [2] [3] [4] [5]Documentation and Specification Updates
README.md,SPEC.md) to describe the new browser-based extraction flow: all supported file types (TXT, CSV, TSV, JSON, PDF, XLSX, DOCX) are parsed in a browser Web Worker, with no file or password sent to the server, and no server-side or sidecar parsing. [1] [2] [3] [4]AI Provider and Endpoint Policy
README.md,SPEC.md) [1] [2]README.md,SPEC.md) [1] [2] [3] [4]Deployment and Compose Changes
README.md,SPEC.md) [1] [2] [3]These changes significantly simplify the codebase and deployment, centralize all import file parsing in the browser, and clarify AI provider usage and security.