[Dallas98]

This pull request introduces several improvements and refactorings across the backend, frontend, and Python runtime. The main highlights are the refactoring of document loading logic in the Python backend for better maintainability and extensibility, improvements to dataset file pagination and selection in the frontend, and minor enhancements to backend APIs and Docker configuration.

Python Backend: Document Loader Refactor

• Introduced a new UniversalDocLoader class in document_loaders.py to centralize and simplify the logic for loading various document types (TXT, JSON, CSV, Markdown, Word, PDF), replacing the previous ad-hoc loader selection in generation_service.py. This makes it easier to add support for new formats and improves code maintainability. [1] [2] [3] [4]
• Updated the document loading logic in GenerationService to use the new load_documents utility, and removed the now-unnecessary _build_loader method and related imports. [1] [2] [3] [4]

Frontend: Dataset File Pagination and Selection

• Refactored the file pagination logic in DatasetFileTransfer.tsx to improve page and pageSize handling, reset pagination when the dataset changes, and ensure correct file fetching and state updates. [1] [2] [3]
• Improved file selection logic: fixed issues with select-all/deselect-all, ensured correct handling of file IDs, and simplified row selection event handlers. [1] [2]
• Added a comment clarifying that Ant Design Table pagination is 1-based to prevent off-by-one errors.

Backend API: Dataset File Prefix Default

• Set a default value of an empty string for the prefix parameter in the getDatasetFiles API endpoint, ensuring consistent behavior when the parameter is not provided.

Other Improvements

• Updated the Python Dockerfile to download additional NLTK data required for text processing.
• Added a module docstring to app/common/__init__.py for clarity.…ment loading support