Conversation
- models: syllabus uploaded per course+instructor, sha256-deduped file store - settings: SYLLABUS/OLLAMA config defaults, MEDIA_ROOT, CELERY_BROKER_URL, ADMINS - deps: celery, pypdf, pypdfium2, python-docx
- pypdf/python-docx text extraction; pypdfium2 page render + vision OCR for scans - qwen3.8 chat analysis: course-match/legitimacy verdict + markdown summary - comparison run against current primary when a course+instructor gains a second different syllabus; recommendation resolves primary, both files kept - failed analysis marks syllabus failed with error_message
- POST courses/<id>/syllabi: multipart upload, sha256 dedup on SyllabusFile,
idempotent per (course, instructor, file), enqueues analysis task
- GET lists per course; PATCH syllabi/<id> staff-only (summary/verdict/primary)
- GET syllabi/<id>/download streams original file to logged-in users
- course detail + course_instructors now return {id, name} instructor objects
- user/status exposes is_staff; admin models editable with reject action
- ruff-format spider crawler tests and the legacy review import test, whose pre-existing deviations made every pre-commit run fail - compose.dev.yaml: worker service (`celery -A website.celery worker`, Ollama on host via host.containers.internal by default), backend and worker share the syllabus_media:/app/media volume - Containerfile: create /app/media and chown to nonroot
- CourseSyllabiAPI/SyllabusDetailAPI: GenericAPIView+Mixins need explicit
get/post/patch methods or they 405
- refresh Syllabus row after delay() so eager execution (tests) is visible
- move FAILED bookkeeping out of the task atomic block: a re-raise inside
rolled the failed status back before Celery saw the error
- CourseSerializer.get_instructors: term-agnostic {id, name} pairs so
syllabus uploads can target any instructor who taught the course
- eager_media fixture pushes task_always_eager onto the Celery app config, not just Django settings (celery.py snapshots settings at import time) - staff_client gets its own APIClient so force_authenticate does not bleed between clients; download-auth checks use a fresh anonymous client - SyllabusFileFactory sha256 unique per instance (unique constraint) - fake ollama/extraction monkeypatches cover analyze, compare, OCR paths
Django processes never imported website.celery, so @shared_task resolved to Celery's default bare app and delay() tried amqp://guest@localhost:5672 instead of the configured redis broker. Import the app from website/__init__ so every process picks up the redis broker. Ollama /api/chat returns content nested under message.content, not top level, so verdict parsing always saw an empty string; parse the nested field. qwen3 thinking models emit the final answer into message.thinking under JSON format, leaving content empty - disable thinking in options. Also log empty-content responses with done_reason for diagnostics. Drop NUM_CTX from 262144 to 32768: a 27B Q4 model plus a 262k-token KV cache exceeds the 24GB GPU, spilling layers to CPU and stalling inference.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Contributor
Author
|
This PR has local ollama request so pytest fail, this pr is just for reference about the syllabus upload/download and analyse function. |
The runtime fix moved content extraction from the top-level response key to message.content; the canned fixture still returned top-level content, so eager-mode analysis saw an empty body.
CodeQL flagged the user-derived syllabus_id in a log call (log injection) and the two empty JSONDecodeError handlers. The URL regex already limits syllabus_id to digits; strip CR/LF anyway and explain both pass branches.
gc_offerings.parse_gc_offerings already extracts term, section, and instructor names, but import_gc_courses only created Course rows and dropped the rest. Write Instructor (get_or_create by unique name) and CourseOffering (get_or_create by unique (term, course, section)), then instructors.set() so re-runs sync instructor changes instead of accumulating stale bindings. Co-Authored-By: Claude Code <noreply@anthropic.com>
crawl_timetable and crawl_medians tasks are commented out, so their beat entries dispatched to nonexistent tasks every night. Add crawl_gc_course_offerings at 1:15AM instead; its CrawledData handler chains the import task automatically. Co-Authored-By: Claude Code <noreply@anthropic.com>
- Syllabus gains a rejected status; the analysis task moves uploads whose verdict match_score is below SYLLABUS.MIN_MATCH_SCORE (60) to it instead of analyzed. The row is kept for audit but hidden from course pages (frontend filters it). - SyllabusFile rows whose last referencing syllabus is rejected or deleted move to media/recycle/ via a new syllabus_files helper (shared files stay in place), wired through a post_delete signal so Django admin delete_selected recycles too. - SyllabusDetailAPI accepts staff DELETE.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds syllabus upload, AI analysis, versioning, and download to the course detail flow. Logged-in users upload a syllabus (PDF or DOCX) for a course + instructor pairing; a Celery worker extracts the text, runs it through the local Ollama model (
qwen3.8) to check whether it actually matches the course and looks like a legitimate syllabus, and stores a markdown summary plus a structured verdict. Duplicate files are deduplicated by sha256. When multiple syllabi exist for the same course/instructor pair, the worker compares them and marks oneis_primary. Staff can edit the AI summary, adjust the verdict, and flip the primary flag in Django admin or via the PATCH API.Nothing here touches the review/vote/course-list endpoints except one serializer shape change called out below.
Changes
1. Models (
feat(web), migration0014)Syllabus(apps/web/models/syllabus.py): links aCourse+Instructorpair to aSyllabusFile. Carries a lifecyclestatus(pending→processing→analyzed/failed),summary_md,verdict(JSON),comparison(JSON, multi-version diff),is_primary,error_message, anduploaded_by. Unique constraint on(course, instructor, file)so re-uploading the same bytes is a no-op.SyllabusFile(apps/web/models/syllabus_file.py): the binary blob, keyed by uniquesha256— dedup is at the file level, independent of which course/instructor it was uploaded for.apps/web/admin.py: admin registration withsummary_md/verdict/is_primaryediting, and arejectaction that marks a syllabus failed without deleting the file.2. API (
feat(web),apps/web/views.py+urls.py)New endpoints:
/api/courses/<id>/syllabi//api/courses/<id>/syllabi/file(PDF/DOCX ≤ 20 MB) +instructorid; queues analysis/api/courses/<id>/instructors/[{id, name}])/api/syllabi/<id>/summary_md,verdict,is_primary/api/syllabi/<id>/download/Content-Dispositionfilename)Permissions: upload and download are open to any logged-in user; editing is staff-only (not uploader-only); there is deliberately no DELETE endpoint — rejection is a status change in admin.
3. Analysis pipeline (
feat(web))apps/web/syllabus_analysis.py— text extraction (pypdfium2 for PDF, python-docx for DOCX), then two Ollama/api/chatcalls withformat: "json":analyze()returns{match_score, matches_course_content, is_legitimate, flags, summary_md}— the model judges whether the document is a real syllabus and whether it matches the named course.compare()runs when an analyzed syllabus already exists for the pair: diffs old vs new and decides whether the new file supersedes the old (keep_oldsemantics,is_primaryflip).response["message"]["content"]and"think": Falseis passed in options — with qwen3 thinking models + JSON format, the answer otherwise lands inmessage.thinkingandcontentcomes back empty.apps/web/tasks.py—process_syllabustask wraps analyze/compare; status bookkeeping (FAILED, extracted-text cache) happens outside thetransaction.atomicblock so a rollback can't silently lose the failure record.SYLLABUSandOLLAMAconfig blocks inwebsite/settings.pyDEFAULTS (model,NUM_CTX, max file size, etc.), documented inconfig.yaml.example. Env overrides:OLLAMA__BASE_URL,OLLAMA__NUM_CTX, …4. Celery wiring fix (
fix(web))website/__init__.py: import the configured Celery app on Django startup. Without this,@shared_taskin web views binds to Celery's default no-op amqp app and.delay()fails with[Errno 111] amqp://guest@localhost:5672— the tasks were registered but the broker URL was never applied.5. Infrastructure
Containerfile:mkdir -p /app/media && chown -R nonroot:nonroot /app/mediabeforeUSER nonrootso the media volume is writable.compose.dev.yaml: newworkerservice (celery,syllabus_mediavolume,OLLAMA__BASE_URLpointing at the host Ollama) andsyllabus_mediavolume on the backend service.pyproject.toml: addscelery,pypdf,pypdfium2,python-docx(+uv.lock).6. Tests (
test(web))17 new tests in
apps/web/tests/test_syllabus.pycovering upload (dedup, size limit, bad type), list, download, staff PATCH, and the full analyze/compare/primary-selection flow with a fake Ollama client. Full suite: 85 passed.CourseSerializer(course detail —/api/courses/<id>/) now returnsinstead of the previous list of name strings, because the upload flow needs instructor ids.
CourseSearchSerializer(course list) is unchanged. The frontend course-detail page must renderinstructor.name(old code renders[object Object]); frontend PR follows in Tech-JI/CourseFront.Not in this PR
compose.yamlon thefix/production-deploybranch, which is not pushed to origin.