fix(spider): parse restructured GC course detail pages - #67
Merged
Merged
Conversation
GC site moved course detail fields (topics as <p>s, not <li>s) inside nested wrapper divs, breaking the direct-children parser: only course code/title survived, and each crawl overwrote stored descriptions and topics with empty values. Parse the flattened text stream of the detail block, grouping lines by section marker, so field extraction no longer depends on child layout or list tags. Skip import of pages whose structure was not recognized so future layout changes cannot wipe existing course data again. Verified against live GC pages: 6 random courses parse correctly (up to 18 topics, descriptions, prereqs, instructors, credits); the previous parser extracted none of these fields. Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The GC course site restructured its course detail pages: the labelled fields (
Instructors,Credits,Pre-requisites,Description,Course Topics) are now wrapped in nested divs inside the detail block, and topics are listed as plain<p>s instead of<li>s.The current parser (
_crawl_course_data) iterates only the direct children of the detail block and extracts topics viafind_all("li"), so after the restructure it captured nothing but the<h2>course code/title — and becauseimport_departmentwholesale-overwrites on each crawl, stored descriptions/topics kept getting wiped with empty values. Verified live: all 412 courses in the DB havecourse_topics: null/ empty descriptions.Fix
_crawl_course_datato parse the flattened text stream of the detail block (get_text(separator="\n")), grouping lines by section marker. Extraction no longer depends on the wrapper-div layout or list tags.import_department: when a page carries none of the known section markers, skip the import instead of overwriting stored data — so a future layout change can't silently wipe course fields again. Records without the new key default to "recognized" for backward compatibility.Verification
Ran the patched parser (through the repo's own Django env) against live GC pages, 6 random courses — all parsed correctly:
The old parser produced only code/title for the same pages. No DB writes during testing; ruff check/format clean (3 pre-existing warnings on untouched lines).
Note
Not covered here: the crawler's base domain on
mainis stillwww.ji.sjtu.edu.cn; the canonical-domain switch togc.sjtu.edu.cnlives on the local deploy branch and should land separately.