Repository navigation
Keep URL import working on hostile HTML - #168
Conversation
A numeric character past Unicode's range crashed the import, and unclosed meta or nav tags made the old regular expressions retry until the request died. Parsing now walks the page once and skips characters it cannot decode. Co-authored-by: Gáborjáni-Szabó Ádám <Programcode@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reachedEnable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. Next included review available in 2 minutes. View limit details
Note 🎁 Summarized by CodeRabbit Free
Comment |
The forward scan re-ran indexOf for the far close tag at every nesting level, and a stray < with no > rescanned the rest of the page for each character, so a 1 MB page of unclosed tags took minutes. Searches now remember their last hit, and tag stripping in the text pass cannot run past the next <. Adds 1 MB regression tests for each hostile shape.
|
Deployment failed for project debatelab-staging with the following error: Learn More: https://vercel.com/adam-gaborjanis-projects?upgradeToPro=build-rate-limit |
|
Deployment failed for project debate-navigator with the following error: Learn More: https://vercel.com/adam-gaborjanis-projects?upgradeToPro=build-rate-limit |
Importing a page could fail in two ways that showed up as a toast. A numeric character like
�threw insideString.fromCodePointand that exception text was shown to the student. A page with thousands of unclosed<metaor<navtags could also stall the function on a slow regular expression (called out as security-edge-7 in the October review).The HTML helpers now walk the document once. Bad code points become empty text instead of throwing. Unclosed tags are skipped instead of being re-scanned. The import deadline is no longer cleared before that parse.
No UI change. A page that used to crash now imports, or fails with the existing import error rather than a parser exception.
Verified locally (GitHub Actions is out of minutes, so this is not a CI pass):
deno test supabase/functions/import-source/(23 passed), including the bad-entity case and the 8,000 unclosed meta / 4,000 unclosed nav cases finishing in under half a secondNot run:
npm run verifyor a live URL import in the browser.