diff --git a/CHANGELOG.md b/CHANGELOG.md index 481b631..f17ceac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -141,6 +141,41 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). each other -- see [The three write paths fail differently](docs/caching.md#the-three-write-paths-fail-differently). +### Fixed + +- **Google and DeepL translations in HTML mode no longer come back + double-escaped.** Both vendors return entity-escaped text -- an + apostrophe as `'`, a quote as `"`, an ampersand as `&` -- and + the pipeline decoded entities on the way in but never on the way out, so + the renderer escaped the vendor's own `&` a second time and a reader saw + `didn't` on the page. English is full of apostrophes, so in practice + every Google or DeepL translation into English was affected somewhere. + `TranslationDiff::Translation::Response.build` now decodes a provider's + reply the same way it already decoded the source, symmetrically, for + every provider -- named entities, and both the decimal (`'`) and hex + (`'`) numeric forms, are decoded; an entity neither decoder + recognizes is left exactly as it arrived. See [How it + works](docs/how-it-works.md). +- **Behaviour change: a literal `<` in a source sentence now renders as + `<`.** Decoding the fix above exposed a second bug: a provider's own + `<` now decoded to a bare `<`, and a bare `<` in front of a letter + reads as an opening tag -- a provider could inject markup into the + rendered document. A translated `<` that is not shaped like a tag is now + escaped on render instead. `if a < b then stop.` used to come back with + the bare `<` exactly as written; it now comes back + `if a < b then stop.`, the correct HTML encoding of that character and + identical once a browser renders it -- but visible to anything comparing + output byte-for-byte against an earlier release. `>` is untouched: a + stray `>` never opens anything a parser would honour. See [How it + works](docs/how-it-works.md#html). +- **A warm cache keeps serving the corrupted text after you upgrade.** A + cache entry's key is derived from the source sentence, not from the value + stored under it, so an entry written before this fix is served exactly as + it was written until it expires -- upgrading alone does not clear it. + Give the configuration a new `cache_namespace`, or let `cache_ttl` lapse, + to force every sentence to be retranslated under the fix. See + [Caching](docs/caching.md). + ### Security - `Configuration#inspect` and `Provider#inspect` print `[FILTERED]` in place diff --git a/docs/caching.md b/docs/caching.md index af70ddc..5834f64 100644 --- a/docs/caching.md +++ b/docs/caching.md @@ -17,6 +17,15 @@ its own Redis database). The key format is left alone here on purpose: changing its shape invalidates every entry already cached, everywhere, at once. +**A cache entry written before a bug fix keeps serving what the bug +produced.** The key above is built from the source sentence, never from the +value stored under it, so fixing what a provider's reply decodes to does not +invalidate what is already cached -- an entry written under the HTML-entity +double-escaping fixed in the Unreleased CHANGELOG entry is served exactly as +it was written until it expires. Give the configuration a new +`cache_namespace`, or let `cache_ttl` lapse, to force every sentence to be +retranslated under the fix. + Both read and write the same cache, keyed per provider, so switching one never serves you the other's translations. diff --git a/docs/how-it-works.md b/docs/how-it-works.md index 6406eaf..3bd71e5 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -59,9 +59,29 @@ Everything below is a collaborator one of the two drives. shape. `TranslationDiff::Markup` is the small module underneath steps 2, 3 and 6: it -decodes entity references on the way to a provider, encodes `&` and `<` again -on the way out, and escapes a `<` that opens no tag so `ox` cannot read the -rest of the sentence as markup. +decodes entity references on the way to a provider and, in +`TranslationDiff::Translation::Response.build`, on the way back too, for +every provider -- Google and DeepL both return HTML-escaped text, and +without the second decode a vendor's own `&` was escaped a second time, so +`didn't` came back as `didn't`. Named entities, and both the decimal +(`'`) and hex (`'`) numeric forms, are decoded; an entity neither +decoder recognizes, or one that would decode to invalid UTF-8, is left +exactly as it arrived. + +Decoding a reply raw would make `<` a bare `<`, and `ox` reads a bare `<` +in front of a letter as an opening tag -- a provider's own `<b attack` +would become a real `` element. So a reply is escaped the same way +a source document's own bare angles already are, before it is decoded, and +`Segment#render` re-encodes a translated sentence with +`Markup.encode_translation`: `&` is always escaped, and so is a `<` that is +not shaped like a tag -- a source document's own bare `<` is untouched by +this. **This is a behaviour change:** `if a < b then stop.` used to come +back with the bare `<` exactly as written; it now comes back +`if a < b then stop.`, the correct HTML encoding of that character, +rendering identically in a browser but visible to anything comparing output +byte-for-byte against an earlier release. `>` is left alone -- a stray `>` +never opens anything a parser would honour, so there is nothing to protect +it from. *NOTE:* if `:from` is not specified or equal to nil, then the provider's `#detect` will be called once with a sample of text up to 100 characters long to determine the language, and `#translate` will be called separately with the entire text. Try to specify `:from` explicitly to save the extra call -- it also improves segmentation, since the segmenter only sees a language when `:from` is given (see [The segmenter contract](contracts.md#the-segmenter-contract)). @@ -93,3 +113,23 @@ You can pass HTML as like as plain text: ```ruby TranslationDiff.translate("Black", from: "en", to: "es") ``` + +Nothing marks a `
` or `` block as code. The scanner's `OPAQUE`
+list (see [The steps](#the-steps) above) excludes only `