diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 18d74c3..46837bb 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -11,7 +11,7 @@ jobs: strategy: fail-fast: false matrix: - ruby: ["3.2", "3.3", "3.4", "4.0"] + ruby: ["3.4", "4.0"] steps: - uses: actions/checkout@v4 diff --git a/.rubocop.yml b/.rubocop.yml index 4127e85..dea43ba 100644 --- a/.rubocop.yml +++ b/.rubocop.yml @@ -1,5 +1,5 @@ AllCops: - TargetRubyVersion: 3.2 + TargetRubyVersion: 3.4 NewCops: enable Style/Documentation: @@ -21,6 +21,11 @@ Layout/LineLength: Gemspec/DevelopmentDependencies: EnforcedStyle: gemspec +# Ruby 3.4 chills string literals in files without this comment, and the +# supported floor is now 3.4. +Style/FrozenStringLiteralComment: + Enabled: false + # Request takes its collaborators as keyword arguments -- values, from:, to:, # provider:, config: and the provider's own options. The confusion this cop # guards against is positional: six named arguments at a call site read diff --git a/CHANGELOG.md b/CHANGELOG.md index 3910600..68320c1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -75,6 +75,46 @@ described below. Everything here is relative to `deepl_diff` 2.2.0. configured above 600 seconds is enforced over 600 seconds instead -- up to six times more eager than the configuration reads. Keep `rate_interval` within that range, or expect a tighter effective window than configured. +- Providers must inherit `TranslationDiff::Provider`. A duck-typed object is + no longer accepted: the base class supplies the transport, the + configuration check and the capability defaults, and a provider without + them is a provider that fails in the ways this library has already been + bitten by twice. +- `provider.translate(texts, from:, to:, **options)` is now + `provider.translate(request)`, taking a `Translation::Request` and + returning a `Translation::Response`. The response carries the detected + source language and, where the provider reports it, the characters billed. +- `max_request_size` and `max_batch_size` move from provider methods to + `Capabilities`. +- `TranslationDiff::Providers::Naming` is gone; the registry stamps + `cache_key` and `Provider` implements it. +- Every provider normalises language codes to the casing its own vendor + documents and accepts either casing from the caller: DeepL upper-cases, the + other five lower-case. A code carrying a script or region subtag + (`"zh-Hans"`, `"pt-BR"`) is passed through untouched. `Provider#language` + is the shared rule and `self.language_case` selects the casing, so a + provider of your own gets it by inheriting. Previously only Google and + DeepL normalised at all -- Amazon Translate rejected `"EN"`/`"RU"` and + LibreTranslate answered 400, on every call, for anyone who followed the + README's "switch provider by changing `config.provider`" with DeepL-style + codes -- and DeepL upper-cased subtags too, corrupting `"zh-Hans"`. +- `deepl-rb` and `google-cloud-translate-v2` are no longer used at all. + `faraday` and `faraday-retry` become runtime dependencies; `aws-sigv4` is + required lazily by the Amazon provider only. +- `config.deepl_host` is renamed `config.deepl_api_base`, matching the + `_api_base` name every other provider uses. There is no alias: a + configuration still setting `deepl_host` raises `NoMethodError` on + `TranslationDiff.configure`. Rename it. +- A provider returning the wrong number of translations now raises + `TranslationDiff::ResponseError`, not `TranslationDiff::Request::Error`. + `Request::Error` still exists, and still means "`from:` is missing and the + provider cannot detect"; a `rescue TranslationDiff::Request::Error` written + to catch a short response no longer catches one. Both are + `TranslationDiff::Error`, so a rescue of the base class is unaffected. +- A provider returning a well-formed response that carries no translation for + one input -- Azure answers 200 for a batch where a single string failed -- + also raises `TranslationDiff::ResponseError`, naming the position. It + previously reached `Spacing.restore` and died there as `NoMethodError`. ### Removed @@ -86,15 +126,29 @@ described below. Everything here is relative to `deepl_diff` 2.2.0. ### Added - A Google provider: `config.provider = :google` translates through Cloud - Translation v2 (Basic), on the `google-cloud-translate-v2` gem, required - lazily so an application using DeepL never needs it installed. It declares - `google_api_key` and `google_project_id`; an API key alone is enough, and - with none configured the gem reads `TRANSLATE_KEY`/`GOOGLE_CLOUD_KEY` or - falls back to application default credentials. The provider asks for - `format: :html`, which the tokenizer's output requires -- a `notranslate` - span is handed over with its tags -- and downcases bare language codes so - a configuration written for DeepL (`"EN"`) keeps working, leaving - subtagged codes such as `"zh-Hans"` alone. + Translation v2 (Basic) over HTTP directly, with no Google gem installed. It + declares `google_api_key`, `google_project_id` and `google_api_base`; an API + key alone is enough. With no key configured it reads `TRANSLATE_KEY` and + then `GOOGLE_CLOUD_KEY`, and `google_project_id` falls back to + `TRANSLATE_PROJECT` -- the variables `google-cloud-translate-v2` used to + read on your behalf. Application default credentials are **not** supported: + that path lived in the gem that is gone, and an application relying on ADC + must now configure an API key. The provider asks for `format: :html`, which + the tokenizer's output requires -- a `notranslate` span is handed over with + its tags -- and downcases bare language codes so a configuration written for + DeepL (`"EN"`) keeps working, leaving subtagged codes such as `"zh-Hans"` + alone. +- Environment-variable credential fallbacks, read by this library now that the + vendor SDKs that read them are gone: `DEEPL_AUTH_KEY` for `deepl_api_key`, + `TRANSLATE_KEY` then `GOOGLE_CLOUD_KEY` for `google_api_key`, and + `TRANSLATE_PROJECT` for `google_project_id`. Each is read on use rather than + at load, so setting one after requiring the gem still works, and an + explicitly configured value always wins. The Amazon provider deliberately + has no environment fallback: `aws-sigv4` is handed explicit credentials and + this library does not implement the AWS credential chain. +- A provider declares a default for one of its options by writing + `key => default` in `configuration_options` instead of a bare symbol; a + callable default is evaluated on every read. - `TranslationDiff::Configuration`, a declarative settings object built through the `option(key, default)` macro. Options fall back to their default until assigned, treat a blank string as unset, and support a @@ -181,6 +235,22 @@ described below. Everything here is relative to `deepl_diff` 2.2.0. told apart by an argument's value. A provider with no `detect` makes `from:` required and raises a clear error when it is missing, instead of `NoMethodError`. +- Four new providers: `TranslationDiff::Providers::Azure` (`:azure`), + `TranslationDiff::Providers::ModernMT` (`:modernmt`), + `TranslationDiff::Providers::LibreTranslate` (`:libretranslate`), and + `TranslationDiff::Providers::Amazon` (`:amazon`), Amazon Translate, signed + with `aws-sigv4` rather than headed. Every provider's limits, HTML + support, `notranslate` handling, detection and billing reporting are + declared through `Capabilities` and measured against the vendor rather + than assumed -- see the provider table in the README. +- An error hierarchy for everything a provider's transport can do wrong: + `TranslationDiff::ConfigurationError`, `TranslationDiff::ProviderError` + (and its `AuthenticationError`, `QuotaExceededError`, + `InvalidRequestError`, `ServiceError` and `RateLimitError` subclasses), + `TranslationDiff::TransportError`, `TranslationDiff::ResponseError` and + `TranslationDiff::InvalidProviderError`, all under `TranslationDiff::Error`. +- `config.open_timeout`, `config.timeout` and `config.max_retries`, read by + every HTTP provider's connection and retry policy. ### Changed @@ -218,6 +288,9 @@ described below. Everything here is relative to `deepl_diff` 2.2.0. first check, and raises `TranslationDiff::Error` naming the gem to add when it is missing. Previously the bare constant surfaced a raw `NameError` instead of the message the Redis path already raises. +- DeepL's batch limit was declared as 300 sentences per request; DeepL + documents 50. The request-size limit (1,700 escaped characters) was + already correct and is unchanged. ## [2.2.0] - 2026-09-07 diff --git a/Gemfile b/Gemfile index cd64097..281e85e 100644 --- a/Gemfile +++ b/Gemfile @@ -1,15 +1,7 @@ -# frozen_string_literal: true - source "https://rubygems.org" gemspec -# Not a runtime dependency of the gem (see the gemspec) -- the DeepL -# provider requires it lazily at build time. It is only here so the test -# suite, which exercises that provider against the real deepl-rb objects, -# has it available. -gem "deepl-rb", "~> 3.9", require: false - # Not runtime dependencies of the gem (see the gemspec) -- Configuration# # redis_pool requires them lazily, and RedisCacheStore/RedisRateLimiter # duck-type against whatever a caller's connection pool yields. They are @@ -26,9 +18,16 @@ gem "redis-namespace", "~> 1.11", require: false # stand-in, has it available. gem "ratelimit", "~> 1.1", require: false -# Not a runtime dependency of the gem (see the gemspec) -- the Google -# provider requires it lazily at build time, so an application using DeepL -# never needs it installed. It is only here so the test suite, which -# exercises that provider's build path against the real -# Google::Cloud::Translate::V2 objects, has it available. -gem "google-cloud-translate-v2", "~> 1.2", require: false +# Not a runtime dependency of the gem (see the gemspec) -- lib/ only ever +# needs CGI.escape, which cgi/escape (in Ruby's default load path) still +# provides. This is here because the Google provider's test decodes the +# query string it sent, and Ruby 4.0 removed CGI.parse from the default +# load path; "install cgi gem" is Ruby's own suggested fix. +gem "cgi", "~> 0.5", require: false + +# Not a runtime dependency of the gem (see the gemspec) -- the Amazon +# provider requires it lazily when it signs its first request, so an +# application using another provider never needs it installed. It is here so +# the test suite, which signs against the real library rather than a +# stand-in, has it available. +gem "aws-sigv4", "~> 1.12", require: false diff --git a/README.md b/README.md index 0f2c84a..e8c2177 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ # TranslationDiff A translation cache that helps translate only changes between revisions of -long texts. It ships with DeepL and Google Cloud Translation providers, but -any translation service can be plugged in by implementing a small provider -contract -- this gem has no hard dependency on either of them, or on any -other provider. +long texts. It ships with six providers -- DeepL, Google Cloud Translation, +Azure AI Translator, ModernMT, LibreTranslate and Amazon Translate -- but any +translation service can be plugged in by subclassing a small base class; see +[Providers](#providers). **TranslationDiff** based on [GoogleTranslateDiff](https://github.com/gzigzigzeo/google_translate_diff) @@ -21,21 +21,24 @@ Much better approach is to try to translate every repeated structural element (s ## Dependencies -This gem loads two: [`ox`](https://github.com/ohler55/ox) to walk the HTML, and -[`pragmatic_segmenter`](https://github.com/diasks2/pragmatic_segmenter), which backs -the default sentence segmenter and has zero dependencies of its own. See [Segmenters -and the segmenter contract](#segmenters-and-the-segmenter-contract) below if you want -to avoid the second dependency. - -Everything else is duck typed and supplied by you: `deepl-rb` only if you use -the DeepL provider (the default) and `google-cloud-translate-v2` only if you -use the Google one, each required lazily the first time it is needed, with a -clear error if it is missing. The same is true of `redis` and -`connection_pool` once you configure `redis_url`, and of `ratelimit` on the -first check once you configure `rate_limit`. `redis-namespace` (for the Redis -cache store) goes one step further: this gem never requires it at all, so -your application must `require` it itself before using the Redis-backed -store. See [Getting started](#getting-started) below. +This gem loads four at require time: [`ox`](https://github.com/ohler55/ox) to +walk the HTML; [`pragmatic_segmenter`](https://github.com/diasks2/pragmatic_segmenter), +which backs the default sentence segmenter and has zero dependencies of its +own (see [Segmenters and the segmenter contract](#segmenters-and-the-segmenter-contract) +below if you want to avoid it); and [`faraday`](https://github.com/lostisland/faraday) +with [`faraday-retry`](https://github.com/lostisland/faraday-retry), the HTTP +transport every REST-backed provider (DeepL, Google, Azure, ModernMT, +LibreTranslate) inherits and owns directly -- none of them wraps a +vendor-supplied SDK any more. + +Everything else is duck typed and supplied by you: `aws-sigv4`, required +lazily the first time the Amazon provider signs a request, with a clear +error if it is missing. The same is true of `redis` and `connection_pool` +once you configure `redis_url`, and of `ratelimit` on the first check once +you configure `rate_limit`. `redis-namespace` (for the Redis cache store) +goes one step further: this gem never requires it at all, so your +application must `require` it itself before using the Redis-backed store. +See [Getting started](#getting-started) below. ## Installation @@ -64,8 +67,11 @@ end TranslationDiff.translate("Привет.", from: "ru", to: "en") ``` -Both of those have sensible defaults, so with `DEEPL_AUTH_KEY` and `REDIS_URL` -in the environment there is nothing to configure at all. Without `REDIS_URL` +`deepl_api_key` is required -- the provider checks for it at build time and +raises `TranslationDiff::ConfigurationError` naming what is missing, rather +than failing on the first real request. Leave it unset and `DEEPL_AUTH_KEY` +is read instead, on use rather than at load, so the variable may be exported +after this gem is required. `redis_url` is optional: without it the cache lives in the process, which means the library runs before any infrastructure does. @@ -74,7 +80,7 @@ of your own**: ```ruby TranslationDiff.configure do |config| - config.provider = :deepl # or any object satisfying the provider contract + config.provider = :deepl # or any TranslationDiff::Provider of your own -- see Providers config.cache = :redis # or any object satisfying the cache store contract config.segmenter = :pragmatic # or any object satisfying the segmenter contract end @@ -89,7 +95,7 @@ at all, so an unset environment variable never has to be special-cased. | Option | Default | Meaning | | --- | --- | --- | -| `provider` | `:deepl` | The translation provider: a registered name or an object satisfying the [provider contract](#the-provider-contract). | +| `provider` | `:deepl` | The translation provider: a registered name or a `TranslationDiff::Provider` of your own. See [Providers](#providers). | | `cache` | `nil` | The cache store: a registered name or an object satisfying the [cache store contract](#the-cache-store-contract). `nil` means "choose for me" -- see below. | | `cache_ttl` | `604_800` (one week) | Seconds a Redis cache entry is kept. Only meaningful for `RedisCacheStore`; `MemoryCacheStore` evicts by size instead. | | `cache_namespace` | `"translation-diff"` | Prefix applied to every Redis key this gem writes -- both cache entries and the rate limiter's own bookkeeping. | @@ -103,24 +109,33 @@ at all, so an unset environment variable never has to be special-cased. | `segmenter` | `:pragmatic` | The sentence segmenter: a registered name or an object satisfying the [segmenter contract](#segmenters-and-the-segmenter-contract). | | `instrumenter` | `nil` | Anything satisfying `ActiveSupport::Notifications`' `#instrument(name, payload) { }` interface. See [Instrumentation and logging](#instrumentation-and-logging). | | `logger` | `nil` | A standard `Logger`. Receives one `debug` line per provider resolution, naming the provider class -- never content and never a credential. See [Instrumentation and logging](#instrumentation-and-logging). | +| `open_timeout` | `5` | Seconds an HTTP-backed provider waits to open a connection before raising `TranslationDiff::TransportError`. | +| `timeout` | `30` | Seconds an HTTP-backed provider waits for a response before raising `TranslationDiff::TransportError`. | +| `max_retries` | `3` | Retries `faraday-retry` attempts on a transport failure or a `429`/`500`/`502`/`503`/`504` response, with exponential backoff. `faraday-retry` honours a `Retry-After` header itself, so a `429` usually exhausts its retries before `TranslationDiff::RateLimitError` is ever raised. | -The `:deepl` provider declares two options of its own, registered the moment +Every provider declares its own configuration options, registered the moment `translation_diff` is required: -| Option | Default | Meaning | +| Provider | Options | Meaning | | --- | --- | --- | -| `deepl_api_key` | `nil` | Forwarded to `deepl-rb` as `auth_key`. Left unset, `deepl-rb` reads `DEEPL_AUTH_KEY` from the environment itself. | -| `deepl_host` | `nil` | Overrides `deepl-rb`'s automatic free/paid host selection (from the `:fx` suffix on the key). Rarely needed. | - -The `:google` provider declares two of its own, on the same terms: - -| Option | Default | Meaning | -| --- | --- | --- | -| `google_api_key` | `nil` | Forwarded to `google-cloud-translate-v2` as `key`. Left unset, that gem reads `TRANSLATE_KEY` or `GOOGLE_CLOUD_KEY` from the environment itself, and falls back to application default credentials when there is no key at all. | -| `google_project_id` | `nil` | Only consulted on the credentials path; an API key needs no project. Left unset, the gem reads `TRANSLATE_PROJECT`. | +| `:deepl` | `deepl_api_key` (required) | Sent as `DeepL-Auth-Key`. Falls back to `ENV["DEEPL_AUTH_KEY"]`. | +| | `deepl_api_base` | Overrides the automatic free/paid host selection (from the `:fx` suffix on the key). Rarely needed. | +| `:google` | `google_api_key` (required) | Sent as the `key` query parameter. Falls back to `ENV["TRANSLATE_KEY"]`, then `ENV["GOOGLE_CLOUD_KEY"]`. | +| | `google_project_id` | Declared for a future credentials path; not currently read -- an API key needs no project. Falls back to `ENV["TRANSLATE_PROJECT"]`. | +| | `google_api_base` | Overrides the default `https://translation.googleapis.com`. | +| `:azure` | `azure_api_key` (required) | Sent as `Ocp-Apim-Subscription-Key`. | +| | `azure_region` | Sent as `Ocp-Apim-Subscription-Region`. Required by a multi-service Azure resource; a single-service resource needs no region. | +| | `azure_api_base` | Overrides the default `https://api.cognitive.microsofttranslator.com`. | +| `:modernmt` | `modernmt_api_key` (required) | Sent as `MMT-ApiKey`. | +| | `modernmt_api_base` | Overrides the default `https://api.modernmt.com`. | +| `:libretranslate` | `libretranslate_api_base` (required) | Every instance is self-hosted; there is no default to fall back to. | +| | `libretranslate_api_key` | Sent as `api_key` in the request body. Most instances do not require one. | +| `:amazon` | `amazon_access_key_id`, `amazon_secret_access_key`, `amazon_region` (all required) | Used to sign each request with `aws-sigv4`. No environment fallback: this library does not implement the AWS credential chain, so `AWS_ACCESS_KEY_ID` and friends are not read. | +| | `amazon_session_token` | For temporary credentials. | +| | `amazon_api_base` | Overrides the default `https://translate..amazonaws.com`. | A provider you register yourself can declare its own options the same way -- -see [Registering your own provider](#registering-your-own-provider) below. +see [Writing a provider](#writing-a-provider) below. **Configure once, before the first translation.** `provider`, `cache`, `segmenter` and `rate_limiter` each resolve to a collaborator on first use @@ -147,136 +162,149 @@ source and target language codes, a digest of the provider options that call passed (`formality:`, a glossary id, ...), and a digest of the sentence itself. `RedisCacheStore` prefixes all of that with `cache_namespace`. -**`deepl_host` is deliberately not part of the key.** Two configurations -pointing `deepl_host` at different endpoints share cache entries. For DeepL's -own free and paid hosts that is correct -- they return the same translations --- but a self-hosted or proxied endpoint may not, and it would be served, and -would serve, the real service's entries. Give such a configuration its own -`cache_namespace` (or its own Redis database). The key format is left alone -here on purpose: changing its shape invalidates every entry already cached, -everywhere, at once. - -### The DeepL provider - -`config.provider = :deepl` is the default. It sends `tag_handling: :html` -and `tag_handling_version: "v2"` with every translation, because what -reaches a provider is not plain text: a `notranslate` span arrives whole, -tags included. DeepL honours `class="notranslate"` and `translate="no"` -only under HTML tag handling -- without it, in DeepL's own words, "tags are -treated as regular text". - -That failure was a quiet one, worth knowing about if you translated with an -older version: DeepL leaves the tags themselves alone either way, so the -markup looks untouched and only the protected content comes back changed. - -``` -"Bold Mountain is a good place." -no tag handling -> "Болд-Маунтин — отличное место." -tag_handling -> "Bold Mountain — это хорошее место." -``` - -Both values are overridable per call, as any provider option is. Under HTML -tag handling DeepL defaults `split_sentences` to `nonewlines`; this library -sends one sentence at a time, so that changes nothing. - -### The Google provider - -`config.provider = :google` translates through Cloud Translation v2 (Basic). -It needs the `google-cloud-translate-v2` gem, which this gem requires lazily -the first time the provider is built: - -```ruby -gem "google-cloud-translate-v2", "~> 1.2" -``` - -```ruby -TranslationDiff.configure do |config| - config.provider = :google - config.google_api_key = ENV["GOOGLE_TRANSLATE_KEY"] -end -``` - -An API key is the whole setup -- no project id, no service account. Leave -`google_api_key` unset and the gem reads `TRANSLATE_KEY` or -`GOOGLE_CLOUD_KEY` itself; with no key anywhere it falls back to application -default credentials, which is the path where `google_project_id` matters. - -Two things this provider does on your behalf, both of which would otherwise -be silent problems: - -- **It asks for HTML**, which is Google's own default and what this gem has - always sent. What reaches a provider is not plain text: a `notranslate` - span arrives whole, tags included -- that is how the tokenizer marks - content the provider must leave alone -- and entities such as `&` - stay in the text it emits. Asking for `text` makes Google translate the - protected span and drop its markup: - - ``` - "Bold Mountain is a good place." - format: text -> "Болд Маунтин — хорошее место." - format: html -> "Bold Mountain — хорошее место." - ``` - - The cost is that Google escapes its own output: a literal apostrophe - returns as `'`. Inside the HTML fragment these values usually are, - that renders as an apostrophe and is correct. Inside a value that never - had any markup in it, it is noise -- pass `format: :text` per call when - translating bare strings. -- **It downcases bare language codes.** Google's codes are lowercase, and a - configuration written against DeepL says `"EN"`. Codes carrying a subtag - -- `"zh-Hans"`, `"zh-CN"`, `"pt-BR"` -- are passed through untouched, - because the casing of a script or region subtag is its own. - -Its limits are Google's documented ones: 128 strings per request (a hard -limit -- a larger batch is rejected), and 5,000 characters per request (the -documented recommendation, well under the hard 100 KB ceiling). Chunker -measures the URL-escaped form of each string, which is never smaller than -its UTF-8 byte count, so a chunk within that bound in escaped characters is -within it in bytes as well. - -`google_project_id` is not part of the cache key, on the same reasoning as -`deepl_host` above: it selects an account to bill, not a translation. - -## Registering your own provider +**No provider's `*_api_base` option is part of the key.** Two configurations +pointing `deepl_api_base` (or any other provider's `_api_base`) at different +endpoints share cache entries. For DeepL's own free and paid hosts that is +correct -- they return the same translations -- but a self-hosted or proxied +endpoint may not, and it would be served, and would serve, the real +service's entries. Give such a configuration its own `cache_namespace` (or +its own Redis database). The key format is left alone here on purpose: +changing its shape invalidates every entry already cached, everywhere, at +once. + +## Providers + +Every provider declares what it can do through +`TranslationDiff::Capabilities` -- there is nowhere else these numbers live, +so this table is generated from the same source the library reads at +runtime: + +| Provider | `config.provider` | Auth option(s) | Batch size | Request size (escaped chars) | HTML support | `notranslate` | Detects language | Reports billing | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Null | `:null` | none | 1,000,000 | 1,000,000 | no | no | no | no | +| DeepL | `:deepl` (default) | `deepl_api_key` | 50 | 1,700 | yes (`tag_handling`) | yes | yes | yes | +| Google | `:google` | `google_api_key` | 128 | 5,000 | yes (`format`) | yes | yes | no | +| Azure | `:azure` | `azure_api_key` | 1,000 | 50,000 | yes (`textType`) | yes | yes | yes | +| ModernMT | `:modernmt` | `modernmt_api_key` | 128 | 5,000 | yes (`format`) | no | yes | yes | +| LibreTranslate | `:libretranslate` | `libretranslate_api_base` | 50 | 5,000 | yes (`format`) | no | yes | no | +| Amazon | `:amazon` | `amazon_access_key_id`, `amazon_secret_access_key`, `amazon_region` | 1 | 10,000 | no | no | yes | no | + +**Every keyword other than `from:`, `to:`, `provider:` and `config:` is +forwarded to the provider, and every provider applies them the same way: +its own defaults first, then your options, then the fields the request cannot +do without.** So `formality: :less` overrides a default, and a keyword +colliding with the language pair or the texts themselves is overridden rather +than obeyed. + +**`usage.billed_characters` is `nil` when the provider said nothing about +billing and a number -- `0` included -- when it said something.** All three +providers that report billing follow that rule; the other four always answer +`nil`. + +**Language codes are normalised per vendor, so switching provider needs no +other change.** A bare code (`"EN"`, `:ru`) is cased the way the vendor +documents it -- DeepL takes upper case, every other provider here takes lower +case -- whichever casing you wrote. A code carrying a script or region subtag +(`"zh-Hans"`, `"pt-BR"`) is passed through untouched, because the casing of a +subtag is its own. A provider of your own gets the same rule from +`TranslationDiff::Provider#language`; declare `def self.language_case = +:upcase` if your vendor wants upper case. + +"Request size" is what `Chunker` measures: the URL-escaped form of each +string (`CGI.escape(text).size`), which is never smaller than its UTF-8 byte +count. "HTML support" names the provider option that turns HTML handling on +-- every vendor spells it differently, which is exactly what +`Capabilities#html` is for. A provider whose "Detects language" column says +no makes `from:` required; passing it makes every provider's `#detect` call +unnecessary regardless of whether it has one. + +**Amazon translates one text per call and honours no `notranslate`.** There +is no batch form of `TranslateText`, so a hundred sentences are a hundred +requests -- slow, but correct, and `Capabilities#max_batch_size` reflects +it. Amazon also has no HTML mode: a `notranslate` span reaches it as plain +text and is translated like everything else, tags and all. Both facts are +worth weighing before your bill and your brand names arrive, not after. + +**LibreTranslate does not honour `notranslate` either -- measured, not +assumed.** Its HTML format preserves markup, but probing a real instance +(`docker run libretranslate/libretranslate --load-only en,ru`) with +`Bold Mountain is a good place.` came back +with the span tag intact and its content translated anyway -- "Bold +Mountain" became "Смелая гора". The tags survive; what they were meant to +protect does not. + +ModernMT's `notranslate: false` is the conservative default rather than a +measurement: it documents an HTML format but says nothing about +`class="notranslate"`, and no key was available to probe it. A capability +that under-promises costs a warning; one that over-promises costs a +customer's protected content reaching a competitor's brand voice. + +### Writing a provider Any translation service can be a provider -- no change to this gem's own -code is required. Registering a provider also declares the options it needs, -so `config.yandex_api_key` below does not exist until `YandexProvider` -is registered: +code is required. Subclass `TranslationDiff::HTTPProvider` for a REST +service; it owns the Faraday connection, retries, timeouts and turns HTTP +status codes into this library's error hierarchy, and asks only for three +seams per operation: the URL, how to render a request, how to parse a +reply. Subclass `TranslationDiff::Provider` directly for anything that +reaches its service some other way -- signed requests, another gem, an +LLM client -- and implement `#translate` outright, the way +`TranslationDiff::Providers::Amazon` does. + +Registering a provider also declares the options it needs, so +`config.yandex_api_key` below does not exist until `YandexProvider` is +registered: ```ruby -class YandexProvider - # Declares this provider's own configuration options. TranslationDiff::Providers.register - # adds each one to TranslationDiff::Configuration as a side effect. +class YandexProvider < TranslationDiff::HTTPProvider + # Declares this provider's own configuration options. + # TranslationDiff::Providers.register adds each one to + # TranslationDiff::Configuration as a side effect. def self.configuration_options = %i[yandex_api_key] - def self.build(config) = new(config.yandex_api_key) + def self.configuration_requirements = %i[yandex_api_key] + + # What this provider can do, checked once by the pipeline for chunking, + # detection and cache-key safety. + def self.capabilities + TranslationDiff::Capabilities.new( + max_request_size: 10_000, max_batch_size: 100, max_text_size: nil, + html: :format, notranslate: true, detects_language: true, + reports_billing: false + ) + end + + def api_base = "https://translate.api.cloud.yandex.net" + def headers = { "Authorization" => "Api-Key #{config.yandex_api_key}" } + def translate_url = "translate/v2/translate" - def initialize(api_key) - @client = SomeYandexClient.new(key: api_key) + # The three seams: build the request body, decode the reply. + def render_translate_payload(request) + { format: "HTML", texts: request.texts, targetLanguageCode: request.to.to_s } + .tap { |body| body[:sourceLanguageCode] = request.from.to_s unless request.from.nil? } end - # Required: translate an array of strings, return one string per input, in - # the same order. Provider-specific options (formality, glossary, ...) - # arrive through **options untouched. - def translate(texts, from:, to:, **options) - texts.map { |text| @client.translate(text, from: from, to: to)[:text] } + def parse_translate_response(body, _headers, request) + translations = Array(body["translations"]) + + TranslationDiff::Translation::Response.build( + request: request, + texts: translations.map { |t| t["text"] }, + detected_source: translations.first&.dig("detectedLanguageCode")&.downcase + ) end - # Required: this provider's own request- and batch-size limits. - def max_request_size = 30_000 - def max_batch_size = 128 + def detect_url = "translate/v2/detect" - # Optional: omit entirely if the provider has no detection endpoint, or if - # callers of this gem always pass `from:` explicitly. def detect(text) - @client.detect(text)[:language] + response = post(detect_url, { text: text }) + response.body["languageCode"]&.downcase end - # cache_key is optional too: TranslationDiff::Providers.register mixes a - # module into any provider that does not define its own #cache_key, and - # that module stamps every instance built through the registry with its - # registered name -- nothing here needs to supply one by hand. See - # "Provider objects and cache_key" below for what happens without it. + # cache_key is optional: TranslationDiff::Providers.register stamps every + # instance built through the registry with its registered name, and + # Provider#cache_key falls back to that. Define it yourself only if this + # provider will also be instantiated and assigned directly, bypassing the + # registry -- see "Provider objects and cache_key" below. end TranslationDiff::Providers.register(:yandex, YandexProvider) @@ -287,6 +315,12 @@ TranslationDiff.configure do |config| end ``` +`translate_url`/`render_translate_payload`/`parse_translate_response` are +the three seams `HTTPProvider#translate` calls in order; `detect` is +entirely optional -- omit it (and leave `capabilities.detects_language: +false`) if the provider has no detection endpoint, or if callers of this +gem always pass `from:` explicitly. + **Provider names must be unique.** `TranslationDiff::Providers.register` overwrites whatever was previously registered under that name, silently -- there is no error for registering `:deepl` twice. This is deliberate: a @@ -304,6 +338,21 @@ the log to say so. Registering over an existing name is not a way to substitute a service -- give the replacement its own name, or clear the cache (`cache_namespace` is the cheapest way to do that). +**An option can declare a default.** A bare symbol in +`configuration_options` declares an option with no default. Writing +`key => default` instead declares one, and a callable default is evaluated on +every read rather than at load time -- which is what lets an environment +variable work when the application exports it after requiring this gem: + +```ruby +def self.configuration_options + [:yandex_api_base, { yandex_api_key: -> { ENV.fetch("YANDEX_API_KEY", nil) } }] +end +``` + +An explicitly configured value always wins over a default, and a default that +resolves to a blank string reads as unset -- the same rule assignment follows. + **Option names are unique too, and enforced.** Two providers declaring the same `configuration_options` name would share one accessor on `TranslationDiff::Configuration`, which would hand one service's credential @@ -316,64 +365,20 @@ and a Rails reload both re-run registration. `TranslationDiff::Providers.names` lists every registered provider; `TranslationDiff::Providers.registered?(:yandex)` checks one. -## The provider contract - -`config.provider` accepts either a registered name (`:deepl`, `:google`, -`:null`, or anything you registered yourself) or an object of your own that satisfies -this contract directly, bypassing the registry entirely: - -```ruby -# Translates an array of strings and returns an array of strings, one per -# input, in the same order. Provider-specific options (e.g. `formality:`) -# arrive through **options and are passed straight through to the provider. -def translate(texts, from:, to:, **options); end - -# Detects the source language of a single string and returns it. Optional: -# omit this method entirely if the provider has no detection endpoint, or if -# callers of this gem always pass `from:` explicitly. When `detect` is -# missing and `from:` is not given, TranslationDiff raises rather than -# guessing. -def detect(text); end - -# The largest single request the provider accepts, in characters of the -# escaped form -- which is what the chunker measures (CGI.escape(text).size, -# not String#size). For Cyrillic and other non-Latin text this is 6 to 9 -# times the raw character count. Declaring the provider's raw character -# limit here will either waste most of the budget (if you under-report) or -# raise Chunker::Error on text the provider would actually have accepted -# (if you over-report). Used to split long texts into multiple requests. -def max_request_size; end - -# The largest number of strings the provider accepts in one batched request. -# Used to split large arrays into multiple requests. -def max_batch_size; end - -# A short, stable, non-empty string identifying this provider. Used to -# namespace cache keys, so that switching providers does not return one -# provider's cached translations for another. Not required when a provider -# is only ever built through TranslationDiff::Providers.register -- see -# below. -def cache_key; end -``` - -`test/support/provider_contract.rb` is the executable form of this contract: -include `ProviderContract` in a test class that defines `#provider`, and it -verifies `translate`, `max_request_size` and `max_batch_size` behave as -documented above. - -Two providers ship with this gem: `TranslationDiff::Providers::DeepL` (the -default, registered as `:deepl`), wrapping the -[`deepl-rb`](https://github.com/wikiti/deepl-rb) gem (not a dependency of -this one -- required at build time); and `TranslationDiff::Providers::Null` -(`:null`), which hands back exactly what it was given, for tests and for -wiring up a pipeline before a real provider is available. +`test/support/provider_contract.rb` and `test/support/http_provider_contract.rb` +are the executable form of the provider contract: include `ProviderContract` +(and, for an `HTTPProvider` subclass, `HTTPProviderContract`) in a test class +that defines `#provider`, and they verify a provider inherits +`TranslationDiff::Provider`, that `#translate` preserves order and returns +one string per input, and that its declared capabilities are internally +consistent (a provider claiming `notranslate` must also claim an HTML mode). ### Provider objects and cache_key A provider built through `TranslationDiff::Providers.build` (which is what happens when `config.provider` is a symbol) is stamped with its registered -name automatically, and never needs to define `cache_key` itself -- the -registry mixes in a module that supplies it. +name automatically, and never needs to define `cache_key` itself -- +`Provider#cache_key` falls back to that stamped name. A provider object assigned straight to `config.provider` never passes through the registry, so it gets no name and **must define `cache_key` @@ -638,31 +643,17 @@ same guarantee applies to it as to instrumentation payloads: no log line this library writes carries the text being translated, its translation, or a credential. -**deepl-rb has request logging of its own, and this library deliberately -does not enable it.** `config.logger` is never passed to `deepl-rb`. Given a -logger, `deepl-rb` writes a `Request details:` line at DEBUG holding the full -`Authorization: DeepL-Auth-Key ...` header and the request payload -- your API -key and the text being translated. Forwarding this gem's logger into it would -break the guarantee above at the exact moment someone raises the log level to -diagnose a problem, which is why the provider does not. - -If you want that log anyway, ask for it explicitly: build the `DeepL::API` -yourself, wrap it in the provider, and assign the object. - -```ruby -api = DeepL::API.new( - DeepL::Configuration.new(auth_key: ENV["DEEPL_AUTH_KEY"], logger: verbose_logger) -) - -provider = TranslationDiff::Providers::DeepL.new(api) -provider.name = :deepl # the cache key a registry-built provider gets for free - -TranslationDiff.configure { |config| config.provider = provider } -``` - -Everything `verbose_logger` then receives -- source text, translations, and -the auth key -- goes wherever it writes. Point it somewhere disposable, not at -the application log, and do not leave it on. +**No HTTP-backed provider ever receives `config.logger`, and there is no way +to opt one in.** `TranslationDiff::HTTPProvider` installs no logging +middleware on its Faraday connection and never passes a logger to it -- this +is enforced by `test/support/http_provider_contract.rb`, not merely +documented. Earlier versions wrapped `deepl-rb`, which logged a +`Request details:` line at DEBUG holding the full +`Authorization: DeepL-Auth-Key ...` header and the request payload -- your +API key and the text being translated -- if you gave it a logger of its own. +Owning the transport directly closed that door rather than working around +it: nothing this library builds writes source text, a translation, or a +credential anywhere, and no configuration option reopens that. ## Errors @@ -671,22 +662,43 @@ so rescuing the gem's failures in one clause is a single `rescue TranslationDiff ``` TranslationDiff::Error -├── TranslationDiff::Request::Error # e.g. provider returned the wrong number of -│ # translations, from: missing and the -│ # provider cannot detect, cache_key -│ # missing on an assigned provider object -├── TranslationDiff::Cache::Error # provider options have no stable -│ # serialisation for the cache key -├── TranslationDiff::Chunker::Error # a single value is larger than the -│ # provider's max_request_size +├── TranslationDiff::ConfigurationError # a provider is missing a required option +├── TranslationDiff::ProviderError # the service answered and said no +│ ├── AuthenticationError # 401/403 +│ ├── RateLimitError # 429, once faraday-retry's own retries +│ │ # are exhausted -- carries #retry_after +│ │ # when the service sent one +│ ├── QuotaExceededError # 456 +│ ├── InvalidRequestError # any other 4xx +│ └── ServiceError # 5xx, or anything else +├── TranslationDiff::TransportError # nobody answered: connection failed, +│ # timed out, or TLS failed +├── TranslationDiff::ResponseError # the answer was well-formed HTTP but broke +│ # this library's contract -- a body that +│ # is not JSON, a provider that returned +│ # the wrong number of translations, or one +│ # that returned no translation for an input +├── TranslationDiff::InvalidProviderError # a class registered without inheriting +│ # TranslationDiff::Provider +├── TranslationDiff::Request::Error # from: missing and the provider cannot +│ # detect, cache_key missing on an +│ # assigned provider object +├── TranslationDiff::Cache::Error # provider options have no stable +│ # serialisation for the cache key +├── TranslationDiff::Chunker::Error # a single value is larger than the +│ # provider's declared max_request_size ├── TranslationDiff::Segmenters::Pragmatic::Error -│ # Pragmatic computed offsets that -│ # violate its own postcondition -- -│ # not raised by ordinary use +│ # Pragmatic computed offsets that +│ # violate its own postcondition -- +│ # not raised by ordinary use └── TranslationDiff::RedisRateLimiter::RateLimitExceeded - # the configured rate_limit was exceeded + # the configured rate_limit was exceeded ``` +`ProviderError` and its subclasses carry `#provider` (the registered name) +and `#status` (the HTTP status code), so a caller can log or branch on which +service and which response caused the failure without parsing the message. + `TranslationDiff::Registry` -- which backs the provider, cache store and segmenter registries -- also raises `TranslationDiff::Error` directly (not a dedicated subclass) for an unknown name, listing what is actually @@ -728,11 +740,13 @@ TranslationDiff.translate("Black", from: "en", to: "es") ## Very long texts -Every provider limits how large a single request or a single batch can be. -Providers state their own limits through `#max_request_size` and -`#max_batch_size`; if your text is longer than that, TranslationDiff splits it -into multiple requests automatically. The DeepL provider, for example, caps -requests at 1700 characters and batches at 300 sentences. +Every provider limits how large a single request or a single batch can be, +declared through `TranslationDiff::Capabilities#max_request_size` and +`#max_batch_size`; if your text is longer than that, TranslationDiff splits +it into multiple requests automatically. See the [provider +table](#providers) for each built-in provider's actual numbers -- DeepL, for +example, caps requests at 1,700 escaped characters and batches at 50 +sentences. ## Former name and upgrading @@ -752,6 +766,15 @@ configured, also read the upgrading note in never actually enforcing your threshold before 3.1.0, and it starts doing so now. See [CHANGELOG.md](CHANGELOG.md) for the full list of breaking changes. +**If you registered a custom provider,** it must now subclass +`TranslationDiff::Provider` (or `TranslationDiff::HTTPProvider`), declare +`self.capabilities`, and implement `#translate(request)` taking a +`TranslationDiff::Translation::Request` and returning a +`TranslationDiff::Translation::Response` -- the duck-typed +`#translate(texts, from:, to:, **options)` plus `#max_request_size` and +`#max_batch_size` methods are no longer read at all. See [Writing a +provider](#writing-a-provider). + ## Development After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment. diff --git a/Rakefile b/Rakefile index 34f1378..d4ffd43 100644 --- a/Rakefile +++ b/Rakefile @@ -1,5 +1,3 @@ -# frozen_string_literal: true - require "bundler/gem_tasks" require "rake/testtask" diff --git a/lib/translation_diff.rb b/lib/translation_diff.rb index a3a4b94..2011962 100644 --- a/lib/translation_diff.rb +++ b/lib/translation_diff.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - require "cgi/escape" require "digest/md5" require "forwardable" @@ -9,14 +7,27 @@ require "translation_diff/version" require "translation_diff/error" +require "translation_diff/errors" +require "translation_diff/capabilities" +require "translation_diff/translation/usage" +require "translation_diff/translation/request" +require "translation_diff/translation/response" require "translation_diff/registry" require "translation_diff/configuration" require "translation_diff/configuration/provider_option_owners" +require "translation_diff/provider" +require "translation_diff/http_provider" require "translation_diff/providers" require "translation_diff/providers/null" + require "translation_diff/providers/deepl" require "translation_diff/providers/google" +require "translation_diff/providers/azure" +require "translation_diff/providers/modernmt" +require "translation_diff/providers/libretranslate" +require "translation_diff/providers/amazon" + require "translation_diff/segmenters" require "translation_diff/segmenters/simple" require "translation_diff/segmenters/pragmatic" @@ -39,20 +50,13 @@ def config = @config ||= Configuration.new def configure = yield(config) - # Tests need this, and without it one test's configuration leaks into - # every test that runs after it. + # Without this, one test's configuration leaks into every test that runs after it. def reset! = @config = nil - # An isolated copy of the configuration with the same entry point, for - # per-tenant or per-request settings. The global configuration is left - # alone. - # - # tenant = TranslationDiff.context { |c| c.deepl_api_key = key } - # tenant.translate("Hello.", from: "en", to: "ru") + # An isolated copy of the configuration with the same entry point, for per-tenant settings. def context(&) = Context.new(config.copy.tap(&)) - # `provider:` and `config:` are reserved; every other keyword is - # forwarded to the provider untouched. + # `provider:` and `config:` are reserved; every other keyword is forwarded to the provider. def translate(values, from: nil, to: nil, provider: nil, **) Request.new(values, from: from, to: to, provider: provider, config: config, **).call end diff --git a/lib/translation_diff/cache.rb b/lib/translation_diff/cache.rb index 1f00872..15afd92 100644 --- a/lib/translation_diff/cache.rb +++ b/lib/translation_diff/cache.rb @@ -1,16 +1,10 @@ -# frozen_string_literal: true - class TranslationDiff::Cache class Error < TranslationDiff::Error; end - # An application uses a handful of distinct option sets, so 32 bits of - # digest is ample to keep them apart; the full 128-bit MD5 would just - # bloat every key in a cache that may hold millions of them. + # 32 bits of digest is ample to keep option sets apart without bloating every key in the cache. DIGEST_LENGTH = 8 - # `store` is the cache store this instance reads and writes through. It has - # no reader: #store is already the public method that writes a chunk of - # translations back, and an attr_reader would silently replace it. + # No attr_reader for `store`: #store is already the public method that writes translations back. def initialize(from, to, provider:, store:, options: {}) @from = from @to = to @@ -27,9 +21,12 @@ def cached_and_missing(values) [cached, missing] end + # Indexes rather than shifting: #store is public and takes an outside array, which is not ours to consume. def store(values, cached, updates) + update = -1 + cached.map.with_index do |value, index| - value || store_value(values[index], updates.shift) + value || store_value(values[index], updates[update += 1]) end end @@ -48,26 +45,19 @@ def key(value) [provider, language(from), language(to), options_digest, hash].compact.join(":") end - # "EN" and :en are the same language; without this they are two entries - # for identical work. The collision argument for #key depends on none of - # its segments containing a colon: from/to are caller-supplied, so this - # normalisation must not introduce one. + # "EN" and :en are the same language; also must never introduce a colon, the key-join separator. def language(code) code.to_s.downcase end - # Two calls differing only in formality or glossary are two different - # translations and must not share a key. + # Two calls differing only in formality or glossary must not share a key. def options_digest return @options_digest if defined?(@options_digest) @options_digest = options.empty? ? nil : Digest::MD5.hexdigest(canonical(options))[0, DIGEST_LENGTH] end - # Object#inspect is not a stable serialisation: Ruby 3.4 changed how - # symbol-keyed hashes render, and an object without its own #inspect embeds - # a memory address. Either would silently change every cache key and make - # the application pay for every translation a second time. + # Object#inspect isn't stable: Ruby 3.4 changed hash rendering, and default #inspect embeds an address. def canonical(value) case value when Hash then value.sort_by { |key, _| key.to_s }.map { |key, item| "#{key}=#{canonical(item)}" }.join(",") diff --git a/lib/translation_diff/capabilities.rb b/lib/translation_diff/capabilities.rb new file mode 100644 index 0000000..e76f27c --- /dev/null +++ b/lib/translation_diff/capabilities.rb @@ -0,0 +1,9 @@ +# Declared, not discovered -- duck-typing left notranslate silently broken on two providers. +TranslationDiff::Capabilities = Data.define(:max_request_size, :max_batch_size, + :max_text_size, :html, :notranslate, + :detects_language, :reports_billing) do + def html? = html != :none + def notranslate? = notranslate + def detects_language? = detects_language + def reports_billing? = reports_billing +end diff --git a/lib/translation_diff/chunker.rb b/lib/translation_diff/chunker.rb index 0a12a73..60268dd 100644 --- a/lib/translation_diff/chunker.rb +++ b/lib/translation_diff/chunker.rb @@ -1,13 +1,9 @@ -# frozen_string_literal: true - class TranslationDiff::Chunker class Error < TranslationDiff::Error; end Chunk = Struct.new(:texts, :escaped_size) - # No defaults. With one provider a default looks meaningful; with two it - # silently lies about the second, which is the class of bug this file - # already had once. + # No defaults: a default silently lies about the second provider, a bug this file already had once. def initialize(values, limit:, count_limit:) @values = values @limit = limit @@ -43,9 +39,7 @@ def next_chunk?(tail, value) tail.texts.size >= count_limit end - # What the limit is about is the size of the request that goes over the - # wire, so every measurement here is of the escaped form. Mixing it with - # String#size lets a chunk of non-ASCII text run several times over. + # The limit is on wire size, so measure the escaped form -- String#size lets non-ASCII text run over. def escaped_size(text) CGI.escape(text).size end diff --git a/lib/translation_diff/configuration.rb b/lib/translation_diff/configuration.rb index 493fde1..944b5bd 100644 --- a/lib/translation_diff/configuration.rb +++ b/lib/translation_diff/configuration.rb @@ -1,24 +1,4 @@ -# frozen_string_literal: true - -# Every setting this library has, declared in one place with its default. -# -# TranslationDiff.configure do |config| -# config.deepl_api_key = ENV["DEEPL_API_KEY"] -# config.redis_url = ENV["REDIS_URL"] -# end -# -# Options are declared with `option`, which generates a reader that falls back -# to the default and a writer that normalises a blank string to nil -- so an -# unset environment variable behaves as though the option was never touched. -# -# A default may be a literal or a callable. A callable is invoked on read, not -# at load time, so `-> { ENV["REDIS_URL"] }` reflects the environment when the -# value is needed rather than when this file was required. -# -# Provider-specific options such as `deepl_api_key` are NOT declared here. -# A provider declares its own and registers them through -# TranslationDiff::Providers.register, which keeps the core ignorant of any -# particular translation service. +# Every declared setting in one place; callable defaults are invoked on read, not at load time. class TranslationDiff::Configuration class << self def option(key, default = nil) @@ -35,14 +15,11 @@ def option(key, default = nil) options << key end - # Declares the options a provider needs, remembering which provider - # declared each one. See ProviderOptionOwners (configuration/ - # provider_option_owners.rb) for the conflict rules and the - # all-or-nothing guarantee. - def register_provider_options(keys, provider) - keys = Array(keys).map(&:to_sym) - provider_option_owners.claim(keys, provider) - keys.each { |key| option(key) } + # See ProviderOptionOwners for the conflict rules and the all-or-nothing guarantee. + def register_provider_options(declared, provider) + declared = normalise_declarations(declared) + provider_option_owners.claim(declared.keys, provider) + declared.each { |key, default| option(key, default) } end def options = @options ||= [] @@ -50,6 +27,15 @@ def defaults = @defaults ||= {} private + # `:key` declares an option with no default; `{ key => default }` declares one, and a callable is read lazily. + def normalise_declarations(declared) + entries = declared.is_a?(Hash) ? [declared] : Array(declared) + + entries.each_with_object({}) do |entry, result| + entry.is_a?(Hash) ? result.merge!(entry.transform_keys(&:to_sym)) : result[entry.to_sym] = nil + end + end + def provider_option_owners = @provider_option_owners ||= ProviderOptionOwners.new end @@ -67,19 +53,11 @@ def provider_option_owners = @provider_option_owners ||= ProviderOptionOwners.ne option :segmenter, :pragmatic option :instrumenter, nil option :logger, nil + option :open_timeout, 5 + option :timeout, 30 + option :max_retries, 3 - # Values are copied; memoised collaborators (`provider_instance`, - # `cache_store`, `segmenter_instance`, `rate_limiter_instance` and - # `redis_pool`) are deliberately not -- `copy` walks `self.class.options` - # only, which never includes those readers' instance variables, so a copy - # builds its own provider, store, rate limiter and connection pool from - # its own values instead of inheriting the original's. This matters for - # `rate_limiter_instance` in particular: a tenant context that sets its - # own `cache_namespace` must not be rate-limited against its parent's - # namespace just because the parent had already resolved a limiter before - # the copy was made. An object the caller assigned is an option value and - # is therefore shared -- which is correct: someone who hands us one - # connection pool means one connection pool. + # Memoised collaborators aren't copied, or a tenant's own cache_namespace leaks its parent's rate limiter. def copy self.class.new.tap do |other| self.class.options.each do |key| @@ -88,15 +66,13 @@ def copy end end - # The provider actually used. `provider` holds what the caller set -- a - # symbol or an object -- and this turns it into an instance once. + # Guarded, unlike cache/segmenter/rate_limiter: only the provider gained a base class to check against. def provider_instance - @provider_instance ||= resolve(provider, TranslationDiff::Providers) + @provider_instance ||= + TranslationDiff::Providers.ensure_provider!(resolve(provider, TranslationDiff::Providers)) end - # `cache` unset means "choose for me": Redis when a URL is configured, - # otherwise the in-process store, so the library works before anything is - # running. + # Unset `cache` means Redis when a URL is configured, otherwise in-process -- works before anything runs. def cache_store @cache_store ||= resolve(cache || (redis_url ? :redis : :memory), TranslationDiff::Stores) end @@ -105,18 +81,7 @@ def segmenter_instance @segmenter_instance ||= resolve(segmenter, TranslationDiff::Segmenters.registry) end - # `rate_limiter` holds what the caller set -- an object, or nil -- exactly - # like `provider`, `cache` and `segmenter` hold theirs. nil, not a null - # object: Request checks for nil and skips the whole rate-limiting path, - # which is the common case and should cost nothing. - # - # The assigned object when there is one; nil when no `rate_limit` - # threshold was ever configured; a RedisRateLimiter built from this - # config's own values otherwise. Memoised under its own instance - # variable, like `provider_instance`, `cache_store` and - # `segmenter_instance`, so a copy builds its own limiter from its own - # settings instead of inheriting one built for a different config's - # namespace or connection pool. + # nil, not a null object: Request checks for nil and skips rate-limiting entirely -- costs nothing normally. def rate_limiter_instance return rate_limiter unless rate_limiter.nil? return nil if rate_limit.nil? @@ -124,9 +89,7 @@ def rate_limiter_instance @rate_limiter_instance ||= TranslationDiff::RedisRateLimiter.build(self) end - # One pool for the cache store and the rate limiter both. Callers used to - # build this themselves and pass it to each, keeping the namespaces in step - # by hand. + # One pool shared by the cache store and the rate limiter; callers used to build and pass it by hand. def redis_pool @redis_pool ||= build_redis_pool end @@ -146,14 +109,17 @@ def build_redis_pool '`gem "redis-namespace"` to your Gemfile.' end + # A default is resolved on every read, and a blank one is unset -- the rule the writer already applies. def read(key) value = instance_variable_get(:"@#{key}") return value unless value.nil? default = self.class.defaults[key] - default.respond_to?(:call) ? default.call : default + blank_to_nil(default.respond_to?(:call) ? default.call : default) end + def blank_to_nil(value) = value.is_a?(String) && value.strip.empty? ? nil : value + # The symbol-or-object rule, implemented once for all three extension points. def resolve(value, registry) value.is_a?(Symbol) || value.is_a?(String) ? registry.build(value, self) : value diff --git a/lib/translation_diff/configuration/provider_option_owners.rb b/lib/translation_diff/configuration/provider_option_owners.rb index df4f0f5..55e7759 100644 --- a/lib/translation_diff/configuration/provider_option_owners.rb +++ b/lib/translation_diff/configuration/provider_option_owners.rb @@ -1,29 +1,10 @@ -# frozen_string_literal: true - -# Tracks which provider declared each provider-specific configuration option -# name, so `Configuration.option` -- which returns early on a name it -# already knows -- never lets two providers silently share one accessor. If -# it did, a credential set for one provider would be handed to the other; -# that is a packaging conflict a human has to resolve, so #claim raises and -# names both. -# -# The same provider re-declaring its own options is not a conflict: a -# defensive double `require` and a Rails development reload both re-run -# registration, and a reload yields a *new* class object for the same -# constant, which is why identity is not the only test. Nor is a subclass of -# the declaring provider a conflict -- a provider subclassed to point it at -# a different host or account is meant to share its parent's options. +# Tracks which provider declared each option; two silently sharing one accessor would leak a credential. class TranslationDiff::Configuration::ProviderOptionOwners def initialize @owners = {} end - # All-or-nothing: every key is checked for a conflict before any of them - # is recorded as owned. Checking and recording key by key would leave the - # options before a conflicting one already attributed to a provider that - # then failed to register -- so a later, legitimate registration of one - # of those names would be refused, blaming a provider that was never - # registered. + # All-or-nothing: recording key by key would attribute earlier keys to a provider that then failed. def claim(keys, provider) validate!(keys, provider) keys.each { |key| @owners[key] = provider } @@ -40,10 +21,7 @@ def available?(key, provider) owner.nil? || same_provider?(owner, provider) end - # `<=>` returns non-nil (0 for equal, -1/1 for either direction) exactly - # when `owner` and `provider` sit on one inheritance chain, and nil for - # two unrelated classes -- which is also how it answers "or is not a - # Module at all", since only Modules define `<=>` this way. + # `<=>` is non-nil exactly when both sit on one inheritance chain -- also true if `provider` isn't a Module. def same_provider?(owner, provider) owner.equal?(provider) || (!owner.name.nil? && owner.name == provider.name) || diff --git a/lib/translation_diff/context.rb b/lib/translation_diff/context.rb index 5440e6c..e0c94b0 100644 --- a/lib/translation_diff/context.rb +++ b/lib/translation_diff/context.rb @@ -1,8 +1,4 @@ -# frozen_string_literal: true - -# An isolated configuration scope offering the same entry point as the -# TranslationDiff module itself, for multi-tenant applications and -# per-request overrides. Created with TranslationDiff.context. +# An isolated configuration scope with the same entry point as TranslationDiff itself. class TranslationDiff::Context attr_reader :config diff --git a/lib/translation_diff/error.rb b/lib/translation_diff/error.rb index c231e62..f733c1e 100644 --- a/lib/translation_diff/error.rb +++ b/lib/translation_diff/error.rb @@ -1,6 +1,2 @@ -# frozen_string_literal: true - -# Common ancestor for every error this gem raises. Without it, a caller who -# wants to rescue anything TranslationDiff can throw has to list four -# unrelated classes; with it, `rescue TranslationDiff::Error` is enough. +# Common ancestor for every error this gem raises, so `rescue TranslationDiff::Error` is enough. class TranslationDiff::Error < StandardError; end diff --git a/lib/translation_diff/errors.rb b/lib/translation_diff/errors.rb new file mode 100644 index 0000000..974d48d --- /dev/null +++ b/lib/translation_diff/errors.rb @@ -0,0 +1,35 @@ +# No error carries the text being translated -- errors are logged, and this library handles other people's content. +module TranslationDiff + class ConfigurationError < Error; end + + class ProviderError < Error + attr_reader :provider, :status + + def initialize(message, provider: nil, status: nil) + super(message) + @provider = provider + @status = status + end + end + + class AuthenticationError < ProviderError; end + class QuotaExceededError < ProviderError; end + class InvalidRequestError < ProviderError; end + class ServiceError < ProviderError; end + + class RateLimitError < ProviderError + # Faraday's retry middleware already honours this header; it's here for a caller scheduling its own retry. + attr_reader :retry_after + + def initialize(message, provider: nil, status: nil, retry_after: nil) + super(message, provider: provider, status: status) + @retry_after = retry_after + end + end + + class TransportError < Error; end + class ResponseError < Error; end + + # Its own class, not the generic Error, so rescuing "wrong shape" can't also swallow an option-name collision. + class InvalidProviderError < Error; end +end diff --git a/lib/translation_diff/http_provider.rb b/lib/translation_diff/http_provider.rb new file mode 100644 index 0000000..b591578 --- /dev/null +++ b/lib/translation_diff/http_provider.rb @@ -0,0 +1,119 @@ +require "faraday" +require "faraday/retry" +require "json" + +# Every HTTP provider inherits this; no logging middleware, ever -- lines must carry no source text or credential. +class TranslationDiff::HTTPProvider < TranslationDiff::Provider + RETRY_STATUSES = [429, 500, 502, 503, 504].freeze + + # Faraday raises these when nobody answered, as opposed to answering "no". + TRANSPORT_FAILURES = [Faraday::ConnectionFailed, Faraday::TimeoutError, Faraday::SSLError].freeze + + def api_base = raise NotImplementedError, "#{self.class} must implement #api_base" + def headers = {} + + def translate_url = raise NotImplementedError, "#{self.class} must implement #translate_url" + + def render_translate_payload(_request) + raise NotImplementedError, "#{self.class} must implement #render_translate_payload" + end + + def parse_translate_response(_body, _headers, _request) + raise NotImplementedError, "#{self.class} must implement #parse_translate_response" + end + + def translate(request) + response = post(translate_url, render_translate_payload(request)) + parse_translate_response(response.body, response.headers, request) + end + + def connection = @connection ||= build_connection + + private + + # Mirrors the shape a Faraday::Response would give if its own JSON middleware were still in the stack. + Decoded = Data.define(:status, :headers, :body) + private_constant :Decoded + + def post(url, payload) + raw = connection.post(url, payload) + response = Decoded.new(status: raw.status, headers: raw.headers, body: decode(raw)) + raise_for_status!(response) + response + rescue *TRANSPORT_FAILURES => e + # The message is the transport's, never the payload's: the payload is the customer's text. + raise TranslationDiff::TransportError, "#{self.class}: #{e.class}: #{e.message}" + end + + # Faraday's JSON middleware passes parser options positionally, which json 3 (default on Ruby 4.x) removed. + def decode(response) + body = response.body + return body unless body.is_a?(String) + return body if body.strip.empty? + return body unless json?(response) + + JSON.parse(body) + rescue JSON::ParserError => e + raise TranslationDiff::ResponseError, + "#{self.class} returned a body that is not JSON: #{e.message[0, 200]}" + end + + def json?(response) = response.headers["content-type"].to_s.match?(/\bjson\b/) + + # The block is how a test swaps in Faraday's test adapter; Amazon overrides it too, to sign the body as sent. + def build_connection(&) + Faraday.new(url: api_base, headers: headers) do |faraday| + faraday.request :json + faraday.request :retry, retry_options + adapt(faraday, &) + apply_timeouts(faraday) + end + end + + def adapt(faraday, &block) + block ? block.call(faraday) : faraday.adapter(Faraday.default_adapter) + end + + def apply_timeouts(faraday) + faraday.options.open_timeout = config.open_timeout + faraday.options.timeout = config.timeout + end + + # faraday-retry reads Retry-After itself, which is why a 429 usually never reaches #raise_for_status!. + def retry_options + { max: config.max_retries, interval: 0.5, backoff_factor: 2, interval_randomness: 0.5, + retry_statuses: RETRY_STATUSES, methods: %i[post get], + exceptions: TRANSPORT_FAILURES + [Faraday::RetriableResponse] } + end + + def raise_for_status!(response) + status = response.status + return if status < 400 + + raise error_class(status).new(error_message(response), **error_options(response)) + end + + def error_class(status) + case status + when 401, 403 then TranslationDiff::AuthenticationError + when 429 then TranslationDiff::RateLimitError + when 456 then TranslationDiff::QuotaExceededError + when 400..499 then TranslationDiff::InvalidRequestError + else TranslationDiff::ServiceError + end + end + + def error_options(response) + options = { provider: name, status: response.status } + return options unless response.status == 429 + + options.merge(retry_after: response.headers["Retry-After"]&.to_i) + end + + # Truncated: an untruncated provider error body can be a whole HTML error page. + def error_message(response) + body = response.body + text = body.is_a?(Hash) ? (body["message"] || body["error"] || body.to_s) : body.to_s + "#{self.class} responded #{response.status}: #{text.to_s[0, 300]}" + end +end diff --git a/lib/translation_diff/instrumentation.rb b/lib/translation_diff/instrumentation.rb index c4522c7..b251e3e 100644 --- a/lib/translation_diff/instrumentation.rb +++ b/lib/translation_diff/instrumentation.rb @@ -1,26 +1,11 @@ -# frozen_string_literal: true - -# Emits events to whatever the application configured as its instrumenter -- -# ActiveSupport::Notifications satisfies the interface as-is. When nothing is -# configured, #instrument yields (if given a block) and returns, so call -# sites never branch on whether instrumentation is on. -# -# Payloads carry counts, language codes and provider names. They never carry -# the text being translated, its translation, or a credential: this library -# handles other people's content, and an instrumenter usually writes -# somewhere that content must not go. +# Payloads carry counts, language codes and provider names -- never the text, its translation, or a credential. module TranslationDiff::Instrumentation - SUFFIX = ".translation_diff" + SUFFIX = ".translation_diff".freeze - # Both methods are private: they are internal plumbing for the class that - # includes this module, not part of its public surface. `include` ignores - # the includer's own `private` keyword, so the visibility has to be - # declared here. + # `include` ignores the includer's own `private` keyword, so visibility has to be declared here. private - # A point event (no block) reports a fact that already happened -- a cache - # hit/miss tally, say -- rather than wrapping work, so it only yields when - # a block was actually given. + # A point event (no block) reports a fact that already happened, e.g. a cache hit/miss tally. def instrument(name, payload = {}) instrumenter = config.instrumenter return yield if instrumenter.nil? && block_given? diff --git a/lib/translation_diff/linearizer.rb b/lib/translation_diff/linearizer.rb index b99e6e8..3e2f913 100644 --- a/lib/translation_diff/linearizer.rb +++ b/lib/translation_diff/linearizer.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - class TranslationDiff::Linearizer class << self def linearize(struct, array = []) diff --git a/lib/translation_diff/memory_cache_store.rb b/lib/translation_diff/memory_cache_store.rb index 69844a9..d7871aa 100644 --- a/lib/translation_diff/memory_cache_store.rb +++ b/lib/translation_diff/memory_cache_store.rb @@ -1,15 +1,4 @@ -# frozen_string_literal: true - -# The default cache: a bounded LRU in the current process, so the library -# works the moment it is required and without Redis running. -# -# Ruby hashes keep insertion order, so "least recently used" is delete and -# reinsert on every touch, and eviction is a shift of the first pair. -# -# NOT thread-safe, and deliberately so -- a lock here would be a tax on the -# single-threaded case to make the multi-threaded one merely less wrong. A -# process that needs a cache shared between threads or machines sets -# `redis_url` and gets TranslationDiff::RedisCacheStore instead. +# The default cache, a bounded in-process LRU. NOT thread-safe, deliberately -- set `redis_url` for that. class TranslationDiff::MemoryCacheStore def self.build(config) = new(max_size: config.cache_max_size) diff --git a/lib/translation_diff/provider.rb b/lib/translation_diff/provider.rb new file mode 100644 index 0000000..413977b --- /dev/null +++ b/lib/translation_diff/provider.rb @@ -0,0 +1,76 @@ +# Connects this library to one translation service; knows nothing about HTTP itself -- that's HTTPProvider. +class TranslationDiff::Provider + # A subclass that forgets to declare capabilities under-promises, not over-promises: smaller batches, not silent risk. + DEFAULT_CAPABILITIES = TranslationDiff::Capabilities.new( + max_request_size: 1_000, max_batch_size: 1, max_text_size: nil, + html: :none, notranslate: false, detects_language: false, reports_billing: false + ).freeze + + # A bare alphabetic code is cased the way the vendor documents; anything with a subtag is left alone. + BARE_LANGUAGE_CODE = /\A[A-Za-z]{2,3}\z/ + + # Stamped by the registry at build time. See #cache_key. + attr_accessor :name + + attr_reader :config + + def initialize(config) + @config = config + ensure_configured! + end + + # Callers write whichever casing their old configuration used; the vendor gets the one it documents. + def language(value) + code = value.to_s + return nil if code.empty? + return code unless code.match?(BARE_LANGUAGE_CODE) + + self.class.language_case == :upcase ? code.upcase : code.downcase + end + + # nil means the provider reported no billing at all; 0 means it reported zero. Both are claims. + def billed_characters(reported) + values = reported.compact + values.empty? ? nil : values.sum + end + + def translate(_request) = raise NotImplementedError, "#{self.class} must implement #translate" + + # Only called when `capabilities.detects_language?`. + def detect(_text) = raise NotImplementedError, "#{self.class} must implement #detect" + + # Raising when never stamped, rather than falling back to "", is deliberate: "" would merge namespaces silently. + def cache_key + return name.to_s unless name.nil? + + raise TranslationDiff::Error, + "#{self.class} has no cache key: it was instantiated directly instead of being " \ + "built through the registry. Build it through TranslationDiff::Providers.build, " \ + "or give #{self.class} its own #cache_key." + end + + class << self + # The casing this vendor documents for a bare code. DeepL upcases; everyone else takes lower case. + def language_case = :downcase + + def configuration_options = [] + + # Checked once, at build time, so a caller learns what to set before a vendor's own exception does. + def configuration_requirements = [] + + def capabilities = DEFAULT_CAPABILITIES + + def build(config) = new(config) + end + + private + + def ensure_configured! + missing = self.class.configuration_requirements.reject { |key| config.public_send(key) } + return if missing.empty? + + raise TranslationDiff::ConfigurationError, + "Provider #{self.class} is missing #{missing.join(', ')}. " \ + "Set #{missing.size == 1 ? 'it' : 'them'} in TranslationDiff.configure." + end +end diff --git a/lib/translation_diff/providers.rb b/lib/translation_diff/providers.rb index e57d27e..4d5c917 100644 --- a/lib/translation_diff/providers.rb +++ b/lib/translation_diff/providers.rb @@ -1,64 +1,46 @@ -# frozen_string_literal: true - -# Translation providers, by name. This is the one registry that does more -# than look a class up: registering a provider also declares that provider's -# configuration options, which is what keeps names like `deepl_api_key` out -# of TranslationDiff::Configuration and out of this library's core. -# -# TranslationDiff::Providers.register(:google, GoogleTranslateProvider) -# -# TranslationDiff.configure do |config| -# config.provider = :google -# config.google_api_key = ENV["GOOGLE_API_KEY"] -# end -# -# Nothing in lib/ changes to make that work. +# Translation providers, by name; registering one also declares its configuration options. module TranslationDiff::Providers - # Supplies the cache-key segment that keeps one provider's cached - # translations from being served to another. A provider built through the - # registry is stamped with its registered name and needs nothing else; a - # provider object assigned straight to `config.provider` never passed - # through here, so it must define #cache_key itself. - # - # Raising when `name` is unset -- rather than falling back to "" -- is - # deliberate: `cache_key` is a segment of every cache key this provider - # ever reads or writes, so a quietly empty one would let two different - # providers share the same cache namespace instead of failing loudly the - # first time anyone tries to use an unstamped instance. - module Naming - attr_accessor :name - - def cache_key - if name.nil? - raise TranslationDiff::Error, - "#{self.class} has no cache key: it was instantiated directly instead of being " \ - "built through the registry. Either build it through " \ - "TranslationDiff::Providers.build (which stamps #name for you), or give " \ - "#{self.class} its own #cache_key." - end - - name.to_s - end - end + # Said once, so a provider refused by name, by class or as an object is refused in the same words. + CONTRACT = "The base class supplies the transport, the configuration check and the " \ + "capability defaults, so a provider that skips it has none of them.".freeze class << self - # Options are declared before the registry entry is written, so a - # provider whose option names collide with another's raises without - # having replaced anything under `name`. + # Options are declared before the registry entry is written, so a name collision raises without replacing. def register(name, klass) - klass.include(Naming) unless klass.method_defined?(:cache_key) + ensure_provider_class!(klass) TranslationDiff::Configuration.register_provider_options(klass.configuration_options, klass) registry.register(name, klass) end + # Guards registration. `klass < Provider` alone raises NoMethodError for an instance or a non-Module. + def ensure_provider_class!(klass) + return klass if klass.is_a?(Class) && klass < TranslationDiff::Provider + + raise TranslationDiff::InvalidProviderError, + "#{describe(klass)} cannot be registered as a provider: the registry takes a " \ + "class inheriting TranslationDiff::Provider. #{CONTRACT}" + end + + # Guards the other two ways a provider arrives: assigned to config.provider, or passed as `provider:`. + def ensure_provider!(instance) + return instance if instance.is_a?(TranslationDiff::Provider) + + raise TranslationDiff::InvalidProviderError, + "#{describe(instance)} cannot be used as a provider: it does not inherit " \ + "TranslationDiff::Provider. #{CONTRACT}" + end + def build(name, config) - registry.build(name, config).tap do |provider| - provider.name = name.to_sym if provider.respond_to?(:name=) - end + registry.build(name, config).tap { |provider| provider.name = name.to_sym } end def registered?(name) = registry.registered?(name) def names = registry.names def registry = @registry ||= TranslationDiff::Registry.new("provider") + + private + + # Never #to_s on a non-Module: an arbitrary object renders its own content, or an address. + def describe(value) = value.is_a?(Module) ? value.to_s : "an instance of #{value.class}" end end diff --git a/lib/translation_diff/providers/amazon.rb b/lib/translation_diff/providers/amazon.rb new file mode 100644 index 0000000..8278655 --- /dev/null +++ b/lib/translation_diff/providers/amazon.rb @@ -0,0 +1,114 @@ +# Amazon Translate: no batch API (one text per call), no HTML mode, and requests are signed, not just headed. +class TranslationDiff::Providers::Amazon < TranslationDiff::HTTPProvider + SERVICE = "translate".freeze + TARGET = "AWSShineFrontendService_20170701.TranslateText".freeze + CONTENT_TYPE = "application/x-amz-json-1.1".freeze + + # Amazon's own way of asking for detection; reaches Comprehend under the hood, in regions that have it. + AUTO = "auto".freeze + + def self.capabilities + TranslationDiff::Capabilities.new( + max_request_size: 10_000, max_batch_size: 1, max_text_size: 10_000, + html: :none, notranslate: false, detects_language: true, reports_billing: false + ) + end + + # Deliberately no environment fallback: this library does not implement the AWS credential chain. + def self.configuration_options + %i[amazon_access_key_id amazon_secret_access_key amazon_session_token + amazon_region amazon_api_base] + end + + def self.configuration_requirements + %i[amazon_access_key_id amazon_secret_access_key amazon_region] + end + + def api_base = config.amazon_api_base || "https://#{SERVICE}.#{config.amazon_region}.amazonaws.com" + + # Detected language is the first one Amazon reported: every text in a chunk shares a source language. + def translate(request) + detected = nil + texts = request.texts.map do |text| + body = call(text, request) + detected ||= body["SourceLanguageCode"]&.downcase + body["TranslatedText"] + end + + TranslationDiff::Translation::Response.build( + request: request, texts: texts, detected_source: detected, + usage: TranslationDiff::Translation::Usage.new(characters: request.texts.sum(&:size)) + ) + end + + def detect(text) + call(text, TranslationDiff::Translation::Request.new(texts: [text], from: nil, to: "en")) + .fetch("SourceLanguageCode", nil)&.downcase + end + + private + + def call(text, request) + payload = request.options.transform_keys(&:to_s).merge( + "Text" => text, + "SourceLanguageCode" => request.from.nil? ? AUTO : language(request.from), + "TargetLanguageCode" => language(request.to) + ) + + post_signed(JSON.generate(payload)).body + end + + def post_signed(body) + raw = connection.post("/", body, signed_headers(body)) + response = decoded_response(raw) + raise_for_status!(response) + response + rescue *TRANSPORT_FAILURES => e + # The message is the transport's, never the payload's: the payload is the customer's text. + raise TranslationDiff::TransportError, "#{self.class}: #{e.class}: #{e.message}" + end + + # Reuses the base's own decoding rather than a second, Faraday-middleware-based path (see HTTPProvider#decode). + def decoded_response(raw) = Decoded.new(status: raw.status, headers: raw.headers, body: decode(raw)) + + # Required here, not at load time, so an application using another provider never needs it installed. + def signer + @signer ||= begin + require_sigv4 + Aws::Sigv4::Signer.new( + service: SERVICE, region: config.amazon_region, + access_key_id: config.amazon_access_key_id, + secret_access_key: config.amazon_secret_access_key, + session_token: config.amazon_session_token + ) + end + end + + def require_sigv4 + require "aws-sigv4" + rescue LoadError + raise TranslationDiff::Error, + "provider is :amazon but the `aws-sigv4` gem is not available. " \ + 'Add `gem "aws-sigv4"` to your Gemfile.' + end + + def signed_headers(body) + signature = signer.sign_request( + http_method: "POST", url: "#{api_base}/", body: body, + headers: { "Content-Type" => CONTENT_TYPE, "X-Amz-Target" => TARGET } + ) + + signature.headers.merge("Content-Type" => CONTENT_TYPE, "X-Amz-Target" => TARGET) + end + + # The signature covers the body exactly as sent, so this omits `faraday.request :json` unlike the base class. + def build_connection(&) + Faraday.new(url: api_base, headers: headers) do |faraday| + faraday.request :retry, retry_options + adapt(faraday, &) + apply_timeouts(faraday) + end + end +end + +TranslationDiff::Providers.register(:amazon, TranslationDiff::Providers::Amazon) diff --git a/lib/translation_diff/providers/azure.rb b/lib/translation_diff/providers/azure.rb new file mode 100644 index 0000000..9cc4d04 --- /dev/null +++ b/lib/translation_diff/providers/azure.rb @@ -0,0 +1,69 @@ +# Azure AI Translator, REST v3.0: cheapest per character, most generous per request (1,000 strings/50,000 chars). +class TranslationDiff::Providers::Azure < TranslationDiff::HTTPProvider + HOST = "https://api.cognitive.microsofttranslator.com".freeze + API_VERSION = "3.0".freeze + + # Azure spells HTML handling `textType`, and under it honours `class=notranslate` like DeepL and Google do. + DEFAULT_TEXT_TYPE = "html".freeze + + def self.capabilities + TranslationDiff::Capabilities.new( + max_request_size: 50_000, max_batch_size: 1_000, max_text_size: 50_000, + html: :textType, notranslate: true, detects_language: true, reports_billing: true + ) + end + + def self.configuration_options = %i[azure_api_key azure_region azure_api_base] + def self.configuration_requirements = %i[azure_api_key] + + def api_base = config.azure_api_base || HOST + + # A multi-service resource needs the region header; a single-service one rejects nothing without it. + def headers + { "Ocp-Apim-Subscription-Key" => config.azure_api_key.to_s } + .tap { |h| h["Ocp-Apim-Subscription-Region"] = config.azure_region if config.azure_region } + end + + # Azure takes the language pair in the query string, so the URL is built per request, not a constant. + def translate_url = "translate" + + def translate(request) + response = post(url_for(request), render_translate_payload(request)) + parse_translate_response(response.body, response.headers, request) + end + + def render_translate_payload(request) = request.texts.map { |text| { Text: text } } + + def parse_translate_response(body, headers, request) + results = Array(body) + + TranslationDiff::Translation::Response.build( + request: request, + texts: results.map { |result| result.dig("translations", 0, "text") }, + detected_source: results.first&.dig("detectedLanguage", "language")&.downcase, + usage: TranslationDiff::Translation::Usage.new( + characters: request.texts.sum(&:size), + billed_characters: headers["x-metered-usage"]&.to_i + ) + ) + end + + def detect(text) + response = post("detect?api-version=#{API_VERSION}", [{ Text: text }]) + response.body.dig(0, "language")&.downcase + end + + private + + # Defaults, then caller options, then mandatory fields: a caller must not displace the language pair. + def url_for(request) + params = { "textType" => DEFAULT_TEXT_TYPE } + .merge(request.options.transform_keys(&:to_s)) + .merge("api-version" => API_VERSION, "to" => language(request.to)) + params["from"] = language(request.from) unless request.from.nil? + + "#{translate_url}?#{URI.encode_www_form(params)}" + end +end + +TranslationDiff::Providers.register(:azure, TranslationDiff::Providers::Azure) diff --git a/lib/translation_diff/providers/deepl.rb b/lib/translation_diff/providers/deepl.rb index a67b09f..6283a3e 100644 --- a/lib/translation_diff/providers/deepl.rb +++ b/lib/translation_diff/providers/deepl.rb @@ -1,81 +1,77 @@ -# frozen_string_literal: true - -# Talks to DeepL through deepl-rb's per-instance objects rather than its -# module-level shortcuts, so this library never calls DeepL.configure and -# never mutates another gem's global state. Two behaviours come for free by -# using their Configuration: it reads DEEPL_AUTH_KEY when no key is given, -# and it picks the free or the paid host from the key's ":fx" suffix. -# -# deepl-rb is not a dependency of this gem. It is required at build time, so -# an application using a different provider never needs it installed. -class TranslationDiff::Providers::DeepL - # DeepL requires a target language even when only the detection is - # wanted, so the provider picks one rather than making the caller do it. - DETECTION_TARGET = "EN" - - MAX_REQUEST_SIZE = 1700 - MAX_BATCH_SIZE = 300 - - # What arrives here is not plain text, despite having been through the - # Tokenizer. A notranslate span is handed over whole, tags included -- - # that is how the tokenizer marks content the provider must leave alone. - # DeepL honours `class="notranslate"` (and `translate="no"`) only under - # HTML tag handling; without it, in DeepL's own words, "tags are treated - # as regular text". The failure is quiet, because DeepL leaves the tags - # themselves alone either way and only the protected content changes: - # - # "Bold Mountain is a good place." - # no tag_handling -> "Болд-Маунтин — отличное место." - # tag_handling -> "Bold Mountain — это хорошее место." - # - # v2 is the tag handling algorithm DeepL's documentation recommends. - # Note that under HTML tag handling DeepL defaults `split_sentences` to - # `nonewlines`; this library sends one sentence at a time, so that - # changes nothing here. +# Talks to DeepL's REST API directly, not deepl-rb: it logged the auth key at DEBUG and defaulted notranslate off. +class TranslationDiff::Providers::DeepL < TranslationDiff::HTTPProvider + PAID_HOST = "https://api.deepl.com".freeze + FREE_HOST = "https://api-free.deepl.com".freeze + + # A key ending in :fx is a free-plan key, and the free plan lives on its own host. + FREE_KEY_SUFFIX = ":fx".freeze + + # DeepL honours class="notranslate" only under HTML tag handling -- otherwise content translates, tags survive. DEFAULT_OPTIONS = { tag_handling: :html, tag_handling_version: "v2" }.freeze - def self.configuration_options = %i[deepl_api_key deepl_host] - - # `config.logger` is deliberately NOT forwarded into DeepL::Configuration. - # deepl-rb logs the whole request at DEBUG -- a "Request details:" line - # carrying the Authorization header, DeepL auth key and all, followed by - # the payload, which is the text being translated. This library's logger - # carries a guarantee that no line it writes holds translated text, source - # text, or a credential; handing it to a gem that logs payloads would break - # that guarantee silently, at the exact moment someone turns DEBUG on to - # diagnose a problem. Anyone who wants deepl-rb's own request log can build - # the DeepL::API themselves, wrap it in this provider, and assign that to - # `config.provider` -- see "Instrumentation and logging" in the README. - def self.build(config) - require "deepl" - - settings = { auth_key: config.deepl_api_key, host: config.deepl_host }.compact - new(::DeepL::API.new(::DeepL::Configuration.new(settings))) - rescue LoadError - raise TranslationDiff::Error, - "provider is :deepl but the `deepl-rb` gem is not available. " \ - 'Add `gem "deepl-rb"` to your Gemfile.' + # 50 texts / 128 KiB are DeepL's documented per-request limits; max_batch_size was wrong before (it said 300). + def self.capabilities + TranslationDiff::Capabilities.new( + max_request_size: 1_700, max_batch_size: 50, max_text_size: nil, + html: :tag_handling, notranslate: true, detects_language: true, reports_billing: true + ) end - def initialize(api) - @api = api + # DEEPL_AUTH_KEY is what deepl-rb read on our behalf; the callable keeps it read on use, not at load. + def self.configuration_options + [:deepl_api_base, { deepl_api_key: -> { ENV.fetch("DEEPL_AUTH_KEY", nil) } }] end - def translate(texts, from:, to:, **options) - Array(request(texts, from, to, DEFAULT_OPTIONS.merge(options))).map(&:text) + def self.configuration_requirements = %i[deepl_api_key] + + # DeepL is the one vendor documenting upper-case codes. + def self.language_case = :upcase + + # DeepL requires a target language even when only detection is wanted, so the provider picks one. + DETECTION_TARGET = "EN".freeze + + def api_base + config.deepl_api_base || (free_key? ? FREE_HOST : PAID_HOST) end - def detect(text) - request(text, nil, DETECTION_TARGET).detected_source_language.downcase + def headers = { "Authorization" => "DeepL-Auth-Key #{config.deepl_api_key}" } + + def translate_url = "v2/translate" + + def render_translate_payload(request) + DEFAULT_OPTIONS + .merge(request.options) + .merge(text: request.texts, target_lang: language(request.to)) + .tap { |payload| payload[:source_lang] = language(request.from) unless request.from.nil? } end - def max_request_size = MAX_REQUEST_SIZE - def max_batch_size = MAX_BATCH_SIZE + def parse_translate_response(body, _headers, request) + translations = Array(body["translations"]) + + TranslationDiff::Translation::Response.build( + request: request, + texts: translations.map { |t| t["text"] }, + detected_source: translations.first&.dig("detected_source_language")&.downcase, + usage: usage_for(request, translations) + ) + end + + # DeepL has no detection endpoint; translating a sample and reading the source it reports is the only way. + def detect(text) + request = TranslationDiff::Translation::Request.new(texts: [text], from: nil, + to: DETECTION_TARGET) + translate(request).detected_source + end private - def request(text, from, to, options = {}) - ::DeepL::Requests::Translate.new(@api, text, from, to, options).request + def free_key? = config.deepl_api_key.to_s.end_with?(FREE_KEY_SUFFIX) + + def usage_for(request, translations) + TranslationDiff::Translation::Usage.new( + characters: request.texts.sum(&:size), + billed_characters: billed_characters(translations.map { |t| t["billed_characters"] }) + ) end end diff --git a/lib/translation_diff/providers/google.rb b/lib/translation_diff/providers/google.rb index 58d2b67..020e4a8 100644 --- a/lib/translation_diff/providers/google.rb +++ b/lib/translation_diff/providers/google.rb @@ -1,100 +1,52 @@ -# frozen_string_literal: true +# Talks to Cloud Translation v2 directly, not google-cloud-translate-v2, which pulled in grpc for one POST. +class TranslationDiff::Providers::Google < TranslationDiff::HTTPProvider + HOST = "https://translation.googleapis.com".freeze -# Talks to Google Cloud Translation v2 (Basic) through the API object the -# google-cloud-translate-v2 gem builds, rather than its module-level -# shortcuts, so this library never mutates another gem's global state. Two -# behaviours come for free by using their constructor: it reads TRANSLATE_KEY -# and GOOGLE_CLOUD_KEY when no key is given, and it falls back to application -# default credentials when there is no key at all. -# -# google-cloud-translate-v2 is not a dependency of this gem. It is required -# at build time, so an application using a different provider never needs it -# installed. -class TranslationDiff::Providers::Google - # Google's own numbers. The batch limit is a hard one -- "the maximum - # number of strings is 128" -- and a larger request is rejected outright. - # The size limit is the documented recommendation of 5K characters per - # request, well under the hard 100K-byte ceiling. Chunker measures the - # URL-escaped form, which is never smaller than the UTF-8 byte count, so - # staying under this in escaped characters keeps every request under it in - # bytes too. - MAX_REQUEST_SIZE = 5_000 - MAX_BATCH_SIZE = 128 - - # What arrives here is not plain text, despite having been through the - # Tokenizer. A notranslate span is handed over whole, tags included -- - # that is how the tokenizer marks content the provider must leave alone -- - # and HTML entities such as `&` stay in the text it emits. Asking - # Google for `text` makes it translate the protected span and drop the - # markup around it entirely: - # - # "Bold Mountain is a good place." - # format: text -> "Болд Маунтин — хорошее место." - # format: html -> "Bold Mountain — хорошее место." - # - # So `html` it is, which is also what this gem sent for its whole life - # before the provider seam existed. The cost is that Google escapes its - # own output -- a literal apostrophe returns as "'" -- which is - # correct inside the HTML fragment these values usually are, and noise - # inside a value that never had markup in it. A caller translating bare - # strings can pass `format: :text` per call. + # Verified against the live API: `text` format translates the protected span and drops its markup. DEFAULT_FORMAT = :html - # A bare alphabetic code is downcased, so a configuration written for - # DeepL ("EN") keeps working against Google, whose codes are lowercase. - # Anything else is passed through untouched: "zh-Hans", "zh-CN" and - # "pt-BR" carry script and region subtags whose casing is their own, and a - # blanket downcase would corrupt them. - BARE_LANGUAGE_CODE = /\A[A-Za-z]{2,3}\z/ - - def self.configuration_options = %i[google_api_key google_project_id] - - # `config.logger` is deliberately not forwarded: the gem takes no logger, - # and this library's own guarantee -- that no line it writes holds source - # text, translated text or a credential -- is easiest to keep by never - # handing the logger to a gem that has not made the same promise. - def self.build(config) - require "google/cloud/translate/v2" - - settings = { key: config.google_api_key, project_id: config.google_project_id }.compact - new(::Google::Cloud::Translate::V2.new(**settings)) - rescue LoadError - raise TranslationDiff::Error, - "provider is :google but the `google-cloud-translate-v2` gem is not available. " \ - 'Add `gem "google-cloud-translate-v2"` to your Gemfile.' + # Google's documented limits: 128 strings/request, 5,000 chars recommended (hard ceiling 100 KB). + def self.capabilities + TranslationDiff::Capabilities.new( + max_request_size: 5_000, max_batch_size: 128, max_text_size: nil, + html: :format, notranslate: true, detects_language: true, reports_billing: false + ) end - def initialize(api) - @api = api + # TRANSLATE_KEY then GOOGLE_CLOUD_KEY, the order google-cloud-translate-v2 read them in. + def self.configuration_options + [:google_api_base, + { google_api_key: -> { ENV.fetch("TRANSLATE_KEY", nil) || ENV.fetch("GOOGLE_CLOUD_KEY", nil) }, + google_project_id: -> { ENV.fetch("TRANSLATE_PROJECT", nil) } }] end - def translate(texts, from:, to:, **options) - settings = { from: language(from), to: language(to), format: DEFAULT_FORMAT }.merge(options) - - results = @api.translate(*texts, **settings) - # One text yields a bare Translation, not a one-element array. `Array()` - # is not usable to even that out: Translation would have to be trusted - # never to define #to_a or #to_ary, and if it ever did, `Array()` would - # quietly splat one translation into several strings instead of raising. - results = [results] unless results.is_a?(Array) + def self.configuration_requirements = %i[google_api_key] - results.map(&:text) - end + def api_base = config.google_api_base || HOST + def translate_url = "language/translate/v2?key=#{CGI.escape(config.google_api_key.to_s)}" + def detect_url = "language/translate/v2/detect?key=#{CGI.escape(config.google_api_key.to_s)}" - def detect(text) - @api.detect(text).language + def render_translate_payload(request) + { format: DEFAULT_FORMAT } + .merge(request.options) + .merge(q: request.texts, target: language(request.to)) + .tap { |payload| payload[:source] = language(request.from) unless request.from.nil? } end - def max_request_size = MAX_REQUEST_SIZE - def max_batch_size = MAX_BATCH_SIZE - - private + def parse_translate_response(body, _headers, request) + translations = Array(body.dig("data", "translations")) - def language(value) - code = value.to_s - return nil if code.empty? + TranslationDiff::Translation::Response.build( + request: request, + texts: translations.map { |t| t["translatedText"] }, + detected_source: translations.first&.dig("detectedSourceLanguage")&.downcase, + usage: TranslationDiff::Translation::Usage.new(characters: request.texts.sum(&:size)) + ) + end - code.match?(BARE_LANGUAGE_CODE) ? code.downcase : code + def detect(text) + response = post(detect_url, { q: [text] }) + response.body.dig("data", "detections", 0, 0, "language")&.downcase end end diff --git a/lib/translation_diff/providers/libretranslate.rb b/lib/translation_diff/providers/libretranslate.rb new file mode 100644 index 0000000..5a2728a --- /dev/null +++ b/lib/translation_diff/providers/libretranslate.rb @@ -0,0 +1,55 @@ +# The only free, self-hosted provider here; base URL is required (everyone runs their own), API key is optional. +class TranslationDiff::Providers::LibreTranslate < TranslationDiff::HTTPProvider + DEFAULT_FORMAT = "html".freeze + + # The API's own way of asking for detection: `source` is required, and "auto" means "work it out". + AUTO = "auto".freeze + + # Observed 2026-09-09 via Docker: LibreTranslate's HTML format preserves markup but translates content anyway. + LIBRETRANSLATE_HONOURS_NOTRANSLATE = false + + # LibreTranslate publishes no per-request limits; these are this library's own conservative numbers. + def self.capabilities + TranslationDiff::Capabilities.new( + max_request_size: 5_000, max_batch_size: 50, max_text_size: nil, + html: :format, notranslate: LIBRETRANSLATE_HONOURS_NOTRANSLATE, + detects_language: true, reports_billing: false + ) + end + + def self.configuration_options = %i[libretranslate_api_key libretranslate_api_base] + def self.configuration_requirements = %i[libretranslate_api_base] + + def api_base = config.libretranslate_api_base + def translate_url = "translate" + + def render_translate_payload(request) + { format: DEFAULT_FORMAT } + .merge(request.options) + .merge(q: request.texts, target: language(request.to), + source: request.from.nil? ? AUTO : language(request.from)) + .tap { |payload| payload[:api_key] = config.libretranslate_api_key if config.libretranslate_api_key } + end + + def parse_translate_response(body, _headers, request) + translated = body["translatedText"] + detected = body["detectedLanguage"] + detected = detected.first if detected.is_a?(Array) + + TranslationDiff::Translation::Response.build( + request: request, + texts: translated.is_a?(Array) ? translated : [translated].compact, + detected_source: detected.is_a?(Hash) ? detected["language"]&.downcase : nil, + usage: TranslationDiff::Translation::Usage.new(characters: request.texts.sum(&:size)) + ) + end + + def detect(text) + payload = { q: text } + payload[:api_key] = config.libretranslate_api_key if config.libretranslate_api_key + + post("detect", payload).body.dig(0, "language")&.downcase + end +end + +TranslationDiff::Providers.register(:libretranslate, TranslationDiff::Providers::LibreTranslate) diff --git a/lib/translation_diff/providers/modernmt.rb b/lib/translation_diff/providers/modernmt.rb new file mode 100644 index 0000000..b3bc68d --- /dev/null +++ b/lib/translation_diff/providers/modernmt.rb @@ -0,0 +1,66 @@ +# ModernMT: adaptive translation with translation memories. +class TranslationDiff::Providers::ModernMT < TranslationDiff::HTTPProvider + HOST = "https://api.modernmt.com".freeze + + # ModernMT spells its formats as MIME types. + DEFAULT_FORMAT = "text/html".freeze + + # Unverified, not observed: no key was available to probe it; false is the safe assumption either way. + MODERNMT_HONOURS_NOTRANSLATE = false + + # 128 texts is documented; the character limit is not, so Google's 5,000 recommendation is borrowed. + def self.capabilities + TranslationDiff::Capabilities.new( + max_request_size: 5_000, max_batch_size: 128, max_text_size: nil, + html: :format, notranslate: MODERNMT_HONOURS_NOTRANSLATE, + detects_language: true, reports_billing: true + ) + end + + def self.configuration_options = %i[modernmt_api_key modernmt_api_base] + def self.configuration_requirements = %i[modernmt_api_key] + + def api_base = config.modernmt_api_base || HOST + def headers = { "MMT-ApiKey" => config.modernmt_api_key.to_s } + def translate_url = "translate" + + def render_translate_payload(request) + { format: DEFAULT_FORMAT } + .merge(request.options) + .merge(q: request.texts, target: language(request.to)) + .tap { |payload| payload[:source] = language(request.from) unless request.from.nil? } + end + + # One text comes back as an object rather than a one-element array, so the envelope is always coerced. + def parse_translate_response(body, _headers, request) + results = results_from(body) + + TranslationDiff::Translation::Response.build( + request: request, + texts: results.map { |r| r["translation"] }, + detected_source: results.first&.dig("detectedLanguage")&.downcase, + usage: usage_for(request, results) + ) + end + + def detect(text) + request = TranslationDiff::Translation::Request.new(texts: [text], from: nil, to: "en") + translate(request).detected_source + end + + private + + def results_from(body) + data = body["data"] + data.is_a?(Array) ? data : [data].compact + end + + def usage_for(request, results) + TranslationDiff::Translation::Usage.new( + characters: request.texts.sum(&:size), + billed_characters: billed_characters(results.map { |r| r["billedCharacters"] }) + ) + end +end + +TranslationDiff::Providers.register(:modernmt, TranslationDiff::Providers::ModernMT) diff --git a/lib/translation_diff/providers/null.rb b/lib/translation_diff/providers/null.rb index f344fe8..2882676 100644 --- a/lib/translation_diff/providers/null.rb +++ b/lib/translation_diff/providers/null.rb @@ -1,20 +1,19 @@ -# frozen_string_literal: true - -# Hands back what it was given. For tests, and for wiring a pipeline up -# before a real provider is available. -class TranslationDiff::Providers::Null - def self.configuration_options = [] - def self.build(_config) = new +# Hands back what it was given -- for tests, and for wiring a pipeline up before a real provider is available. +class TranslationDiff::Providers::Null < TranslationDiff::Provider + # Deliberately not detecting: this is the provider that proves the optional branch works. + def self.capabilities + TranslationDiff::Capabilities.new( + max_request_size: 1_000_000, max_batch_size: 1_000_000, max_text_size: nil, + html: :none, notranslate: false, detects_language: false, reports_billing: false + ) + end - # No #detect on purpose: detection is optional in the contract, and this - # is the provider that proves the optional branch works. - # rubocop:disable-next Lint/UnusedMethodArgument - def translate(texts, from:, to:, **options) - texts.map(&:to_s) + def translate(request) + TranslationDiff::Translation::Response.build( + request: request, texts: request.texts.map(&:to_s) + ) end - def max_request_size = 1_000_000 - def max_batch_size = 1_000_000 def cache_key = "null" end diff --git a/lib/translation_diff/redis_cache_store.rb b/lib/translation_diff/redis_cache_store.rb index 576a744..9dde87d 100644 --- a/lib/translation_diff/redis_cache_store.rb +++ b/lib/translation_diff/redis_cache_store.rb @@ -1,15 +1,12 @@ -# frozen_string_literal: true - class TranslationDiff::RedisCacheStore ONE_WEEK = 60 * 60 * 24 * 7 - DEFAULT_NAMESPACE = "translation-diff" + DEFAULT_NAMESPACE = "translation-diff".freeze def self.build(config) new(config.redis_pool, timeout: config.cache_ttl, namespace: config.cache_namespace) end - # `connection_pool` is anything answering to #with, and what it yields is - # anything Redis::Namespace accepts. Neither gem is a dependency of this one. + # `connection_pool` is duck-typed to #with; neither connection_pool nor redis-namespace is a hard dependency. def initialize(connection_pool, timeout: ONE_WEEK, namespace: DEFAULT_NAMESPACE) @connection_pool = connection_pool @timeout = timeout diff --git a/lib/translation_diff/redis_rate_limiter.rb b/lib/translation_diff/redis_rate_limiter.rb index c70f64c..fd42ed1 100644 --- a/lib/translation_diff/redis_rate_limiter.rb +++ b/lib/translation_diff/redis_rate_limiter.rb @@ -1,16 +1,12 @@ -# frozen_string_literal: true - class TranslationDiff::RedisRateLimiter class RateLimitExceeded < TranslationDiff::Error; end DEFAULT_THRESHOLD = 8000 DEFAULT_INTERVAL = 60 - DEFAULT_NAMESPACE = "translation-diff" + DEFAULT_NAMESPACE = "translation-diff".freeze - # Ratelimit counts per subject. This library limits the provider as a - # whole rather than per caller, so there is exactly one subject and it - # only has to be stable. - SUBJECT = "call" + # This library limits the provider as a whole rather than per caller, so there is exactly one subject. + SUBJECT = "call".freeze def self.build(config) new(config.redis_pool, @@ -19,8 +15,7 @@ def self.build(config) namespace: config.cache_namespace) end - # `connection_pool` is anything answering to #with, and what it yields is - # anything Ratelimit accepts. Neither gem is a dependency of this one. + # `connection_pool` is duck-typed to #with; neither connection_pool nor ratelimit is a hard dependency. def initialize(connection_pool, threshold: DEFAULT_THRESHOLD, interval: DEFAULT_INTERVAL, @@ -46,11 +41,7 @@ def check(size) attr_reader :connection_pool, :threshold, :interval, :namespace - # `ratelimit` is not a dependency of this gem, so it is required here, at - # the first check, rather than at load time -- an application that - # configures no `rate_limit` never needs it installed. Naming the bare - # constant instead surfaced its absence as a raw NameError; this raises the - # same "add this gem" TranslationDiff::Error the Redis path already does. + # Required at first check, not load time; naming the bare constant instead would raise a raw NameError. def ratelimit_class require "ratelimit" ::Ratelimit diff --git a/lib/translation_diff/registry.rb b/lib/translation_diff/registry.rb index 9d9d90b..9d98fcc 100644 --- a/lib/translation_diff/registry.rb +++ b/lib/translation_diff/registry.rb @@ -1,15 +1,6 @@ -# frozen_string_literal: true - -# Maps a short symbol to a class that knows how to build itself from a -# Configuration. Three of these exist -- providers, cache stores and -# segmenters -- so that every extension option in this library can accept -# either a symbol naming a built-in or an object the caller supplies. -# -# The only thing the three kinds have in common is that a registered class -# answers `build(config)`. That is deliberately the whole contract. +# Maps a symbol to a class that builds itself from a Configuration; the whole contract is answering `build(config)`. class TranslationDiff::Registry - # `kind` appears in the error message for an unknown name, so it should be - # the singular noun a reader would use: "provider", "cache store". + # `kind` appears in the unknown-name error message, so it should be a singular noun: "provider". def initialize(kind) @kind = kind @entries = {} diff --git a/lib/translation_diff/request.rb b/lib/translation_diff/request.rb index d967e80..b193084 100644 --- a/lib/translation_diff/request.rb +++ b/lib/translation_diff/request.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - class TranslationDiff::Request extend Forwardable include TranslationDiff::Instrumentation @@ -30,15 +28,16 @@ def call attr_reader :values, :options, :to, :config - # The provider for this call: the `provider:` keyword when the caller gave - # one, otherwise whatever the configuration resolves to. + # The `provider:` keyword when the caller gave one, otherwise whatever the configuration resolves to. def api @api ||= (@provider.nil? ? config.provider_instance : resolve_provider(@provider)) .tap { |provider| log("provider #{provider.class}") } end def resolve_provider(value) - value.is_a?(Symbol) || value.is_a?(String) ? TranslationDiff::Providers.build(value, config) : value + return TranslationDiff::Providers.build(value, config) if value.is_a?(Symbol) || value.is_a?(String) + + TranslationDiff::Providers.ensure_provider!(value) end def rate_limiter = config.rate_limiter_instance @@ -47,53 +46,41 @@ def from @from ||= detect_language end - # A detected language arrives as a String while :to is usually a Symbol, so - # the two have to be compared on equal footing or the short circuit never - # fires and the text gets translated into its own language. + # A detected language is a String while :to is usually a Symbol -- without casecmp? this never short-circuits. def same_language? !to.nil? && from.to_s.casecmp?(to.to_s) end - # Covers values holding no translatable text at all: "", nil, an empty - # collection, or a scalar the tokenizer has nothing to say about. + # Covers "", nil, an empty collection, or a scalar the tokenizer has nothing to say about. def nothing_to_translate? text_tokens_texts.all?(&:empty?) end + def capabilities = api.class.capabilities + def detect_language - raise Error, "Pass from: -- #{api.class} cannot detect the source language" unless api.respond_to?(:detect) + unless capabilities.detects_language? + raise Error, "Pass from: -- provider #{provider_cache_key} cannot detect the source language" + end api.detect(text_tokens_texts.join(" ")[0..100]) end - # Extracts flat text array - # => "Name", "Good boy" - # - # #values might be something like { name: "Name", bio: "Good boy" } def texts @texts ||= linearize(values) end - # Converts each array item to token list - # => [..., [["", :markup], ["Good", :text], ...]] def tokens @tokens ||= texts.map do |value| TranslationDiff::Tokenizer.tokenize(value, segmenter: config.segmenter_instance, language: source_language) end end - # The segmenter's language, not the resolved one: `from` triggers - # auto-detection the first time it is called, and detection builds its - # sample from the segmented text, so asking `from` here would be circular. - # Only a language the caller actually passed is usable at this point -- - # everything else genuinely doesn't know yet, and nil is the honest - # answer. + # Not the resolved `from`: detection builds its sample from the segmented text, so asking `from` here is circular. def source_language @from&.to_s end - # Extracts text tokens from token list - # => { ..., "1_1" => "Good", 1_3 => "Boy", ... } def text_tokens @text_tokens ||= extract_text_tokens.to_h end @@ -106,25 +93,18 @@ def extract_text_tokens end end - # Extracts values from text tokens - # => [ ..., "Good", "Boy", ... ] def text_tokens_texts @text_tokens_texts ||= linearize(text_tokens).map(&:to_s).map(&:strip) end - # Splits things requires translations to per-request chunks - # (groups less 2k sym) - # => [[ ..., "Good", "Boy", ... ]] def chunks @chunks ||= TranslationDiff::Chunker.new( text_tokens_texts, - limit: api.max_request_size, - count_limit: api.max_batch_size + limit: capabilities.max_request_size, + count_limit: capabilities.max_batch_size ).call end - # Translates/loads from cache values from each chunk - # => [[ ..., "Horoshiy", "Malchik", ... ]] def chunks_translated @chunks_translated ||= chunks.map do |chunk| cached, missing = cache.cached_and_missing(chunk) @@ -137,15 +117,11 @@ def chunks_translated end end - # Restores indexes for translated tokens - # => { ..., "1_1" => "Horoshiy", 1_3 => "Malchik", ... } def text_tokens_translated @text_tokens_translated ||= restore(text_tokens, chunks_translated.flatten) end - # Restores tokens translated + adds same spacing as in source token - # => [[..., [ "Horoshiy", :text ], ...]] # rubocop:disable-next Metrics/AbcSize def tokens_translated @tokens_translated ||= tokens.dup.tap do |tokens| @@ -161,13 +137,10 @@ def restore_spacing(source_value, value) TranslationDiff::Spacing.restore(source_value, value) end - # Restores texts from tokens - # [..., "Horoshiy Malchik", ...] def texts_translated @texts_translated ||= tokens_translated.map.with_index do |group, index| source = texts[index] - # Only strings are rebuilt from tokens. Anything else has no tokens to - # rebuild from; nil keeps collapsing to "" the way it always has. + # Only strings are rebuilt from tokens; nil keeps collapsing to "" the way it always has. next source unless source.nil? || source.is_a?(String) group.map { |value, type| type == :text ? value : fix_ascii(value) }.join @@ -181,17 +154,15 @@ def translation def call_api(values) check_rate_limit(values) - translations = instrument("request", provider: provider_cache_key, - batch: values.size, - characters: values.sum(&:size)) do - api.translate(values, from: from, to: to, **options) + request = TranslationDiff::Translation::Request.new( + texts: values, from: from, to: to, options: options + ) + response = instrument("request", provider: provider_cache_key, batch: values.size, + characters: values.sum(&:size)) do + api.translate(request) end - return translations if translations.size == values.size - - # Letting a short response through means shifting nils into the results, - # which surfaces much later as a NoMethodError far from the cause. - raise Error, - "Provider returned #{translations.size} translations for #{values.size} values" + # Dup'd: the array is the provider's own, and handing it to a collaborator makes it the collaborator's too. + response.texts.dup end def cache @@ -200,11 +171,7 @@ def cache ) end - # A provider built through the registry is stamped with its name. An object - # assigned straight to `config.provider` never passed through the registry, - # so it has to supply this itself -- without it two providers' translations - # would share cache entries and a caller would be served the wrong service's - # answer. + # An object assigned straight to `config.provider` never passed through the registry's stamping. def provider_cache_key key = api.cache_key if api.respond_to?(:cache_key) return key unless key.nil? || key.to_s.strip.empty? diff --git a/lib/translation_diff/segmenters.rb b/lib/translation_diff/segmenters.rb index 7e178ac..5d2a5f2 100644 --- a/lib/translation_diff/segmenters.rb +++ b/lib/translation_diff/segmenters.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - module TranslationDiff::Segmenters # Segmenters, by name. `config.segmenter = :simple` resolves through here. def self.registry = @registry ||= TranslationDiff::Registry.new("segmenter") diff --git a/lib/translation_diff/segmenters/pragmatic.rb b/lib/translation_diff/segmenters/pragmatic.rb index 6314509..a147d0e 100644 --- a/lib/translation_diff/segmenters/pragmatic.rb +++ b/lib/translation_diff/segmenters/pragmatic.rb @@ -1,63 +1,14 @@ -# frozen_string_literal: true - require "pragmatic_segmenter" -# Splits a string into sentence-sized cache units using the -# `pragmatic_segmenter` gem's per-language rule sets. This is the default -# segmenter: measured against the Golden Rules corpus -- the -# `context "Golden Rules" do` block of each of the 10 per-language spec -# files on diasks2/pragmatic_segmenter, 80 exemplars in total; a sample of -# the same corpus is in test/translation_diff/golden_rules_test.rb -- this -# class scores 76/80 against TranslationDiff::Segmenters::Simple's 47/80 on -# the same corpus -- and the gap is worst on exactly the languages Simple -# cannot reason about at all: Arabic, Hindi, Armenian, Greek, because they -# have no letter case for Simple's central "does the next letter look -# lowercase" rule to use. -# -# `pragmatic_segmenter` returns sentence strings, not offsets, and drops the -# whitespace between them. This gem reassembles the source document from -# offsets (see TranslationDiff::Segmenters::Simple), so the strings have to -# be turned back into split points by finding each one in the source, in -# order, starting the search where the previous one left off. -# -# That recovery is only as good as the assumption that each returned -# sentence still appears verbatim in the text pragmatic_segmenter was given -# -- and pinned version 0.3.24's cleaner does not preserve that in every -# case. It collapses runs of three or more spaces inside a sentence, it -# respaces "Ph.D." into "Ph. D.", and (language-specific) its Japanese rules -# delete a "\n" that follows "の". None of these are rare: an English -# sentence mentioning a degree, or HTML indented with more than two spaces, -# hits one of them routinely. #recover_offsets does not raise when this -# happens. It recovers every offset it can verify, in order, and stops at -# the first sentence it cannot -- but the boundary at the end of the last -# sentence it did verify is not thrown away with the rest: it was matched -# character for character, so it is still emitted, and only the genuinely -# unverifiable remainder becomes one final unit. This is a coarsening, not -# a fallback: every offset this class ever emits has been proved to exist -# at that position in the source, so the document reassembles exactly -# either way -- the last cache unit is just bigger when recovery stops -# early, not the whole node. +# Default segmenter: scores 76/80 on the Golden Rules corpus vs Simple's 47/80 (golden_rules_test.rb). class TranslationDiff::Segmenters::Pragmatic - # Raised only if #split_offsets itself computed offsets that violate its - # own postcondition (start at 0, strictly increasing, all within the - # text) -- not by ordinary use of pragmatic_segmenter, however it rewrites - # a sentence. Kept as public API: it is documented in the README's error - # tree, and a caller may already rescue it. + # Raised only if computed offsets violate their own postcondition, never by ordinary sentence rewriting. class Error < TranslationDiff::Error; end - # pragmatic_segmenter defaults to English rules when no language is given. - # Without this, Russian (among others) mis-segments: it treats "Проф." as - # a full sentence and stops there instead of reading through to the next - # real terminator. Passing the caller's language avoids that; see - # TranslationDiff::Request for where the language comes from and why it is - # sometimes nil despite this. - DEFAULT_LANGUAGE = "en" - - # A single newline -- one with neither a preceding nor a following "\n" -- - # is incidental source formatting almost everywhere this gem is used (HTML - # indentation, hand-wrapped prose), not a paragraph break. A run of two or - # more newlines is left alone: that is a real paragraph break, and - # pragmatic_segmenter already handles it correctly. See #shadow_newlines. + # Without a language, Russian mis-segments: it treats "Проф." as a full sentence and stops there. + DEFAULT_LANGUAGE = "en".freeze + + # A lone "\n" is incidental source formatting, not a paragraph break; a run of two or more is left alone. SINGLE_NEWLINE = /(? offsets.last offsets end - # Walks the sentences in order, returning the starts of every one located - # (see #locate), the cursor left after the last one located, and whether - # the walk stopped before exhausting every sentence. + # Returns the starts of every sentence located, the cursor after the last one, and whether the walk stopped early. def walk(shadow, sentences) cursor = 0 starts = [] @@ -175,16 +78,7 @@ def walk(shadow, sentences) [starts, cursor, false] end - # Returns [start, next_cursor] for a sentence found at or after cursor; - # :skip for an empty sentence, which #recover_offsets passes over without - # ending the walk (an empty pattern would "match" at cursor itself and - # never advance past it -- distinct from not being found, which should - # end the walk, not stall it); or nil if a non-empty sentence cannot be - # found there at all, which does end the walk. A located non-empty - # sentence always advances past cursor by construction (String#index - # never returns a position before where the search started), so there is - # no further case to guard here -- #assert_valid_offsets is the actual - # backstop if that ever stops holding. + # :skip for empty (an empty match would never advance the cursor); nil if a non-empty sentence isn't found. def locate(shadow, sentence, cursor) return :skip if sentence.empty? @@ -194,11 +88,7 @@ def locate(shadow, sentence, cursor) [start, start + sentence.length] end - # The postcondition every caller of #split_offsets depends on: offsets - # start at 0, strictly increase, and every one is a valid index into the - # text. #recover_offsets is built to guarantee this by construction: this - # asserts it rather than trusting that construction forever, since a wrong - # offset here would silently corrupt the document it feeds back into. + # Asserts the postcondition rather than trusting construction forever: a wrong offset would silently corrupt. def assert_valid_offsets(text, offsets) return if offsets.first.zero? && offsets.each_cons(2).all? { |a, b| a < b } && diff --git a/lib/translation_diff/segmenters/simple.rb b/lib/translation_diff/segmenters/simple.rb index 3288cde..441cdc1 100644 --- a/lib/translation_diff/segmenters/simple.rb +++ b/lib/translation_diff/segmenters/simple.rb @@ -1,48 +1,22 @@ -# frozen_string_literal: true - -# Splits a string into sentence-sized cache units, with no runtime -# dependency beyond Ruby's own Unicode data. -# -# A wrong boundary never corrupts the document -- the tokenizer slices by -# offset and joins the pieces back together -- but the two kinds of error are -# not equal. A missed boundary just makes the cache unit bigger: still a -# coherent, correctly translated piece of text. A false boundary sends half a -# sentence to the provider on its own, and it comes back wrong. So this class -# is deliberately conservative: it only splits on strong signals, and every -# guard below exists to turn a would-be split back off, never to add one. -# -# Its central rule -- "the next visible character is lowercase, so do not -# split" -- has no meaning in scripts without case, such as Arabic, Hindi or -# Hebrew, so it fares worse than TranslationDiff::Segmenters::Pragmatic (the -# default) on those languages. It exists for callers who want zero extra -# dependencies and translate only from cased scripts. +# No runtime dependency, but deliberately conservative: every guard exists to turn a split off, never on. class TranslationDiff::Segmenters::Simple - # A period, question mark, exclamation mark or ellipsis, run together - # (`?!`, `!!!`) so a whole run is treated as a single terminator; or a - # run of CJK terminators, which need no trailing whitespace and no guards - # -- those scripts have no case ambiguity and no abbreviation periods. + # CJK terminators need no trailing whitespace and no guards -- those scripts have no case or abbreviations. TERMINATOR = /[.!?…]+|[。！？]+/ CJK_TERMINATOR = /\A[。！？]/ - # A short list of words that end in "." without ending a sentence. - # Matched case-insensitively against the word immediately before the - # period. A caller can extend this constant to cover their own domain. + # Words that end in "." without ending a sentence, matched case-insensitively; extend for your own domain. ABBREVIATIONS = %w[ т.е. т.д. т.п. см. рис. стр. гр. ул. г. руб. проф. акад. тыс. млн. млрд. им. Mr. Mrs. Ms. Dr. Prof. St. etc. e.g. i.e. vs. approx. No. im. fig. Fig. vol. p. pp. ].map(&:downcase).freeze - # The trailing run of letters, digits and periods in a preceding word -- - # what is actually compared against ABBREVIATIONS and checked for being a - # single initial. Strips leading punctuation such as an opening quote so - # `"Dr. Smith` still guards on "Dr.". + # Strips leading punctuation like an opening quote so `"Dr. Smith` still guards on "Dr.". WORD_TAIL = /[\p{L}\p{N}.]+\z/ def self.build(_config) = new - # language: is part of the shared segmenter contract but is ignored here -- - # this segmenter's rules (case, digits, punctuation) are language-neutral. + # language: is part of the shared segmenter contract but ignored here -- these rules are language-neutral. # rubocop:disable-next Lint/UnusedMethodArgument def split_offsets(text, language: nil) offsets = [0] @@ -67,17 +41,13 @@ def boundary_for(text, match) end end - # CJK terminators need no trailing whitespace to split, but if whitespace - # does follow, it is attached to the sentence that just ended rather than - # left as a leading gap on the next one. + # If whitespace follows, it's attached to the sentence that just ended, not left as a leading gap. def cjk_boundary(text, run_end) boundary = skip_whitespace(text, run_end) boundary if boundary < text.length end - # Latin terminators only count as a candidate when followed by whitespace; - # a bare "." with nothing after it is not a sentence break, it is a string - # that stops mid-thought. + # A bare "." with nothing after it is not a sentence break, it is a string that stops mid-thought. def latin_boundary(text, match) run_end = match.end(0) return unless whitespace?(text[run_end]) @@ -128,8 +98,7 @@ def url_or_email?(word_before, run) token.include?("://") || token.include?("@") end - # The maximal run of non-whitespace characters immediately before the - # terminator, i.e. the "word" the terminator is attached to. + # The maximal run of non-whitespace characters immediately before the terminator. def word_before(text, run_start) start = run_start start -= 1 while start.positive? && !whitespace?(text[start - 1]) diff --git a/lib/translation_diff/spacing.rb b/lib/translation_diff/spacing.rb index 5d24793..85f0aef 100644 --- a/lib/translation_diff/spacing.rb +++ b/lib/translation_diff/spacing.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - # Adds same count leading-trailing spaces left has to the right class TranslationDiff::Spacing class << self diff --git a/lib/translation_diff/stores.rb b/lib/translation_diff/stores.rb index 1fcbd07..5193f94 100644 --- a/lib/translation_diff/stores.rb +++ b/lib/translation_diff/stores.rb @@ -1,5 +1,2 @@ -# frozen_string_literal: true - -# Cache stores, by name. `config.cache = :redis` resolves through here; -# assigning an object bypasses it entirely. +# Cache stores, by name; assigning an object to `config.cache` bypasses this entirely. TranslationDiff::Stores = TranslationDiff::Registry.new("cache store") diff --git a/lib/translation_diff/tokenizer.rb b/lib/translation_diff/tokenizer.rb index f3199e7..91e2188 100644 --- a/lib/translation_diff/tokenizer.rb +++ b/lib/translation_diff/tokenizer.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - class TranslationDiff::Tokenizer < Ox::Sax SKIP = %i[script style].freeze INNER_SPANS = %i[notranslate span end_span end_notranslate].freeze @@ -30,15 +28,7 @@ def start_element(name) start_markup(name) end - # Ox reports a comment, a doctype and a CDATA section as events of their - # own, each with its own byte position, and each needs a handler here. - # Without one the bytes it covers belong to no token: a leading comment - # disappears from the rebuilt string outright, and one in the middle of a - # sentence leaks its ", 0 + assert_operator capabilities.max_batch_size, :>, 0 + assert_includes [true, false], capabilities.notranslate? end - def test_max_request_size_is_a_positive_integer - assert_kind_of Integer, provider.max_request_size - assert_operator provider.max_request_size, :>, 0 - end + # Google and DeepL both shipped with this broken, in different ways, before the capability existed. + def test_notranslate_is_only_claimed_with_an_html_mode + capabilities = provider.class.capabilities - def test_max_batch_size_is_a_positive_integer - assert_kind_of Integer, provider.max_batch_size - assert_operator provider.max_batch_size, :>, 0 + assert capabilities.html?, "claims notranslate without an html mode" if capabilities.notranslate? end end diff --git a/test/support/stubbed_provider.rb b/test/support/stubbed_provider.rb new file mode 100644 index 0000000..0f0f32f --- /dev/null +++ b/test/support/stubbed_provider.rb @@ -0,0 +1,39 @@ +require "faraday" + +# Builds a provider whose Faraday connection answers from a stub and records what was sent. +module StubbedProvider + def requests = @requests ||= [] + + def stub_provider(route:, body:, status: 200, headers: {}, name: :test) + stubs = build_stubs(route: route, body: body, status: status, headers: headers) + + provider_class.new(config).tap do |built| + built.name = name + attach_stub(built, stubs) + end + end + + def sent = JSON.parse(requests.first.body) + def query = CGI.parse(requests.first.url.query.to_s) + + private + + def build_stubs(route:, body:, status:, headers:) + recorder = requests + Faraday::Adapter::Test::Stubs.new do |stub| + stub.post(route) do |env| + # Faraday's test adapter mutates this env's body in place for the response -- dup it now, or lose the request. + recorder << env.dup + rendered = body.respond_to?(:call) ? body.call(env) : body + [status, { "Content-Type" => "application/json" }.merge(headers), + rendered.is_a?(String) ? rendered : rendered.to_json] + end + end + end + + def attach_stub(provider, stubs) + provider.instance_variable_set(:@connection, provider.send(:build_connection) do |faraday| + faraday.adapter :test, stubs + end) + end +end diff --git a/test/test_helper.rb b/test/test_helper.rb index deb188f..039a2c5 100644 --- a/test/test_helper.rb +++ b/test/test_helper.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - require "simplecov" SimpleCov.start @@ -8,8 +6,7 @@ require "minitest/autorun" -# Stands in for a connection_pool. The gem only ever calls #with on whatever -# it is handed, so this is the whole contract. +# Stands in for a connection_pool: the gem only ever calls #with on whatever it is handed. class FakeConnectionPool def initialize(connection) @connection = connection @@ -20,8 +17,7 @@ def with end end -# Any test that configures anything must reset afterwards, or its settings -# leak into every test that runs after it. +# Any test that configures anything must reset afterwards, or its settings leak into later tests. class ConfiguredTest < Minitest::Test def setup = TranslationDiff.reset! def teardown = TranslationDiff.reset! diff --git a/test/translation_diff/cache_test.rb b/test/translation_diff/cache_test.rb index 6203d03..2c345db 100644 --- a/test/translation_diff/cache_test.rb +++ b/test/translation_diff/cache_test.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - require "test_helper" class CacheTest < Minitest::Test @@ -20,9 +18,7 @@ def write(_key, value) end end - # A store that answers with fixed, caller-chosen results regardless of the - # keys it is asked about, to pin the positional contract between the keys - # sent and the results read back. + # Answers with fixed results regardless of keys asked, to pin the positional contract. class PositionalStore def initialize(responses) @responses = responses @@ -37,8 +33,7 @@ def setup @store = RecordingStore.new end - # Two providers writing to one store used to collide: switching DeepL for - # another provider silently returned DeepL's translations. + # Two providers writing to one store used to collide, silently returning DeepL's translations for another. def test_the_provider_is_part_of_the_key key_for(provider: "deepl") key_for(provider: "google") @@ -46,8 +41,7 @@ def test_the_provider_is_part_of_the_key refute_equal @store.keys[0], @store.keys[1] end - # formality: :less used to share a key with the default, so whichever - # translated first won. + # formality: :less used to share a key with the default, so whichever translated first won. def test_the_provider_options_are_part_of_the_key key_for(options: {}) key_for(options: { formality: :less }) @@ -62,8 +56,7 @@ def test_the_options_digest_is_order_independent assert_equal @store.keys[0], @store.keys[1] end - # "EN" and :en are the same language and used to produce two entries for - # identical work. + # "EN" and :en are the same language and used to produce two entries for identical work. def test_the_language_codes_are_normalised key_for(from: "EN", to: "RU") key_for(from: :en, to: :ru) @@ -78,8 +71,7 @@ def test_leading_and_trailing_space_does_not_change_the_key assert_equal @store.keys[0], @store.keys[1] end - # nil and "" are different values; #to_s would collide them, so the digest - # must be built from a serialisation that keeps them apart. + # nil and "" are different values; #to_s would collide them. def test_nil_and_empty_string_option_values_produce_different_keys key_for(options: { a: nil }) key_for(options: { a: "" }) @@ -87,8 +79,7 @@ def test_nil_and_empty_string_option_values_produce_different_keys refute_equal @store.keys[0], @store.keys[1] end - # An Array-valued option (e.g. glossary_ids: %w[a b]) exercises the Array - # branch of #canonical, which no other test in this file reaches. + # Exercises the Array branch of #canonical, which no other test in this file reaches. def test_an_array_option_value_is_part_of_the_digest key_for(options: { glossary_ids: %w[a b] }) key_for(options: { glossary_ids: %w[a c] }) @@ -96,16 +87,12 @@ def test_an_array_option_value_is_part_of_the_digest refute_equal @store.keys[0], @store.keys[1] end - # An option value with no stable serialisation (no #inspect of its own, - # or one that embeds a memory address) must not be allowed to silently - # produce an unreproducible cache key. + # An option value with no stable serialisation must not silently produce an unreproducible cache key. def test_an_unsupported_option_value_raises assert_raises(TranslationDiff::Cache::Error) { key_for(options: { a: Object.new }) } end - # cached_and_missing pairs the store's response with the requested values - # by position, trusting the store to return results in key order. A - # database-backed store answering `WHERE key IN (...)` will not. + # Trusts the store to return results in key order; a `WHERE key IN (...)` store will not. def test_cached_and_missing_pairs_results_positionally store = PositionalStore.new(["cached one", nil, "cached three"]) @@ -116,6 +103,42 @@ def test_cached_and_missing_pairs_results_positionally assert_equal ["two"], missing end + # #store is public and takes an outside array; shifting it emptied the caller's own array. + def test_store_does_not_consume_the_updates_it_is_given + updates = %w[один три] + cache = TranslationDiff::Cache.new(:en, :ru, provider: "deepl", store: @store) + + cache.store(%w[one two three], [nil, "два", nil], updates) + + assert_equal %w[один три], updates + end + + def test_store_fills_the_gaps_in_key_order + cache = TranslationDiff::Cache.new(:en, :ru, provider: "deepl", store: @store) + + assert_equal %w[один два три], + cache.store(%w[one two three], [nil, "два", nil], %w[один три]) + end + + # The exact keys a translation is stored under, verified equal to main's -- move one and every user + # re-translates their whole corpus, so the digest of the options and of the sentence are pinned too. + # rubocop:disable-next Metrics/MethodLength + def test_the_keys_are_the_ones_users_already_have_in_their_caches + key_for(value: "text", from: :en, to: :ru, provider: "deepl") + key_for(value: " text ", from: "EN", to: "RU", provider: "deepl") + key_for(value: "text", from: :en, to: :ru, provider: "deepl", options: { formality: :less }) + key_for(value: "Hello there.", from: :en, to: :"pt-BR", provider: "google", + options: { glossary_ids: %w[a b], nested: { z: 1, a: nil } }) + key_for(value: "Ein Satz.", from: :de, to: :en, provider: "azure", + options: { a: "", b: 2, c: true }) + + assert_equal ["deepl:en:ru:1cb251ec0d568de6a929b520c4aed8d1", + "deepl:en:ru:1cb251ec0d568de6a929b520c4aed8d1", + "deepl:en:ru:80df90b8:1cb251ec0d568de6a929b520c4aed8d1", + "google:en:pt-br:2744ebab:9d6a2963872077db674a27a39c492e61", + "azure:de:en:da1fef03:84f8c5b939a540fa8da132c838a8d61d"], @store.keys + end + private def key_for(value: "text", from: :en, to: :ru, provider: "deepl", options: {}) diff --git a/test/translation_diff/capabilities_test.rb b/test/translation_diff/capabilities_test.rb new file mode 100644 index 0000000..47ad19b --- /dev/null +++ b/test/translation_diff/capabilities_test.rb @@ -0,0 +1,27 @@ +require "test_helper" + +class CapabilitiesTest < Minitest::Test + def capabilities(**overrides) + TranslationDiff::Capabilities.new(max_request_size: 5_000, max_batch_size: 128, max_text_size: nil, + html: :format, notranslate: true, detects_language: true, + reports_billing: false, **overrides) + end + + # Declaring the predicates once stops the codebase asking `detects_language` in one place, `?` in another. + def test_it_answers_in_predicates + assert_predicate capabilities, :html? + assert_predicate capabilities, :notranslate? + assert_predicate capabilities, :detects_language? + refute_predicate capabilities, :reports_billing? + end + + # `html` holds the option name that turns HTML on, different per vendor, so :none says "cannot". + def test_html_is_false_only_when_the_provider_has_no_html_mode + refute_predicate capabilities(html: :none), :html? + assert_predicate capabilities(html: :tag_handling), :html? + end + + def test_it_is_frozen_so_a_provider_cannot_be_mutated_at_runtime + assert_predicate capabilities, :frozen? + end +end diff --git a/test/translation_diff/chunker_test.rb b/test/translation_diff/chunker_test.rb index 70e2542..fd30ce6 100644 --- a/test/translation_diff/chunker_test.rb +++ b/test/translation_diff/chunker_test.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - require "test_helper" class ChunkerTest < Minitest::Test @@ -8,7 +6,7 @@ class ChunkerTest < Minitest::Test LONG = "a" * 10 MEDIUM = "a" * 7 - SHORT = "x" + SHORT = "x".freeze OVERSIZED = "a" * 30 CASES = { @@ -42,14 +40,11 @@ def test_raises_when_a_single_value_exceeds_the_limit assert_match(/Too long part/, error.message) end - # The limit is about the size of the request that goes over the wire, and - # CGI.escape inflates Cyrillic sixfold. Measuring the raw String#size - # anywhere here let chunks of non-ASCII text run several times over. + # CGI.escape inflates Cyrillic sixfold; measuring raw String#size let chunks of non-ASCII text run over. def test_measures_non_ascii_values_by_their_escaped_size value = "я" * 3 - # Three characters raw, eighteen escaped. Measured raw, both values fit - # in one chunk of 20; measured as sent, they cannot. + # Measured raw, both values fit in one chunk of 20; measured as sent, they cannot. assert_equal 3, value.size assert_equal 18, CGI.escape(value).size assert_equal [[value], [value]], chunk([value, value]) diff --git a/test/translation_diff/configuration_test.rb b/test/translation_diff/configuration_test.rb index 1cf1cce..d6bb527 100644 --- a/test/translation_diff/configuration_test.rb +++ b/test/translation_diff/configuration_test.rb @@ -1,10 +1,25 @@ -# frozen_string_literal: true - require "test_helper" +require "support/env_stub" class ConfigurationTest < Minitest::Test - # A hand-written double for TranslationDiff::Registry: this project's - # Minitest (6.0) dropped minitest/mock, so there is no Minitest::Mock here. + include EnvStub + + # A hand-written double for TranslationDiff::Registry: Minitest 6.0 dropped minitest/mock. + # Duck-typed collaborators, deliberately inheriting nothing: the provider is the only tightened one. + class FakeStore + def read_multi(keys) = [nil] * keys.size + def write(_key, value) = value + end + + class FakeSegmenter + def split_offsets(_text, **) = [] + end + + class FakeRateLimiter + # The real limiter raises when the threshold is passed and returns nothing useful otherwise. + def check(_size) = nil + end + ResolvingRegistry = Struct.new(:answer) do attr_reader :asked @@ -46,8 +61,7 @@ def test_assigning_a_blank_string_stores_nil_so_an_unset_env_var_behaves_as_unse end def test_assigning_false_is_kept_and_not_treated_as_unset - # test_flag is registered here purely to exercise `option`; it is not a - # production option and mutating class-level state with it is harmless. + # test_flag is registered here purely to exercise `option`; it is not a production option. TranslationDiff::Configuration.option(:test_flag, true) @config.test_flag = false @@ -64,9 +78,7 @@ def test_copy_carries_values_and_leaves_the_original_alone assert_equal 120, copy.cache_ttl end - # Stand-ins for two provider classes. register_provider_options is handed - # the class so it can tell "the same provider declaring its options again" - # from "another provider claiming a name that is already taken". + # Stand-ins for two provider classes, to distinguish redeclaring from a name collision. class AcmeOptionOwner def self.configuration_options = %i[acme_api_key] end @@ -76,9 +88,7 @@ def self.configuration_options = %i[contested_key] end def test_register_provider_options_adds_readers_and_writers - # acme_api_key is registered here purely to exercise - # register_provider_options; it is not a production option and mutating - # class-level state with it is harmless. + # acme_api_key is registered here purely to exercise register_provider_options; not a production option. TranslationDiff::Configuration.register_provider_options(%i[acme_api_key], AcmeOptionOwner) @config.acme_api_key = "secret" @@ -86,8 +96,7 @@ def test_register_provider_options_adds_readers_and_writers assert_includes TranslationDiff::Configuration.options, :acme_api_key end - # A double `require` and a Rails development reload both re-run - # registration; neither is a conflict. + # A double `require` and a Rails development reload both re-run registration; neither is a conflict. def test_the_same_provider_may_redeclare_its_own_options TranslationDiff::Configuration.register_provider_options(%i[acme_repeat_key], AcmeOptionOwner) TranslationDiff::Configuration.register_provider_options(%i[acme_repeat_key], AcmeOptionOwner) @@ -95,9 +104,7 @@ def test_the_same_provider_may_redeclare_its_own_options assert_includes TranslationDiff::Configuration.options, :acme_repeat_key end - # Without this, `option` returns early on the already-declared name and the - # two providers silently share one accessor -- so the credential set for - # one is handed to the other. + # Without this, `option` returns early and two providers silently share one accessor. def test_a_different_provider_declaring_a_declared_option_raises TranslationDiff::Configuration.register_provider_options(%i[contested_key], AcmeOptionOwner) @@ -110,9 +117,7 @@ def test_a_different_provider_declaring_a_declared_option_raises assert_match(/RivalOptionOwner/, error.message) end - # A subclass wanting its parent's options is the one case where sharing - # the accessor is correct -- subclassing a provider to point it at a - # different host or account is an obvious thing to want. + # A subclass wanting its parent's options is the one case where sharing the accessor is correct. class AcmeSubclassOwner < AcmeOptionOwner; end def test_a_subclass_of_the_declaring_provider_may_redeclare_its_options @@ -122,8 +127,7 @@ def test_a_subclass_of_the_declaring_provider_may_redeclare_its_options assert_includes TranslationDiff::Configuration.options, :acme_subclass_key end - # An unrelated class is still refused, even though a subclass is now - # allowed -- the guard only relaxes for an actual inheritance relationship. + # The guard only relaxes for an actual inheritance relationship; an unrelated class is still refused. def test_an_unrelated_class_claiming_a_subclassable_providers_option_still_raises TranslationDiff::Configuration.register_provider_options(%i[acme_unrelated_key], AcmeOptionOwner) @@ -134,16 +138,12 @@ def test_an_unrelated_class_claiming_a_subclassable_providers_option_still_raise assert_match(/acme_unrelated_key/, error.message) end - # A provider that conflicts on its *second* option, once its first - # (fresh_option) has already been checked. + # Conflicts on its *second* option, once its first (fresh_option) has already been checked. class IntruderOptionOwner def self.configuration_options = %i[fresh_option owned_by_acme] end - # register_provider_options used to declare and record ownership key by - # key, so a provider whose *second* option conflicted still left its first - # option declared and owned by the class that failed to register. Fixed to - # check every key before mutating any of them. + # Used to declare ownership key by key, leaving the first option owned by a class that failed to register. def test_a_conflict_on_a_later_option_leaves_no_partial_state TranslationDiff::Configuration.register_provider_options(%i[owned_by_acme], AcmeOptionOwner) @@ -154,22 +154,115 @@ def test_a_conflict_on_a_later_option_leaves_no_partial_state refute_includes TranslationDiff::Configuration.options, :fresh_option - # If the failed attempt had already declared or claimed :fresh_option, - # this would raise, blaming a provider that was never registered. + # If the failed attempt had already claimed :fresh_option, this would raise, blaming the wrong provider. TranslationDiff::Configuration.register_provider_options(%i[fresh_option], RivalOptionOwner) assert_includes TranslationDiff::Configuration.options, :fresh_option end + # A provider needing no default keeps the bare-symbol form; one needing a default declares `key => default`. + class DefaultingOptionOwner + def self.configuration_options + [:defaulting_bare, { defaulting_keyed: -> { ENV.fetch("DEFAULTING_TEST_VAR", nil) } }] + end + end + + def test_a_provider_declares_bare_symbols_and_defaults_in_one_list + TranslationDiff::Configuration.register_provider_options( + DefaultingOptionOwner.configuration_options, DefaultingOptionOwner + ) + + assert_includes TranslationDiff::Configuration.options, :defaulting_bare + assert_includes TranslationDiff::Configuration.options, :defaulting_keyed + assert_nil TranslationDiff::Configuration.new.defaulting_bare + end + + # Evaluated on read, not at load: an application that sets the variable after requiring us still gets it. + def test_a_declared_callable_default_is_evaluated_on_every_read + TranslationDiff::Configuration.register_provider_options( + DefaultingOptionOwner.configuration_options, DefaultingOptionOwner + ) + + with_env("DEFAULTING_TEST_VAR" => "from-the-environment") do + assert_equal "from-the-environment", TranslationDiff::Configuration.new.defaulting_keyed + end + end + + def test_an_assigned_value_wins_over_a_declared_default + TranslationDiff::Configuration.register_provider_options( + DefaultingOptionOwner.configuration_options, DefaultingOptionOwner + ) + config = TranslationDiff::Configuration.new + config.defaulting_keyed = "assigned" + + with_env("DEFAULTING_TEST_VAR" => "from-the-environment") do + assert_equal "assigned", config.defaulting_keyed + end + end + + # The blank rule the writer already applies: a variable exported empty means unset, not an empty credential. + def test_a_blank_default_reads_as_unset + TranslationDiff::Configuration.register_provider_options( + DefaultingOptionOwner.configuration_options, DefaultingOptionOwner + ) + + with_env("DEFAULTING_TEST_VAR" => " ") do + assert_nil TranslationDiff::Configuration.new.defaulting_keyed + end + end + + class KeyedConflictOwner + def self.configuration_options = [:keyed_fresh_option, { owned_by_acme: -> { "x" } }] + end + + # Ownership is claimed for every key however it was written, so a keyed clash still leaves no partial state. + def test_a_conflict_on_a_keyed_option_leaves_no_partial_state + TranslationDiff::Configuration.register_provider_options(%i[owned_by_acme], AcmeOptionOwner) + + assert_raises(TranslationDiff::Error) do + TranslationDiff::Configuration.register_provider_options( + KeyedConflictOwner.configuration_options, KeyedConflictOwner + ) + end + + refute_includes TranslationDiff::Configuration.options, :keyed_fresh_option + end + def test_registering_an_option_twice_does_not_clobber_the_first_default - # shared_option is not a production option; mutating class-level state - # with it is harmless. + # shared_option is not a production option; mutating class-level state with it is harmless. TranslationDiff::Configuration.option(:shared_option, "first") TranslationDiff::Configuration.option(:shared_option, "second") assert_equal "first", TranslationDiff::Configuration.new.shared_option end + # Providers.register refuses a class that skips the base class; assigning an object bypassed that entirely. + def test_an_assigned_provider_object_that_is_not_a_provider_is_refused + @config.provider = Object.new + + error = assert_raises(TranslationDiff::InvalidProviderError) { @config.provider_instance } + + assert_match(/TranslationDiff::Provider/, error.message) + end + + def test_an_assigned_provider_object_that_is_a_provider_is_used_as_is + provider = TranslationDiff::Providers::Null.new(TranslationDiff::Configuration.new) + @config.provider = provider + + assert_same provider, @config.provider_instance + end + + # Only the provider gained a base class; these three are still genuinely duck-typed. + def test_the_other_extension_points_stay_duck_typed + @config.cache = FakeStore.new + @config.segmenter = FakeSegmenter.new + @config.rate_limiter = FakeRateLimiter.new + + assert_instance_of FakeStore, @config.cache_store + assert_instance_of FakeSegmenter, @config.segmenter_instance + assert_instance_of FakeRateLimiter, @config.rate_limiter_instance + end + def test_resolve_builds_from_a_registry_for_a_symbol built = Object.new registry = ResolvingRegistry.new(built) @@ -244,14 +337,7 @@ def test_the_default_rate_limiter_is_memoised assert_same @config.rate_limiter_instance, @config.rate_limiter_instance end - # `rate_limiter_instance` follows the same rule as `provider_instance`, - # `cache_store` and `segmenter_instance`: a config that never had an - # object assigned starts from nothing on `copy` and builds its own - # limiter. This is the case that used to be broken -- a copy inherited an - # already-built limiter, which would silently rate-limit a tenant against - # its parent's namespace. See - # `test_a_copy_that_changes_its_namespace_gets_a_rate_limiter_using_that_namespace` - # below for the scenario that actually surfaces the bleed. + # Used to be broken: a copy inherited an already-built limiter and rate-limited a tenant against its parent's. def test_copy_does_not_share_a_built_default_rate_limiter @config.rate_limit = 100 original_limiter = @config.rate_limiter_instance @@ -259,12 +345,7 @@ def test_copy_does_not_share_a_built_default_rate_limiter refute_same original_limiter, @config.copy.rate_limiter_instance end - # An explicitly assigned object is different: it is the option's own - # value, exactly like an assigned `cache` or `provider`, and `copy` - # carries option values over on purpose -- "someone who hands us one - # object means one object." Only the *default build* must not survive a - # copy; an object the caller supplied is never rebuilt in the first - # place, so there is nothing for a copy to get wrong. + # An assigned object is an option value, and `copy` carries option values over on purpose. def test_copy_shares_an_assigned_rate_limiter_object limiter = Object.new @config.rate_limiter = limiter @@ -272,9 +353,7 @@ def test_copy_shares_an_assigned_rate_limiter_object assert_same limiter, @config.copy.rate_limiter_instance end - # The regression this whole area was fixed for: a tenant context that - # sets its own `cache_namespace` must get a rate limiter scoped to that - # namespace, not one built for -- and still carrying -- its parent's. + # The regression this whole area was fixed for. def test_a_copy_that_changes_its_namespace_gets_a_rate_limiter_using_that_namespace @config.rate_limit = 100 @config.redis_url = "redis://localhost:6379" diff --git a/test/translation_diff/context_test.rb b/test/translation_diff/context_test.rb index beb72e0..9f7c062 100644 --- a/test/translation_diff/context_test.rb +++ b/test/translation_diff/context_test.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - require "test_helper" class ContextTest < Minitest::Test @@ -7,8 +5,7 @@ def setup TranslationDiff.reset! TranslationDiff.configure do |c| c.provider = :null - # Pinned rather than left to the default so a developer with REDIS_URL - # set in their environment does not have these tests reach for a socket. + # Pinned so a developer with REDIS_URL set doesn't have these tests reach for a socket. c.cache = :memory end end @@ -42,7 +39,9 @@ def test_two_contexts_build_separate_collaborators end def test_a_context_translates_through_its_own_configuration - context = TranslationDiff.context { |c| c.provider = :null } + context = TranslationDiff.context do |c| + c.provider = TranslationDiff::Providers::Null.new(TranslationDiff::Configuration.new) + end assert_equal "Hello.", context.translate("Hello.", from: "en", to: "ru") end diff --git a/test/translation_diff/errors_test.rb b/test/translation_diff/errors_test.rb new file mode 100644 index 0000000..aa72eda --- /dev/null +++ b/test/translation_diff/errors_test.rb @@ -0,0 +1,42 @@ +require "test_helper" + +class ErrorsTest < Minitest::Test + # A caller who wants to rescue anything this gem raises writes one rescue. + def test_every_error_descends_from_the_common_ancestor + [TranslationDiff::ConfigurationError, TranslationDiff::ProviderError, + TranslationDiff::AuthenticationError, TranslationDiff::RateLimitError, + TranslationDiff::QuotaExceededError, TranslationDiff::InvalidRequestError, + TranslationDiff::ServiceError, TranslationDiff::TransportError, + TranslationDiff::ResponseError].each do |klass| + assert_operator klass, :<, TranslationDiff::Error + end + end + + # Rescuing "the provider said no" must not also catch a config mistake or a socket timeout. + def test_provider_errors_are_a_family_of_their_own + [TranslationDiff::AuthenticationError, TranslationDiff::RateLimitError, + TranslationDiff::QuotaExceededError, TranslationDiff::InvalidRequestError, + TranslationDiff::ServiceError].each do |klass| + assert_operator klass, :<, TranslationDiff::ProviderError + end + + refute_operator TranslationDiff::ConfigurationError, :<, TranslationDiff::ProviderError + refute_operator TranslationDiff::TransportError, :<, TranslationDiff::ProviderError + end + + def test_a_provider_error_carries_the_provider_and_the_status + error = TranslationDiff::RateLimitError.new("slow down", provider: :deepl, status: 429, + retry_after: 30) + + assert_equal :deepl, error.provider + assert_equal 429, error.status + assert_equal 30, error.retry_after + assert_equal "slow down", error.message + end + + def test_retry_after_is_nil_when_the_provider_did_not_say + error = TranslationDiff::RateLimitError.new("slow down", provider: :deepl, status: 429) + + assert_nil error.retry_after + end +end diff --git a/test/translation_diff/golden_rules_test.rb b/test/translation_diff/golden_rules_test.rb index ae2315b..a8d7ff2 100644 --- a/test/translation_diff/golden_rules_test.rb +++ b/test/translation_diff/golden_rules_test.rb @@ -1,22 +1,6 @@ -# frozen_string_literal: true - require "test_helper" -# Eleven exemplars from the "Golden Rules", the de-facto benchmark for -# sentence segmentation, adapted from the `context "Golden Rules" do` block -# of each per-language spec file under spec/pragmatic_segmenter/languages/ -# on https://github.com/diasks2/pragmatic_segmenter (MIT licence, Copyright -# (c) 2015 Kevin S. Dias). The full corpus (80 exemplars across 10 -# languages) lives outside this repo; this sample exists so a future change -# to the default segmenter cannot quietly regress segmentation quality -# without a test noticing here first. -# -# It deliberately weights the languages that have no letter case -- Arabic -# (two exemplars), Greek, Hindi -- and includes one Japanese exemplar, -# because those are exactly the languages TranslationDiff::Segmenters::Simple -# cannot reason about (its central rule, "does the next letter look -# lowercase", has no meaning for them) and where Pragmatic earns its place -# as the default. +# Adapted from diasks2/pragmatic_segmenter's Golden Rules (MIT, Copyright (c) 2015 Kevin S. Dias). class GoldenRulesTest < Minitest::Test EXEMPLARS = [ { language: "en", text: "Hello World. My name is Jonas.", @@ -43,9 +27,7 @@ class GoldenRulesTest < Minitest::Test "يقول معارضو الرئيس الإيراني إن الطريقة التي اعلنت بها النتائج كانت مثيرة للاستغراب." ] }, { language: "ar", - # Contains U+202A/U+202C (left-to-right embedding) around the - # abbreviation's period -- a real bidi-formatting shape, and a good - # stress test for offset recovery finding a sentence verbatim. + # Contains U+202A/U+202C (left-to-right embedding) around the abbreviation's period, a real bidi shape. text: "وقال د‪.‬ ديفيد ريدي و الأطباء الذين كانوا يعالجونها في مستشفى برمنجهام إنها كانت " \ "تعاني من أمراض أخرى. وليس معروفا ما اذا كانت قد توفيت بسبب اصابتها بأنفلونزا الخنازير.", expected: [ @@ -81,10 +63,7 @@ def test_pragmatic_the_default_segmenter_matches_the_golden_rules_sample end end - # Simple is not held to the Golden Rules' exact boundaries -- it cannot be, - # for languages without letter case -- but it must never corrupt the - # document while trying. This is the structural guarantee that still has - # to hold when a caller opts into the zero-dependency segmenter. + # Simple isn't held to exact boundaries, but must never corrupt the document while trying. def test_simple_still_reconstructs_every_exemplar_even_where_it_under_or_over_splits segmenter = TranslationDiff::Segmenters::Simple.new diff --git a/test/translation_diff/http_provider_test.rb b/test/translation_diff/http_provider_test.rb new file mode 100644 index 0000000..51dad96 --- /dev/null +++ b/test/translation_diff/http_provider_test.rb @@ -0,0 +1,134 @@ +require "test_helper" +require "faraday" + +class HTTPProviderTest < Minitest::Test + # A provider that exists only to exercise the base class. + class Echo < TranslationDiff::HTTPProvider + def api_base = "https://echo.test" + def headers = { "X-Echo" => "1" } + def translate_url = "v1/translate" + def render_translate_payload(request) = { "q" => request.texts } + + def parse_translate_response(body, _headers, request) + TranslationDiff::Translation::Response.build(request: request, texts: body["translations"]) + end + end + + def setup + @config = TranslationDiff::Configuration.new + end + + def request(texts = %w[one]) + TranslationDiff::Translation::Request.new(texts: texts, from: "en", to: "ru") + end + + # Faraday's test adapter: no webmock, no network, same middleware stack the real connection has. + def provider_for(status:, body:, headers: {}) + stubs = Faraday::Adapter::Test::Stubs.new do |stub| + stub.post("/v1/translate") { [status, headers, body] } + end + Echo.new(@config).tap do |provider| + provider.instance_variable_set(:@connection, provider.send(:build_connection) do |faraday| + faraday.adapter :test, stubs + end) + end + end + + def test_it_posts_the_rendered_payload_and_parses_the_reply + provider = provider_for(status: 200, body: { "translations" => %w[один] }.to_json, + headers: { "Content-Type" => "application/json" }) + + assert_equal %w[один], provider.translate(request).texts + end + + def test_a_401_becomes_an_authentication_error_naming_the_provider + provider = provider_for(status: 401, body: "nope") + provider.name = :echo + + error = assert_raises(TranslationDiff::AuthenticationError) { provider.translate(request) } + + assert_equal :echo, error.provider + assert_equal 401, error.status + end + + def test_a_400_becomes_an_invalid_request_error + provider = provider_for(status: 400, body: "bad") + + assert_raises(TranslationDiff::InvalidRequestError) { provider.translate(request) } + end + + def test_a_456_becomes_a_quota_error + provider = provider_for(status: 456, body: "out of quota") + + assert_raises(TranslationDiff::QuotaExceededError) { provider.translate(request) } + end + + def test_a_500_becomes_a_service_error + provider = provider_for(status: 500, body: "boom") + + assert_raises(TranslationDiff::ServiceError) { provider.translate(request) } + end + + # Retries are turned off; what is asserted here is the mapping, not the retrying. + def test_a_429_becomes_a_rate_limit_error_carrying_retry_after + @config.max_retries = 0 + provider = provider_for(status: 429, body: "slow down", headers: { "Retry-After" => "17" }) + + error = assert_raises(TranslationDiff::RateLimitError) { provider.translate(request) } + + assert_equal 17, error.retry_after + end + + def test_a_connection_failure_becomes_a_transport_error + @config.max_retries = 0 + stubs = Faraday::Adapter::Test::Stubs.new do |stub| + stub.post("/v1/translate") { raise Faraday::ConnectionFailed, "no route" } + end + provider = Echo.new(@config) + provider.instance_variable_set(:@connection, provider.send(:build_connection) do |faraday| + faraday.adapter :test, stubs + end) + + assert_raises(TranslationDiff::TransportError) { provider.translate(request) } + end + + # No line this library writes may carry source text or a credential. + def test_no_logging_middleware_is_installed_even_when_a_logger_is_configured + @config.logger = Logger.new(StringIO.new) + handlers = Echo.new(@config).connection.builder.handlers + + refute_includes handlers.map(&:name), "Faraday::Response::Logger" + end + + def test_the_configured_timeouts_reach_the_connection + @config.open_timeout = 2 + @config.timeout = 7 + connection = Echo.new(@config).connection + + assert_equal 2, connection.options.open_timeout + assert_equal 7, connection.options.timeout + end + + def test_the_subclass_headers_are_sent + connection = Echo.new(@config).connection + + assert_equal "1", connection.headers["X-Echo"] + end + + def test_it_decodes_a_json_body_without_faradays_middleware + provider = provider_for(status: 200, body: { "translations" => %w[один] }.to_json, + headers: { "Content-Type" => "application/json" }) + + assert_equal %w[один], provider.translate(request).texts + end + + # An error page from a proxy is HTML, and the error path must survive it. + def test_a_non_json_error_body_still_produces_the_mapped_error + provider = provider_for(status: 500, body: "gateway", + headers: { "Content-Type" => "text/html" }) + + error = assert_raises(TranslationDiff::ServiceError) { provider.translate(request) } + + assert_match(/gateway/, error.message) + end +end diff --git a/test/translation_diff/instrumentation_test.rb b/test/translation_diff/instrumentation_test.rb index b31b047..c6bd826 100644 --- a/test/translation_diff/instrumentation_test.rb +++ b/test/translation_diff/instrumentation_test.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - require "test_helper" class InstrumentationTest < ConfiguredTest @@ -14,11 +12,7 @@ def instrument(name, payload) end end - # Assignable via `config.rate_limiter =`, same as `config.cache =`. Always - # lets the call through, so `check_rate_limit` has something to call - # without needing a real Redis connection -- and so the `rate_limit` event - # fires on every translation in this file, alongside `translate`, `cache` - # and `request`. + # Always lets the call through, so the `rate_limit` event fires without a real Redis connection. class FakeRateLimiter def check(_size) = nil end @@ -28,9 +22,7 @@ def setup @recorder = Recorder.new TranslationDiff.configure do |c| c.provider = :null - # Pinned so a developer with REDIS_URL set does not have these tests - # resolve the Redis store and open a real socket -- the same reason - # context_test.rb pins it. It weakens no assertion here. + # Pinned so a developer with REDIS_URL set doesn't have these tests open a real socket. c.cache = :memory c.instrumenter = @recorder c.rate_limiter = FakeRateLimiter.new @@ -80,10 +72,7 @@ def test_the_rate_limit_event_carries_the_provider_and_a_character_count ALL_EVENT_NAMES = %w[translate.translation_diff cache.translation_diff request.translation_diff rate_limit.translation_diff].sort.freeze - # A guard that only checked payload content would pass even if an event - # quietly stopped firing -- asserting the full set of names first makes - # sure every event this library emits is actually present and inspected, - # not just whichever ones happened to show up. + # A guard that only checked payload content would pass even if an event quietly stopped firing. def test_no_payload_ever_contains_the_text_being_translated secret = "Zaphod Beeblebrox is president." TranslationDiff.translate(secret, from: "en", to: "ru") diff --git a/test/translation_diff/linearizer_test.rb b/test/translation_diff/linearizer_test.rb index b78ee95..be37589 100644 --- a/test/translation_diff/linearizer_test.rb +++ b/test/translation_diff/linearizer_test.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - require "test_helper" class LinearizerTest < Minitest::Test diff --git a/test/translation_diff/memory_cache_store_test.rb b/test/translation_diff/memory_cache_store_test.rb index 1571a15..bb9ee05 100644 --- a/test/translation_diff/memory_cache_store_test.rb +++ b/test/translation_diff/memory_cache_store_test.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - require "test_helper" require "support/cache_store_contract" diff --git a/test/translation_diff/provider_test.rb b/test/translation_diff/provider_test.rb new file mode 100644 index 0000000..124b9b1 --- /dev/null +++ b/test/translation_diff/provider_test.rb @@ -0,0 +1,113 @@ +require "test_helper" + +class ProviderTest < Minitest::Test + class Bare < TranslationDiff::Provider + end + + class Demanding < TranslationDiff::Provider + def self.configuration_options = %i[demanding_key demanding_secret demanding_region] + def self.configuration_requirements = %i[demanding_key demanding_secret] + end + + def setup + TranslationDiff::Configuration.register_provider_options( + Demanding.configuration_options, Demanding + ) + @config = TranslationDiff::Configuration.new + end + + class Upcasing < TranslationDiff::Provider + def self.language_case = :upcase + end + + # The rule lived in google.rb and nowhere else; four providers shipped without it. + def test_a_bare_code_is_cased_the_way_the_vendor_documents_it + downcasing = Bare.new(@config) + upcasing = Upcasing.new(@config) + + assert_equal "ru", downcasing.language("RU") + assert_equal "ru", downcasing.language(:ru) + assert_equal "RU", upcasing.language("ru") + assert_equal "RU", upcasing.language(:RU) + end + + # A script or region subtag has its own casing; a blanket transform corrupts it. + def test_a_subtagged_code_passes_through_untouched + downcasing = Bare.new(@config) + upcasing = Upcasing.new(@config) + + %w[zh-Hans pt-BR EN-GB sr-Latn-RS].each do |code| + assert_equal code, downcasing.language(code) + assert_equal code, upcasing.language(code) + end + end + + # nil so a provider can leave the field out of the payload entirely rather than sending "". + def test_an_absent_code_is_nil + assert_nil Bare.new(@config).language(nil) + assert_nil Bare.new(@config).language("") + end + + def test_providers_downcase_unless_they_say_otherwise + assert_equal :downcase, TranslationDiff::Provider.language_case + assert_equal :upcase, TranslationDiff::Providers::DeepL.language_case + + %w[Google Azure ModernMT LibreTranslate Amazon].each do |name| + assert_equal :downcase, TranslationDiff::Providers.const_get(name).language_case + end + end + + def test_a_provider_without_requirements_builds + assert_instance_of Bare, Bare.new(@config) + end + + # The old behaviour raised a vendor SDK's own error, which never named the option this library needs set. + def test_it_names_every_missing_option_at_once + error = assert_raises(TranslationDiff::ConfigurationError) { Demanding.new(@config) } + + assert_match(/demanding_key/, error.message) + assert_match(/demanding_secret/, error.message) + assert_match(/TranslationDiff.configure/, error.message) + end + + # An option that is declared but not required must not appear in the error. + def test_it_does_not_demand_optional_options + @config.demanding_key = "k" + @config.demanding_secret = "s" + + assert_instance_of Demanding, Demanding.new(@config) + end + + def test_translate_raises_until_a_subclass_implements_it + request = TranslationDiff::Translation::Request.new(texts: %w[one], from: "en", to: "ru") + + assert_raises(NotImplementedError) { Bare.new(@config).translate(request) } + end + + def test_detect_raises_until_a_subclass_implements_it + assert_raises(NotImplementedError) { Bare.new(@config).detect("etwas") } + end + + # A quietly empty cache_key would let two providers share a namespace and serve the wrong translations. + def test_cache_key_is_the_registered_name + provider = Bare.new(@config) + provider.name = :bare + + assert_equal "bare", provider.cache_key + end + + def test_cache_key_raises_when_the_provider_was_never_stamped + error = assert_raises(TranslationDiff::Error) { Bare.new(@config).cache_key } + + assert_match(/registry/, error.message) + end + + def test_the_default_capabilities_are_conservative + capabilities = TranslationDiff::Provider.capabilities + + refute_predicate capabilities, :html? + refute_predicate capabilities, :notranslate? + refute_predicate capabilities, :detects_language? + refute_predicate capabilities, :reports_billing? + end +end diff --git a/test/translation_diff/providers/amazon_test.rb b/test/translation_diff/providers/amazon_test.rb new file mode 100644 index 0000000..5671961 --- /dev/null +++ b/test/translation_diff/providers/amazon_test.rb @@ -0,0 +1,169 @@ +require "test_helper" +require "support/provider_contract" +require "support/http_provider_contract" +require "support/env_stub" +require "faraday" +require "aws-sigv4" + +class AmazonProviderTest < Minitest::Test + include ProviderContract + include HTTPProviderContract + include EnvStub + + attr_reader :config, :requests + + def setup + TranslationDiff.reset! + @config = TranslationDiff::Configuration.new + @config.amazon_access_key_id = "AKIAEXAMPLE" + @config.amazon_secret_access_key = "secret" + @config.amazon_region = "eu-central-1" + @requests = [] + end + + # No AWS key was available: shaped from Amazon's "TranslateText" reference (read 2026-09-09), not observed. + def provider(texts: nil) + built = TranslationDiff::Providers::Amazon.new(config) + built.name = :amazon + built.instance_variable_set(:@connection, built.send(:build_connection) do |faraday| + faraday.adapter :test, build_stubs(texts) + end) + built + end + + def build_stubs(texts) + recorder = @requests + Faraday::Adapter::Test::Stubs.new do |stub| + stub.post("/") { |env| respond_to_translate(env, texts, recorder) } + end + end + + def respond_to_translate(env, texts, recorder) + # Faraday's test adapter mutates this env's body in place for the response -- dup it now, or lose the request. + recorder << env.dup + body = JSON.parse(env.body) + translated = texts&.shift || "#{body['Text']}-ru" + [200, { "Content-Type" => "application/x-amz-json-1.1" }, + { "TranslatedText" => translated, "SourceLanguageCode" => "en", + "TargetLanguageCode" => "ru" }.to_json] + end + + # Deliberate: aws-sigv4 takes explicit credentials and this library does not implement the credential chain. + def test_it_reads_no_aws_environment_variables + with_env("AWS_ACCESS_KEY_ID" => "AKIAENV", "AWS_SECRET_ACCESS_KEY" => "secret", + "AWS_REGION" => "eu-west-1") do + fresh = TranslationDiff::Configuration.new + + assert_nil fresh.amazon_access_key_id + assert_nil fresh.amazon_secret_access_key + assert_nil fresh.amazon_region + end + end + + # Amazon Translate rejected "EN"/"RU" outright, on every call. + def test_it_downcases_a_bare_language_code_whichever_casing_the_caller_used + provider.translate(translation_request(%w[one], from: "EN", to: "RU")) + body = JSON.parse(requests.first.body) + + assert_equal "en", body["SourceLanguageCode"] + assert_equal "ru", body["TargetLanguageCode"] + end + + def test_it_leaves_a_subtagged_code_untouched + provider.translate(translation_request(%w[one], from: "zh-Hans", to: "pt-BR")) + body = JSON.parse(requests.first.body) + + assert_equal "zh-Hans", body["SourceLanguageCode"] + assert_equal "pt-BR", body["TargetLanguageCode"] + end + + # Every provider applies defaults, then caller options, then mandatory fields. Amazon had the last two swapped. + def test_a_caller_option_cannot_displace_a_mandatory_field + request = TranslationDiff::Translation::Request.new( + texts: %w[one], from: :en, to: :ru, + options: { TargetLanguageCode: "de", Text: "somebody else's text" } + ) + provider.translate(request) + body = JSON.parse(requests.first.body) + + assert_equal "ru", body["TargetLanguageCode"] + assert_equal "one", body["Text"] + end + + def test_a_caller_option_that_displaces_nothing_still_reaches_amazon + request = TranslationDiff::Translation::Request.new( + texts: %w[one], from: :en, to: :ru, options: { Settings: { "Formality" => "FORMAL" } } + ) + provider.translate(request) + + assert_equal({ "Formality" => "FORMAL" }, JSON.parse(requests.first.body)["Settings"]) + end + + def test_the_endpoint_is_regional + assert_equal "https://translate.eu-central-1.amazonaws.com", + TranslationDiff::Providers::Amazon.new(config).api_base + end + + def test_missing_credentials_are_named_before_any_request + config.amazon_secret_access_key = nil + + error = assert_raises(TranslationDiff::ConfigurationError) do + TranslationDiff::Providers::Amazon.new(config) + end + + assert_match(/amazon_secret_access_key/, error.message) + end + + def test_it_sends_the_json_rpc_target_header + provider.translate(translation_request(%w[one])) + + assert_equal "AWSShineFrontendService_20170701.TranslateText", + requests.first.request_headers["X-Amz-Target"] + end + + # Runs against the real aws-sigv4 library, not a stand-in, so this is real evidence signing works. + def test_it_signs_the_request + provider.translate(translation_request(%w[one])) + authorization = requests.first.request_headers["Authorization"] + + assert_match(/\AAWS4-HMAC-SHA256 Credential=AKIAEXAMPLE/, authorization) + assert_match(/Signature=[0-9a-f]{64}\z/, authorization) + end + + def test_it_sends_one_text_per_call_because_the_api_has_no_batch + provider(texts: %w[один два три]).translate(translation_request(%w[one two three])) + + assert_equal 3, requests.size + sent = requests.map { |r| JSON.parse(r.body)["Text"] } + + assert_equal %w[one two three], sent + end + + def test_it_returns_the_translations_in_the_order_they_were_asked_for + response = provider(texts: %w[один два три]).translate(translation_request(%w[one two three])) + + assert_equal %w[один два три], response.texts + end + + def test_a_missing_source_language_becomes_auto + provider.translate(translation_request(%w[one], from: nil)) + + assert_equal "auto", JSON.parse(requests.first.body)["SourceLanguageCode"] + end + + def test_it_reports_the_source_language_amazon_resolved + response = provider.translate(translation_request(%w[one], from: nil)) + + assert_equal "en", response.detected_source + end + + # The capability is the warning: Amazon has no HTML mode, so a notranslate span sent to it WILL be translated. + def test_it_claims_neither_html_nor_notranslate + capabilities = TranslationDiff::Providers::Amazon.capabilities + + refute_predicate capabilities, :html? + refute_predicate capabilities, :notranslate? + assert_equal 1, capabilities.max_batch_size + assert_equal 10_000, capabilities.max_text_size + end +end diff --git a/test/translation_diff/providers/azure_test.rb b/test/translation_diff/providers/azure_test.rb new file mode 100644 index 0000000..d0597cf --- /dev/null +++ b/test/translation_diff/providers/azure_test.rb @@ -0,0 +1,212 @@ +require "test_helper" +require "support/provider_contract" +require "support/http_provider_contract" +require "support/stubbed_provider" +require "faraday" +require "cgi" + +class AzureProviderTest < Minitest::Test + include ProviderContract + include HTTPProviderContract + include StubbedProvider + + # No Azure key was available: shaped from Microsoft's v3 "Translate" reference (read 2026-09-09), not observed. + BODY = [ + { "detectedLanguage" => { "language" => "en", "score" => 1.0 }, + "translations" => [{ "text" => "один", "to" => "ru" }] }, + { "detectedLanguage" => { "language" => "en", "score" => 1.0 }, + "translations" => [{ "text" => "два", "to" => "ru" }] } + ].freeze + + attr_reader :config + + def setup + TranslationDiff.reset! + @config = TranslationDiff::Configuration.new + @config.azure_api_key = "test-key" + end + + def provider_class = TranslationDiff::Providers::Azure + + # Left nil, `body:` echoes back whatever texts were sent, so ProviderContract's count check never trips. + def provider(body: nil, status: 200, headers: {}) + stub_provider(route: "/translate", body: body || method(:echo_translations), + status: status, headers: headers, name: :azure) + end + + def test_the_default_api_base_is_the_documented_host + assert_equal "https://api.cognitive.microsofttranslator.com", + TranslationDiff::Providers::Azure.new(config).api_base + end + + def test_the_api_base_option_overrides_the_default + config.azure_api_base = "https://azure.internal" + + assert_equal "https://azure.internal", TranslationDiff::Providers::Azure.new(config).api_base + end + + def test_it_sends_the_key_in_the_documented_header + assert_equal "test-key", + TranslationDiff::Providers::Azure.new(config).headers["Ocp-Apim-Subscription-Key"] + end + + # A single-service key needs no region and a multi-service one does. + def test_the_region_header_appears_only_when_configured + refute TranslationDiff::Providers::Azure.new(config).headers.key?("Ocp-Apim-Subscription-Region") + + config.azure_region = "westeurope" + + assert_equal "westeurope", + TranslationDiff::Providers::Azure.new(config).headers["Ocp-Apim-Subscription-Region"] + end + + def test_a_missing_key_is_named_before_any_request + config.azure_api_key = nil + + error = assert_raises(TranslationDiff::ConfigurationError) do + TranslationDiff::Providers::Azure.new(config) + end + + assert_match(/azure_api_key/, error.message) + end + + def test_the_languages_travel_in_the_query_string_not_the_body + provider.translate(translation_request(%w[one two])) + + assert_equal ["3.0"], query["api-version"] + assert_equal ["en"], query["from"] + assert_equal ["ru"], query["to"] + end + + # Azure rejected "&to=RU"; a caller keeping DeepL-style codes hit it on every call. + def test_it_downcases_a_bare_language_code_whichever_casing_the_caller_used + provider.translate(translation_request(%w[one], from: "EN", to: "RU")) + + assert_equal ["en"], query["from"] + assert_equal ["ru"], query["to"] + end + + def test_it_leaves_a_subtagged_code_untouched + provider.translate(translation_request(%w[one], from: "zh-Hans", to: "pt-BR")) + + assert_equal ["zh-Hans"], query["from"] + assert_equal ["pt-BR"], query["to"] + end + + # Defaults, then caller options, then mandatory fields -- the order the other five already used. + def test_a_caller_option_cannot_displace_a_mandatory_query_parameter + request = TranslationDiff::Translation::Request.new( + texts: %w[one], from: :en, to: :ru, + options: { "to" => "de", "api-version" => "1.0", "textType" => "plain" } + ) + provider.translate(request) + + assert_equal ["ru"], query["to"] + assert_equal ["3.0"], query["api-version"] + assert_equal ["plain"], query["textType"] + end + + def test_it_omits_from_when_none_was_given + provider.translate(translation_request(%w[one], from: nil)) + + refute query.key?("from") + end + + # The body is an array of objects with a capital-T Text key. + def test_it_wraps_each_text_in_the_documented_object + provider.translate(translation_request(%w[one two])) + + assert_equal [{ "Text" => "one" }, { "Text" => "two" }], sent + end + + def test_it_asks_for_html + provider.translate(translation_request(%w[one])) + + assert_equal ["html"], query["textType"] + end + + def test_it_flattens_one_translation_per_input + response = provider(body: BODY).translate(translation_request(%w[one two])) + + assert_equal %w[один два], response.texts + assert_equal "en", response.detected_source + end + + def test_it_reads_the_billed_characters_from_the_metered_usage_header + response = provider(body: BODY, headers: { "X-metered-usage" => "6" }) + .translate(translation_request(%w[one two])) + + assert_equal 6, response.usage.billed_characters + end + + # Absent, the header must yield nil rather than 0 -- 0 is a false claim about billing, not "unknown". + def test_billed_characters_is_nil_when_the_header_is_absent + response = provider(body: BODY).translate(translation_request(%w[one two])) + + assert_nil response.usage.billed_characters + end + + # The same convention the other two billing providers now follow. + def test_a_reported_zero_is_zero_not_unknown + response = provider(body: BODY, headers: { "X-metered-usage" => "0" }) + .translate(translation_request(%w[one two])) + + assert_equal 0, response.usage.billed_characters + end + + def test_a_short_response_raises_rather_than_shifting_nils_into_the_results + short = [BODY.first] + + assert_raises(TranslationDiff::ResponseError) do + provider(body: short).translate(translation_request(%w[one two])) + end + end + + def test_detect_returns_the_language_azure_reports + detector = stub_provider(route: "/detect", body: [{ "language" => "en", "score" => 1.0 }], + name: :azure) + + assert_equal "en", detector.detect("something") + end + + # Azure answers 200 for a batch where one string failed, carrying `error` in place of `translations`. + def test_a_per_string_failure_in_the_middle_of_a_batch_raises_rather_than_caching_nil + body = [ + { "translations" => [{ "text" => "один", "to" => "ru" }] }, + { "error" => { "code" => 400_050, "message" => "The input is too long." } }, + { "translations" => [{ "text" => "три", "to" => "ru" }] } + ] + + error = assert_raises(TranslationDiff::ResponseError) do + provider(body: body).translate(translation_request(%w[one two three])) + end + + assert_match(/position 1/, error.message) + end + + def test_its_limits_are_azures_documented_ones + capabilities = TranslationDiff::Providers::Azure.capabilities + + assert_equal 1_000, capabilities.max_batch_size + assert_equal 50_000, capabilities.max_request_size + assert_equal 50_000, capabilities.max_text_size + end + + def test_it_claims_html_and_notranslate + capabilities = TranslationDiff::Providers::Azure.capabilities + + assert_predicate capabilities, :html? + assert_predicate capabilities, :notranslate? + assert_predicate capabilities, :detects_language? + assert_predicate capabilities, :reports_billing? + end + + private + + def echo_translations(env) + JSON.parse(env.body).map do |item| + { "detectedLanguage" => { "language" => "en", "score" => 1.0 }, + "translations" => [{ "text" => item["Text"], "to" => "ru" }] } + end + end +end diff --git a/test/translation_diff/providers/deepl_test.rb b/test/translation_diff/providers/deepl_test.rb index 7dfb5a7..a0a778b 100644 --- a/test/translation_diff/providers/deepl_test.rb +++ b/test/translation_diff/providers/deepl_test.rb @@ -1,130 +1,219 @@ -# frozen_string_literal: true - require "test_helper" require "support/provider_contract" - -# `TranslationDiff::Providers::DeepL.build` only requires "deepl" lazily, at -# call time, so whether ::DeepL is already defined when this file runs -# depends on test order -- Minitest randomises it. Requiring it explicitly -# here means this file's constant references (::DeepL::Exceptions::Error -# below) don't depend on some other test file having required "deepl" first. -require "deepl" +require "support/http_provider_contract" +require "support/stubbed_provider" +require "support/env_stub" +require "faraday" class DeepLProviderTest < Minitest::Test include ProviderContract + include HTTPProviderContract + include StubbedProvider + include EnvStub + + # A real response body, captured from api-free.deepl.com on 2026-09-09. + TRANSLATE_BODY = { + "translations" => [ + { "detected_source_language" => "EN", "text" => "один", "billed_characters" => 3 }, + { "detected_source_language" => "EN", "text" => "два", "billed_characters" => 3 } + ] + }.freeze + + attr_reader :config + + def setup + TranslationDiff.reset! + @config = TranslationDiff::Configuration.new + @config.deepl_api_key = "test-key:fx" + end + + def provider_class = TranslationDiff::Providers::DeepL + + # Left nil, `body:` echoes back whatever texts were sent, so ProviderContract's count check never trips. + def provider(body: nil, status: 200, headers: {}) + stub_provider(route: "/v2/translate", body: body || method(:echo_translations), + status: status, headers: headers, name: :deepl) + end - # The provider now issues DeepL::Requests::Translate itself, one layer - # below where the old adapter's fake client stood. Stubbing - # DeepL::Requests::Translate is not possible without a mocking library - # (Minitest 6.0 dropped minitest/mock), so this subclasses the provider - # instead and overrides its private #request method -- the smallest - # honest seam available without one, and it still exercises every line - # of #translate and #detect. - class FakeDeepL < TranslationDiff::Providers::DeepL - Text = Struct.new(:text, :detected_source_language) - - attr_reader :calls - - def initialize(*) - super(:unused_api) - @calls = [] + # deepl-rb read DEEPL_AUTH_KEY on our behalf; an app that set it and configured nothing kept working. + def test_the_deepl_auth_key_environment_variable_supplies_an_unset_key + with_env("DEEPL_AUTH_KEY" => "env-key:fx") do + assert_equal "env-key:fx", TranslationDiff::Configuration.new.deepl_api_key end + end - private + # The default is a callable, so setting the variable after this file was required still works. + def test_the_environment_is_read_on_use_not_at_load_time + built = TranslationDiff::Configuration.new - def request(text, from, to, options = {}) - @calls << [text, from, to, options] - Array(text).map { |value| Text.new("#{value}-translated", "EN") } - .then { |texts| text.is_a?(Array) ? texts : texts.first } + with_env("DEEPL_AUTH_KEY" => "set-after-the-fact") do + assert_equal "set-after-the-fact", built.deepl_api_key end end - def provider - FakeDeepL.new + def test_an_explicitly_configured_key_wins_over_the_environment + with_env("DEEPL_AUTH_KEY" => "env-key") do + assert_equal "test-key:fx", config.deepl_api_key + end end - def test_translate_unwraps_the_text_of_each_result - assert_equal %w[one-translated two-translated], provider.translate(%w[one two], from: :en, to: :ru) + def test_a_blank_environment_variable_is_not_a_key + with_env("DEEPL_AUTH_KEY" => " ") do + assert_nil TranslationDiff::Configuration.new.deepl_api_key + end end - def test_translate_passes_provider_options_through - fake = FakeDeepL.new + # The scenario: ensure_configured! raised ConfigurationError at boot for an app that only set the variable. + def test_the_environment_variable_satisfies_the_configuration_requirement + with_env("DEEPL_AUTH_KEY" => "env-key") do + assert_instance_of TranslationDiff::Providers::DeepL, + TranslationDiff::Providers::DeepL.new(TranslationDiff::Configuration.new) + end + end - fake.translate(%w[one], from: :en, to: :ru, formality: :less) + # DeepL documents upper-case codes; a caller writing Google-style lower case must still work. + def test_it_upcases_a_bare_language_code_whichever_casing_the_caller_used + provider.translate(translation_request(%w[one], from: "en", to: "ru")) - assert_equal({ formality: :less }, fake.calls.first.last.slice(:formality)) + assert_equal "EN", sent["source_lang"] + assert_equal "RU", sent["target_lang"] end - # What reaches a provider is not plain text: Tokenizer hands a notranslate - # span over with its tags. DeepL honours `class="notranslate"` only under - # `tag_handling: html`; without it, per DeepL's own documentation, "tags - # are treated as regular text" -- and the protected content is translated - # while the tags survive, which is exactly the shape of bug nobody spots. - def test_translate_asks_for_html_tag_handling - fake = FakeDeepL.new + # The casing of a script or region subtag is its own; upcasing "pt-BR" corrupts it. + def test_it_leaves_a_subtagged_code_untouched + provider.translate(translation_request(%w[one], from: "zh-Hans", to: "pt-BR")) - fake.translate(%w[one], from: :en, to: :ru) + assert_equal "zh-Hans", sent["source_lang"] + assert_equal "pt-BR", sent["target_lang"] + end - assert_equal({ tag_handling: :html, tag_handling_version: "v2" }, - fake.calls.first.last.slice(:tag_handling, :tag_handling_version)) + def test_a_free_key_selects_the_free_host + assert_equal "https://api-free.deepl.com", TranslationDiff::Providers::DeepL.new(config).api_base end - def test_translate_lets_the_caller_override_the_tag_handling - fake = FakeDeepL.new + def test_a_paid_key_selects_the_paid_host + config.deepl_api_key = "test-key" - fake.translate(%w[one], from: :en, to: :ru, tag_handling: :xml) + assert_equal "https://api.deepl.com", TranslationDiff::Providers::DeepL.new(config).api_base + end + + def test_the_api_base_option_overrides_both + config.deepl_api_base = "https://deepl.internal" - assert_equal :xml, fake.calls.first.last[:tag_handling] + assert_equal "https://deepl.internal", TranslationDiff::Providers::DeepL.new(config).api_base end - def test_detect_downcases_the_language - assert_equal "en", provider.detect("etwas") + def test_it_authenticates_with_the_deepl_scheme + assert_equal "DeepL-Auth-Key test-key:fx", + TranslationDiff::Providers::DeepL.new(config).headers["Authorization"] end - # DeepL has no detection endpoint, so the provider supplies a target of - # its own rather than making the caller invent one. - def test_detect_supplies_its_own_target_language - fake = FakeDeepL.new + def test_a_missing_key_is_named_before_any_request + config.deepl_api_key = nil + + error = assert_raises(TranslationDiff::ConfigurationError) do + TranslationDiff::Providers::DeepL.new(config) + end + + assert_match(/deepl_api_key/, error.message) + end - fake.detect("etwas") + def test_it_sends_the_texts_and_the_language_pair + provider.translate(translation_request(%w[one two])) - assert_equal [["etwas", nil, "EN", {}]], fake.calls + assert_equal %w[one two], sent["text"] + assert_equal "EN", sent["source_lang"] + assert_equal "RU", sent["target_lang"] end - def test_build_sends_a_free_key_to_the_free_host - config = TranslationDiff::Configuration.new - config.deepl_api_key = "abc:fx" + # DeepL wants upper-case language codes; a caller writing "en" must work. + def test_it_upcases_the_language_codes + provider.translate(translation_request(%w[one two], from: "en", to: "ru")) + + assert_equal "EN", sent["source_lang"] + assert_equal "RU", sent["target_lang"] + end + + def test_it_omits_the_source_language_when_none_was_given + provider.translate(translation_request(%w[one two], from: nil)) + + refute sent.key?("source_lang") + end - provider = TranslationDiff::Providers::DeepL.build(config) - host = provider.instance_variable_get(:@api).configuration.host + # DeepL honours class="notranslate" only under HTML tag handling; otherwise content translates, tags survive. + def test_it_asks_for_html_tag_handling + provider.translate(translation_request(%w[one two])) - assert_equal "https://api-free.deepl.com", host + assert_equal "html", sent["tag_handling"] + assert_equal "v2", sent["tag_handling_version"] end - # Regression test for a content and credential leak. deepl-rb logs the - # whole request at DEBUG -- the Authorization header, DeepL auth key and - # all, plus the text being translated -- so forwarding this library's - # `config.logger` into DeepL::Configuration wrote customers' content and - # the API key into the application log the moment anyone turned DEBUG on. - # This gem guarantees its own log lines carry none of that, so the logger - # must not cross into deepl-rb. - def test_build_does_not_forward_the_logger_into_deepl_rb - config = TranslationDiff::Configuration.new - config.deepl_api_key = "abc:fx" - config.logger = Object.new + def test_a_caller_option_overrides_a_default + provider.translate(translation_request(%w[one two], tag_handling: "xml", formality: "less")) + + assert_equal "xml", sent["tag_handling"] + assert_equal "less", sent["formality"] + end + + def test_it_returns_the_translations_in_order + response = provider(body: TRANSLATE_BODY).translate(translation_request(%w[one two])) + + assert_equal %w[один два], response.texts + end + + def test_it_reports_the_detected_source_and_the_billed_characters + response = provider(body: TRANSLATE_BODY).translate(translation_request(%w[one two], from: nil)) + + assert_equal "en", response.detected_source + assert_equal 6, response.usage.billed_characters + end + + # nil means "the provider did not say"; a reported 0 is a claim, and mapping it to nil erased one. + def test_a_reported_zero_is_zero_not_unknown + body = { "translations" => [{ "text" => "один", "billed_characters" => 0 }, + { "text" => "два", "billed_characters" => 0 }] } + response = provider(body: body).translate(translation_request(%w[one two])) + + assert_equal 0, response.usage.billed_characters + end + + def test_billed_characters_is_nil_when_no_translation_reported_it + body = { "translations" => [{ "text" => "один" }, { "text" => "два" }] } + response = provider(body: body).translate(translation_request(%w[one two])) + + assert_nil response.usage.billed_characters + end + + def test_a_short_response_raises_rather_than_shifting_nils_into_the_results + short = { "translations" => [{ "text" => "один" }] } + + assert_raises(TranslationDiff::ResponseError) do + provider(body: short).translate(translation_request(%w[one two])) + end + end + + # DeepL has no detection endpoint; it detects by translating a sample and reading the reported source. + def test_detect_returns_the_language_deepl_reports + assert_equal "en", provider.detect("something") + end + + def test_its_batch_limit_is_deepls_documented_fifty + assert_equal 50, TranslationDiff::Providers::DeepL.capabilities.max_batch_size + end - provider = TranslationDiff::Providers::DeepL.build(config) + def test_it_claims_html_and_notranslate + capabilities = TranslationDiff::Providers::DeepL.capabilities - assert_nil provider.instance_variable_get(:@api).configuration.logger + assert_predicate capabilities, :html? + assert_predicate capabilities, :notranslate? + assert_predicate capabilities, :detects_language? + assert_predicate capabilities, :reports_billing? end - def test_build_raises_when_no_key_is_available - original = ENV.fetch("DEEPL_AUTH_KEY", nil) - ENV["DEEPL_AUTH_KEY"] = nil - config = TranslationDiff::Configuration.new + private - assert_raises(::DeepL::Exceptions::Error) { TranslationDiff::Providers::DeepL.build(config) } - ensure - ENV["DEEPL_AUTH_KEY"] = original + def echo_translations(env) + texts = JSON.parse(env.body)["text"] + { "translations" => texts.map { |t| { "text" => t, "detected_source_language" => "EN" } } } end end diff --git a/test/translation_diff/providers/google_test.rb b/test/translation_diff/providers/google_test.rb index 5af5640..d434729 100644 --- a/test/translation_diff/providers/google_test.rb +++ b/test/translation_diff/providers/google_test.rb @@ -1,178 +1,187 @@ -# frozen_string_literal: true - require "test_helper" require "support/provider_contract" - -# `TranslationDiff::Providers::Google.build` only requires the gem lazily, at -# call time, so whether ::Google::Cloud::Translate::V2 is already defined when -# this file runs depends on test order -- Minitest randomises it. Requiring it -# explicitly here means this file's constant references don't depend on some -# other test file having required it first. -require "google/cloud/translate/v2" +require "support/http_provider_contract" +require "support/stubbed_provider" +require "support/env_stub" +require "faraday" +require "cgi" class GoogleProviderTest < Minitest::Test include ProviderContract + include HTTPProviderContract + include StubbedProvider + include EnvStub - # Stands in for Google::Cloud::Translate::V2::Api. The single/array return - # asymmetry is copied deliberately from the real thing: Translation - # .from_gapi_list and Detection.from_gapi both return a bare object rather - # than a one-element array when they were given one text, and a provider - # that forgets that hands Request a Translation where it expects an Array. - class FakeApi - Translation = Struct.new(:text) - Detection = Struct.new(:language) + # Shaped from the Cloud Translation v2 REST reference read 2026-09-09: translations nest under "data". + TRANSLATE_BODY = { + "data" => { "translations" => [ + { "translatedText" => "один", "detectedSourceLanguage" => "en" }, + { "translatedText" => "два", "detectedSourceLanguage" => "en" } + ] } + }.freeze - attr_reader :calls + attr_reader :config + + def setup + TranslationDiff.reset! + @config = TranslationDiff::Configuration.new + @config.google_api_key = "test-key" + end + + def provider_class = TranslationDiff::Providers::Google + + # Left nil, `body:` echoes back whatever texts were sent, so ProviderContract's count check never trips. + def provider(body: nil, status: 200, headers: { "Content-Type" => "application/json; charset=UTF-8" }) + stub_provider(route: "/language/translate/v2", body: body || method(:echo_translations), + status: status, headers: headers, name: :google) + end - def initialize - @calls = [] + # google-cloud-translate-v2 read these on our behalf, TRANSLATE_KEY first. + def test_translate_key_supplies_an_unset_api_key + with_env("TRANSLATE_KEY" => "from-translate-key", "GOOGLE_CLOUD_KEY" => nil) do + assert_equal "from-translate-key", TranslationDiff::Configuration.new.google_api_key end + end - def translate(*text, **options) - @calls << [text, options] - unwrap(text.map { |value| Translation.new("#{value}-translated") }) + def test_google_cloud_key_is_the_second_fallback + with_env("TRANSLATE_KEY" => nil, "GOOGLE_CLOUD_KEY" => "from-google-cloud-key") do + assert_equal "from-google-cloud-key", TranslationDiff::Configuration.new.google_api_key end + end - def detect(*text) - @calls << [text, {}] - unwrap(text.map { Detection.new("en") }) + def test_translate_key_wins_over_google_cloud_key + with_env("TRANSLATE_KEY" => "first", "GOOGLE_CLOUD_KEY" => "second") do + assert_equal "first", TranslationDiff::Configuration.new.google_api_key end + end - private + def test_an_explicitly_configured_key_wins_over_both_variables + with_env("TRANSLATE_KEY" => "first", "GOOGLE_CLOUD_KEY" => "second") do + assert_equal "test-key", config.google_api_key + end + end - def unwrap(results) = results.size == 1 ? results.first : results + def test_translate_project_supplies_an_unset_project_id + with_env("TRANSLATE_PROJECT" => "a-project") do + assert_equal "a-project", TranslationDiff::Configuration.new.google_project_id + end end - def provider = TranslationDiff::Providers::Google.new(FakeApi.new) + def test_it_downcases_a_bare_language_code_whichever_casing_the_caller_used + provider.translate(translation_request(%w[one], from: "EN", to: "RU")) - def test_translate_unwraps_the_text_of_each_result - assert_equal %w[one-translated two-translated], - provider.translate(%w[one two], from: :en, to: :ru) + assert_equal "en", sent["source"] + assert_equal "ru", sent["target"] end - # The API hands back a bare Translation, not a one-element array, when it - # was given one text. Request counts the results against the values it - # sent, so a provider that passes that through fails the count check. - def test_translate_returns_an_array_for_a_single_text - assert_equal %w[one-translated], provider.translate(%w[one], from: :en, to: :ru) - end + def test_it_leaves_a_subtagged_code_untouched + provider.translate(translation_request(%w[one], from: "zh-Hans", to: "pt-BR")) - # What reaches a provider is not plain text: Tokenizer hands over a - # notranslate span with its tags intact, and leaves entities such as - # `&` in the text it emits. Asking for plain text makes Google - # translate the protected span and drop its markup -- verified against - # the live API. `html` is what this gem's tokenizer contract requires, - # and what it has always sent. - def test_translate_asks_for_html - api = FakeApi.new + assert_equal "zh-Hans", sent["source"] + assert_equal "pt-BR", sent["target"] + end - TranslationDiff::Providers::Google.new(api).translate(%w[one], from: :en, to: :ru) + def test_the_key_travels_in_the_query_string + provider.translate(translation_request(%w[one])) - assert_equal :html, api.calls.first.last[:format] + assert_equal ["test-key"], query["key"] end - def test_translate_lets_the_caller_override_the_format - api = FakeApi.new - - TranslationDiff::Providers::Google.new(api).translate(%w[one], from: :en, to: :ru, format: :text) + def test_it_sends_the_texts_and_the_language_pair + provider.translate(translation_request(%w[one two])) - assert_equal :text, api.calls.first.last[:format] + assert_equal %w[one two], sent["q"] + assert_equal "en", sent["source"] + assert_equal "ru", sent["target"] end - # A configuration written against DeepL says "EN"; Google's codes are - # lowercase. - def test_translate_downcases_bare_language_codes - api = FakeApi.new + def test_it_asks_for_html + provider.translate(translation_request(%w[one])) - TranslationDiff::Providers::Google.new(api).translate(%w[one], from: "EN", to: "RU") - - assert_equal({ from: "en", to: "ru" }, api.calls.first.last.slice(:from, :to)) + assert_equal "html", sent["format"] end - # "zh-Hans", "zh-CN" and "pt-BR" carry subtags whose casing is their own; - # a blanket downcase would corrupt them. - def test_translate_passes_subtagged_codes_through_untouched - api = FakeApi.new - - TranslationDiff::Providers::Google.new(api).translate(%w[one], from: "en", to: "zh-Hans") + def test_a_caller_may_ask_for_plain_text + provider.translate(translation_request(%w[one], format: :text)) - assert_equal "zh-Hans", api.calls.first.last[:to] + assert_equal "text", sent["format"] end - # No source language means "detect it", which the API does when `source` - # is absent. Sending "" instead would be rejected. - def test_translate_omits_the_source_language_when_none_is_given - api = FakeApi.new + # Google's codes are lower case, but "zh-Hans" carries a subtag whose casing is its own. + def test_it_downcases_bare_codes_and_leaves_subtagged_ones_alone + provider.translate(translation_request(%w[one], from: "EN", to: "zh-Hans")) - TranslationDiff::Providers::Google.new(api).translate(%w[one], from: nil, to: :ru) - - assert_nil api.calls.first.last[:from] + assert_equal "en", sent["source"] + assert_equal "zh-Hans", sent["target"] end - def test_translate_passes_provider_options_through - api = FakeApi.new - - TranslationDiff::Providers::Google.new(api).translate(%w[one], from: :en, to: :ru, model: "nmt") + def test_it_omits_the_source_language_when_none_was_given + provider.translate(translation_request(%w[one], from: nil)) - assert_equal "nmt", api.calls.first.last[:model] + refute sent.key?("source") end - def test_translate_sends_every_text_in_one_call - api = FakeApi.new + def test_it_parses_the_nested_data_envelope + body = { "data" => { "translations" => [ + { "translatedText" => "один", "detectedSourceLanguage" => "en" } + ] } } + response = provider(body: body).translate(translation_request(%w[one], from: nil)) + + assert_equal %w[один], response.texts + assert_equal "en", response.detected_source + end - TranslationDiff::Providers::Google.new(api).translate(%w[one two three], from: :en, to: :ru) + def test_it_returns_the_translations_in_order + response = provider(body: TRANSLATE_BODY).translate(translation_request(%w[one two])) - assert_equal 1, api.calls.size - assert_equal %w[one two three], api.calls.first.first + assert_equal %w[один два], response.texts end - def test_detect_returns_the_language - assert_equal "en", provider.detect("etwas") + def test_a_short_response_raises_rather_than_shifting_nils_into_the_results + short = { "data" => { "translations" => [{ "translatedText" => "один" }] } } + + assert_raises(TranslationDiff::ResponseError) do + provider(body: short).translate(translation_request(%w[one two])) + end end - # Both numbers are Google's own, and both are load-bearing: Chunker uses - # them to decide where to split, and a batch over 128 is rejected outright. - def test_the_limits_are_the_documented_ones - assert_equal 128, provider.max_batch_size - assert_equal 5_000, provider.max_request_size + def test_the_api_base_option_overrides_the_default + config.google_api_base = "https://google.internal" + + assert_equal "https://google.internal", TranslationDiff::Providers::Google.new(config).api_base end - def test_build_passes_the_configured_key_to_the_api - config = TranslationDiff::Configuration.new - config.google_api_key = "abc" + def test_a_missing_key_is_named_before_any_request + config.google_api_key = nil - provider = TranslationDiff::Providers::Google.build(config) + error = assert_raises(TranslationDiff::ConfigurationError) do + TranslationDiff::Providers::Google.new(config) + end - assert_equal "abc", provider.instance_variable_get(:@api).service.key + assert_match(/google_api_key/, error.message) end - def test_build_passes_the_configured_project_id_to_the_api - config = TranslationDiff::Configuration.new - config.google_api_key = "abc" - config.google_project_id = "a-project" + def test_its_batch_limit_is_googles_documented_one_hundred_twenty_eight + assert_equal 128, TranslationDiff::Providers::Google.capabilities.max_batch_size + end - provider = TranslationDiff::Providers::Google.build(config) + def test_it_claims_html_and_notranslate + capabilities = TranslationDiff::Providers::Google.capabilities - assert_equal "a-project", provider.instance_variable_get(:@api).service.project_id + assert_predicate capabilities, :html? + assert_predicate capabilities, :notranslate? + assert_predicate capabilities, :detects_language? end - # Without a key the gem falls through to application default credentials, - # which need a project id it cannot find in a test environment. - def test_build_raises_when_no_key_is_available - original = ENV.to_hash.slice("TRANSLATE_KEY", "GOOGLE_CLOUD_KEY", "TRANSLATE_PROJECT") - original.each_key { |key| ENV[key] = nil } - config = TranslationDiff::Configuration.new - - assert_raises(StandardError) { TranslationDiff::Providers::Google.build(config) } - ensure - original&.each { |key, value| ENV[key] = value } + def test_google_reports_no_billing + refute_predicate TranslationDiff::Providers::Google.capabilities, :reports_billing? end - def test_it_is_registered_under_its_own_name - config = TranslationDiff::Configuration.new - config.google_api_key = "abc" + private - assert TranslationDiff::Providers.registered?(:google) - assert_equal "google", TranslationDiff::Providers.build(:google, config).cache_key + def echo_translations(env) + texts = JSON.parse(env.body)["q"] + translations = texts.map { |t| { "translatedText" => t, "detectedSourceLanguage" => "en" } } + { "data" => { "translations" => translations } } end end diff --git a/test/translation_diff/providers/libretranslate_test.rb b/test/translation_diff/providers/libretranslate_test.rb new file mode 100644 index 0000000..f576f3b --- /dev/null +++ b/test/translation_diff/providers/libretranslate_test.rb @@ -0,0 +1,122 @@ +require "test_helper" +require "support/provider_contract" +require "support/http_provider_contract" +require "support/stubbed_provider" +require "faraday" + +class LibreTranslateProviderTest < Minitest::Test + include ProviderContract + include HTTPProviderContract + include StubbedProvider + + attr_reader :config + + def setup + TranslationDiff.reset! + @config = TranslationDiff::Configuration.new + @config.libretranslate_api_base = "https://libretranslate.test" + end + + def provider_class = TranslationDiff::Providers::LibreTranslate + + # Left nil, `body:` echoes back whatever texts were sent, so ProviderContract's count check never trips. + def provider(body: nil, status: 200, headers: {}) + stub_provider(route: "/translate", body: body || method(:echo_translations), + status: status, headers: headers, name: :libretranslate) + end + + # Everyone self-hosts this one, so base URL is the requirement and key is the option -- the reverse of the rest. + def test_the_api_base_is_required_and_the_key_is_not + config.libretranslate_api_base = nil + + error = assert_raises(TranslationDiff::ConfigurationError) do + TranslationDiff::Providers::LibreTranslate.new(config) + end + + assert_match(/libretranslate_api_base/, error.message) + end + + def test_the_key_travels_in_the_body_when_set + config.libretranslate_api_key = "test-key" + provider.translate(translation_request(%w[one])) + + assert_equal "test-key", sent["api_key"] + end + + def test_the_key_is_absent_from_the_body_when_unset + provider.translate(translation_request(%w[one])) + + refute sent.key?("api_key") + end + + # source is required by the API, and "auto" is how detection is asked for. + def test_a_missing_source_language_becomes_auto + provider.translate(translation_request(%w[one], from: nil)) + + assert_equal "auto", sent["source"] + end + + # LibreTranslate answered 400 for "EN"/"RU" on every call. + def test_it_downcases_a_bare_language_code_whichever_casing_the_caller_used + provider.translate(translation_request(%w[one], from: "EN", to: "RU")) + + assert_equal "en", sent["source"] + assert_equal "ru", sent["target"] + end + + def test_it_leaves_a_subtagged_code_untouched + provider.translate(translation_request(%w[one], from: "zh-Hans", to: "pt-BR")) + + assert_equal "zh-Hans", sent["source"] + assert_equal "pt-BR", sent["target"] + end + + def test_it_asks_for_html + provider.translate(translation_request(%w[one])) + + assert_equal "html", sent["format"] + end + + def test_it_reads_an_array_response + body = { "translatedText" => %w[один два], + "detectedLanguage" => [{ "language" => "en", "confidence" => 92.0 }] } + response = provider(body: body).translate(translation_request(%w[one two])) + + assert_equal %w[один два], response.texts + assert_equal "en", response.detected_source + end + + # A single q comes back as a bare string, not a one-element array. + def test_it_reads_a_single_string_response + body = { "translatedText" => "один", "detectedLanguage" => { "language" => "en" } } + response = provider(body: body).translate(translation_request(%w[one])) + + assert_equal %w[один], response.texts + end + + def test_a_short_response_raises_rather_than_shifting_nils_into_the_results + short = { "translatedText" => "один" } + + assert_raises(TranslationDiff::ResponseError) do + provider(body: short).translate(translation_request(%w[one two])) + end + end + + def test_detect_returns_the_language_libretranslate_reports + detector = stub_provider(route: "/detect", body: [{ "language" => "en", "confidence" => 92.0 }], + name: :libretranslate) + + assert_equal "en", detector.detect("something") + end + + def test_its_batch_limit_is_this_librarys_own_conservative_choice + assert_equal 50, TranslationDiff::Providers::LibreTranslate.capabilities.max_batch_size + end + + private + + def echo_translations(env) + texts = JSON.parse(env.body)["q"] + { "translatedText" => texts, "detectedLanguage" => texts.map { { "language" => "en" } } } + end +end diff --git a/test/translation_diff/providers/modernmt_test.rb b/test/translation_diff/providers/modernmt_test.rb new file mode 100644 index 0000000..b501cb7 --- /dev/null +++ b/test/translation_diff/providers/modernmt_test.rb @@ -0,0 +1,109 @@ +require "test_helper" +require "support/provider_contract" +require "support/http_provider_contract" +require "support/stubbed_provider" +require "faraday" + +class ModernMTProviderTest < Minitest::Test + include ProviderContract + include HTTPProviderContract + include StubbedProvider + + # Shaped from modernmt.com/api's reference (read 2026-09-09), not a live call: no key was available. + BODY = { "data" => [ + { "translation" => "один", "billedCharacters" => 3, "characters" => 3, "detectedLanguage" => "en" }, + { "translation" => "два", "billedCharacters" => 3, "characters" => 3, "detectedLanguage" => "en" } + ] }.freeze + + attr_reader :config + + def setup + TranslationDiff.reset! + @config = TranslationDiff::Configuration.new + @config.modernmt_api_key = "test-key" + end + + def provider_class = TranslationDiff::Providers::ModernMT + + # Left nil, `body:` echoes back whatever texts were sent, so ProviderContract's count check never trips. + def provider(body: nil, status: 200, headers: {}) + stub_provider(route: "/translate", body: body || method(:echo_translations), + status: status, headers: headers, name: :modernmt) + end + + def test_it_sends_the_key_in_the_documented_header + assert_equal "test-key", + TranslationDiff::Providers::ModernMT.new(config).headers["MMT-ApiKey"] + end + + def test_it_sends_the_texts_and_the_language_pair_in_the_body + provider.translate(translation_request(%w[one two])) + + assert_equal %w[one two], sent["q"] + assert_equal "en", sent["source"] + assert_equal "ru", sent["target"] + end + + def test_it_downcases_a_bare_language_code_whichever_casing_the_caller_used + provider.translate(translation_request(%w[one], from: "EN", to: "RU")) + + assert_equal "en", sent["source"] + assert_equal "ru", sent["target"] + end + + def test_it_leaves_a_subtagged_code_untouched + provider.translate(translation_request(%w[one], from: "zh-Hans", to: "pt-BR")) + + assert_equal "zh-Hans", sent["source"] + assert_equal "pt-BR", sent["target"] + end + + def test_it_asks_for_html_by_mime_type + provider.translate(translation_request(%w[one])) + + assert_equal "text/html", sent["format"] + end + + def test_it_unwraps_the_data_envelope + response = provider(body: BODY).translate(translation_request(%w[one two])) + + assert_equal %w[один два], response.texts + assert_equal "en", response.detected_source + assert_equal 6, response.usage.billed_characters + end + + # Same convention as DeepL and Azure: a reported 0 is a claim, not "unknown". + def test_a_reported_zero_is_zero_not_unknown + body = { "data" => [{ "translation" => "один", "billedCharacters" => 0 }, + { "translation" => "два", "billedCharacters" => 0 }] } + response = provider(body: body).translate(translation_request(%w[one two])) + + assert_equal 0, response.usage.billed_characters + end + + def test_billed_characters_is_nil_when_no_result_reported_it + body = { "data" => [{ "translation" => "один" }, { "translation" => "два" }] } + response = provider(body: body).translate(translation_request(%w[one two])) + + assert_nil response.usage.billed_characters + end + + # One text comes back as an object, not a one-element array, which would hand the pipeline a bare Hash. + def test_a_single_text_comes_back_unwrapped_and_is_still_a_list + single = { "data" => { "translation" => "один", "detectedLanguage" => "en" } } + response = provider(body: single).translate(translation_request(%w[one])) + + assert_equal %w[один], response.texts + end + + def test_its_batch_limit_is_the_documented_maximum + assert_equal 128, TranslationDiff::Providers::ModernMT.capabilities.max_batch_size + end + + private + + def echo_translations(env) + texts = JSON.parse(env.body)["q"] + { "data" => texts.map { |text| { "translation" => text, "detectedLanguage" => "en" } } } + end +end diff --git a/test/translation_diff/providers/null_test.rb b/test/translation_diff/providers/null_test.rb index 6a7cb89..567c563 100644 --- a/test/translation_diff/providers/null_test.rb +++ b/test/translation_diff/providers/null_test.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - require "test_helper" require "support/provider_contract" @@ -7,10 +5,16 @@ class NullProviderTest < Minitest::Test include ProviderContract def provider - TranslationDiff::Providers::Null.new + TranslationDiff::Providers::Null.new(TranslationDiff::Configuration.new) end def test_translate_returns_the_input_unchanged - assert_equal %w[one two], provider.translate(%w[one two], from: :en, to: :ru) + request = TranslationDiff::Translation::Request.new(texts: %w[one two], from: :en, to: :ru) + + assert_equal %w[one two], provider.translate(request).texts + end + + def test_cache_key_is_null + assert_equal "null", provider.cache_key end end diff --git a/test/translation_diff/providers_test.rb b/test/translation_diff/providers_test.rb index 8ba0c9f..64e2c27 100644 --- a/test/translation_diff/providers_test.rb +++ b/test/translation_diff/providers_test.rb @@ -1,60 +1,48 @@ -# frozen_string_literal: true - require "test_helper" class ProvidersTest < Minitest::Test - # A provider defined entirely outside this library, to prove that adding a - # translation service requires no change to lib/. - class AcmeProvider + # Defined entirely outside this library, to prove adding a translation service requires no change to lib/. + class AcmeProvider < TranslationDiff::Provider def self.configuration_options = %i[acme_token] - def self.build(config) = new(config.acme_token) - - attr_accessor :name - def initialize(token) - @token = token + def self.capabilities + TranslationDiff::Capabilities.new( + max_request_size: 1_000, max_batch_size: 10, max_text_size: nil, + html: :none, notranslate: false, detects_language: false, reports_billing: false + ) end - def translate(texts, from:, to:, **_options) - texts.map { |text| "#{@token}:#{from}-#{to}:#{text}" } + def translate(request) + TranslationDiff::Translation::Response.build( + request: request, + texts: request.texts.map { |text| "#{config.acme_token}:#{request.from}-#{request.to}:#{text}" } + ) end - - def max_request_size = 1_000 - def max_batch_size = 10 end - # Two providers that both want the same option name. Registering the - # second must raise rather than hand it the first one's accessor. - class ConflictingProviderA + # Registering the second must raise rather than hand it the first one's accessor. + class ConflictingProviderA < TranslationDiff::Provider def self.configuration_options = %i[shared_provider_token] - def self.build(_config) = new end - class ConflictingProviderB + class ConflictingProviderB < TranslationDiff::Provider def self.configuration_options = %i[shared_provider_token] - def self.build(_config) = new end - # A subclass wanting AcmeProvider's own option -- subclassing a provider - # to point it at a different host or account is an obvious thing to want. + # A subclass wanting AcmeProvider's own option. class SubclassOfAcmeProvider < AcmeProvider; end - # The second-key-conflicts shape: PartialB would declare :partial_own_key - # successfully if checked eagerly, but conflicts with PartialA's - # :partial_shared_key on its second option. - class PartialProviderA + # PartialB would declare :partial_own_key successfully if checked eagerly, but conflicts on its second. + class PartialProviderA < TranslationDiff::Provider def self.configuration_options = %i[partial_shared_key] - def self.build(_config) = new end - class PartialProviderB + class PartialProviderB < TranslationDiff::Provider def self.configuration_options = %i[partial_own_key partial_shared_key] - def self.build(_config) = new end - class PartialProviderC + class PartialProviderC < TranslationDiff::Provider def self.configuration_options = %i[partial_own_key] - def self.build(_config) = new end def setup @@ -69,8 +57,9 @@ def test_registering_declares_the_providers_own_options_on_the_configuration def test_building_produces_a_working_provider provider = TranslationDiff::Providers.build(:acme, @config) + request = TranslationDiff::Translation::Request.new(texts: %w[hi], from: :en, to: :ru) - assert_equal ["T:en-ru:hi"], provider.translate(["hi"], from: :en, to: :ru) + assert_equal ["T:en-ru:hi"], provider.translate(request).texts end def test_the_registered_name_becomes_the_cache_key @@ -78,25 +67,23 @@ def test_the_registered_name_becomes_the_cache_key end def test_the_built_in_providers_are_registered - assert TranslationDiff::Providers.registered?(:deepl) assert TranslationDiff::Providers.registered?(:null) + assert TranslationDiff::Providers.registered?(:deepl) + assert TranslationDiff::Providers.registered?(:google) end def test_null_keeps_its_own_cache_key assert_equal "null", TranslationDiff::Providers.build(:null, @config).cache_key end - # A defensive double `require` and a Rails development reload both re-run - # registration, so the same provider redeclaring its own options must stay - # silent. #setup has already registered AcmeProvider once. + # A defensive double `require` and a Rails reload both re-run registration; redeclaring must stay silent. def test_registering_the_same_provider_twice_is_not_a_conflict TranslationDiff::Providers.register(:acme, AcmeProvider) assert_includes TranslationDiff::Configuration.options, :acme_token end - # Rails reloading yields a *new* class object under the same constant, so - # identity alone would make an ordinary development reload raise. + # Rails reloading yields a *new* class object under the same constant, so identity alone would raise. def test_a_reloaded_class_of_the_same_name_is_not_a_conflict Object.const_set(:ReloadedProvider, reloadable_provider_class) TranslationDiff::Providers.register(:reloaded, ReloadedProvider) @@ -112,9 +99,7 @@ def test_a_reloaded_class_of_the_same_name_is_not_a_conflict Object.send(:remove_const, :ReloadedProvider) if Object.const_defined?(:ReloadedProvider) end - # Reproduces the credential crossing: `Configuration.option` returns early - # on a name it already knows, so without this guard ConflictingProviderB - # would be handed the accessor -- and the value -- ConflictingProviderA set. + # Without this guard, ConflictingProviderB would be handed the accessor and value ConflictingProviderA set. def test_a_second_provider_claiming_a_declared_option_name_raises TranslationDiff::Providers.register(:conflict_a, ConflictingProviderA) @@ -128,10 +113,7 @@ def test_a_second_provider_claiming_a_declared_option_name_raises refute TranslationDiff::Providers.registered?(:conflict_b) end - # AcmeProvider (registered as :acme in #setup) owns :acme_token. A - # subclass inherits that option and must still be registerable under its - # own name -- this used to raise, since a subclass claiming its inherited - # option looked identical to an unrelated class claiming a taken one. + # Used to raise: a subclass claiming its inherited option looked identical to an unrelated class claiming it. def test_a_subclass_of_a_registered_provider_may_be_registered TranslationDiff::Providers.register(:acme_subclass, SubclassOfAcmeProvider) @@ -139,12 +121,10 @@ def test_a_subclass_of_a_registered_provider_may_be_registered assert_includes TranslationDiff::Configuration.options, :acme_token end - # An unrelated class is still refused for the very option a subclass may - # now share -- the relaxation is specific to an inheritance relationship. + # The relaxation is specific to an inheritance relationship; an unrelated class is still refused. def test_an_unrelated_class_claiming_a_subclassable_option_still_raises - unrelated = Class.new do + unrelated = Class.new(TranslationDiff::Provider) do def self.configuration_options = %i[acme_token] - def self.build(_config) = new end error = assert_raises(TranslationDiff::Error) do @@ -155,10 +135,7 @@ def self.build(_config) = new refute TranslationDiff::Providers.registered?(:acme_unrelated) end - # Registration must be all-or-nothing: PartialProviderB conflicts on its - # second option, so it must not leave its first option declared, owned by - # PartialProviderB, or itself registered -- and a later, legitimate - # provider claiming that first option name must succeed. + # Registration must be all-or-nothing: a conflict on the second option must not leave the first declared. def test_a_failed_registration_leaves_no_partial_option_state TranslationDiff::Providers.register(:partial_a, PartialProviderA) @@ -175,24 +152,71 @@ def test_a_failed_registration_leaves_no_partial_option_state assert_includes TranslationDiff::Configuration.options, :partial_own_key end - # A provider instantiated directly, bypassing TranslationDiff::Providers.build, - # never gets its #name stamped. Falling back to "" there would let two such - # providers share the same cache namespace silently, so this must raise - # instead. - def test_cache_key_raises_when_the_provider_was_never_built_through_the_registry - provider = AcmeProvider.new("T") + # A duck-typed object cannot be given the transport, the requirement check, or the capability defaults. + def test_registering_a_class_that_is_not_a_provider_raises + not_a_provider = Class.new do + def self.configuration_options = [] + def self.build(_config) = new + end + + error = assert_raises(TranslationDiff::InvalidProviderError) do + TranslationDiff::Providers.register(:impostor, not_a_provider) + end + + assert_match(/TranslationDiff::Provider/, error.message) + refute TranslationDiff::Providers.registered?(:impostor) + end + + # `klass < Provider` raised NoMethodError for an instance -- the first thing someone writing a provider hits. + def test_registering_an_instance_rather_than_a_class_raises_the_registry_error + instance = TranslationDiff::Providers::Null.new(TranslationDiff::Configuration.new) + + error = assert_raises(TranslationDiff::InvalidProviderError) do + TranslationDiff::Providers.register(:impostor_instance, instance) + end + + assert_match(/TranslationDiff::Provider/, error.message) + refute TranslationDiff::Providers.registered?(:impostor_instance) + end + + # `klass < Provider` raised ArgumentError for a non-Module, and NoMethodError for nil or a symbol. + def test_registering_a_non_module_raises_the_registry_error + [nil, :deepl, 42, Object.new].each do |value| + assert_raises(TranslationDiff::InvalidProviderError) do + TranslationDiff::Providers.register(:impostor_value, value) + end + end + + refute TranslationDiff::Providers.registered?(:impostor_value) + end + + # The message names the class, never the object: an arbitrary #to_s renders its own content or an address. + def test_the_registry_error_names_the_class_and_not_the_object + secret = Struct.new(:token).new("s3cret") - error = assert_raises(TranslationDiff::Error) { provider.cache_key } + error = assert_raises(TranslationDiff::InvalidProviderError) do + TranslationDiff::Providers.register(:impostor_secret, secret) + end - assert_match(/registry/, error.message) + refute_match(/s3cret/, error.message) + end + + # A caller rescuing "this class cannot be a provider" must not also accidentally swallow an option collision. + def test_an_option_collision_raises_the_generic_error_not_the_invalid_provider_one + TranslationDiff::Providers.register(:collision_a, ConflictingProviderA) + + error = assert_raises(TranslationDiff::Error) do + TranslationDiff::Providers.register(:collision_b, ConflictingProviderB) + end + + refute_kind_of TranslationDiff::InvalidProviderError, error end private def reloadable_provider_class - Class.new do + Class.new(TranslationDiff::Provider) do def self.configuration_options = %i[reloaded_token] - def self.build(_config) = new end end end diff --git a/test/translation_diff/redis_cache_store_test.rb b/test/translation_diff/redis_cache_store_test.rb index ce76cc9..73c3029 100644 --- a/test/translation_diff/redis_cache_store_test.rb +++ b/test/translation_diff/redis_cache_store_test.rb @@ -1,19 +1,7 @@ -# frozen_string_literal: true - require "test_helper" require "support/cache_store_contract" -# The gem depends on neither redis nor redis-namespace at runtime -- it just -# calls into whatever the application supplies. This stand-in applies the -# namespace the way redis-namespace does, so the keys reaching Redis can be -# asserted on. -# -# TranslationDiff::Configuration#redis_pool requires the real "redis" gem -# lazily, at call time, so whether ::Redis is already defined when this file -# runs depends on test order -- Minitest randomises it. Requiring it -# explicitly here, and nesting this stand-in inside the real class instead of -# declaring a fake top-level `Redis` module, means this file never collides -# with -- or races -- that real constant. +# Nested inside the real Redis class, requiring it explicitly, so this never races Configuration's lazy require. require "redis" class Redis::Namespace @@ -34,11 +22,7 @@ def setex(key, timeout, value) class RedisCacheStoreTest < Minitest::Test include CacheStoreContract - # `values`, when given, forces every #mget to return it regardless of the - # keys asked for -- what the namespacing tests below use to inspect the - # keys reaching Redis without needing real storage behind them. Without - # it, #mget and #setex behave like a real key/value store, which is what - # the shared CacheStoreContract needs. + # `values`, when given, forces #mget to return it regardless of keys asked, to inspect keys without real storage. class FakeRedis attr_reader :calls diff --git a/test/translation_diff/redis_rate_limiter_test.rb b/test/translation_diff/redis_rate_limiter_test.rb index 17e2f44..17c5ae5 100644 --- a/test/translation_diff/redis_rate_limiter_test.rb +++ b/test/translation_diff/redis_rate_limiter_test.rb @@ -1,19 +1,10 @@ -# frozen_string_literal: true - require "test_helper" -# RedisRateLimiter requires "ratelimit" lazily, at the first check, so this -# file exercises the production integration against the real Ratelimit class -# rather than a stand-in. The stand-in this file used to define hid a real -# defect: the gem's signature is `add(subject, count)`, so `add(size)` was -# counting under a subject named after the character count while `exceeded?` -# read a subject nothing ever incremented -- the limit never fired. +# A prior stand-in here hid a real defect: `add(size)` counted under the wrong subject and the limit never fired. require "ratelimit" class RedisRateLimiterTest < Minitest::Test - # An in-memory Redis server implementing exactly the commands ratelimit 1.1 - # issues, plus the two Lua scripts it loads (interpreted here rather than - # run). No socket is opened; nothing here is a stub of the gem under test. + # An in-memory Redis server implementing exactly the commands ratelimit 1.1 issues; no socket is opened. class FakeRedisServer attr_reader :hashes, :expiries, :count_spans @@ -86,8 +77,7 @@ def test_check_counts_under_a_custom_namespace assert_equal({ "ratelimit:tenant-42:call" => 7 }, server.totals) end - # Ratelimit's own rule is `count >= threshold`, so the check that carries - # the count over the line still passes and the next one raises. + # Ratelimit's own rule is `count >= threshold`, so the check that carries the count over the line still passes. def test_check_raises_once_the_threshold_is_passed server = FakeRedisServer.new @@ -107,10 +97,7 @@ def test_check_uses_the_default_threshold assert_raises(TranslationDiff::RedisRateLimiter::RateLimitExceeded) { limiter(server).check(1) } end - # Ratelimit buckets five seconds at a time, so the number of buckets its - # count script sweeps is the interval divided by five -- exactly, since - # both intervals here are multiples of five. That is the only observable - # the interval has. + # Ratelimit buckets five seconds at a time, so buckets swept is the interval divided by five. def test_check_looks_back_over_the_default_interval server = FakeRedisServer.new @@ -127,11 +114,7 @@ def test_check_looks_back_over_a_custom_interval assert_equal [120], server.count_spans end - # The gem used to name the bare `Ratelimit` constant, so an application - # that had not installed it got a raw NameError rather than the "add this - # gem" message the Redis path raises. The singleton `require` here is the - # narrowest way to simulate the gem being absent: it shadows Kernel#require - # for this one object only. + # Naming the bare `Ratelimit` constant used to raise a raw NameError instead of this gem's own message. def test_a_missing_ratelimit_gem_raises_a_translation_diff_error limiter = limiter(FakeRedisServer.new) limiter.define_singleton_method(:require) { |_name| raise LoadError } diff --git a/test/translation_diff/registry_test.rb b/test/translation_diff/registry_test.rb index 881b2ec..f2fce90 100644 --- a/test/translation_diff/registry_test.rb +++ b/test/translation_diff/registry_test.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - require "test_helper" class RegistryTest < Minitest::Test diff --git a/test/translation_diff/request_test.rb b/test/translation_diff/request_test.rb index e7e4878..ebce31c 100644 --- a/test/translation_diff/request_test.rb +++ b/test/translation_diff/request_test.rb @@ -1,24 +1,29 @@ -# frozen_string_literal: true - require "test_helper" class RequestTest < ConfiguredTest - # A minimal adapter. Records what it was asked to translate so the call - # can be asserted on, and answers with a canned response. - class FakeApi - attr_reader :calls, :max_request_size, :max_batch_size + # Records what it was asked to translate, and answers with a canned response. + class FakeApi < TranslationDiff::Provider + CAPABILITIES = TranslationDiff::Capabilities.new( + max_request_size: 1_000_000, max_batch_size: 1_000_000, max_text_size: nil, + html: :none, notranslate: false, detects_language: true, reports_billing: false + ).freeze + + def self.capabilities = CAPABILITIES + + attr_reader :calls - def initialize(response, detected: nil, max_request_size: 1_000_000, max_batch_size: 1_000_000) + def initialize(response, detected: nil) + super(TranslationDiff::Configuration.new) @response = response @detected = detected - @max_request_size = max_request_size - @max_batch_size = max_batch_size @calls = [] end - def translate(texts, from:, to:, **options) - @calls << [texts, from, to, options] - @response.shift(texts.size) + def translate(request) + @calls << [request.texts, request.from, request.to, request.options] + TranslationDiff::Translation::Response.build( + request: request, texts: @response.shift(request.texts.size) + ) end def detect(text) @@ -29,6 +34,31 @@ def detect(text) def cache_key = "fake" end + # Proves the generalisation took effect: capabilities capping the batch at one must change the chunking. + class NarrowBatchApi < FakeApi + def self.capabilities + TranslationDiff::Capabilities.new( + max_request_size: 1_000_000, max_batch_size: 1, max_text_size: nil, + html: :none, notranslate: false, detects_language: true, reports_billing: false + ) + end + end + + # Registered under :echo to exercise resolving a provider by name through the registry. + class EchoProvider < TranslationDiff::Provider + def self.capabilities + TranslationDiff::Capabilities.new( + max_request_size: 1_000_000, max_batch_size: 1_000_000, max_text_size: nil, + html: :none, notranslate: false, detects_language: false, reports_billing: false + ) + end + + def translate(request) = TranslationDiff::Translation::Response.build(request: request, texts: request.texts) + + def cache_key = "echo" + end + TranslationDiff::Providers.register(:echo, EchoProvider) unless TranslationDiff::Providers.registered?(:echo) + # Always misses, so every value reaches the API. class FakeCacheStore attr_reader :writes @@ -46,25 +76,17 @@ def write(key, value) end end - # A provider object assigned straight to `config.provider` never passed - # through the registry, so nothing stamped it with a name. This one defines - # its own #cache_key and returns an empty segment from it, which is the case - # TranslationDiff::Providers::Naming cannot catch. + # An object assigned straight to `config.provider` never passed through the registry's stamping. class NamelessApi < FakeApi def cache_key = "" end - # A whitespace-only key is just as blank as an empty one -- it must not - # slip past the guard and cache translations under a segment that looks - # empty to anyone reading the store. + # A whitespace-only key is just as blank as an empty one; it must not slip past the guard. class WhitespaceNamedApi < FakeApi def cache_key = " " end - # Already has every key cached, regardless of what it is asked for. Proves - # the all-cached short circuit in Request#chunks_translated: the gem's - # headline behaviour is serving a translation from cache without calling - # the adapter at all. + # Proves the all-cached short circuit: serving a translation from cache without calling the adapter at all. class AllCachedStore def initialize(responses) @responses = responses @@ -125,10 +147,7 @@ def test_translates_text_around_markup_and_leaves_the_markup_alone assert_equal [[%w[One Black So Red that], :en, :ru, {}]], api.calls end - # True of the public interface, but not new: 2.1.0 already protected the - # caller's hash from mutation by dup-ing it in the initializer. Keyword - # arguments keep that guarantee for a different reason (a fresh hash per - # call), but this test alone cannot tell the two implementations apart. + # True of the public interface, but not new: 2.1.0 already protected the caller's hash by dup-ing it. def test_repeated_calls_leave_the_callers_options_hash_alone options = { from: :en, to: :ru } @@ -140,18 +159,14 @@ def test_repeated_calls_leave_the_callers_options_hash_alone assert_equal({ from: :en, to: :ru }, options) end - # This is what actually changed: the initializer declares one positional - # parameter now, so a single positional options hash -- which is what every - # pre-task caller passed -- is no longer accepted. + # The initializer now declares one positional parameter, so a positional options hash is no longer accepted. def test_the_positional_options_hash_is_no_longer_accepted assert_raises(ArgumentError) do TranslationDiff::Request.new("text", { from: :en, to: :ru }) end end - # A detected language comes back as a String while :to is usually a Symbol, - # so the source == target short circuit never fired and the text was paid - # for and translated into its own language. + # A detected language is a String while :to is usually a Symbol -- without casecmp? this never short-circuits. def test_skips_the_translation_when_the_detected_language_is_the_target api = FakeApi.new([], detected: "RU") configure_with(api) @@ -162,20 +177,54 @@ def test_skips_the_translation_when_the_detected_language_is_the_target assert_equal [[:detect, "привет"]], api.calls end - def test_raises_when_from_is_missing_and_the_adapter_cannot_detect - configure_with(TranslationDiff::Providers::Null.new) + # Chunker's limits used to come from two provider methods; they now come from the declared capabilities. + # rubocop:disable-next Metrics/AbcSize, Metrics/MethodLength + def test_chunking_uses_the_providers_declared_capabilities + narrow = Class.new(TranslationDiff::Provider) do + def self.capabilities + TranslationDiff::Capabilities.new( + max_request_size: 20, max_batch_size: 1, max_text_size: nil, + html: :none, notranslate: false, detects_language: false, reports_billing: false + ) + end + + attr_reader :batches + + def initialize(config) + super + @batches = [] + end + + def translate(request) + @batches << request.texts + TranslationDiff::Translation::Response.build(request: request, texts: request.texts) + end + + def cache_key = "narrow" + end + + provider = narrow.new(TranslationDiff::Configuration.new) + TranslationDiff.translate("One. Two. Three.", from: "en", to: "ru", provider: provider) + + assert(provider.batches.all? { |batch| batch.size == 1 }, + "expected one text per request, got #{provider.batches.inspect}") + end + # The old check was `respond_to?(:detect)`, satisfied by inheriting the base class's raising stub. + def test_a_provider_that_cannot_detect_says_so_before_it_is_called error = assert_raises(TranslationDiff::Request::Error) do - TranslationDiff::Request.new("text", to: :ru).call + TranslationDiff.translate("Some text.", to: "ru", provider: :null) end assert_match(/cannot detect/, error.message) + assert_match(/null/, error.message) end + # The count check now lives in Translation::Response.build, so it holds for every provider, hence ResponseError. def test_raises_when_the_api_returns_fewer_translations_than_asked_for configure_with(FakeApi.new(%w[Один])) - error = assert_raises(TranslationDiff::Request::Error) do + error = assert_raises(TranslationDiff::ResponseError) do TranslationDiff::Request.new({ a: "One", b: "Two" }, from: :en, to: :ru).call end @@ -203,8 +252,7 @@ def test_passes_through_nil_untouched assert_empty api.calls end - # Scalars nested in a structure are passed through too, while nil keeps - # collapsing to "" the way it always has. + # Scalars nested in a structure are passed through too, while nil keeps collapsing to "". def test_passes_nested_scalars_through_and_still_blanks_out_nils configure_with(FakeApi.new(%w[Один])) @@ -213,10 +261,9 @@ def test_passes_nested_scalars_through_and_still_blanks_out_nils assert_equal({ a: "Один", n: 42, skip: "" }, result) end - # Proves the generalisation took effect rather than merely being - # described: an adapter declaring tiny limits must change the batching. + # Proves the generalisation took effect: a provider declaring tiny limits must change the batching. def test_batches_according_to_the_limits_the_adapter_declares - api = FakeApi.new(%w[Один Два], max_batch_size: 1) + api = NarrowBatchApi.new(%w[Один Два]) configure_with(api) TranslationDiff::Request.new({ a: "One", b: "Two" }, from: :en, to: :ru).call @@ -224,9 +271,7 @@ def test_batches_according_to_the_limits_the_adapter_declares assert_equal 2, api.calls.size, "one call per text at a batch size of 1" end - # Every fake cache store elsewhere in this file always misses, so this is - # the only Request-level test exercising a cache hit: a translation served - # from the store without ever reaching the adapter. + # The only Request-level test exercising a cache hit; every other fake store in this file always misses. def test_serves_a_translation_from_cache_without_calling_the_adapter api = FakeApi.new([]) configure_with(api, AllCachedStore.new(["Какая-то строка"])) @@ -237,22 +282,27 @@ def test_serves_a_translation_from_cache_without_calling_the_adapter assert_empty api.calls end - # `provider:` picks the provider for one call. A name is built through the - # registry against this call's configuration; the configured provider is - # left untouched and unused. + # `provider:` picks the provider for one call; the configured provider is left untouched and unused. def test_the_provider_keyword_overrides_the_configured_provider_for_one_call api = FakeApi.new(%w[Один]) configure_with(api) - result = TranslationDiff::Request.new("One", from: :en, to: :ru, provider: :null).call + result = TranslationDiff::Request.new("One", from: :en, to: :ru, provider: :echo).call assert_equal "One", result assert_empty api.calls end - # An empty cache-key segment would put this provider's translations in the - # same namespace as every other provider's, and a caller would be served - # another service's answer. Refusing is the only safe response. + # The `provider:` keyword is the third way to supply one; it must fail in the same words as the other two. + def test_a_provider_object_passed_for_one_call_that_is_not_a_provider_is_refused + error = assert_raises(TranslationDiff::InvalidProviderError) do + TranslationDiff::Request.new("One", from: :en, to: :ru, provider: Object.new).call + end + + assert_match(/TranslationDiff::Provider/, error.message) + end + + # An empty cache-key segment would put this provider's translations in every other provider's namespace. def test_a_provider_whose_cache_key_is_empty_is_refused_rather_than_sharing_a_namespace configure_with(NamelessApi.new(%w[Один])) @@ -275,7 +325,6 @@ def test_a_provider_whose_cache_key_is_whitespace_is_refused_rather_than_sharing private - # Translates `values` from :en to :ru against fakes. # Returns the translation and the API fake, so the call can be asserted on. def translate(values, response) api = FakeApi.new(response) @@ -284,9 +333,7 @@ def translate(values, response) [TranslationDiff::Request.new(values, from: :en, to: :ru).call, api] end - # The two collaborators every test here needs, assigned as configuration - # options. An object assigned to `provider` or `cache` is used as-is, so a - # fake goes in exactly where a registered name would. + # An object assigned to `provider` or `cache` is used as-is, so a fake goes in exactly where a name would. def configure_with(api, store = FakeCacheStore.new) TranslationDiff.configure do |config| config.provider = api diff --git a/test/translation_diff/segmenters/pragmatic_test.rb b/test/translation_diff/segmenters/pragmatic_test.rb index d2e4a02..065e153 100644 --- a/test/translation_diff/segmenters/pragmatic_test.rb +++ b/test/translation_diff/segmenters/pragmatic_test.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - require "test_helper" class PragmaticSegmenterTest < Minitest::Test @@ -11,16 +9,13 @@ class PragmaticSegmenterTest < Minitest::Test ["Смеркалось. Ворчало. Кричало.", "ru"], ["Набор «Солнечная механика» от 4М — это 6 экспериментов.\n\n" \ "Юному изобретателю предстоит воочию посмотреть на чудеса.", "ru"], - # Multiple blank lines: more than one blank line in a single gap, and - # more than one such gap in the same text. Untouched by shadowing -- - # SINGLE_NEWLINE only matches a "\n" with no adjoining "\n". + # Multiple blank lines, untouched by shadowing: SINGLE_NEWLINE only matches "\n" with no adjoining "\n". ["First paragraph.\n\n\nSecond paragraph.\n\n\n\nThird paragraph.", "en"], ["見て。すごい！次はどうなる？", "ja"], ["Проф. Иванов пришёл домой. Было поздно.", "ru"], ["سؤال وجواب: ماذا حدث؟ طرح الكثير من التساؤلات.", "ar"], ["Ի՞նչ ես մտածում: Ոչինչ:", "hy"], - # Single newlines (shadowed) and a blank-line paragraph break (not - # shadowed), together, so the invariant is exercised against both paths. + # Single newlines (shadowed) and a blank-line paragraph break (not), together, exercising both paths. ["The cat sat on the mat\nand looked at the moon. It was content.\n\n" \ "A new paragraph starts here.", "en"], ["これは父の\n家です。それはペンです。", "ja"] @@ -60,9 +55,7 @@ def test_cjk_terminators_split_without_a_language_hint assert_equal [0, "見て。".length], @segmenter.split_offsets(text) end - # This is the trap the brief calls out by name: without a language, - # pragmatic_segmenter falls back to English rules, and English rules read - # "Проф." as a complete sentence on its own. + # Without a language, pragmatic_segmenter falls back to English rules, which read "Проф." as a full sentence. def test_without_a_language_russian_abbreviations_are_mis_segmented text = "Проф. Иванов пришёл домой. Было поздно." offsets = @segmenter.split_offsets(text) @@ -70,9 +63,7 @@ def test_without_a_language_russian_abbreviations_are_mis_segmented assert_equal [0, "Проф. ".length, "Проф. Иванов пришёл домой. ".length], offsets end - # The same text, with the language supplied, segments correctly -- proving - # the language argument is actually threaded through to pragmatic_segmenter - # rather than merely accepted and ignored. + # Proves the language argument is actually threaded through to pragmatic_segmenter, not merely accepted. def test_with_the_language_russian_abbreviations_are_respected text = "Проф. Иванов пришёл домой. Было поздно." offsets = @segmenter.split_offsets(text, language: "ru") @@ -88,15 +79,7 @@ def test_blank_lines_between_sentences_are_attached_to_the_first_sentence assert_equal [0, "Набор «Солнечная механика» от 4М — это 6 экспериментов.\n\n".length], offsets end - # pragmatic_segmenter treats essentially any single newline as a sentence - # boundary candidate, independent of punctuation -- confirmed on ordinary, - # punctuation-free, line-wrapped prose, including with whitespace on both - # sides of the newline (not just a newline glued to non-whitespace). A - # false split is the harmful kind of error this whole gem exists to avoid, - # and it is common: HTML text nodes routinely carry incidental newlines - # from source formatting. Shadowing single newlines before segmenting - # fixes this while leaving the original text -- newline included -- in - # the output. + # pragmatic_segmenter treats any single newline as a sentence boundary, confirmed false on wrapped prose. def test_a_single_newline_with_no_punctuation_does_not_split_the_sentence text = "Some text \n continues here without any punctuation at the break" assert_equal [0], @segmenter.split_offsets(text) @@ -109,10 +92,7 @@ def test_a_single_newline_between_two_real_sentences_is_preserved_as_a_boundary_ assert_equal [0, "This is a sentence\ncut off by a line wrap. ".length], offsets end - # A blank-line run is a real paragraph break, not incidental formatting, - # and shadowing deliberately leaves it alone -- pragmatic_segmenter already - # handles it correctly (also covered by the reconstruction invariant above, - # and by TokenizerTest's own blank-line case). + # A blank-line run is a real paragraph break; shadowing deliberately leaves it alone. def test_a_blank_line_paragraph_break_still_splits text = "Первое предложение.\n\nВторое предложение." offsets = @segmenter.split_offsets(text, language: "ru") @@ -120,9 +100,7 @@ def test_a_blank_line_paragraph_break_still_splits assert_equal [0, "Первое предложение.\n\n".length], offsets end - # Shadowing fixes the real trigger reported in the previous round: this no - # longer raises, and the newline survives in the output exactly as it - # appeared in the source. + # Shadowing fixes the real trigger reported in the previous round: this no longer raises. def test_the_japanese_newline_after_a_common_particle_now_segments_instead_of_raising text = "これは父の\n家です。それはペンです。" offsets = @segmenter.split_offsets(text, language: "ja") @@ -130,15 +108,7 @@ def test_the_japanese_newline_after_a_common_particle_now_segments_instead_of_ra assert_equal [0, "これは父の\n家です。".length], offsets end - # H1 (fix round 2): pragmatic_segmenter's cleaner rewrites the sentence it - # hands back in ways shadowing does not touch -- collapsing runs of three - # or more spaces, respacing "Ph.D." into "Ph. D.", deleting a formatting - # artefact outright. None of these are rare (an English sentence naming a - # degree, or HTML indented with more than two spaces, hits one of them - # routinely), and none of them may abort translation any more: recovery - # stops at the first sentence it cannot verify and the remainder of the - # text stands as one final unit -- a coarsening, not a failure. Each case - # below is a real, reproduced trigger, not a hypothetical. + # H1 (fix round 2): pragmatic_segmenter's cleaner respaces "Ph.D." into "Ph. D.", a real reproduced trigger. def test_ph_d_no_longer_aborts_and_the_original_text_is_untouched text = "He has a Ph.D. in physics. It took years." assert_equal [0], @segmenter.split_offsets(text, language: "en") @@ -159,22 +129,13 @@ def test_a_newline_plus_a_two_space_indent_no_longer_aborts assert_equal [0], @segmenter.split_offsets(text, language: "en") end - # The inline-formatting artefact pragmatic_segmenter deletes outright - # (lib/pragmatic_segmenter/cleaner/rules.rb, InlineFormattingRule) is what - # the previous round used to prove the (now-removed) raise fired on real - # behaviour. It now proves the opposite: recovery still stops cleanly - # instead of guessing, and the whole node survives as one unit. + # pragmatic_segmenter's InlineFormattingRule deletes this artefact outright; recovery now stops cleanly instead. def test_a_deleted_formatting_artefact_no_longer_aborts text = "This is a sentence{b^>3barbaz" \ - "" + "".freeze # source => expected tokens CASES = { @@ -56,10 +54,7 @@ class TokenizerTest < Minitest::Test "text_split_into_sentences" => [ "! Киловольт. Смеркалось. Ворчало. Кричало.", [ - # Pragmatic does not treat a lone terminator with no preceding - # content as a sentence of its own, so "!" stays merged with - # "Киловольт." -- a missed boundary, which only makes the cache - # unit bigger, not a false one. + # A lone terminator with no preceding content stays merged: a missed boundary, not a false one. ["! Киловольт. ", :text], ["", :markup], ["Смеркалось. ", :text], @@ -106,9 +101,7 @@ class TokenizerTest < Minitest::Test ["", :markup] ] ], - # Ox reports a comment, a doctype and a CDATA section as their own SAX - # events. Each one needs a handler: without it the bytes it covers are - # attributed to no token at all and vanish from the rebuilt string. + # Without a handler, the bytes a comment/doctype/CDATA event covers vanish from the rebuilt string. "an_html_comment" => [ " Visible text.", [ @@ -158,8 +151,6 @@ class TokenizerTest < Minitest::Test private - # The segmenter these expectations were written against, and the one the - # configuration still defaults to. Passed explicitly now that the tokenizer - # takes its segmenter as a collaborator rather than reaching for a global. + # Passed explicitly now that the tokenizer takes its segmenter as a collaborator, not a global. def segmenter = TranslationDiff::Segmenters::Pragmatic.new end diff --git a/test/translation_diff/translation/response_test.rb b/test/translation_diff/translation/response_test.rb new file mode 100644 index 0000000..1ce7f7e --- /dev/null +++ b/test/translation_diff/translation/response_test.rb @@ -0,0 +1,85 @@ +require "test_helper" + +class TranslationResponseTest < Minitest::Test + def request(texts = %w[one two]) + TranslationDiff::Translation::Request.new(texts: texts, from: "en", to: "ru") + end + + def test_a_request_defaults_its_options_to_an_empty_hash + assert_empty TranslationDiff::Translation::Request.new(texts: %w[one], from: nil, to: "ru").options + end + + def test_build_returns_the_texts_it_was_given + response = TranslationDiff::Translation::Response.build(request: request, texts: %w[один два]) + + assert_equal %w[один два], response.texts + end + + # A short response would shift nils into the results, surfacing much later as a distant NoMethodError. + def test_build_raises_when_the_provider_returned_the_wrong_number_of_texts + error = assert_raises(TranslationDiff::ResponseError) do + TranslationDiff::Translation::Response.build(request: request, texts: %w[один]) + end + + assert_match(/1/, error.message) + assert_match(/2/, error.message) + end + + # A nil translation used to reach Spacing.restore and die there as NoMethodError, naming nothing. + def test_build_raises_when_a_translation_is_not_a_string + error = assert_raises(TranslationDiff::ResponseError) do + TranslationDiff::Translation::Response.build(request: request, texts: ["один", nil]) + end + + assert_match(/position 1/, error.message) + assert_match(/NilClass/, error.message) + end + + # Azure returns 200 for a batch where one element carries `error` instead of `translations`. + def test_build_raises_for_a_nil_in_the_middle_of_a_batch + batch = request(%w[one two three]) + + error = assert_raises(TranslationDiff::ResponseError) do + TranslationDiff::Translation::Response.build(request: batch, texts: ["один", nil, "три"]) + end + + assert_match(/position 1/, error.message) + end + + def test_build_names_only_the_first_offending_position + batch = request(%w[one two three]) + + error = assert_raises(TranslationDiff::ResponseError) do + TranslationDiff::Translation::Response.build(request: batch, texts: [nil, nil, 42]) + end + + assert_match(/position 0/, error.message) + refute_match(/position 2/, error.message) + end + + # The offending value is the customer's text or a provider's error object; neither belongs in a message. + def test_build_names_the_class_but_never_the_offending_value + error = assert_raises(TranslationDiff::ResponseError) do + TranslationDiff::Translation::Response.build(request: request, texts: ["один", { "error" => "s3cret" }]) + end + + refute_match(/s3cret/, error.message) + assert_match(/Hash/, error.message) + end + + def test_detected_source_and_usage_default_to_nil + response = TranslationDiff::Translation::Response.build(request: request, texts: %w[один два]) + + assert_nil response.detected_source + assert_nil response.usage + end + + def test_usage_carries_what_the_provider_reported_and_nil_for_the_rest + usage = TranslationDiff::Translation::Usage.new(characters: 40, billed_characters: 40) + + assert_equal 40, usage.characters + assert_equal 40, usage.billed_characters + assert_nil usage.tokens + assert_nil usage.model + end +end diff --git a/test/translation_diff_test.rb b/test/translation_diff_test.rb index 52bd8ed..0b4f449 100644 --- a/test/translation_diff_test.rb +++ b/test/translation_diff_test.rb @@ -1,5 +1,3 @@ -# frozen_string_literal: true - require "test_helper" class TranslationDiffTest < ConfiguredTest diff --git a/translation_diff.gemspec b/translation_diff.gemspec index 83289fe..bf8b2a6 100644 --- a/translation_diff.gemspec +++ b/translation_diff.gemspec @@ -1,5 +1,3 @@ -# frozen_string_literal: true - lib = File.expand_path("lib", __dir__) $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib) require "translation_diff/version" @@ -21,7 +19,7 @@ small edit costs the price of the edit, not the whole text. ) spec.homepage = "https://github.com/Halvanhelv/translation_diff" spec.license = "MIT" - spec.required_ruby_version = ">= 3.2" + spec.required_ruby_version = ">= 3.4" if spec.respond_to?(:metadata) spec.metadata["allowed_push_host"] = "https://rubygems.org" @@ -45,11 +43,14 @@ small edit costs the price of the edit, not the whole text. spec.add_development_dependency "rubocop", "~> 1.90" spec.add_development_dependency "simplecov", "~> 1.2" - # The only two gems this one loads. `ox` walks HTML; `pragmatic_segmenter` - # backs the default sentence segmenter and has zero dependencies of its - # own. Everything else -- the DeepL client, the connection pool, and - # whatever backs the cache and the rate limiter -- is supplied by the - # application and duck typed, so it stays out of the gemspec. + # `ox` walks HTML; `pragmatic_segmenter` backs the default sentence + # segmenter and has zero dependencies of its own; `faraday` and + # `faraday-retry` are the HTTP transport every REST provider inherits. + # Everything else -- the connection pool, and whatever backs the cache and + # the rate limiter -- is supplied by the application and duck typed, so it + # stays out of the gemspec. + spec.add_dependency "faraday", "~> 2.9" + spec.add_dependency "faraday-retry", "~> 2.2" spec.add_dependency "ox", "~> 2.14" spec.add_dependency "pragmatic_segmenter", "~> 0.3" end