Add API to extract terms from provided text by flodolo · Pull Request #4516 · mozilla/pontoon

flodolo · 2026-09-11T08:23:45Z

Opening up for review since it's self-contained, while the second part (extract from file) will touch and likely refactor some code being reviewed in #4511.

flodolo · 2026-09-14T18:39:03Z

(solving merge conflicts)

mathjazz · 2026-09-16T11:52:46Z



+@pytest.fixture
+def terminology_extraction_setup():


Extraction means candidate discovery in terminology lingo, whereas this API matches against an existing glossary.

mathjazz · 2026-09-16T12:49:37Z


    def get_translation_text(self, obj):
+        if obj.do_not_translate:
+            return obj.text


This also changes the existing endpoint /api/v2/search/terminology/, but I guess it's the right behaviour and possibly also not impacting the output.

Good point, forgot to call it out in the PR.

Note: it doesn't impact search/terminology, because that doesn't return do-no-translate terms (they don't have an entity when created, so no translations).

return queryset.filter(translations__locale__code=value)

I wonder if that behavior should be changed, but in case it's probably a follow-up?

mathjazz · 2026-09-16T12:51:33Z

+| Parameter | Description                                               |
+| --------- | --------------------------------------------------------- |
+| `locale`  | Locale code                                               |
+| `text`    | Text to extract terminology from, at most 2048 characters |


Technically this is TERMINOLOGY_API_MAX_CHARS characters. But we can also leave it out in the docs.

mathjazz · 2026-09-16T12:55:23Z

+        if not text.strip():
+            errors["text"] = ["This field is required."]
+        elif len(text) > TERMINOLOGY_API_MAX_CHARS:
+            errors["text"] = [
+                f"Text exceeds maximum length of {TERMINOLOGY_API_MAX_CHARS} characters."
+            ]


Nit: These two conditions are independent, so the elif misreports a 5000-character whitespace-only text as {"text": ["This field is required."]} instead of the length error.

mathjazz · 2026-09-16T13:01:32Z

+    path(
+        "terminology/extract-from-text/",
+        views.TermExtractFromTextView.as_view(),
+        name="term-extract-from-text",


Can we settle the path before this ships? "Extract" in terminology parlance means discovering candidate terms from a corpus, whereas this does the inverse — matching text against the existing glossary. Also, -from-text just repeats the text parameter.

Suggested change

name="term-extract-from-text",

"terminology/matches/",

views.TermMatchListView.as_view(),

name="term-matches",

The reason for using from-text was that this would be coupled with from-file. But at this point, the idea is to leave that part alone for now, and probably have a more generic analysis that does more than just terminology.

On that point, I think I'm going to split the original issue to make that clear, and have this PR fix the existing one.

mathjazz · 2026-09-16T13:06:13Z

        return qs


+class TermExtractFromTextView(generics.ListAPIView):


Shall we add a throttle class here? Each call SELECTs every Term, prefetches their TermTranslations, and runs one re.search per term over up to 2048 chars.

mathjazz · 2026-09-16T13:07:14Z

    return "\n".join(text_parts)


+def join_text_fragments(texts: Iterable[str]) -> str:


This has no production caller. What did you have in mind for it?

D'oh. It's a left-over from local iterations where I was testing "upload from file" (this would be needed for complex Fluent strings), making sure that the structure could be build on top.

mathjazz · 2026-09-16T13:18:37Z

+        )
+    ).order_by("text", "id")
+
+    return terms.for_string(text)


The Prefetch above will not prevent the TermTranslation rows to load for every candidate term rather than the matches — a text matching 1 of 1000 terms still pulls 1000 rows and discards them.

I'd suggest something like this:

matched = Term.objects.for_string(text) return ( Term.objects.filter(pk__in=[t.pk for t in matched]) .prefetch_related(Prefetch(...)) .order_by("text", "id") )

Using slightly different approach that should use one less query?

prefetch_related_objects( terms, Prefetch( "translations", queryset=TermTranslation.objects.filter(locale=locale), to_attr="filtered_translations", ), )

mathjazz · 2026-09-16T13:23:27Z

+
+Extract the terms appearing in a text, with their translation in a given locale.
+
+Unlike [`/api/v2/search/terminology/`](#json-mode), which looks up terms by name, this


Suggested change

Unlike [`/api/v2/search/terminology/`](#json-mode), which looks up terms by name, this

Unlike [`/api/v2/search/terminology/`](#/search/search_terminology_list), which looks up terms by name, this

mathjazz

Great work! Just left one minor suggestion inline for the README consistency.

mathjazz · 2026-09-17T06:29:53Z

+The endpoint is rate limited per user, or per IP address for anonymous requests, with a
+burst limit of 60 calls per minute and a sustained limit of 600 calls per hour by default
+(configurable via `API_TERMINOLOGY_THROTTLE_BURST` and
+`API_TERMINOLOGY_THROTTLE_SUSTAINED`). Calls over the limit are rejected with `429`.


Like for the Upload throttle variables, I'd mention these are independent:
#4503 (comment)

API to extract terms from a string

272057c

flodolo requested a review from mathjazz September 11, 2026 08:23

flodolo changed the title ~~Add API to extract terms from a string~~ Add API to extract terms from provided text Sep 11, 2026

Merge main and resolve conflicts

5906f0a

mathjazz reviewed Sep 16, 2026

View reviewed changes

Address review comments

f5952a3

flodolo requested a review from mathjazz September 17, 2026 04:43

mathjazz reviewed Sep 17, 2026

View reviewed changes

mathjazz approved these changes Sep 17, 2026

View reviewed changes

Update README.md

99a5f65

flodolo merged commit 22dd6c8 into mozilla:main Sep 17, 2026
7 checks passed

flodolo deleted the issue4512_api_terms_strings branch September 17, 2026 06:45

-        name="term-extract-from-text",
+        "terminology/matches/",
+        views.TermMatchListView.as_view(),
+        name="term-matches",

		return qs


		class TermExtractFromTextView(generics.ListAPIView):

		return "\n".join(text_parts)


		def join_text_fragments(texts: Iterable[str]) -> str:


		Extract the terms appearing in a text, with their translation in a given locale.

		Unlike [`/api/v2/search/terminology/`](#json-mode), which looks up terms by name, this

Uh oh!

Conversation

flodolo commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flodolo commented Sep 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flodolo Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mathjazz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

flodolo commented Sep 11, 2026 •

edited

Loading

flodolo Sep 16, 2026 •

edited

Loading