[SharePoint BUG] RefinementToken broken for accented values — Search REST API

[Jsss1010]

Target SharePoint environment

SharePoint Online

What SharePoint development model, framework, SDK or API is this about?

SharePoint REST API

Developer environment

None

What browser(s) / client(s) have you tested

• 💥 Internet Explorer
• 💥 Microsoft Edge
• 💥 Google Chrome
• 💥 FireFox
• 💥 Safari
• mobile (iOS/iPadOS)
• mobile (Android)
• not applicable
• other (enter in the "Additional environment details" area below)

Additional environment details

• browser version
• SPFx version
• Node.js version
• etc

Describe the bug / error

Summary

The RefinementToken returned by /_api/search/postquery in RefinementResults does not work as-is in RefinementFilters when the refinement value contains non-ASCII characters such as accented characters.

The request returns 0 results silently, with no error.

Root cause

The API appears to be asymmetric in how it encodes and decodes refinement tokens:

• Token emission: the value is encoded using the hex representation of its UTF-8 bytes (é → c3a9)
• Filter matching: the token appears to be decoded using Latin-1 / character-code semantics (é → e9)

As a result, the RefinementToken returned by the API cannot be reused as-is when it contains a multi-byte UTF-8 character.

This does not affect pure ASCII values because, for characters below 0x80, UTF-8 and Latin-1/character-code representations are identical.

This explains why values such as Security work while Sécurité fails.

The same behavior can be reproduced directly through the SharePoint Search REST API, without any client-side application logic.

Tested and ruled out: NFC/NFD normalization, lowercase, diacritic stripping,Culture: 1036 (fr-FR), double UTF-8 encoding — all returned 0. Only the charcode re-encoding (or the string(mode="phrase") FQL fallback) works.

Impact

Any refinement filter (RefinableStringXX) on a non-ASCII value breaks if the raw API token is passed back — so most FR/DE/ES/etc. taxonomies are affected.

Fix applied on our side

Decodes the API token (hex UTF-8 → text), then re-encodes it as charcode hex before putting it in RefinementFilters. Falls back to string(mode="phrase") for characters outside Latin-1 (Arabic, CJK...).

Steps to reproduce

1. Create or use a SharePoint Online Managed Metadata field mapped to a refinable managed property, for example RefinableStringXX.

2. Add a taxonomy value containing an accented character, for example:
   Sécurité

3. Execute a Search REST API request using /_api/search/postquery with:

   {
     "request": {
       "Querytext": "*",
       "QueryTemplate": "{searchTerms} PromotedState:2",
       "Refiners": "RefinableStringXX",
       "RowLimit": 10
     }
   }

4. In the RefinementResults, locate the refinement corresponding to Sécurité.

5. Copy the RefinementToken returned by SharePoint. (starting by ǂǂ....)

6. Use the returned RefinementToken directly in RefinementFilters.

7. The request completes successfully but returns 0 results

Same request, only the encoding of é inside the token changes:
...437962657273c3a96375726... (UTF-8) → 0 results
...437962657273e96375726... (charcode) → 10 results ✔ (matches the expected RefinementCount)

Expected behavior

The RefinementToken returned by RefinementResults should be directly reusable in RefinementFilters.

A token returned by: RefinementResults should produce the corresponding refinement results when passed unchanged to:

RefinementFilters

Non-ASCII characters should be handled consistently between token generation and token matching.

[patrikhellgren]

Here are a few other threads about the same issue:

microsoft-search/pnp-modern-search#4926
microsoft-search/pnp-modern-search#4901

I have an ongoing support case with Microsoft about this and also posted here, https://sharepoint.stackexchange.com/questions/317496/issue-with-the-sharepoint-search-api-and-refiners-containing-accented-characters, for increased visibilty.

[Ashlesha-MSFT]

Hello jSSSSssss (@Jsss1010)
Thank you for bringing this issue to our attention. We will look into it and get back to you shortly.

[mmm-str]

We are experiencing the same issue with Swedish characters (ö,å,ä).

[Ashlesha-MSFT]

jSSSSssss (@Jsss1010)
Thanks for the detailed repro steps and the additional context from the linked threads. We attempted to reproduce this in an internal test tenant:

• Created a Managed Metadata column mapped to a taxonomy term containing an accented character (Sécurité), added it to list items, and mapped the corresponding crawled property to a RefinableString managed property.
• Confirmed base content crawling works correctly for the list (items are indexed and searchable).
• However, the taxonomy-column-specific crawled property has not yet surfaced in our test tenant even after multiple forced reindexes and an extended wait, so we have not yet been able to complete the RefinementResults → RefinementFilters round-trip test end-to-end to directly observe the 0-result behavior described.
• Given that other reporters describe this as intermittent (working in some environments/timeframes but not others), this may depend on specific search infrastructure/tenant state rather than being reliably reproducible via a fresh minimal setup.

I will tag the engineering team internally for confirmation on the backend encoding/decoding behavior and will update this thread once we hear back.

[msznwska]

Ashlesha-MSFT
We are experiencing this in most tenants we have checked (Sweden), at least 5 production tenants and 2 dev. I've been searching for a working tenants to verify workaround and only found 1 production and 1 dev that work correctly. In affected tenants the issue is consistent, happens all the time and as far as we know for all users.

But there is a separate issue that seems to be affecting investigation - crawled properties for newly created columns stopped appearing at all in some tenants. In some, previously existing properties disappeared as well. We have active support requests with Microsoft Support for both the refinement issue and the crawled properties.

jSSSSssss (@Jsss1010)
Thank you so much for sharing your findings!