.NET 11 fails to decode certain base64 inputs

[JustArchi]

Description

I've observed a serious regression in one of my unit tests after updating the project to target .NET 11. I've isolated minimal reproducible one-liner in the meantime to aid debugging.

Reproduction Steps

dotnet new console, then in Program.cs:

Console.WriteLine($"OK: {Convert.FromBase64String("KDHC3rsY8+CmiswnXJcE5e5dRfd=").Length}");

Expected behavior

Should work fine with OK: 20 output, which is the case when running this against <TargetFramework>net10.0</TargetFramework>

Actual behavior

Unhandled exception. System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
   at System.Buffers.Text.Base64.DecodeFromChars(ReadOnlySpan`1 source)
   at Program.<Main>$(String[] args) in Program.cs:line 1

Happens with <TargetFramework>net11.0</TargetFramework>

As a side note, not every valid base64 input results in this regression, but I've found at least one that does, which you can hardcode for unit test later on if deemed wanted.

Regression?

Yes, worked fine in .NET 10

Known Workarounds

None for now.

Configuration

.NET SDK:
 Version:           11.0.100-rc.1.26425.128
 Commit:            3551975be0
 Workload version:  11.0.100-rc.1.26458.5
 MSBuild version:   18.11.0-1.26425.128+3551975be

Runtime Environment:
 OS Name:     debian
 OS Version:  
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/11.0.100-rc.1.26425.128

.NET workloads installed:
Brak zainstalowanych obciążeń do wyświetlenia.
Skonfigurowano używanie workload sets podczas instalowania nowych manifestów.

Host:
  Version:      11.0.0-rc.1.26425.128
  Architecture: x64
  Commit:       3551975be0

.NET SDKs installed:
  11.0.100-rc.1.26425.128 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 10.0.12 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 11.0.0-rc.1.26425.128 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 10.0.12 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 11.0.0-rc.1.26425.128 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  DOTNET_CLI_TELEMETRY_OPTOUT              [true]
  DOTNET_NOLOGO                            [true]

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

Other information

I don't see any user error, neither a mention in the changelog regarding that - especially considering this is a valid base64 input that worked before.

I'd recommend to take a look into it before full release, as it seems rather serious to me and is a blocker right now for several of my projects.

[EgorBo]

Seems like either intentional or unintentional fallout from #121044

[EgorBo]

Perhaps @stephentoub @MihaZupan remember the reason behind forcing both encoders and decoders to reject the non-zero unused bits? While the spec doesn't require the rejection for the decoder.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

[vcsjones]

Seems like either intentional or unintentional fallout from #121044

Based on the pull request, the change is intentional (though I won't speak to if it was a good idea or not).

Console.WriteLine($"OK: {Convert.FromBase64String("KDHC3rsY8+CmiswnXJcE5e5dRfd=").Length}");

The last quartet in this base64 is not canonically encoded. It should be KDHC3rsY8+CmiswnXJcE5e5dRfc=. In .NET 10, both of these things decode to the same thing, but the first line uses non-canonical encoding.

Console.WriteLine($"OK: {Convert.ToHexString(Convert.FromBase64String("KDHC3rsY8+CmiswnXJcE5e5dRfd="))}");
Console.WriteLine($"OK: {Convert.ToHexString(Convert.FromBase64String("KDHC3rsY8+CmiswnXJcE5e5dRfc="))}");

@JustArchi How was this base64 value created? It's not correctly encoded, but a more lax decoder permits it. .NET 11 was changed to be more strict about incorrect encodings.

[JustArchi]

@vcsjones I don't know specific details in regards to what encoder was used and how that value was created itself, only the context that it was not "artificially" created to cause a regression by me, but rather is a legit shared secret key that Steam platform initialized one 2FA authenticator with, i.e. used by actual Steam mobile app to generate TOTP tokens for login attempts (custom solution, not a typical google authenticator).

I believe more people will be affected by this in the final release, as I stumbled upon it rather casually, and correct me if I'm wrong, but right now there's no way to "normalize" such input in a way .NET base64 decoder accepts it, i.e. it requires from me rolling out totally custom base64 parser as a workaround - which is a horrible idea.

I'd recommend evaluating whether this change is truly intentional considering context given above, and if yes, consider offering alternative such as FromBase64StringRelaxed function or something similar that would not make it a hard blocker for people like me to upgrade.

The best approach seems to be accepting relaxed input from decoder and generating deterministic output in encoder (different from input if non-canonical input was used, otherwise the same as input).

[vcsjones]

cc @jeffhandley

[MihaZupan]

It was indeed an intentional change (see also #105264, #105262).

As for why we chose to look at this, one argument was that it could cause confusion in setups where different connected systems (e.g. frontend & backend) get confused between each other if one looks at decoded bytes while the other compares raw base64 strings.
E.g. the frontend validated the decoded bytes, then saved the input base64 string. The backend at some other point in time computes some bytes (e.g. checksum) and encodes it into base64. When it checks it against the saved value, there's a mismatch.
More context (MSFT internal): https://github.com/dotnet/designs-microsoft/pull/77#issuecomment-2234323022

The claim at the time the change was made was:

This doesn't seem to be a breaking change, my quick research did not find any encoder, that produces output that doesn't set unused bits to 0. Though it could break tests that randomly generates Base64 encoded text.

If this is causing issues in practice, we may want to reconsider, or offer a way to opt back in.

[JustArchi]

It definitely causes a regression in my project (ASF) and its legit functionality (of generating TOTP codes for Steam platform for certain shared secrets). One could argue it's just a crap implementation on their end, but I have no control over that one, and as long as it meets the RFC, especially considering it worked before - I don't believe breaking otherwise valid setups and paths is warranted for avoiding potential confusion in unrelated systems.

I'm not hard advocate for reverting it entirely though, only for making it possible one way or another to accept such input and work with it without a need of implementing custom parsers for whole .NET 11 lifetime.

[MihaZupan]

FWIW it may not be that common in practice - Base64/Base64Url types were already updated to be stricter here in .NET 9 - #105262.
The .NET 11 change was to do the same for remaining base64-related APIs on Convert.

and as long as it meets the RFC

It doesn't.

https://datatracker.ietf.org/doc/html/rfc4648#section-3.5

These pad bits MUST be set to zero by conforming encoders

[JustArchi]

Alright, then it's a question whether you guys would consider relaxing this requirement, or providing some alternative way to deal with this, since I don't see any other option right now other than writing custom base64 decoder (which wasn't needed before).

I'm not really happy that I need to work with services not following RFC, and probably neither do you, but nonetheless I'd like to request an alternative way for what was previously a working functionality (if total revert is deemed unwanted).

[peters]

Another real-world hit, plus a documentation gap.

We pin SFTP host keys as SHA256:<base64> fingerprints and normalise whatever an operator pastes (with or without padding) through Convert.FromBase64String. A 43-character SHA-256 fingerprint whose last character carries non-zero pad bits decodes on .NET 10 and throws on .NET 11 RC1, so a pin that resolves today would start failing closed after the upgrade. Measured on the same machine (10.0.12 vs 11.0.0-rc.1.26425.128):

Input	.NET 10.0.12	.NET 11 RC1
Convert.FromBase64String("QR==")	[0x41]	FormatException
Convert.TryFromBase64String("QR==", …)	true, 1 byte	false
Convert.FromBase64String("<43 chars, last char non-canonical>=")	32 bytes	FormatException
Base64.IsValid("QR==") / Base64.DecodeFromUtf8	false / InvalidData	unchanged

Two observations for whoever decides on #121044:

1. Whatever the outcome, this is currently undocumented: it is not in the .NET 11 breaking-change list (docs/core/compatibility/11.md in dotnet/docs) and Convert.FromBase64XYZ decoder should reject input when unused bits are not set to 0 #121044 carries no breaking-change label. The Base64/Base64Url tightening in .NET 9 had a doc; the Convert one needs the same so people find it before RC2/GA rather than through failing tests.
2. There is no lenient overload left on Convert, so the only way to keep accepting such input is to canonicalise the last character yourself before decoding (mask the unused low bits of its sextet). That is what we ended up doing; a Base64FormattingOptions-style opt-in would have been preferable to a hand-rolled fix-up.