Repository navigation
fix: Blob/Data setters reject raw binary disguised as base64 - #1957
Open
PrathmeshPawar2110 wants to merge 1 commit into
Open
PrathmeshPawar2110 wants to merge 1 commit into
PrathmeshPawar2110 wants to merge 1 commit into
Conversation
ImageContentBlock.Data, AudioContentBlock.Data, and BlobResourceContents.Blob
are ReadOnlyMemory<byte> properties documented to hold the base64 text
representation of binary data, as UTF-8 bytes — not the raw, unencoded
bytes. Assigning raw bytes directly (instead of going through the
FromBytes factories) is an easy, understandable mistake: nothing fails
at the point of assignment, and the polymorphic JSON writer serializes
those bytes as if they already were base64 text. Most raw binary is not
valid UTF-8, so the writer corrupts it in the process (replacement
characters, escaped control characters), and the corruption only
surfaces later, confusingly, as a client-side FormatException when the
receiving end tries to base64-decode the mangled text.
This isn't a serializer regression — serializing a BlobResourceContents
or ImageContentBlock built correctly (via FromBytes, or with Blob/Data
set to real base64 UTF-8 bytes) already round-trips correctly. The
problem is that misuse of the byte-typed property fails late and
unhelpfully instead of failing immediately at the point of misuse.
Add EncodingUtilities.ValidateBase64Utf8 and call it from the Blob/Data
setters, so assigning non-base64 bytes now throws an ArgumentException
immediately, naming the FromBytes factory to use instead. To avoid
penalizing the common case, the JSON converters' existing fast path
(reusing an unescaped wire value's UTF-8 bytes directly, which are
already known-valid base64 text as read from the document) now goes
through a new internal FromTrustedBase64Utf8 factory that bypasses this
validation, so normal deserialization pays no extra cost.
A few existing tests used a non-base64 placeholder string ("base64data",
or raw byte literals) as filler Data/Blob content; those are updated to
use real base64 text, since that was never the behavior under test.
Fixes modelcontextprotocol#1835
Co-Authored-By: Claude Sonnet 5 <[email protected]>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
This is not a serializer regression. @luisangelrod's investigation on this issue found that
BlobResourceContents.Blob,ImageContentBlock.Data, andAudioContentBlock.Dataare documented to hold the base64 text representation of the data, as UTF-8 bytes — not raw, unencoded binary. The correct construction path for raw bytes is theFromBytesfactory on each type (BlobResourceContents.FromBytes(...),ImageContentBlock.FromBytes(...),AudioContentBlock.FromBytes(...)), which base64-encodes correctly. Serializing an instance built viaFromBytes, or with the property set to real base64 UTF-8 bytes, already round-trips correctly throughMcpJsonUtilities.DefaultOptions.The real problem: assigning raw bytes directly to
Blob/Data(the mistake in the issue's repro) is silent and easy. Nothing fails at the point of assignment. The polymorphic JSON writer serializes those bytes as if they already were base64 text (Utf8JsonWriter.WriteString(name, ReadOnlySpan<byte>)), and since most raw binary isn't valid UTF-8, it gets mangled (escaped control characters, replacement characters) rather than rejected outright. The corruption then only surfaces later, confusingly, as aFormatExceptionon the deserializing end.Fix
Add
EncodingUtilities.ValidateBase64Utf8and call it from theBlob/Datasetters on all three types, so assigning non-base64 bytes now throws anArgumentExceptionimmediately, naming theFromBytesfactory to use instead:To avoid penalizing the common (correct) case, the JSON converters' existing fast path — reusing an unescaped wire value's UTF-8 bytes directly, which are already known-valid base64 text as read from the document — now goes through a new
internalFromTrustedBase64Utf8factory on each type that bypasses this validation, so normal deserialization pays no extra cost.A few existing tests used a non-base64 placeholder string (
"base64data", or a rawbyte[] { 1, 2, 3 }literal) as fillerData/Blobcontent; those are updated to use real base64 text, since base64 fidelity was never what those tests were checking.Fixes #1835
Testing
New tests cover: raw-bytes rejection on all three setters (including a 900,000-byte payload matching the issue's repro size), rejection of well-formed-alphabet-but-wrong-length input, and that the existing
FromBytes/valid-base64-setter/JSON-round-trip paths are unaffected.I also ran the full suite (
dotnet test tests/ModelContextProtocol.Tests/, net10.0): 3313 passed. The non-zero failure count there reproduces in this environment on unrelated branches too — this sandbox's checkout path contains a space (C:\Open Source\csharp-sdk), which breaksStdioClientTransportTests.EscapesCliArgumentsCorrectly'scmd.exeargument parsing regardless of what's being tested, plus some test-host launch flakiness unrelated to this change.Co-Authored-By: Claude Sonnet 5 [email protected]