[Storage] Add structured message CRC64 content validation for azfile by tanyasethi-msft · Pull Request #27261 · Azure/azure-sdk-for-go

tanyasethi-msft · 2026-07-27T19:05:55Z

Summary

Adds structured message (XSM/1.0) reader/writer infrastructure to azfile and azdatalake internal packages
Integrates CRC64 content validation into azfile upload (UploadRange) and download (DownloadStream) paths
Adds TransferValidationTypeComputeStructuredMessageCRC64 option for azfile clients
For uploads: wraps request body with SMEncoder, sets x-ms-structured-body header
For downloads: detects structured body response header and wraps body with SMDecoder for transparent CRC64 validation

Test plan

Unit tests for structured message encode/decode
Integration tests for upload/download with SM CRC64 (in follow-up PR)

azure-pipelines · 2026-07-27T19:06:26Z

Azure Pipelines: Successfully started running 1 pipeline(s). 5 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

Pull request overview

Adds XSM/1.0 CRC64 validation infrastructure and integrates it into Azure Files upload/download operations.

Changes:

Adds streaming structured-message encoding, decoding, and tests.
Exposes CRC64 validation options across azfile APIs.
Integrates structured bodies with uploads, downloads, and retries.

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
`sdk/storage/azfile/internal/shared/structured_message.go`	Implements XSM encoding and decoding.
`sdk/storage/azfile/internal/shared/structured_message_test.go`	Tests azfile XSM infrastructure.
`sdk/storage/azfile/internal/generated/models.go`	Adds structured-body option setters.
`sdk/storage/azfile/internal/exported/transfer_validation_option.go`	Implements validation options.
`sdk/storage/azfile/fileerror/error_codes.go`	Adds unsupported-checksum error.
`sdk/storage/azfile/file/responses.go`	Propagates validation across retries.
`sdk/storage/azfile/file/models.go`	Adds validation configuration fields.
`sdk/storage/azfile/file/constants.go`	Exposes CRC64 validation APIs.
`sdk/storage/azfile/file/client.go`	Integrates encoding and decoding.
`sdk/storage/azdatalake/internal/shared/structured_message.go`	Adds XSM infrastructure for azdatalake.
`sdk/storage/azdatalake/internal/shared/structured_message_test.go`	Tests azdatalake XSM infrastructure.

tanyasethi-msft · 2026-07-27T19:45:59Z

+	totalDataLen := len(data)
+	numSegments := totalDataLen / segmentSize
+	if totalDataLen%segmentSize != 0 {
+		numSegments++
+	}
+	if numSegments == 0 {
+		numSegments = 1
+	}


Added overflow protection to SMEncode matching what NewSMEncoder already does — auto-increases segment size when numSegments exceeds uint16 max.

Copilot

Pull request overview

Copilot reviewed 10 out of 11 changed files in this pull request and generated 8 comments.

Comments suppressed due to low confidence (2)

sdk/storage/azfile/internal/shared/structured_message.go:676

msgLen and bytesRead are recorded but never compared, so the streaming decoder accepts a header whose declared message length does not match the framing it consumed. This omits a structural integrity check already performed by SMDecode; validate the count before every transition to decStateDone.

	d.msgLen = binary.LittleEndian.Uint64(buf[1:9])

sdk/storage/azdatalake/internal/shared/structured_message.go:676

msgLen and bytesRead are recorded but never compared, so the streaming decoder accepts a header whose declared message length does not match the framing it consumed. Validate the count before every transition to decStateDone.

	d.msgLen = binary.LittleEndian.Uint64(buf[1:9])

tanyasethi-msft · 2026-07-27T19:46:39Z

+			body, err = options.TransactionalValidation.Apply(body, uploadRangeOptions)
+			if err != nil {
+				return UploadRangeResponse{}, err
+			}
+			contentLength, err = shared.ValidateSeekableStreamAt0AndGetCount(body)
+			if err != nil {
+				return UploadRangeResponse{}, err
+			}


Fair point about the azfile upload path. The body assignment in UploadRange already uses the return from Apply, so the encoded body does get sent. But will double-check the Optionalbody vs body flow to make sure Content-Length is consistent.

tanyasethi-msft · 2026-07-27T19:46:41Z

+	if resp.StructuredBodyType != nil && *resp.StructuredBodyType != "" {
+		resp.Body = shared.NewSMDecoder(resp.Body)
+	}


Noted — this is a valid concern for chunked downloads. Will investigate the RetryReader EOF behavior with the SM decoder trailer.

tanyasethi-msft · 2026-07-27T19:46:42Z

+func (c TransferValidationTypeCRC64) Apply(rsc io.ReadSeekCloser, cfg generated.TransactionalContentSetter) (io.ReadSeekCloser, error) {
+	buf := make([]byte, 8)
+	binary.LittleEndian.PutUint64(buf, uint64(c))
+	cfg.SetCRC64(buf)
+	return rsc, nil


The CRC64 types for azfile are exported for consistency with azblob's public API surface. Azure Files does support transactional CRC via structured message (that's what this PR adds). The SetCRC64 no-op is for the header-based transactional CRC which Files doesn't support — we could add a guard there, but structured message CRC64 is the primary path.

tanyasethi-msft · 2026-07-27T19:46:44Z

+	d.msgLen = binary.LittleEndian.Uint64(buf[1:9])
+	d.flags = binary.LittleEndian.Uint16(buf[9:11])
+	d.numSegments = binary.LittleEndian.Uint16(buf[11:13])
+	d.hasCRC = d.flags&SMFlagCRC64 != 0


We specifically want to accept the flags as-is from the service response. The server negotiation via x-ms-structured-body header already guarantees CRC64 will be present. Rejecting on missing flag would break forward compatibility if the service ever sends messages with additional properties.

tanyasethi-msft · 2026-07-27T19:46:46Z

+	for i := uint16(1); i <= numSegments; i++ {
+		segment, err := decodeSMSegment(smData, offset, i, hasCRC)


Good catch on the uint16 overflow in the loop counter. Will use a wider counter.

tanyasethi-msft · 2026-07-27T19:46:48Z

+	d.msgLen = binary.LittleEndian.Uint64(buf[1:9])
+	d.flags = binary.LittleEndian.Uint16(buf[9:11])
+	d.numSegments = binary.LittleEndian.Uint16(buf[11:13])
+	d.hasCRC = d.flags&SMFlagCRC64 != 0


Same rationale as above — we accept the flags from the server response as-is.

tanyasethi-msft · 2026-07-27T19:46:52Z

+	if o.TransactionalValidation != nil {
+		if h := exported.GetStructuredBodyType(o.TransactionalValidation); h != "" {
+			opts.StructuredBodyType = &h
+		}
+	}


This follows the same pattern as azblob. The download options accept TransferValidationType for structured message CRC64 specifically. Other validation types are not applicable for downloads and are silently ignored, matching the blob behavior. We could add explicit rejection for non-SM types if preferred.

jhendrixMSFT · 2026-07-27T19:48:56Z

I believe these two azdatalake files were added in error? I see them on the other PR.

You're right — moved the azdatalake SM files to PR #27263 where they belong. This PR is now azfile-only.

Copilot

Pull request overview

Copilot reviewed 10 out of 11 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (10)

sdk/storage/azfile/file/client.go:274

Apply returns the SM-encoded reader, but the generated request reads uploadRangeOptions.Optionalbody, which still points to the original unencoded body set by format. Structured uploads therefore send raw bytes while advertising the encoded content length and structured-body headers. Update the option to use the wrapped reader before sending the request.

			body, err = options.TransactionalValidation.Apply(body, uploadRangeOptions)
			if err != nil {
				return UploadRangeResponse{}, err
			}
			contentLength, err = shared.ValidateSeekableStreamAt0AndGetCount(body)
			if err != nil {
				return UploadRangeResponse{}, err
			}

sdk/storage/azfile/file/models.go:600

Non-structured validation values are silently ignored here. For example, passing TransferValidationTypeMD5 or TransferValidationTypeCRC64 to the new download option sends no validation header and returns an unvalidated body even though the caller requested validation. Reject unsupported types (or use a download-specific option type) instead of treating them as no validation.

	if o.TransactionalValidation != nil {
		if h := exported.GetStructuredBodyType(o.TransactionalValidation); h != "" {
			opts.StructuredBodyType = &h
		}
	}

sdk/storage/azfile/internal/shared/structured_message.go:212

When numSegments is 65535, this uint16 loop wraps from 65535 to 0 and executes again, so SMDecode rejects a valid message produced at the supported segment-count limit. Iterate with a wider integer and convert only the expected segment number.

	for i := uint16(1); i <= numSegments; i++ {
		segment, err := decodeSMSegment(smData, offset, i, hasCRC)

sdk/storage/azdatalake/internal/shared/structured_message.go:212

When numSegments is 65535, this uint16 loop wraps from 65535 to 0 and executes again, so SMDecode rejects a valid message produced at the supported segment-count limit. Iterate with a wider integer and convert only the expected segment number.

	for i := uint16(1); i <= numSegments; i++ {
		segment, err := decodeSMSegment(smData, offset, i, hasCRC)

sdk/storage/azfile/internal/shared/structured_message.go:754

A zero message length disables validation because of the d.msgLen > 0 guard. Changing only the length field to zero leaves all CRCs valid, and the decoder then accepts the corrupted header. Compare the consumed length unconditionally.

	if d.msgLen > 0 && d.bytesRead != int64(d.msgLen) {
		return fmt.Errorf("structured message length mismatch: header says %d, consumed %d bytes", d.msgLen, d.bytesRead)
	}

sdk/storage/azdatalake/internal/shared/structured_message.go:754

A zero message length disables validation because of the d.msgLen > 0 guard. Changing only the length field to zero leaves all CRCs valid, and the decoder then accepts the corrupted header. Compare the consumed length unconditionally.

	if d.msgLen > 0 && d.bytesRead != int64(d.msgLen) {
		return fmt.Errorf("structured message length mismatch: header says %d, consumed %d bytes", d.msgLen, d.bytesRead)
	}

sdk/storage/azfile/internal/shared/structured_message.go:690

If the CRC64 flag is missing or corrupted, the streaming decoder switches to a no-checksum path and returns the payload successfully, even though this decoder is installed to provide CRC64 validation. Require SMFlagCRC64 before reading segments so a flipped flag cannot disable integrity checking.

	d.msgLen = binary.LittleEndian.Uint64(buf[1:9])
	d.flags = binary.LittleEndian.Uint16(buf[9:11])
	d.numSegments = binary.LittleEndian.Uint16(buf[11:13])
	d.hasCRC = d.flags&SMFlagCRC64 != 0

	if d.hasCRC {
		d.msgCRC = crc64.New(CRC64Table)

sdk/storage/azdatalake/internal/shared/structured_message.go:690

If the CRC64 flag is missing or corrupted, the streaming decoder switches to a no-checksum path and returns the payload successfully, even though this decoder is installed to provide CRC64 validation. Require SMFlagCRC64 before reading segments so a flipped flag cannot disable integrity checking.

	d.msgLen = binary.LittleEndian.Uint64(buf[1:9])
	d.flags = binary.LittleEndian.Uint16(buf[9:11])
	d.numSegments = binary.LittleEndian.Uint16(buf[11:13])
	d.hasCRC = d.flags&SMFlagCRC64 != 0

	if d.hasCRC {
		d.msgCRC = crc64.New(CRC64Table)

sdk/storage/azfile/file/client.go:267

The new upload integration is not exercised by the structured-message codec tests, so they cannot detect whether UploadRange actually sends the wrapped body and the required headers/content lengths. Add a client-level test that captures the outgoing request for direct and chunked structured uploads.

	if options != nil && options.TransactionalValidation != nil {
		if _, ok := options.TransactionalValidation.(TransferValidationTypeMD5); !ok {
			body, err = options.TransactionalValidation.Apply(body, uploadRangeOptions)

sdk/storage/azfile/file/client.go:623

A bounded RetryReader stops as soon as its decoded range count reaches zero, before calling the wrapped decoder again. If the final raw bytes exactly fill the caller's buffer (common for 4 MiB chunks copied with a 32 KiB buffer), SMDecoder is left in decStateSegFooter, so neither the final segment CRC nor message CRC is ever read or validated. Ensure structured decoding reaches its trailer before the retry reader reports EOF.

	if resp.StructuredBodyType != nil && *resp.StructuredBodyType != "" {
		resp.Body = shared.NewSMDecoder(resp.Body)
	}

tanyasethi-msft · 2026-07-30T10:29:30Z

+	flags := binary.LittleEndian.Uint16(smData[9:11])
+	numSegments := binary.LittleEndian.Uint16(smData[11:13])
+	hasCRC := flags&SMFlagCRC64 != 0


The SMDecode function is a general-purpose structured message parser. CRC64 validation is enforced at the transport layer — we only install the decoder when the response header says properties=crc64. A corrupted flags field would also cause the msgLen check to fail since the byte counts wouldn't add up without footer/trailer bytes. Hardcoding a CRC64 requirement here would break forward compatibility if the format gains new property types.

Same reasoning as above — the azdatalake decoder is a general-purpose SM parser. CRC64 enforcement happens at the transport level via header negotiation.

jhendrixMSFT · 2026-07-27T20:35:25Z

+				}
+			}
+			if err != nil && d.segRemain > 0 {
+				d.setError(fmt.Errorf("segment %d: %w", d.segIndex, err))


fmt.Errorf("segment %d: %s", d.segIndex, err.Error())

We actually changed this from %s to %w intentionally — using %w preserves the original error type through errors.Is/errors.As, which is important here because RetryReader checks for net.Error to decide if a failure is retryable. With %s or .Error(), the concrete type gets erased and transient network errors during segment reads would stop being retried. That was actually one of the Copilot findings we fixed in this round.

Copilot

Pull request overview

Copilot reviewed 10 out of 11 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (5)

sdk/storage/azfile/file/client.go:267

The wrapped encoder is assigned only to the local body; uploadRangeOptions.Optionalbody still points to the original stream created by format. The generated request reads Optionalbody (zz_file_client.go:2611-2614), so this sends unframed data while advertising structured-message headers and the encoded content length. Update Optionalbody after applying the validation wrapper.

			body, err = options.TransactionalValidation.Apply(body, uploadRangeOptions)

sdk/storage/azfile/internal/shared/structured_message.go:633

Wrapping with %w still makes transient read failures non-retryable here. RetryReader uses a direct err.(net.Error) assertion and exact err == io.ErrUnexpectedEOF comparison (retry_reader.go:137-139), neither of which traverses this wrapper; a premature io.EOF is also wrapped instead of converted to io.ErrUnexpectedEOF. Preserve retry classification by updating those checks to errors.As/errors.Is and mapping premature EOF appropriately, or return the classified errors directly.

			if err != nil && d.segRemain > 0 {
				d.setError(fmt.Errorf("segment %d: %w", d.segIndex, err))
				return totalOut, d.err

sdk/storage/azdatalake/internal/shared/structured_message.go:212

This copy still uses a uint16 loop counter. For a valid message with 65,535 segments, i++ wraps to zero after the last segment and the decoder attempts to parse another segment, so the maximum representable message cannot be decoded. Match the azfile copy by iterating with a wider type.

	for i := uint16(1); i <= numSegments; i++ {
		segment, err := decodeSMSegment(smData, offset, i, hasCRC)

sdk/storage/azfile/internal/shared/structured_message.go:752

The msgLen > 0 guard lets a corrupted zero message-length header bypass validation entirely, even after the decoder consumes a complete nonempty frame. SMDecode rejects the same input, so the streaming decoder should compare every declared length, including zero.

	if d.msgLen > 0 && d.bytesRead != int64(d.msgLen) {

sdk/storage/azdatalake/internal/shared/structured_message.go:752

The msgLen > 0 guard lets a corrupted zero message-length header bypass validation entirely, even after the decoder consumes a complete nonempty frame. SMDecode rejects the same input, so the streaming decoder should compare every declared length, including zero.

	if d.msgLen > 0 && d.bytesRead != int64(d.msgLen) {

… length validation

…azfile

Copilot

Pull request overview

Copilot reviewed 8 out of 9 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

sdk/storage/azfile/internal/shared/structured_message.go:633

%w still produces a wrapper that does not satisfy RetryReader's direct err.(net.Error) check (file/retry_reader.go:137), so transient read failures remain non-retryable despite the attempted fix. Return the underlying error unchanged here, or update RetryReader to use errors.As.

			if err != nil && d.segRemain > 0 {
				d.setError(fmt.Errorf("segment %d: %w", d.segIndex, err))
				return totalOut, d.err

sdk/storage/azfile/internal/shared/structured_message.go:754

The msgLen > 0 guard lets a message whose required length field is zero pass validation as long as its CRCs are valid. SMDecode rejects the same payload because the declared length never matches the bytes consumed; the streaming decoder should compare unconditionally too.

	if d.msgLen > 0 && d.bytesRead != int64(d.msgLen) {
		return fmt.Errorf("structured message length mismatch: header says %d, consumed %d bytes", d.msgLen, d.bytesRead)
	}

sdk/storage/azfile/internal/shared/structured_message.go:243

After validating the trailer, the decoder never advances offset or verifies that the parsed framing consumed the declared message. Appending bytes to a valid payload and updating msgLen therefore makes SMDecode accept trailing garbage. Require the final parsed offset to equal the input length.

		expectedMsgCRC := binary.LittleEndian.Uint64(smData[offset : offset+8])
		actualMsgCRC := msgHasher.Sum64()
		if expectedMsgCRC != actualMsgCRC {
			return SMDecodeResult{}, fmt.Errorf("message trailer CRC64 mismatch (expected 0x%016x, got 0x%016x)", expectedMsgCRC, actualMsgCRC)
		}
	}

Copilot AI review requested due to automatic review settings July 27, 2026 19:05

tanyasethi-msft requested review from gapra-msft, jhendrixMSFT and souravgupta-msft as code owners July 27, 2026 19:05

github-actions Bot added the Storage Storage Service (Queues, Blobs, Files) label Jul 27, 2026

Copilot started reviewing on behalf of tanyasethi-msft July 27, 2026 19:06 View session

Copilot AI reviewed Jul 27, 2026

View reviewed changes

This was referenced Jul 27, 2026

[Storage] Add structured message CRC64 content validation for azdatalake #27263

Open

[Storage] Add content validation integration tests for azfile and azdatalake #27264

Open

Copilot AI review requested due to automatic review settings July 27, 2026 19:30

Copilot started reviewing on behalf of tanyasethi-msft July 27, 2026 19:31 View session

Copilot AI reviewed Jul 27, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 27, 2026 19:43

Copilot started reviewing on behalf of tanyasethi-msft July 27, 2026 19:44 View session

jhendrixMSFT reviewed Jul 27, 2026

View reviewed changes

Copilot AI reviewed Jul 27, 2026

View reviewed changes

jhendrixMSFT reviewed Jul 27, 2026

View reviewed changes

Comment thread sdk/storage/azfile/internal/generated/models.go

jhendrixMSFT reviewed Jul 27, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 28, 2026 06:01

Copilot started reviewing on behalf of tanyasethi-msft July 28, 2026 06:02 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

tanyasethi-msft added 7 commits July 28, 2026 13:29

[Storage] Add structured message infrastructure to azfile

c9a61e3

[Storage] Add structured message upload validation to azfile

612c949

[Storage] Add structured message download validation to azfile

20b4779

Fix copilot suggestions

cfeeae2

[Storage] Fix SMEncode segment overflow, SMDecoder error wrapping and…

74d7531

… length validation

[Storage] Fix uint16 loop counter overflow in SMDecode

da33f80

[Storage] Remove unsupported CRC64 transactional header exports from …

8b1fc33

…azfile

Copilot AI review requested due to automatic review settings July 28, 2026 08:01

tanyasethi-msft force-pushed the storage/content-validation-files-adls branch from c186bd0 to 8b1fc33 Compare July 28, 2026 08:01

Copilot started reviewing on behalf of tanyasethi-msft July 28, 2026 08:02 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

		for i := uint16(1); i <= numSegments; i++ {
		segment, err := decodeSMSegment(smData, offset, i, hasCRC)

Uh oh!

Conversation

tanyasethi-msft commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

azure-pipelines Bot commented Jul 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tanyasethi-msft commented Jul 27, 2026 •

edited

Loading