[python] Keep BlobViewStruct bytes when blob-as-descriptor is true. - #10057
Merged
JingsongLi merged 1 commit intoSep 22, 2026
Merged
Conversation
Resolve get_blob() through the originating table FileIO, and serialize descriptors only on Arrow output. Cache blob-as-descriptor and view output indices so non-view row paths skip per-row option lookup.
Contributor
|
Requirement fit: SUPPORTED. A BlobView must retain its source-table FileIO when |
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
When
blob-as-descriptor=true, keepBlobViewStructbytes through Stage 1 soget_blob()still resolves through the originating table FileIO. Serialize descriptors only at the Arrow/pandas output boundary.Previously Stage 1 replaced view bytes with a plain descriptor. That dropped the upstream-table association, so
get_blob()used the target FileIO to open the URI. Under REST catalog that is the wrong token (403).BlobViewLookup, but does not rewrite view columns whenblob-as-descriptor=true.TableReadconverts view fields to descriptor bytes into_arrow/to_pandas/to_arrow_batch_reader(serial and parallel).to_iterator()is unchanged.blob-as-descriptorand_blob_view_output_indices(positions inread_type, not the widened_scan_read_type) so non-view row paths skip per-row option lookup.Follow-up to #9608. Native read (
read.native.enabled) is unchanged and still defaults to false; view tables should stay on the Python reader until native grows a matching hook.Tests
test_blob_view_as_descriptor_get_blob_uses_upstream_file_io: guarded FileIO;blob-as-descriptor=trueget_blob().to_data()must not read source.blobwith the target FileIO.test_blob_view_as_descriptor_projection_with_predicate_extra_field: projection['picture', 'id']plus a predicate on unprojectedgrp;_blob_view_output_indices == (0,), Arrow has only those two columns, andpictureis descriptor bytes.test_blob_view_fields_resolve_upstream_blob:to_arrowstill returnsBlobDescriptorbytes whenblob-as-descriptor=true.test_blob_view_default_read_uses_upstream_file_io: default (blob-as-descriptor=false) still materializes payload with the upstream FileIO.test_table_read_serializes_only_configured_blob_view_fields: only configured view columns are rewritten; empty indices /blob-as-descriptor=falseskip serialize.test_offset_row_get_blob_view_keeps_per_table_uri_reader:get_blob()keeps the originating table UriReader.test_offset_row_get_blob_v1_resolved_blob_view_field/test_offset_row_get_blob_resolves_null_blob_view/test_offset_row_get_blob_view_struct_without_view_field_indices: view-structget_blob()for v1, null, and missing view-field indices.test_to_iterator_adapters_refresh_blob_view_lookup_after_first_read: mergeto_iteratoradapters copy lookup after the first convert read.