Skip to content

[python] Keep BlobViewStruct bytes when blob-as-descriptor is true. - #10057

Merged
JingsongLi merged 1 commit into
apache:masterfrom
Stephen0421:pypaimon-blob-pr1d-blob-view
Sep 22, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
Stephen0421:pypaimon-blob-pr1d-blob-view

Conversation

@Stephen0421

Copy link
Copy Markdown
Contributor

Purpose

When blob-as-descriptor=true, keep BlobViewStruct bytes through Stage 1 so get_blob() still resolves through the originating table FileIO. Serialize descriptors only at the Arrow/pandas output boundary.

Previously Stage 1 replaced view bytes with a plain descriptor. That dropped the upstream-table association, so get_blob() used the target FileIO to open the URI. Under REST catalog that is the wrong token (403).

  • Convert reader still prescans and builds BlobViewLookup, but does not rewrite view columns when blob-as-descriptor=true.
  • TableRead converts view fields to descriptor bytes in to_arrow / to_pandas / to_arrow_batch_reader (serial and parallel). to_iterator() is unchanged.
  • Cache blob-as-descriptor and _blob_view_output_indices (positions in read_type, not the widened _scan_read_type) so non-view row paths skip per-row option lookup.

Follow-up to #9608. Native read (read.native.enabled) is unchanged and still defaults to false; view tables should stay on the Python reader until native grows a matching hook.

Tests

  • test_blob_view_as_descriptor_get_blob_uses_upstream_file_io: guarded FileIO; blob-as-descriptor=true get_blob().to_data() must not read source .blob with the target FileIO.
  • test_blob_view_as_descriptor_projection_with_predicate_extra_field: projection ['picture', 'id'] plus a predicate on unprojected grp; _blob_view_output_indices == (0,), Arrow has only those two columns, and picture is descriptor bytes.
  • test_blob_view_fields_resolve_upstream_blob: to_arrow still returns BlobDescriptor bytes when blob-as-descriptor=true.
  • test_blob_view_default_read_uses_upstream_file_io: default (blob-as-descriptor=false) still materializes payload with the upstream FileIO.
  • test_table_read_serializes_only_configured_blob_view_fields: only configured view columns are rewritten; empty indices / blob-as-descriptor=false skip serialize.
  • test_offset_row_get_blob_view_keeps_per_table_uri_reader: get_blob() keeps the originating table UriReader.
  • test_offset_row_get_blob_v1_resolved_blob_view_field / test_offset_row_get_blob_resolves_null_blob_view / test_offset_row_get_blob_view_struct_without_view_field_indices: view-struct get_blob() for v1, null, and missing view-field indices.
  • test_to_iterator_adapters_refresh_blob_view_lookup_after_first_read: merge to_iterator adapters copy lookup after the first convert read.

Resolve get_blob() through the originating table FileIO, and serialize
descriptors only on Arrow output. Cache blob-as-descriptor and view
output indices so non-view row paths skip per-row option lookup.
@JingsongLi

Copy link
Copy Markdown
Contributor

Requirement fit: SUPPORTED. A BlobView must retain its source-table FileIO when get_blob() resolves it; eagerly replacing it with descriptor bytes under blob-as-descriptor mode loses that association. The patch defers descriptor serialization to Arrow output and exercises row, Arrow, adapter, and source-IO paths. Implementation: CLEAN in the reviewed production diff. I did not rerun the Python suite locally.

@JingsongLi
JingsongLi merged commit c75ebb8 into apache:master Sep 22, 2026
12 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants