[US-1804] Object tree scalar values: one shared renderer, decoded text strings, binary carve-out

[unidoc-anom]

Description and links

A dictionary-entry scalar row in the object tree tells you a key exists and what type it is, and never what it says. RowSpan number is the whole problem: the cell has a row span, and the tree will not say it is 2. Answering a structural question means chasing object references one at a time with dump object --ref, or leaving for qpdf. JSON has the same hole and states it more plainly, since every scalar node carries valueType and no value key at all.

Underneath that sat a second problem. Three copies of the same scalar switch existed: scalarDisplay (tree.go) behind the tree and the diff, valueEntryFromObject (inspector.go) behind the object detail panel, and writeScalar (objectsource.go) behind dump source. The detail panel does not call scalarDisplay, contrary to what the tree work would naturally assume, so any fix touching one of them leaves the tool saying different things about the same bytes on different screens.

Jira: US-1804

Technical changes

1. Added internal/pdfcore/scalarrender.go, holding two renderings of the same object: scalarRaw (byte-exact, the old scalarDisplay moved verbatim) and scalarText (text strings decoded, everything else identical). semanticLabel, valueEntryFromObject, diffSummarize and the new diffCompare all route through them. writeScalar is deliberately left alone so dump source stays byte-frozen.
2. Gave TreeNode a Value and a ValueRaw, populated in buildTreeNode only for dictionary-entry scalars. ValueRaw is emitted from a content comparison, decoded text against the stored bytes with delimiters and escaping stripped, so an already-ASCII string emits no raw counterpart and its absence carries information rather than duplicating every node.
3. Plain-text dump tree now prints <key> <type> = <value>; --json gains an additive value key on scalar nodes and valueRaw where decoding changed the bytes. The JSON value is never truncated, because the point of --json is piping into jq.
4. Wired textstring.go into the shared helper, so PDF text strings render decoded per ISO 32000-1 7.9.2.2 on all six renderers: dump tree, dump object, diff, the GUI TreePanel, DetailPanel and DiffView. Split ValueEntry.Display (decoded) from ValueEntry.Raw (byte-exact), which were previously written identically.
5. Carved out the binary-carrying strings at the callers, where the key is in hand: signature /Contents and /Cert behind /Type /Sig or /DocTimeStamp or a bare /ByteRange, and filespec /Params /CheckSum by name. They render as <binary, N bytes> rather than a hex blob, and the bytes stay reachable through dump source --ref and dump object --ref. A markup annotation's /Contents is a genuine text string and still decodes.
6. Capped plain-text values at 80 runes with a [truncated: N of M] marker, measured in runes so decoded UTF-16BE is never cut mid-character, and escaped control characters so one node is always exactly one output line. That also fixes dump object, where a /Alt containing newlines rendered its continuation lines at column zero.
7. Rendered the value in the GUI tree row with a fixed element order and a no-drop rule: the value span is the only element that ellipsizes, the label and the three suffixes get flex-shrink-0, and the full value lives in the span's title. Added frontend/src/lib/escapeDisplayValue.ts as the GUI counterpart of the Go escaping, applied to TreePanel, DetailShared and DiffView.

Considerations

Four shapes were on the table before this one. Decoding at the tree call site only, or inside scalarDisplay, would each have been a much smaller change; both were argued from the assumption that the detail panel shared a renderer with the tree, and it does not, so both would have left the tool saying two or three different things about the same object. Shipping values with no decoding at all was internally consistent and cheaper, but then a UTF-16BE /Alt reads as <FEFF0052...> in the primary inspection surface, and the encoding decision has to be made here anyway because the acceptance test must assert something for it.

The consolidation is the part that made this worth doing as one change rather than three. The duplicated switch is exactly the drift risk that motivated a single formatter in the first place.

Two things turned up during review that are worth recording. Decoding nearly entered the diff's equality decision: scalarLeaf took its changed/unchanged status by comparing diffSummarize on both sides, and diffSummarize is a scalarDisplay caller, so <FEFF0041> and (A) would have compared equal and a real byte difference would have reported unchanged. The comparison now runs through diffCompare on the byte-exact rendering while the displayed summaries decode. Separately, a literal string holding unescaped 0x80-0xFF produced invalid UTF-8, which json.Marshal rewrites to U+FFFD, destroying precisely what the raw counterpart exists to preserve; utf8Safe falls back to uppercase hex, the same policy textStringOrRaw already applied to decoded text.

The plain-text line shape changes from RowSpan number to RowSpan number = 2, which breaks an end-anchored grep such as grep 'Type name$'. The only consumers are the CLI's own help text and the cli-tree-dump and cli-object-query suites, updated here as expected churn. Nothing is scripted against it in or out of the repo, and we are pre-1.0, so there is no compatibility shim.

How was this tested?

• go vet ./... clean, golangci-lint run 0 issues
• go test ./... 1049 passed across 13 packages; go test -race ./internal/pdfcore/ ./cmd/cli/ 928 passed
• scripts/test-all.sh all per-module suites under tests/ passed, including the new tests/object-tree-scalar-values/ (51 black-box cases over the built binary) and the existing cli-tree-dump, cli-object-query, cli-output-format-normalization, structural-diff and shared-text-string-decoder
• npx tsc --noEmit clean, npx eslint . clean, npx vitest run 945 passed in 73 files
• Tests were written first and failed for the right reason: 37 of 51 Go cases and 7 of 9 frontend cases were red against a vet-clean tree before any production code was written
• Manual: built the CLI and ran dump tree, dump tree --json, dump object and dump source against testdata/signed.pdf, tagged.pdf, fonts-mixed.pdf, unsigned-sig-field.pdf and pdfa-1b-clean.pdf, plus a hand-assembled PDF covering odd-digit hex, an unpaired surrogate, an undefined escape, a #20 name, a C1 run, a Latin-1 literal and a /Cert array nested inside another array

Not covered by automation: the GUI tree row at narrow widths. Headless tests assert the class names and DOM order, but real layout needs a native build.

What could go wrong?

The binary carve-out is a key-plus-context match, so it depends on signature dictionaries carrying /Type /Sig, /Type /DocTimeStamp or a /ByteRange. A signature dictionary typed with something else and no /ByteRange would have its DER run through the text decoder and land on the row. Nothing outside the signature family carries /ByteRange, so the guard cannot over-carve, but it can under-carve on a shape we have not seen.

A signature /Contents or /Cert array reached as an indirect object cannot be carved out at all. GetObjectDetail recovers the key from the node ID, and an obj:G:N ID has no room for one, so the DER decodes when the value is selected as an object in its own right. The dictionary-entry tree row is unaffected. Fixing it means threading the key through resolveNodeObject or changing the ID scheme, which is its own change.

The decoder inherits pdfcpu's quirks, and they are now visible on six surfaces instead of one. The non-UTF-16BE fallback is Latin-1 despite the function being named CP1252ToUTF8, so a Word-authored curly apostrophe (0x92) decodes to a C1 control rather than the character a reader expects; it is escaped rather than silently invisible, but it is still not the apostrophe. HexLiteralToString runs escape processing over already-decoded bytes, so a 0x5C inside a hex text string is dropped: <415C42> gives AB, not A\B. Both are pinned by tests so the behaviour is at least known.

tests/shared-text-string-decoder/TestDumpObject_RawStringRenderersUnchanged pinned dump object rendering /Title and /UF as raw hex, which is the contract this change deliberately overturns. It was rewritten as TestDumpObject_DisplayIsDecodedAndRawKeepsTheBytes, keeping the recoverability guarantee on the --json raw field. That is a contract change to a prior change's suite, not a mechanical fix, and is worth a reviewer's eye.

One unrelated flake: frontend/src/components/App.drain.test.tsx failed once at line 274, a waitFor on real timers at the 1000 ms default, and passed 28 subsequent runs including a cold-transform full run. If it shows up on a CI leg here, it is that timeout and not this change.

Screenshots/videos (if appropriate)

dump tree --resolve --resolve-depth 10 --depth 10 on a tagged-table fixture, before:

        A
          ColSpan number
          O name
          RowSpan number
        Alt string
        S name

after:

        A
          ColSpan number = 1
          O name = /Table
          RowSpan number = 2
        Alt string = Rapport cell
        S name = /TD

A signature, where the bytes must not be decoded:

        V (5 0 R) Sig
          ByteRange
          Contents string = <binary, 3072 bytes>
          SubFilter name = /adbe.pkcs7.detached

And the JSON contract, where the stored form is recoverable only when decoding changed it:

{"label":"Lang",   "valueType":"string","value":"en-US"}
{"label":"RowSpan","valueType":"number","value":"2"}
{"label":"Alt",    "valueType":"string","value":"Rapport cell","valueRaw":"<FEFF0052006100700070006F00720074002000630065006C006C>"}

jq '.. | select(.valueRaw?)' now surveys every string in a document whose stored form differs from its display form, which previously took one dump source --ref per object.

Checklist

• Tests pass locally
• I targeted the dev branch, not master

[anovik]

LGTM