Skip to content

Allow standardizing a DenseMatrix by shifting the data - #539

Closed
abelianbee wants to merge 1 commit into
Quantco:mainfrom
abelianbee:standardize-dense-by-shifting-data
Closed

abelianbee wants to merge 1 commit into
Quantco:mainfrom
abelianbee:standardize-dense-by-shifting-data

Conversation

@abelianbee

@abelianbee abelianbee commented Sep 11, 2026

Copy link
Copy Markdown

Closes #414.

standardize keeps shift = -mean/std and mult = 1/std on the StandardizedMatrix, so sandwich expands the product into four terms, one of them outer(shift, shift) * sum(d). That is a centered second moment computed from uncentered sums, so when a column's mean is large relative to its standard deviation those terms are orders of magnitude larger than their sum and the significant digits cancel away. No regrouping of the algebra avoids it. Centering the data does.

This adds an opt-in materialize_shift flag to standardize. When set, the shift and multiplier are folded into a copy of the data and the returned matrix carries a zero shift, so the expansion never happens. The default is unchanged. _materialize_standardization returns None by default, so SparseMatrix and CategoricalMatrix ignore the flag and are never densified. StandardizedMatrix holds a reference to the pre-standardization matrix, so unstandardize() stays exact and free, which glum relies on in _glm.py.

Measured against a longdouble oracle, same matrix both ways, on an M1 Max:

n p dtype rel err, expansion rel err, materialized speedup
5,000 1,000 float64 9.1e-09 3.2e-15 1.20x
1,000,000 100 float64 1.2e-07 4.3e-15 1.90x
200,000 50 float64 3.9e-02 4.4e-13 2.36x
5,000 1,000 float32 6.9e+00 2.1e-06 1.11x

Better on both axes, which is worth saying explicitly because the obvious objection is that you pay for a copy. You don't: the expansion does three p x p outer products and an extra transpose_matvec, where the materialized path is one BLAS-3 call on contiguous data.

Tests cover representation equivalence across all seven matrix fixtures with both flag values, operation equivalence for sandwich / matvec / transpose_matvec, and an accuracy regression against a longdouble reference. 5666 passed, 45 skipped, 4 xpassed.

I moved the unreleased changelog section from 4.2.2 to 4.3.0, since this adds functionality. Happy to put it back if you would rather keep the number.

Checklist

  • Added a CHANGELOG.rst entry

Expanding a centered second moment from uncentered sums loses the significant
digits when a column's mean is large relative to its standard deviation. Add an
opt-in materialize_shift flag that folds the shift and multiplier into a copy of
the data instead, so sandwich never forms outer(shift, shift) * sum(d).

Sparse and categorical matrices ignore the flag and are never densified.
unstandardize() stays exact via a reference to the pre-standardization matrix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standardize DenseMatrix by shifting data

1 participant