Stop the graphical Lasso when it cycles instead of converging - #179
Merged
Merged
Conversation
On an ill-conditioned covariance -- typically rank-deficient, as PLNnetwork()
produces when the number of species approaches the number of samples -- the
block coordinate descent settles into a small limit cycle: the convergence
criterion stops decreasing and oscillates just above its threshold forever,
while the solution itself no longer moves. glassoFast spends its whole
10000-sweep budget on these and reports success regardless.
The cycle is now detected after 1000 consecutive sweeps without improving the
best criterion seen so far, and reported as status = "stalled". The patience
comes from a census of 246 penalized problems (Gaussian and count data, p from
10 to 120, along full penalty paths): problems that do converge never went more
than 373 sweeps without progress, while cycling ones went 4000+.
Stopping early costs nothing, because sweeping on does not buy accuracy: the
iterates wander inside the cycle rather than settle. On oaks, the solve stops
after ~1100 sweeps instead of 10000, and both that answer and the 10000-sweep
one sit within 1e-3 of a 50000-sweep grind, with the same number of edges.
PLNnetwork() on oaks goes from 14.4s to 5.6s without covariate and from 126.0s
to 18.2s with the tree covariate, with identical BIC and EBIC selections.
The solver gains a `status` ("converged", "stalled", "max_iter",
"inner_failure", "degenerate"), a `delta` (best criterion relative to its
threshold), and optional `trace` and `stall_patience` arguments. Fits record
`glasso_stalled` alongside `glasso_nonconverged` in their monitoring, and warn
only for the numerical failures: a stalled solve says something about the
problem -- usually too weak a penalty for a nearly rank-deficient covariance --
rather than about the solution.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Running
PLNnetwork()onoaksraises convergence warnings from the graphical Lasso at low penalties, and earlier so when thetreecovariate is added. This is not a regression from the internal solver: glassoFast behaves identically on these problems (same 10001 sweeps, 1.6e-16 difference) -- it simply never reported it.Diagnosis
It is not slow convergence but a limit cycle. The convergence criterion plateaus (4.0e-5, 4.6e-5, 4.9e-5, 2.6e-5, ...) instead of decaying; 200000 sweeps do not converge either. Loosening
thris the wrong lever: 1e-3 and 1e-2 both still fail, and 1e-2 changes the support.The cause is in the model, not the solver: on
oaks, p = 114 and n = 116, so withtreethe residual degrees of freedom are 113 < p and the residual covariance is singular, positive definite only through the S² ridge.Change
The solve stops after
stall_patience = 1000consecutive sweeps without improving its best criterion, and reportsstatus = "stalled". The patience is calibrated on a census of 246 penalized problems (Gaussian and count data, p from 10 to 120, full penalty paths): converging problems never went more than 373 sweeps without progress, cycling ones went 4000+.Stopping early costs nothing, because grinding does not buy accuracy -- the iterates wander inside the cycle. Against a 50000-sweep reference on
oaks: the old 10000-sweep answer is 3.00e-04 away with a different support; the new 1138-sweep answer is 6.98e-04 away with an identical support.A stalled solve no longer warns: it says something about the problem (too weak a penalty for a nearly rank-deficient covariance), not about the solution. Warnings remain for the genuine numerical failures.
Measurements
Optimized build,
OMP_NUM_THREADS=2, same machine, master vs this branch:oaksAbundance ~ 1Abundance ~ 0 + treeSelections are identical on both sides: BIC 0.5561 / EBIC 0.5561 without covariate, BIC 0.4127 / EBIC 0.5670 with
tree.Checks
NOT_CRAN=true, nothing skipped)R CMD check --as-cran: 0 errors, 0 warnings, 1 NOTE (a local compilation flag, not from the package)API
graphical_lasso()gainsstatus,delta, and thetrace/stall_patiencearguments; fits recordglasso_stallednext toglasso_nonconvergedin their monitoring.🤖 Generated with Claude Code