MAINT: Cleanup dead code, address FIXMEs, and fix Kalman filter prediction corner case #9740 #9741
MAINT: Cleanup dead code, address FIXMEs, and fix Kalman filter prediction corner case #9740 #9741un1u3 wants to merge 1 commit intostatsmodels:mainfrom
Conversation
|
@bashtage any updates on this PR |
ChadFulton
left a comment
ChadFulton
left a comment
There was a problem hiding this comment.
Overall seems probably fine. I'm not totally sure how much benefit there is to removing these unused tests. On balance, I suspect cleaning things up is better than keeping around things "in case they're useful", given the relatively limited maintenance hours these days.
On the Kalman filter prediction corner case - what you have looks reasonable to me, but I'd be curious if everything works okay if the situation actually arises. I think the diffuse component is typically centered at zero, so it probably wouldn't cause a problem with the predicted means, though I would assume we would get inf or NaN in the confidence intervals.
Would you be able to add a unit test for the case where we predict prior to the end of diffuse initialization? I think that if you used an UnobservedComponents model with a seasonal component of 12 periods and exact diffuse initialization, then it would still be in the diffuse regime if you had fewer than 12 periods of data, and then called predict.
| num_levels = Xi_vals.size | ||
| for x in Xi_vals: | ||
| if x <= x_u: # FIXME: why a comparison for unordered variables? | ||
| if x <= x_u: |
There was a problem hiding this comment.
Probably shouldn't just remove a FIXME without changing anything
2 participants
PR for Issue #9740
This PR performs a broad cleanup of the statsmodels codebase to improve maintainability and addresses a specific corner case in Kalman filter predictions. Key changes include removing dead code, implementing missing convergence warnings, and refining diffuse initialization logic.
Detailed Changes
1. Bug Fixes & Improvements
kalman_filter.py): Fixed a corner case where dynamic/out-of-sample predictions were incorrectly initialized if the filter hadn't exited the diffuse phase. Now usesInitialization.from_componentswith the diffuse covariance whenpredicted_diffuse_state_covis non-zero.innovations.py): AddedConvergenceWarningtoinnovations_mlewhen the nonlinear optimizer fails (success=False).2. Dead Code Removal (Library)
regressionplots.py: Removed internal helper_partial_regressionwhich was unused and redundant.3. Test Suite Cleanup
test_glm.py: Removed extensive commented-out test blocks relying on obsoleterpy2infrastructure.test_regression.py: Removed multiple unusedcheck_confidenceintervalsmethods and associated FIXME comments.test_var.py: Removed the unusedCheckVARclass and brokengenerate_varfunctions.test_data.py: Removed the commented-outTestDatesclass.4. Documentation & Annotation
kernels.py: Removed a staleFIXMEinaitchison_aitken_cdf. The logic was verified to be correct for joint CDF definitions of encoded discrete variables.Verification Plan
predictnow correctly passesPinfwhen initializing the forecast model during the diffuse phase.pytest.