Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

give common heuristics #22

Open
dpastoor opened this issue Aug 26, 2019 · 3 comments
Open

give common heuristics #22

dpastoor opened this issue Aug 26, 2019 · 3 comments

Comments

@dpastoor
Copy link
Contributor

dpastoor commented Aug 26, 2019

Summary

representative example via PsN:

Successful minimization [ OK ]
No rounding errors [ OK ]
No zero gradients [ OK ]
No final zero gradients [ OK ]
Hessian not reset [ OK ]
Parameter(s) near boundary [ WARNING ]
OMEGA(2,2) 0.000112 0
Covariance step [ OK ]
Large condition number [ WARNING ]
Correlations [ OK ]

To exercise summary:
from: github.com/metrumresearchgroup/babylon/testdata/example-models/nonmem/BQL
bbi summary --json 2

Tests

  • summary_test.go
    • TestSummary
dpastoor added a commit that referenced this issue Aug 26, 2019
lst parsing will still only get the last method, as will default
to the ext file, and lst will be a general fallback.

references #8 #18 #19 #20 #22
@dpastoor
Copy link
Contributor Author

dpastoor commented Sep 16, 2019

  • Successful minimization [ OK ]

search for:

0MINIMIZATION SUCCESSFUL
  • No rounding errors [ OK ]

  • No zero gradients [ OK ]

  • No final zero gradients [ OK ]

  • Hessian not reset [ OK ]
    sample line: CUMULATIVE NO. OF FUNC. EVALS.: 598 RESET HESSIAN, TYPE I
    search text: RESET HESSIAN

  • Parameter(s) near boundary [ WARNING ]
    sample line: 0PARAMETER ESTIMATE IS NEAR ITS BOUNDARY
    search text: PARAMETER ESTIMATE IS NEAR ITS BOUNDARY

  • OMEGA(2,2) 0.000112 0
    I don't think this should have been copied from above?

  • Covariance step [ OK ]
    search text:
    0COVARIANCE STEP OMITTED: NO

  • Large condition number [ WARNING ]
    One way:
    The ratio of the largest eigenvalue to the smallest eigenvalue, referred to as the condition
    number should be less than 1000
    search text: EIGENVALUES OF COR MATRIX OF ESTIMATE
    ... a few lines later:
    2.53E-01 4.29E-01 6.19E-01 7.76E-01 8.79E-01 9.30E-01 1.19E+00 1.31E+00 2.61E+00
    This link leads me to believe the condition number must be calculated:
    Condition Number from get_summary() Appears Inaccurate UUPharmacometrics/xpose#128

  • Correlations [ OK ]
    Maybe check for large numbers (close to 1) in the off-diagonal values of the correlation matrix?
    search text: CORRELATION MATRIX OF ESTIMATE

@david-lyder david-lyder added this to the bbi 2.0.0 milestone Oct 11, 2019
@david-lyder
Copy link
Contributor

After change:

"run_heuristics": {
"large_condition_number": "HeuristicFalse",
"correlations_ok": "HeuristicTrue",
"has_final_zero_gradient": "HeuristicFalse",
"minimization_successful": "HeuristicTrue"
},

@david-lyder
Copy link
Contributor

The above heuristic output is considered complete for version 2.0.0
Further enhancements will be addressed based on feedback from the scientists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants