Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create reformatting support for MODE #355

Open
2 of 29 tasks
bikegeek opened this issue Dec 10, 2024 · 0 comments
Open
2 of 29 tasks

Create reformatting support for MODE #355

bikegeek opened this issue Dec 10, 2024 · 0 comments
Assignees
@bikegeek @CPKalb
Labels
component: METdataio METdataio: General priority: high High Priority reporting: NRL METplus Naval Research Laboratory METplus Project requestor: Navy/NRL Naval Research Laboratory required: FOR DEVELOPMENT RELEASE Required to be completed in the development release for the assigned project type: enhancement Improve something that it is currently doing type: new feature Make it do something new
Milestone
METdataio-3.1.0

Comments

@bikegeek
Copy link
Collaborator

bikegeek commented Dec 10, 2024 •
edited
Loading

Describe the Enhancement

Provide reformatting support for MET MODE output. MODE has two output files: CTS and object attribute. Support formats that can be read in by METcalcpy agg_stat_bootstrap.py module. Refer to the R script agg_stat_boostrap.info_tmpl for creating the config file for agg_stat_bootstrap.py.

Sample XML files

  • on 'seneca'
  • /d1/persona/mkalb/MODE_plots copied to /d1/personal/mwin/MODE_NRL/MODE_from_Tina
  • Begin with supporting plots in the Fcst_vs_Obs directory
  • APCP and REFC variables
    Stats of interest:
  • acsi ( plot_icperts_mode_apcp_afbias.png)

Screenshot 2024-12-10 at 5 52 45 PM

associated XML config file from METviewer:

 `<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<plot_spec>
   <connection>
       <host>mohawk</host>
       <database>mv_rrfse_icpert_stoch_mode_fvo</database>
       <user>******</user>
       <password>******</password>
       <management_system>mariadb</management_system>
   </connection>
   <rscript>/usr/local/R/bin/Rscript</rscript>
   <folders>
       <r_tmpl>/opt/vxwww/tomcat/webapps/metviewer//R_tmpl</r_tmpl>
       <r_work>/opt/vxwww/tomcat/webapps/metviewer//R_work</r_work>
       <plots>/d2/www/dtcenter/met/metviewer_output//plots</plots>
       <data>/d2/www/dtcenter/met/metviewer_output//data</data>
       <scripts>/d2/www/dtcenter/met/metviewer_output//scripts</scripts>
   </folders>
   <plot>
       <template>series_plot.R_tmpl</template>
       <dep>
           <dep1>
               <fcst_var name="APCP_01_ENS_NEP_ge2.54_NBRHD729">
                   <stat>AREARAT_FSA_OSA</stat>
               </fcst_var>
           </dep1>
           <dep2/>
       </dep>
       <series1>
           <field name="model">
               <val>nostoch</val>
               <val>stoch</val>
           </field>
       </series1>
       <series2/>
       <plot_fix/>
       <plot_cond/>
       <indep equalize="false" name="fcst_lead">
           <val label="1" plot_val="">10000</val>
           <val label="2" plot_val="">20000</val>
           <val label="3" plot_val="">30000</val>
           <val label="4" plot_val="">40000</val>
           <val label="5" plot_val="">50000</val>
           <val label="6" plot_val="">60000</val>
           <val label="7" plot_val="">70000</val>
           <val label="8" plot_val="">80000</val>
           <val label="9" plot_val="">90000</val>
           <val label="10" plot_val="">100000</val>
           <val label="11" plot_val="">110000</val>
           <val label="12" plot_val="">120000</val>
           <val label="13" plot_val="">130000</val>
           <val label="14" plot_val="">140000</val>
           <val label="15" plot_val="">150000</val>
           <val label="16" plot_val="">160000</val>
           <val label="17" plot_val="">170000</val>
           <val label="18" plot_val="">180000</val>
           <val label="19" plot_val="">190000</val>
           <val label="20" plot_val="">200000</val>
           <val label="21" plot_val="">210000</val>
           <val label="22" plot_val="">220000</val>
           <val label="23" plot_val="">230000</val>
           <val label="24" plot_val="">240000</val>
           <val label="25" plot_val="">250000</val>
           <val label="26" plot_val="">260000</val>
           <val label="27" plot_val="">270000</val>
           <val label="28" plot_val="">280000</val>
           <val label="29" plot_val="">290000</val>
           <val label="30" plot_val="">300000</val>
           <val label="31" plot_val="">310000</val>
           <val label="32" plot_val="">320000</val>
           <val label="33" plot_val="">330000</val>
           <val label="34" plot_val="">340000</val>
           <val label="35" plot_val="">350000</val>
           <val label="36" plot_val="">360000</val>
       </indep>
       <agg_stat>
           <agg_mode>true</agg_mode>
           <boot_repl>1</boot_repl>
           <boot_random_seed/>
           <boot_ci>perc</boot_ci>
           <eveq_dis>false</eveq_dis>
           <cache_agg_stat>false</cache_agg_stat>
           <circular_block_bootstrap>true</circular_block_bootstrap>
       </agg_stat>
       <plot_stat>median</plot_stat>
       <tmpl>
           <data_file>plot_icperts_mode_apcp_afbias.data</data_file>
           <plot_file>plot_icperts_mode_apcp_afbias.png</plot_file>
           <r_file>plot_icperts_mode_apcp_afbias.R</r_file>
           <title>1h Accum Precipitation NEP (&gt;= 2.54 mm) Object Area Weighted Frequency Bias</title>
           <x_label>Forecast Lead Time (h)</x_label>
           <y1_label>Frequency Bias</y1_label>
           <y2_label/>
           <caption/>
           <job_title>icperts_mode_apcp_afbias</job_title>
           <keep_revisions>false</keep_revisions>
           <listdiffseries1>list()</listdiffseries1>
           <listdiffseries2>list()</listdiffseries2>
       </tmpl>
       <execution_type>Python</execution_type>
       <event_equal>false</event_equal>
       <vert_plot>false</vert_plot>
       <x_reverse>false</x_reverse>
       <num_stats>false</num_stats>
       <indy1_stag>false</indy1_stag>
       <indy2_stag>false</indy2_stag>
       <start_from_zero>false</start_from_zero>
       <grid_on>true</grid_on>
       <sync_axes>false</sync_axes>
       <dump_points1>false</dump_points1>
       <dump_points2>false</dump_points2>
       <log_y1>false</log_y1>
       <log_y2>false</log_y2>
       <varianceinflationfactor>false</varianceinflationfactor>
       <plot_type>png16m</plot_type>
       <plot_height>8.5</plot_height>
       <plot_width>11</plot_width>
       <plot_res>72</plot_res>
       <plot_units>in</plot_units>
       <mar>c(8,4,5,4)</mar>
       <mgp>c(1,1,0)</mgp>
       <cex>1</cex>
       <title_weight>2</title_weight>
       <title_size>1.4</title_size>
       <title_offset>-2</title_offset>
       <title_align>0.5</title_align>
       <xtlab_orient>1</xtlab_orient>
       <xtlab_perp>-0.75</xtlab_perp>
       <xtlab_horiz>0.5</xtlab_horiz>
       <xtlab_freq>0</xtlab_freq>
       <xtlab_size>2</xtlab_size>
       <xlab_weight>1</xlab_weight>
       <xlab_size>3</xlab_size>
       <xlab_offset>2</xlab_offset>
       <xlab_align>0.5</xlab_align>
       <ytlab_orient>1</ytlab_orient>
       <ytlab_perp>0.5</ytlab_perp>
       <ytlab_horiz>0.5</ytlab_horiz>
       <ytlab_size>2</ytlab_size>
       <ylab_weight>1</ylab_weight>
       <ylab_size>3</ylab_size>
       <ylab_offset>-7</ylab_offset>
       <ylab_align>0.5</ylab_align>
       <grid_lty>3</grid_lty>
       <grid_col>#cccccc</grid_col>
       <grid_lwd>1</grid_lwd>
       <grid_x>listX</grid_x>
       <x2tlab_orient>1</x2tlab_orient>
       <x2tlab_perp>1</x2tlab_perp>
       <x2tlab_horiz>0.5</x2tlab_horiz>
       <x2tlab_size>0.8</x2tlab_size>
       <x2lab_size>0.8</x2lab_size>
       <x2lab_offset>-0.5</x2lab_offset>
       <x2lab_align>0.5</x2lab_align>
       <y2tlab_orient>1</y2tlab_orient>
       <y2tlab_perp>0.5</y2tlab_perp>
       <y2tlab_horiz>0.5</y2tlab_horiz>
       <y2tlab_size>1</y2tlab_size>
       <y2lab_size>1</y2lab_size>
       <y2lab_offset>1</y2lab_offset>
       <y2lab_align>0.5</y2lab_align>
       <legend_box>o</legend_box>
       <legend_inset>c(0, -.25)</legend_inset>
       <legend_ncol>3</legend_ncol>
       <legend_size>1.2</legend_size>
       <caption_weight>1</caption_weight>
       <caption_col>#333333</caption_col>
       <caption_size>0.8</caption_size>
       <caption_offset>3</caption_offset>
       <caption_align>0</caption_align>
       <ci_alpha>0.05</ci_alpha>
       <eqbound_low>-0.001</eqbound_low>
       <eqbound_high>0.001</eqbound_high>
       <lines>
           <line color="#000000" line_pos="1" lty="1" lwd="1" type="horiz_line"/>
       </lines>
       <plot_ci>c("none","none")</plot_ci>
       <show_signif>c(FALSE,FALSE)</show_signif>
       <plot_disp>c(TRUE,TRUE)</plot_disp>
       <colors>c("#ff0000FF","#0000ffFF")</colors>
       <pch>c(20,20)</pch>
       <type>c("b","b")</type>
       <lty>c(1,1)</lty>
       <lwd>c(2,2)</lwd>
       <con_series>c(1,1)</con_series>
       <order_series>c(1,2)</order_series>
       <plot_cmd/>
       <legend>c("no stoch","stoch")</legend>
       <create_html>TRUE</create_html>
       <y1_lim>c()</y1_lim>
       <x1_lim>c()</x1_lim>
       <y1_bufr>0.04</y1_bufr>
       <y2_lim>c()</y2_lim>
   </plot>
</plot_spec>
  • afbias
  • matched object centroid distance (m_cendist)
  • matched and unmatched objects intensity (mu_int90)
  • object area (mu_area)
  • object counts (mu_ocounts)

Time Estimate

Estimate the amount of work required here.
3-5 days per sub-issue

Sub-Issues

Consider breaking the enhancement down into sub-issues.

  • Reformat object attribute output (stat_name, stat_values, stat_btcl, stat_btcu, n_stats)
  • Reformat object attribute output for input to METcalcpy agg_stat_bootstrap.py module
  • Reformat the CTS output (stat_name, stat_values, stat_btcl, stat_btcu,??)
  • Generate config file for running METcalcpy agg_stat_bootstrap.py to calculate the boostrap statistics
  • Verify that plots like the above example can be generated
  • Generate tests to verify that reformatting is correct
  • Update User's Guide in METdataio

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

NRL Year 1 7730022

Define the Metadata

Assignee

  • Select engineer(s) Minna Win
  • Select scientist(s) Tina Kalb

Labels

  • Review default alert labels
  • Select component(s)
  • Select priority
  • Select requestor(s)

Milestone and Projects

  • Select Milestone as a METdataio-X.Y.Z version, Consider for Next Release, or Backlog of Development Ideas
  • For a METdataio-X.Y.Z version, select the METdataio-X.Y.Z Development project

Define Related Issue(s)

Consider the impact to the other METplus components.

Enhancement Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding Source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Add any new Python packages to the METplus Components Python Requirements table.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Development issue
    Select: Milestone as the next official version
    Select: METdataio-X.Y.Z Development project for development toward the next official release
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@bikegeek bikegeek added type: enhancement Improve something that it is currently doing priority: high High Priority type: new feature Make it do something new requestor: Navy/NRL Naval Research Laboratory required: FOR DEVELOPMENT RELEASE Required to be completed in the development release for the assigned project METdataio: General component: METdataio reporting: NRL METplus Naval Research Laboratory METplus Project labels Dec 10, 2024
@bikegeek bikegeek added this to the METdataio-3.1.0 milestone Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bikegeek bikegeek

@CPKalb CPKalb

Labels
component: METdataio METdataio: General priority: high High Priority reporting: NRL METplus Naval Research Laboratory METplus Project requestor: Navy/NRL Naval Research Laboratory required: FOR DEVELOPMENT RELEASE Required to be completed in the development release for the assigned project type: enhancement Improve something that it is currently doing type: new feature Make it do something new
Projects
None yet
Milestone
METdataio-3.1.0
Development

No branches or pull requests
2 participants
@bikegeek @CPKalb