File and Folder Structure

[Ovec8hkin]

Hi everyone,

I've been thinking about how best to handle file and folder structure for the assessment so as to keep all of out analyses and outputs organized. I have pushed some code to the 'alt-file-structure' branch of this repository that you can check out that demonstrates a possible directory structure. This is just one of many possible file/folder structures that could be used, but is one that I think keeps everything nicely organized (at least for now).

Full thoughts:

• data/ - This directory should hold any and all data that is direct requested from state/federal agencies or pulled from NOAA servers (ie. survey data). Data that gets output from other analyses (ie. a lot of the SS inputs) are stored somewhere else.
  • data/raw/ - This should be completely raw, unedited, data from various data sources. This folder is untracked and is safe to place confidential data in.
  • data/processed/ - This can be used to place processed data from available data sources. This folder IS tracked, so do not place any confidential data here, but rather cleaned datasets ready for analysis. Ideally, every file in this directory should have an equivalent "raw" data file in data/raw.
• R/ - This directory should hold R scripts and analyses. These can take the form of either runnable .R scripts or .Rmd Rmarkdown documents/notebooks. Note that data outputs/plots should not be stored in this directory. This directory is further broken down into subdirectories based on data type ('bio/', 'comps/', 'discards/', 'landings/', and 'survey/'). Data type specific analyses and script should live in their relevant subdirectories rather than the root R/ directory.
  • R/utils/ - Any utility functions that may be of use across R analyses can go here. For example, plotting code, or helpful data cleaning functions.
• model/ - This directory will (in the future) hold the stock synthesis model code.
• outputs/ - Send all outputs (including plots and potential SS input files) generated by other analyses to this directory. There is no defined directory structure for this directory at this time, so be thoughtful as to directory and file names when you send outputs here.

Again, these are just my personal thoughts on a possible directory structure, so they are open to comment and revision. Folder structure is likely change as more analyses are completed and needs change, but I think this is a good starting point for the data preparation stage.

[haleyoleynik]

I think this sounds great Josh, let's do it! Thanks for putting so much thought into it!

[JaneSullivan-NOAA]

a few notes @Ovec8hkin

• currently nothing from the survey processing scripts is going into data/processed, its all going into outputs/. maybe that's what you wanted?
• i wasn't sure how to reconcile my survey_data.R script with the existing outputs/ structure, so i just stuck all the outputs in data/raw/ so they won't be tracked and put a note at the top of the file that the script outputs weren't used for the assessment.
• i think you'll want to split the afsc triennial survey into two indices per the last assessment (Triennial1 is <= 366 m, Triennial2 > 366 m). this is documented in the assessment and in our notes doc
• i was unable to reproduce afsc_triennial_survey.Rmd. got the following error on Line 62 'Error in if (dim(bio)[1] > 10000) { : argument is of length zero Calls: <Anonymous> ... eval_with_user_handlers -> ...i did not try to troubleshoot this.
• i was also unable to reproduce nwfsc_combo_survey.Rmd. got the following error on Line 155 'Error in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, : invalid font type ... i did not try to troubleshoot this.
• i was also unable to reproduce nwfsc_slope_survey.Rmd. got the following error on Line 62 'Error in round_any(dat$Age, bin_width, floor) : object 'bin_width' not found ... i did not try to troubleshoot this.

[Ovec8hkin]

1. I wasn't intending to save anything to data/processed. All of the survey code right now uses the data in its raw format.
2. I also wasn't sure what to do with survey_data.R so I left it alone.
3. Yes, survey team will discuss.
4. Would have to check that exact problem.
5. Please update your nwfscSurvey library with the most recent GitHub commits (run remotes::install_github('pfmc-assessments/nwfscSurvey). This was the bug I submitted an issue and fix for.
6. Would have to check.

[JaneSullivan-NOAA]

i'm also getting a R/survey/nwfsc_combo_survey_files subdirectory that seems to contain a lot of intermediate files that i'm not sure you want

[Ovec8hkin]

Correct. We should add those file extensions to the .gitignore

[JaneSullivan-NOAA]

ok, i'll let you take care of that

[JaneSullivan-NOAA]

@Ovec8hkin is your branch up-to-date?

[Ovec8hkin]

It is now. PR submitted.

[JaneSullivan-NOAA]

The main branch should now be fully up to date with Josh's folder organization.