SS Models

[Ovec8hkin]

I managed to install, run, and reproduce the 2013 SST assessment from the files that were provided on Google Drive. Everything worked very nicely using SS v 3.24.U. I made an initial commit of all of the outputs to the 2013_assessment branch of the repository, so take a look if interested.

Only major concern I have is what files to actually track using GitHub. It appears that the only files REQUIRED to reproduce the assessment are those with the filetype *.ss. Everything else is pure output from ADMB and SS. Maybe its okay to simply only track the *.ss files and the SS executable? Alternatively, maybe it's best to also track all of the SS output files (*.sso), so that R scripts using the SS_ouptut() function will work out of the box? I have concerns about tracking all of the plot outputs, as my initial commit was ~80mb, which is really large for a git repo. Maybe its sufficient to not track the generate plots/ directory, and force users to run SS_output() and SS_plots() manually on clone to produce the necessary plots?

Looking for suggestions.

@JaneSullivan-NOAA @shipmadison

[Ovec8hkin]

Right now, the 2013_ss_model branch is setup to not track the SS plots/ directory or a selection of ADMB binary file outputs (which we cant interpret anyway).

[MatthVeron]

Thanks Josh for this. Based on my previous experience with SS, I think the most important files to track for transparancy and reproducibility are:

• the 4 input files (Starter.ss, Control.ss, Data.ss, Forecast.ss),
• the .exe,
• the main output files (Report.sso, Forecast-report.sso and the ss.par) and maybe
• Echoinput.sso

As you said, everything else is from ADMB and SS and are not really useful since we can easily run the model (in term of time).
Similarly, I don't think we need to keep track of the /plots/ folder as we can reproduce it quickly as well with those files.

I was also thinking of starting to look at the transition between the two versions, if that's okay for you guys?

[Ovec8hkin]

Thats very helpful, thank you. I haven't spent much time working with SS, so I wasn't sure what precisely was important for reproducibility. Are Report.sso, Forecast-report.sso, and ss.par the only important output files? A bunch of other .sso files are also generated, but its unclear to me right now what they contain and whether or not they are needed for some of the r4ss functions to work correctly.

I was also thinking of starting to look at the transition between the two versions, if that's okay for you guys?

@MatthVeron Go right ahead. I recommend making a branch off the current 2013_ss_model to work on the version transition, since the 2013_ss_model branch isn't merged yet (I think I'll work on merging it either later tonight or tomorrow afternoon, if you want to wait). I am going to ask you too be really diligent about documenting everything that goes into the transition in case we ever get asked about it. Let me know if you run into any problems, as the version transition was something I was going to look into this week as well. Always happy to set up a meeting to discuss.

[MatthVeron]

Yes, I think the most important files are those (and the Echoinput.sso file). I have not yet checked which files are really needed to use the r4ss functions, but I think these are the ones that are used.

I recommend making a branch off the current 2013_ss_model to work on the version transition, since the 2013_ss_model branch isn't merged yet

@Ovec8hkin Thanks for making a pull request. I'll start working making a new brach from the 2013_SS_model so it'll be easy to merge then. Of course, I'll document everything along the process. I'll let you know if I have any problem, and happy to set up a meeting to discuss anyway !

[Ovec8hkin]

Another comment about this:

From a reproducibility standpoint I think it a nice idea to always place the SS executable in each model directory, however, the executable is quite large (2.2mb), so its not really an ideal thing to track. There is documentation in the most recent SS manual about how to add the SS executable to your system $PATH for all major OSs, which alleviates the problem of tracking a bunch of copies of the same SS executable file.

I propose that we create a directory with one copy of each of the SS executables we use (so v 3.24U and v 3.30.21), and then have everyone place those executables on their system $PATH and run SS that way. We can add documentation to that directory about how to add those files to the a system $PATH for anyone who wants to repeat things in the future.

@JaneSullivan-NOAA Do NOAA employees have the ability to modify the system $PATH on their machines, or is that something only a network administrator can do? If it's not easy to do then we can ignore everything above and just have a lot of copies.

[Ovec8hkin]

That sounds a good idea.
Are you thinking about something like a SS_Exe folder on the root and a specific subfolder for each version of SS (model/SS_Exe/V3_24 and model/SS_Exe/V3_30_21) and then we add the documentation in the SS_Exe folder?

Something along those lines. Once I merge in the 2013 model, I will rebranch and play around with the $PATH stuff. It SHOULD be easy, I think, but I will need someone with a Windows machine (and probably a NOAA personnel as well) to confirm that it works for them (if anyone has a linux machine that might also be helpful to cover all the vases with).

@Ovec8hkin we need IT to edit our paths but that's fine, we're used to it, and I think this is a better solution than having dups of the same exe. thanks!

Ok. We'll proceed along those lines then. Thanks.

[MatthVeron]

Something along those lines. Once I merge in the 2013 model, I will rebranch and play around with the $PATH stuff. It SHOULD be easy, I think, but I will need someone with a Windows machine (and probably a NOAA personnel as well) to confirm that it works for them (if anyone has a linux machine that might also be helpful to cover all the vases with).

Sounds good. I can be the Windows user if needed. I was also thinking that another possibility may be to just add a copy.file line in the R script from the Executables folder so everybody is sure about the version of SS that is used. Also, I'm on the transitionning now and just realized that you won't see my brach as I'm working on my fork repo. I'll make a pull request later this afternoon with all the files (I hope to get a model running tonight).

[Ovec8hkin]

Thats actually not a bad idea. We can tell the .gitignore to not track any SS executables not in the main executables directory, and just to tell everyone to always run SS via R (or have SS on your $PATH$). At some point, we should probably sit down and make some kind of generalized run_SS script that just runs an SS model provided as input, that way everyone is running the models the same way and we can control what outputs are kept, where they are sent, what is tracked, etc.

[JaneSullivan-NOAA]

@Ovec8hkin or @MatthVeron - did you all decide on a file structure for the two .exe files? I'm trying to follow the dialogue above... it looks like we just need to copy the exes into each model subdirectory (but they won't be tracked on github - I like this idea). In that case there is no need to edit the system PATH. Correct?

[MatthVeron]

I think that is still in discussion. But that is correct, if we copy the .exe we won't need to get them declared in the PATH. See my comment below for another suggestion.

[MatthVeron]

I totally agree with you about the fact that we should just ask everyone to run SS AND modify input files through R so we can really have a tack of what has been done and what are exactly the differences between version and or models. I think the best way to do it is to build a specific Rscript for each model we want to implement (as you say @Ovec8hkin, those scripts can be based on a general template to work with SS).

Also, through the transition of the 2013 model I finally think that the best idea for the .exe would be to get them stored in a executables folder and then call them from the script itself to run SS. With this way you 1) don't need to get the .exe declared in your PATH which may be easier for users and 2) you know exactly which .exe has been used. I built a structure like this to make the transition (I'm almost done with this - should have a model running today) - I'll make a pull request so you guys will see what I'm taking about. but the idea is to use the r4ss function where you can point to a specific folder for the .exe . So if we use the executable folder as house for all .exe (for each version of SS) we'll just need to track that directory on github and not each .exe in each model folder. This may be easier and more easier for traceability and reproducibility I think.

[JaneSullivan-NOAA]

Loving all this! @MatthVeron when you submit a PR, I'm happy to test it on my machine and merge it in. Really appreciate you both for your work on this.

[Ovec8hkin]

@MatthVeron Sounds good. Make a PR when you're happy with the transition, and I will take a look at it. Then we can move forward with deciding how to deal with the executables and running the models.

[MatthVeron]

Hey @Ovec8hkin and @JaneSullivan-NOAA : Sorry I'm late making this pull request, I got some troubles with my git tonight ahah. Well the results from the transitionning are pretty nice and I think we have a good basis to start the new assessment. Let's talk more tomorrow about this. I think the pull request is detailes enough for you guys to understand what I've done but please let me know if you have any question !

[MatthVeron]

Also, I'm gonna add some stuff on the wiki for the transition and will complete more later if that is ok for you.