Survey Data

[shipmadison]

Hi Survey Data Team!

Starting a thread for discussion here.

@AndreaNOdell @Ovec8hkin

[Ovec8hkin]

@shipmadison @AndreaNOdell If not already done, would be great if everyone could compile their survey analyses into a RMarkdown file with some comments for review at tomorrows meeting.

[shipmadison]

Andrea, please prioritize your qualifying exam! Josh, I'll try and get as much done as I can before our meeting tomorrow, but I'm juggling several Wednesday deadlines already so thanks in advance for your patience.

[JaneSullivan-NOAA]

@Ovec8hkin see my comment in #2. i couldn't get any of your Rmds to compile, and they all had different error msgs. i didn't try to troubleshoot any of them.

[Ovec8hkin]

@AndreaNOdell no problem. Prep for qualifying exams as needed.
@shipmadison sure thing. These analyses aren't terribly complicated to work up and run, and I dont think anyone is waiting on decisions from us for anything (@JaneSullivan-NOAA correct me if Im wrong about that).

[Ovec8hkin]

@Ovec8hkin see my comment in #2. i couldn't get any of your Rmds to compile, and they all had different error msgs. i didn't try to troubleshoot any of them.

@JaneSullivan-NOAA I just saw and responded to that. Will investigate now.

[JaneSullivan-NOAA]

no rush! we can always work with the prelim files i made previously until the official versions are ready

[Ovec8hkin]

Interesting little thing I found today: the scales of the design-based indices being generated by the nwfscSurvey functions are not aligned with those that were reported in the 2013 Assessment Report.

For example, the design-based indices for 2023 from the NWFSCSlope survey are 6,700 - 9,266 (unit not precisely known (metric tons?)), but the indices that were reported in 2013 were on the scale of 30,000. The 2013 assessment document doesn't state what unit those indices are in, but I think that is something important to note and probably ask about.

There are some mild trend deviations between the design-based indices and those reported in 2013, but I believe that is all due to the 2013 indices being calculated by a delta-GLMM.

[JaneSullivan-NOAA]

Great catch @Ovec8hkin. Have you plotted the five new indices against their previously used counterparts (with 95% CI)? If not, we should do that before bringing it to the instructors, so that they have something concrete to work with. If you'd like I'm happy to put that together before we meet with Vlada today.

[Ovec8hkin]

I have not. I was planning on making that a bit of a weekend project, alongside cleaning up my code and thoughts on the survey strata.

[JaneSullivan-NOAA]

I had some time and was curious (and didn't see your message yet!). These are on the log-scale so we could look at the trends a little easier. The trends look very similar to me but the scale is off. One thing I noticed in the 2013 dat file is that at least for the two NWFSC surveys, they used model-based (GLM) indices, which explains the different CIs in the NWFSCcombo survey.
I should also say that I'm still using the output from my survey_data.R script, so if you made different assumptions its possible our indices are a little different. I will update mine to be consistent with yours once the survey subgroup is finished.
Link to the 2013 data
Link to the code to reproduce the figure below (note that you'll have to run the R/survey/survey_data.R script beforehand to get the 2023 assessment data):

[Ovec8hkin]

Thanks for the plotting script. Yes, I did see that the two NWFSC surveys are delta-GLMM derived, which explains some deviations in trends and in CIs, but doesn't explain the scale issue. My initial reaction is that its an issue of units, but that is something we need to be ABSOLUTELY sure about before going much farther. If the relative scale difference between the 2013 and 2023 data was constant, I would feel better about assuming it to be a units problem, but that doesn't initially appear to be the case.

[Ovec8hkin]

@JaneSullivan-NOAA why did you restrict the AFSCSlope survey timeseries to 1997-2001? I can't find anywhere in the text of the 2013 assessment document that says it wasn't used (though their plots seem to imply that), and there data going back to 1988. The spatial resolution of the data pre-1997 is definitely suspect, but seems like something you would account for in the stratification rather than throwing it away all together.

[Ovec8hkin]

Another kind of weird tidbit I found, I had been assuming that the AFSC triennial surveys were divided into Triennial1 and Triennial2 based on year, but it turns out that they are divided by depth. So the Triennial1 survey covers all depths 55-350m (which equates to a timeseries running from 1980-2004), while the Triennial2 survey covers all depths from 350-500m (which equates to a timeseries running from 1995-2004), with the years 1995-2004 having indices for both surveys.

This is definitely something we should investigate. I found a reference online that the 2013 SST assessment was the first one to do this type of divide. The same link states that theres ongoing discussion as to whether the Triennial NEEDS to be split at all. This might be something we want to run sensitivities too.

[JaneSullivan-NOAA]

@Ovec8hkin both of these issues are documented in the google doc notes in the fleet/survey table. For AFSCslope, 1988-1996 data exist, but they have low spatial resolution and were removed (if you look at some of the exploratory plots you can see how sparse the data are). For the Triennial split, they did not sample >366 prior to 1995, hence the split in depths. This is logical for a deep-dwelling species with ontogenetic migration to the slope, though frankly 500 m is still shallow for thornyheads. The link you found documenting some of the data quirks is really interesting! Thanks for sharing. Sadly none of these survey indices have much contrast... in my experience this results in a trendless biomass trajectory, large uncertainty in q, and an overreliance on the most recent survey estimate.

An alternative to doing rounds of individual data sensitivities is exploring the relative influence of each data set to the model result. This can be achieved by iteratively adding indices (there's a nice example of this in the 2022 GOA pollock assessment presentation, see slide 18) or through a leave-on-out type of analysis (in the vein of the 2021 GOA Pacific cod assessment presentaiton, see slides 23-25. These types of diagnostics can tell us which data sets are most informative and potentially for which years.

[Ovec8hkin]

@Ovec8hkin both of these issues are documented in the google doc notes in the fleet/survey table. For AFSCslope, 1988-1996 data exist, but they have low spatial resolution and were removed (if you look at some of the exploratory plots you can see how sparse the data are). For the Triennial split, they did not sample >366 prior to 1995, hence the split in depths. This is logical for a deep-dwelling species with ontogenetic migration to the slope, though frankly 500 m is still shallow for thornyheads. The link you found documenting some of the data quirks is really interesting! Thanks for sharing. Sadly none of these survey indices have much contrast... in my experience this results in a trendless biomass trajectory, large uncertainty in q, and an overreliance on the most recent survey estimate.

Gotcha, thanks. Survey team will discuss further.

An alternative to doing rounds of individual data sensitivities is exploring the relative influence of each data set to the model result. This can be achieved by iteratively adding indices (there's a nice example of this in the 2022 GOA pollock assessment presentation, see slide 18) or through a leave-on-out type of analysis (in the vein of the 2021 GOA Pacific cod assessment presentaiton, see slides 23-25. These types of diagnostics can tell us which data sets are most informative and potentially for which years.

Interesting. We can talk more about the mechanics of this in the future, but good to know their are options beyond factorial level sensitivities.

[Ovec8hkin]

@shipmadison @AndreaNOdell @JaneSullivan-NOAA

I updated the nwfscSurvey package, and that fixed the scaling problem. Everything should be updated on GitHub now. As far as I can tell, it wasn't a problem with the index calculation, but with the underlying data pulling process. Maybe PullCatch.fn wasn't pulling all of the data previously? Regardless, problem solved, though the two slope surveys are still off in scale.

[Ovec8hkin]

@shipmadison @AndreaNOdell

I thought more about our discussion of making nice plots to display the strata boundaries and realized this is a way to simplify the plots and not need shape files. We could just use the west coast map that is used for plotting the geographic distribution of CPUE and then add horizontal lines where the latitudinal strata breaks occur. That wouldn't require a shape file at all. Any takers on wanting to try this out?

Alternatively (and I would have to check this), we might be able to get nice shape files for the west coast directly from some R package (ie. sf), then do some simple spatial calculations with them to generate polygons of the strata. This would be more complicated, but is something we could look into.

[AndreaNOdell]

I finally sorted out github and was going to revisit this, but it looks like you figured it out! Looks and works great on my end with a quick re-run of the code in R. Is there any additional touches on it you were planning to do and would like help with?

[Ovec8hkin]

@AndreaNOdell If you'd like, it might worth it to add a depth plot to the Strata Map to indicate depth strata alongside the map based latitudinal strata. Then again, it sounds like we are going to be using geostatistical indices that are strata agnostic, so maybe these plots aren't useful at all.

[AndreaNOdell]

There are some unexplained differences between the 2013 DB, 2023 DB, and 2023 MB indices for the AFSC and NWFSC slope surveys that should probably be resolved. We've previously discussed the differences between the 2013 and 2023 DB indices for these surveys - 2023 DB indices provide much larger estimates for the AFSC and NWFSC surveys compared to 2013 (see below). Now, when compared with the 2023 MB indices, the 2023 AFSC MB indices align more closely with the 2013 AFSC DB indices. However, the 2023 NWFSC MB indices fall somewhere between the 2013 and 2023 DB indices.

Need to determine if there is an error in calculating the DBI for the AFSC and NWFSC slope surveys.

[Ovec8hkin]

This is a really curious "problem". As we have discussed before, the SS3 model will just rescale q to deal with the scale change so this should have no major effect on model outputs or estimates, but, since we are changing to model-based indices for the first time, it will be best to show that the 2013 DB indices and the 2023 MB indices are similar.

Ian Taylor has noted that there definitely could have been some issues with the 2013 DB and GLMM-based indices as they were calculated by hand, and as such, we should trust the 2023 indices from the nwfscSurvey package more than the 2013 indices. I was ok with this at the time, but the fact that the geostatistical indices don't align perfectly with either the 2013 or the 2023 indices makes me not want to trust either one.

I would propose carefully looking through the nwfscSurvey code and data to ensure that there isn't something strange happening with the 2023 DBIs (weird units, imputed area calculations, etc.). If that yields nothing, we can look more carefully at the 2013 DBI calculations.

[Ovec8hkin]

Alternatively, since its only the two Slope surveys that display this issue, and since those survey timeseries are both short and temporally overlap the Triennial survey, we could explore not using them at all as a sensitivity.

[owenhamel]

I agree with the approach of conducting a sensitivity to leaving them out and seeing if they are providing any additional information to the assessment. Being modelers, and since we want to understand what is going on, we sometimes spend too much time on things like this during assessments, when they could simply be dropped or left as is, with further explorations done when we are not under the time crunch of an assessment.

…

On Tue, Mar 21, 2023 at 1:05 PM Joshua Zahner ***@***.***> wrote: Alternatively, since its only the two Slope surveys that display this issue, and since those survey timeseries are both short and temporally overlap the Triennial survey, we could explore not using them at all as a sensitivity. — Reply to this email directly, view it on GitHub <#7 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWUU7RIGBKM7IBF5FYFZPTW5ICZRANCNFSM6AAAAAAUQWJRQI> . You are receiving this because you are subscribed to this thread.Message ID: <JaneSullivan-NOAA/shortspine_thornyhead_2023/repo-discussions/7/comments/5385522 @github.com>

-- *Owen S. Hamel, Ph.D. (he/him)* *Supervisory Research Fish Biologist* *Northwest Fisheries Science Center* *(206) 697-3102 (C)*

[Ovec8hkin]

I think Petrale Sole in the PADW mentioned leaving them out, and the more I think about it, the more I want to agree with that line of reasoning. They aren't adding any data temporally that we dont get from the Triennial Survey. The timeseries are really short. They dont show any real trend. But removing them significantly simplifies the fleet/survey structure and significantly decreases the total number of selectivity parameters that the model has to estimate, which is probably desirable.

[AndreaNOdell]

Re: data analyses write up

I am trying to recreate some of the information from the 2013 assessment report but am having trouble with it. Specifically, I want to calculate the SST frequency of occurrence in tows to update this specific line

91% of the tows in the NWFSC Combo Survey below 500 m have at least one shortspine thornyhead in the catch (and 97% have at least one longspine).

My code to calculate this is on github in R > survey > SST_surveys_2023.R at the very bottom in the combo survey section. Essentially I am grouping by tows, summing the total number of rows and the number of rows with SST information, The way I have it calculated currently says that 100% of the tows include at least one SST in the tow, which doesn't align with the sentence above. I've tried grouping by the tow column, year x tow, year x vessel x pass x tow, and trawl_id to no avail. I also tried shallower and deeper than 500m in case I was misinterpreting "below" but that also yielded the same result. Can I get a second pair of eyes on this? The issue could be in how I'm grouping "tows".

[AndreaNOdell]

Oh, actually I think I need to incorporate the information from the "subsample_count" column. Let me work on that and I'll update you all here!

[AndreaNOdell]

I updated the code and the problem seems to be fixed. I grouped by year, vessel, pass, and tow which led to 91.3%. A slight increase from the freq. of occurrence for years 2013 and earlier which I calculated as 90.6%