@Vitalis95 in the design above:

First you may wish to install the outfillingR package. Once installed you can use Help > R Package Documentation to get lots of details of the function.

Then in the diagram above:
a) The Data Frame top left is the usual data selector, so we can choose variables needed on the right.
b) The capital letters are for clarity for you. In the dialog they should be our usual Station: etc
c) The controls on the right are just the normal single receivers.
d) On the left, below the data selector Bins: will be a drop down control - same control as in Climatic > PICSA > Crops for Planting Date etc - with the default values above.
e) Count: and Days could be up-down controls with those defaults and allowing integer values, maybe from 1 upwards, but waiting for Emily to reply.
f) For the Omit Months have a checkbox, Maybe labelled Dry Month(s). If checked there is a button Omit Months. Then I hope it will be easy to copy the Omit Months sub-dialog from the Climatic > Check Data > QC Rainfall dialog.
g) The Distribution and also Markov TRUE/FALSE are presumable simple drop down. We have an example in the Define New Filter drop down.
h) Then (not given above) a Checkbox, default unchecked labelled Mode Options. Then an Options button, disabled initially.
i) Then (omitted above) Store Result label and a default name Estimated, that goes up by 1 each new variable. We assume @lilyclements will be changing her function to write a variable.
j) then the usual bottom buttons.
k) Then the R-code will call her function and add the column she produces to the data frame.
l) Then we need to de cide where it goes. Maybe in Climatic > Compare, but I think Climatic > Model, maybe first one there, and just called Outfilling.

@jkmusyoka @lilyclements ,do we need to add station_to_exclude as a default argument? The function won't run without it.

@lilyclements , concerning the target_months argument in the function, it expects a numeric vector specifying the months. Roger suggested copying the Omit Months sub-dialog from the Climatic > Check Data > QC Rainfall dialog but this produces an object as shown below;

filter_months <- instat_calculation$new(type="filter", function_exp="!month_abbr %in% c('Jan','Feb','Mar')", calculated_from=list("zambia_data"="month_abbr"))

Summarising @rdstern's requests for changes in the R code:

1. We want to use "our" version of the Omit Months (see the Rainfall QC dialog)
2. What did Emily say about the Station to Exclude control. Can we omit it? (We have filters in R-Instat)
3. Need a Store Result control at the bottom- the same as in the Calculator and other dialogs. It currently makes a new data frame, despite just producing a single column of the same length. It will be so much easier to use when it just adds to the existing dataframe.
4. Do we need a More button towards a sub-dialog that facilitates changing the other arguments, or can that wait?
5. Have an option of a specified random number seed. The whole thing is comparisons, and this includes comparisons of the different methods - which should use the same sequences.

Just wanted to have this written together somewhere. I can make the changes in the R package

Just to respond to the items above:

1. For now I suggest we stick with Emily's Omit Months, which Vitalis has already included. I think - seeing James' sample data that having it as now is fine and also permits the drop down to give different options.
2. I wasn't clear on the response from Emily. If her Station to exclude is different from a filter, then we may need to change from the current dialog, which only allows a maximum of 1 station to be excluded. (Note if her exclusion just leaves the station(s) out totally, then it is the same as a filter. It may be that she excludes from part of the analysis, e.g. the adjustment phase, while still including it, in the outfilling. That would be different and then we should allow both.)
3. Yes please.
4. Id like to check with Emily, and (at the same time) press her again on other resonable values for the bins, count and days.
5. Yes please. I hope a single number, in an up/down control perhaps from 1 upwards, with enough space for (say) 1 million. Can type in and default is perhaps 099. I wonder what happens now? So do we always want to specify the initial seed, or do we leave the default as now - where it is unknown to us. Then there is an Initial seed checkbox, with the up-down only visible when checked. Default maybe unchecked?

@rdstern

1. Great, thank you.
2. In the R code it looks like it is different to a filter. The station to exclude is in the calibration part, when setting monthly parameters and bits, but when generating the values it does that to the full data.
3. I'm on this now. I've given this as an option to return as a data frame or as a numerical variable. By default it will return as a numerical variable. @Vitalis95 can you change the R code to say that the resulting saved item is now a new column not a new data frame. You will need to update the R function by devtools::install_github("IDEMSInternational/OutfillingR")
4. Great, let's set up a meeting with her to discuss.
5. I was thinking we have it with an "Initial seed" checkbox, and that gives the up-down. Otherwise, no seed runs. What do you think?

@lilyclements

On 2, in the function is it for a maximum of 1 station, or can we have more in your Exclude. The current dialog is just for 0 or 1 station?
On 5 I completely agree. We have a checkbox, default unchecked, and no seed then runs.

@rdstern It can now have multiple stations excluded. Before you pointed that out, it could only have one excluded (so you need to redownload the package)

@lilyclements I see the new version can also output to a variable.
I assume you will also soon include the new random seed, unless that is already there?

My concern now that we can exclude many stations is on our routine use of this function, if it is the only one we have.

Let's take Eastern Province, where there are 5 main stations, then over 100 volunteer stations, and quite some automatic stations. So let's consider the work in 2 stages:
a) First we just use the 5 main stations. We "pretend" sections are missing, e.g. to 2010 and then estimate them. Our aim is the find which satellite algorithm is to be preferred, and also how the tamsat (say) data are to be adjusted in the best way.
b) I presume the optimal adjustment will come from a run of the code with all the data from the 5 stations?
c) Then we want to apply the optimal solution to (say) 30 stations, with short records in Eastern Province? So we now have 35 stations, with 5 used to get the adjustments again, and then they are applied to the other stations. I would like to assume the adjustment is the same, if the randome number seed is the same? I wonder if that is true.

@rdstern setting a seed is there under the parameters “set_seed” (default NULL)

I am unsure if I 100% follow. The monthly parameter (adjustment?) values are the same for all rows (so across the stations) because of how the function is set up, I believe. In the code, the “monthly parameter” values are all set, and then there is a loop through each row of the data using these values to generate new values.

I did check on the generation of random values (e.g. a random binomial distributed value). Setting a single seed does mean that when we randomly generate a variable it does alter for each iteration of the loop (so each row does get a randomly distributed value).
(And between two function calls, it gives the same results - I.e. if I ran do_outfilling twice with the same seed in them then the resulting generated rainfall values are the same).

@rdstern It can now have multiple stations excluded. Before you pointed that out, it could only have one excluded (so you need to redownload the package)

@rdstern , how can we implement this in the dialog?

@Vitalis95 here are my suggestions for minor revisions to the current outfilling dialog. Here it is:

a) Move the Station to Exclude to the left hand side and add a checkbox, default unchecked. Call it Station(s) to Exclude to prepare for a later change, but keep the control as it is for now. (The other changes should be quick, and it will be good to get it working with them. Initially we don't need the stations to exclude option, so leave those changes to next week.)
b) Move the Dry Months control to become the second on the left (with a checkbox) - also as now, with the default unchecked.
c) Add a new Random Seed checkbox under that - so as the third. Default also unchecked. If checked it reveals quite a wide up-down which can give positive integers, and default 999. Typing in, is allowed and 1000000 could be visible. (I understand this is a new argument in the revised @lilyclements code we need to download.)
d) Then have Bins and Count on the left as well. Left-justify the labels and also keep the second parts of the control left justified - as you have now.
e) Move Days, Distribution and Markov to the right-hand side. The 2 drop downs for Distribution and Markov can be narrower as there are no more entries to come there.
f) Add the Store Result Checkbox - with the Position button - same as for the calculator - at the bottom left. Default is checked, i.e. to add a column to the existing data frame

Looking forward!

Once this is all working, the item that remains is to make the Stations to Exclude a control that can exclude multiple stations that you select. It needs a checkbox for each station and we have that elsewhere. It is the control in the Filter from Factors sub-dialog, and also in the filter sub-dialog when you choose a factor to filter on.
So I suggest it changes into a button and calls that same control.

After our meeting with Emily today, it is apparent we want to return two columns: "generated_rainfall" and "outfilled_rainfall".

@Vitalis95 is there a way that we can return two variables from one function into a data frame in R-Instat?
If there isn't, then is it possible in R-Instat to return a data frame, which just replaces our current data frame that we read in? Thanks.

In addition, @rdstern do we want to have as "Stations to Exclude" or "Stations to Include"? I'm not sure which is easier for the user in R-Instat. I can do that we have "everything" by default, but otherwise you state which stations to include.

@Vitalis95 the important change is to produce the 2 variables. We produce multiple variables into the same data frame in some other dialogs, e.g. the Prepare > Column: Numeric > Transform (with multiple options), and the Prepare > Column: Text > Split dialog.

Here is the current dialog:

I suggest you change Store Result(s) into Prefix for Results: Then the default prefix is the name of the estimates variable, e.g. tamsat_ with an underscore here. It produces two variables, with the first being the current one called (here) tamsat_est, and the second being tamsat_out.

@lilyclements can you improve on the est and out addition to the names?

The tamsat_est is the variable currently produced - which is for every day, not just the days missing in the data.

The second is simply ifelse(is.na(rainfall), tamsat_est, rainfall) - maybe coded better?

I've forgotten what happens if you untick the storing checkbox? Lily, do we need to produce a new data frame? If not, then remove the checkbox so it always saves the 2 columns.

We could merge after this, or include the next change (or more) at the same time. The change above doesn't need any change from Lily.

Lily and Emily are suggesting that the Stations to exclude simply becomes Stations:. The checkbox remains, with the default unchecked. The pull-down is replaced by a button, which could be labelled Include . Then the control is easy, because it is the same as the Filter from Factor sub-dialog:

You'll have to discuss with Lily, how to manage whatever it returns!

@rdstern thanks for linking to the similar dialogs - I wasn't sure how to return the R code to have two new columns to join the data frame, but I think this is now done.

I have added in the addition of outfilled_rainfall (as well as generated_rainfall, which is what we had before -- and outfilled_rainfall runs an if statement like you pointed out)

I have also made a change in the stations_to_exclude variable. This is now stations_to_include. If this is NULL, then it will include everything, otherwise, it will only include the stations you state.

Finally, the results were slightly different to what Emily was getting with her code. I have now fixed this after finding where this is coming from.

Next Steps for @rdstern and @Vitalis95:

1. Reupdate the outfillingR package - devtools::install_github("IDEMSInternational/outfillingR")
2. It now returns two columns instead of one. I think this should follow a structure like @rdstern pointed out in the Transform dialog.
3. Change station_to_exclude to be stations_to_include in the R code (and then add an appropriate control like @rdstern has outlined above)