Operations maintained even when data list is overwritten

[edmoss345]

Something that @fagiothree and I discovered while looking at the crisis deployment. We have a scenario where the 'onboarding_data' is initially defined in this workbook in row 7. In this workbook there is a filter operation which filters out various rows.

https://docs.google.com/spreadsheets/d/1M3ohkdFNWzYQdMBdlnPP2_pi3RttKGXAvCUBgYLvuHA/edit?gid=0#gid=0

In the palestine deployment we have made a new 'onboarding_data' data list as shown in the below workbook in row 14.

https://docs.google.com/spreadsheets/d/1tLDsUe-NxOxpxOBuTJmopOnP9dOcmLVHjO2YQxGxe28/edit?gid=1353459808#gid=1353459808

So we are overwriting the data list, but we noticed that the filter operation from the first sheet is still being applied. @geoo89 could you outline what the process is here. I think desired behaviour would be that when we redefine a data list for a deployment, the only filter operations that we want to apply would be those that we specify on the new data list, not keep any existing operations that were in place on the old data list.

[geoo89]

All rows are processed in order of inclusion.

In this case, your sheet_name and your new_name match, so first your onboarding_data is loaded from somewhere (not evident where that is from the sheets you linked) and stored under that name, then the filter is applied to onboarding_data and the result saved as onboarding_data (overwriting the previous version), and then the second filter is applied to onboarding_data and the result saved as onboarding_data (overwriting the previous version again). This is in line with your "desired behavior", because your "old datalist" and "new datalist" are the same.

I don't really know where the original onboarding_data comes from. Is it the same for both deployments? If palestine included an onboarding_data from scratch before filtering it, then the previous filtered result would be overwritten and the palestine filter be applied to that result (assuming that all of this comes after the crisis inclusions).

If both deployments share the same initial dataset that they filter, you should consider storing the result of the filter operation under a different new_name. Then the second filter will still be applied to the original dataset, and overwrite the other filtered dataset.

[istride]

onboarding_data is first defined in parenttext-templates, which is the parent of crisis-base, which is the parent of crisis-palestine. The filtering first occurs in crisis-base, overwriting the original onboarding_data. The intention is for crisis-palestine to redefine onboarding_data (again) with a data sheet from the same repo (not parenttext-templates). This fails because a content index row with type data_sheet and no operation defined is treated as a 'concat' operation, which will fetch the already existing (filtered) definition of onboarding_data rather than load the new sheet.

One workaround might be to force a new sheet to be loaded by using a unique sheet name instead of onboarding_data in the crisis-palestine repo.

type	sheet_name	new_name	data_model
data_sheet	onboarding_data_palestine	onboarding_data	OnboardingStepsModel

[istride]

@geoo89 I have created a failing unit test that replicates this issue.