Revision of project creation workflow : Visibility of data extracts in the project creation workflow

[susmina94]

Describe the bug

I currently face the following issue with the project creation workflow:

1. I select a random form category and complete the project setup but I don't see any data extracts in tasks
2. I upload an AOI where there are no relevant data extracts, I reach the Step 5 but can't use the task splitting algorithm because there are no buildings/road data extracts
3. I can't validate the data extracts before a project is created

I would suggest the following workflow for the project setup.

Step 3 should include:

1. Selection of Feature Category (Rename Form Category to Feature Category)

2. Add a description just below the Feature Category title (Choose the feature type you want to survey)

3. The user will select from the dropdown options that we have as now.

4. Allow the user to select the type of feature category i.e. Point, Line, Polygon

5. The feature type should also be based on the feature category (For eg: If I choose buildings in feature category, I shouldn't have the option to choose line in the feature type options)

6. Now the user should be able to see the extracts as per their selection on the right side of the map
   7.@nrjadkry, the backend should provide the data extracts to the frontend at this point

7. The step 7 is very important because the user should be able to validate their data extracts before they move ahead with the workflow.

8. This will also resolve the conflict where there is no relevant data extract as per the user's feature category
   10 If the data extracts match the user's expectation, they move ahead, if not they can reselect the feature category

Note: If there are no data extracts in that AOI, the user should be displayed the message stating "There are no OSM data extracts around the AOI relevant to the feature category you chose. You may either choose to upload your custom data extracts or select a different feature category. You may still proceed ahead without any data extracts."

Step 4 should include:

1. Selection of form type (Default or custom)
2. The user can either select the default form which provides the user with form relevant to form category selection
3. The user can also add custom xls forms

Step 5 would be the splitting of tasks as it is.

Screenshots

[spwoodcock]

Great write up - thanks Susmina 👍
So the idea would be to swap step 3 and 4 around right?
That makes sense to me.

Step 3

1. Selection of Feature Category (Rename Form Category to Feature Category)

Completely agree on the Form Category --> Feature Category renaming.

2. Add a description just below the Feature Category title (Choose the feature type you want to survey)

What would the description in the Form Category be used for?

5. The feature type should also be based on the feature category (For eg: If I choose buildings in feature category, I shouldn't have the option to choose line in the feature type options)

For this we would need a mapping between feature (form) categories and feature types. {'buildings': 'polygon'} etc. If we had that & the user selects an existing Feature Category we provide, then we could hide the Feature Type selection (select it by default). If the user uploads their own extract, then we can allow them to select the Feature Type themselves.

6. Now the user should be able to see the extracts as per their selection on the right side of the map
   (the backend should provide the data extracts to the frontend at this point)

This needs some thought - as generating the data extract could take some time & will block the user flow. Instead, could we add validation only if the user uploads a custom extract (i.e. just check there are feature present to continue)? Then if the user wants to generate an extract, we can continue the task creation as usual. If the data extract generated is empty, we can halt the task splitting, setting the status of the project to UPDATE_REQUIRED, prompting the user go to back and update the project values (the AOI, or category).

Summary

• Validation of the extract during the project creation may be tricky.
• Perhaps we should instead set the task to UPDATE_REQUIRED if the data extract is empty / not valid when the background tasks run (task splitting etc).
• The user can return the the project, update some values, and re-run the task generation.

Actions

Data Extracts

1. Create a mapping between Category (Form) Types we provide, and their Feature Type (point, line, or polygon).
2. Remove the 'OSM Feature Type' selection in the frontend.
3. If the user selects our predefined Category Type, then pre-select the Feature Type based on the mapping above.
4. If the user uploads their own data extract (i.e. does not select our Category Type), then validate the data extract contains geometries. And also check for what type the geometry is - using this we can also pre-select the Feature Type.

DB Schema - Project Status

Currently we have the projectstatus enum:

    'ARCHIVED',
    'PUBLISHED',
    'DRAFT'

Should we add the additional UPDATE_REQUIRED? Can anyone think of a better name?

Task Splitting

1. Prior to running the task splitting, add validation on the backend to determine if the data extract is valid.
2. If not (i.e. it is empty), then don't continue task generation & set the project status to UPDATE_REQUIRED.
3. Allow the user to return to the project on the frontend, update values, then submit again.

[susmina94]

Reply to Point 2:
The description would make the user clear on why would they use the "select the feature category" field and highlight its use in the project creation?

Agree with Point 5

Reply to Point 6
There are cases where the data extracts around few AOI do not exist in OSM as well. In such cases, the project creation workflow at the moment doesn't get interrupt but the user might be confused why they couldn't see any data extracts in that area.

Also, as a user I would like to know if there are/aren't data extracts in that area before I complete the entire project setup and go back to creating another one after realizing there are no data extracts.

But I do agree it could result in being slow , take some time & will block the user flow.
Any suggestions regarding this workflow maybe from @ivangayton or @robsavoye

[spwoodcock]

2. Ah makes sense - I thought you meant a user provided description. A descriptions to tell help them decide between selecting a feature category vs uploading their own extract is useful 👍

3. We have two options:

Option 1: Validate Data Extract during Project Creation

• When 'Use OSM data extract' is selected, the user clicks a button like 'Get data'.
• This takes some time to process / download via osm-rawdata.
• The extract will be generated as flatgeobuf, so could be viewed immediately in the map on the right (for inspection by the user).
• The advantage is immediate validation by the user, but the disadvantage is waiting time (slows down the project creation flow).
• My concern here is that if we generate the data extract, then the user abandons the project creation, we will have a generated extract that will persist in S3, but isn't linked to a project.

Option 2: Validate Data Extract after Project Creation

• This has worse UX, but may be more practical (testing will let us know).
• The user continues with project creation to the end.
• The data extract is generated and validated.
• If the data extract is not valid, the user is informed (the project flagged), and they have to go back and update the project creation values to try again.

Option 1 is preferred from a UX perspective, but we should test it & address the issue of data extracts generated if the user doesn't complete project creation.

[varun2948]

We should make this data extract visualization thing optional when we dont use custom upload data extract and let user decide whether they want to see the feature with some waiting time or they just want to move on with the task without seeing in the project creation part. What do you think @spwoodcock doing this way?

[spwoodcock]

Great idea @varun2948 - best of both🎉