Upload User and Catchment data

[mahalakshme]

Need:

1. Currently we are not doing validations on data/presence of data. This leads to issues like below:

• Failing of sending Glific messages since username not present (When configured in User Messaging Config) - https://avni.freshdesk.com/a/tickets/3682

2. We expect country code to be mentioned for phone numbers and we dont give user friendly message when it is missing. And neither it is expected when user creates users via webapp. And we have hardcoded to support only Indian mobile numbers.

https://avni.freshdesk.com/a/tickets/3413, https://avni.freshdesk.com/a/tickets/2863, https://avni.freshdesk.com/a/tickets/2494, https://avni.freshdesk.com/a/tickets/2663

3. Sometimes error messages are not understandable for the users eventhough we provide readable error messages.
   https://avni.freshdesk.com/a/tickets/3493

4. Validation on headers not done:
   https://avni.freshdesk.com/a/tickets/3568

5. When a user belongs to multiple user groups, they need to be separated by |. But everywhere else generally the separator is ,
   https://avni.freshdesk.com/a/tickets/3902

6. Other tickets related to error messages and location hierarchy spelling mistakes
   https://avni.freshdesk.com/a/tickets/2496
   https://avni.freshdesk.com/a/tickets/2863, https://avni.freshdesk.com/a/tickets/2494

AC: Phone number related

• Currently we have this regex validation for mobile number "^\\+91[0-9]{10}". Remove it and handle it when receiving the response for AdminCreateUserRequest by catching the exception InvalidParameterException and checking for 'Invalid phone number format' error message. In this case, the error column in CSV need to display as 'Country code not present or incorrect'. This is because validity of mobile numbers are not checked on AWS side.

• Dont make it mandatory to enter + for phone number. If it is not entered, add + before sending the request to AWS cognito and inserting into db. If it is entered as well fine. This is because entering + on excel for a mobile number converts it into exponential format.
• When the mobile numbers are prefixed with 91 or +91 make sure the number of digits that follows are ten in number.
• When values for mandatory fields not present in the uploaded data then display the error in the error column of CSV as Value for 'Mobile number' column is missing in the respective row.

AC: Others

• CSV upload to work with comma separator for user groups as well instead of bar symbol.
• When location in the column 'Location with full hierarchy' is not present in Avni, then error message is 'Provided Location does not exist. Please check for spelling mistakes '%s''. Instead make the message to 'Provided Location does not exist in Avni. Please add it or check for spelling mistakes '%s''' . Reason: When we say doesn't exist, users dont realise we mean in Avni context.
• Even if descriptor row present/not present as part of uploaded data, upload of data should work fine.
• In the location with full hierarchy, the upload should have the same behaviour, independent of the case of the location. Say Kerala->Kasaragod is the location hierarchy, it should work even if the upload file contains, kerala->kasaragod

References:

https://countrycode.org/
https://www.quora.com/How-is-each-country-code-defined
Phone number format as per cognito user pool mentioned here: https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-settings-attributes.html

[vinayvenu]

Need QA scenarios for the phone number changes in behaviour

[petmongrels]

Phone numbers

• https://avni.readme.io/docs/phone-number-formats
• error message for phone numbers will be based on the library error messages
• +91 1111 1111 is invalid phone number, but the error will come only when saving as the front end validator seems to allow it.

Testing

• Glific contact addition
• Create/update users with phone numbers
• Phone numbers in forms

Location with hierarchy

• The code already handles case insensitive matches. The tickets are quite old.

Descriptor row

• not implemented, as the sample story is not done yet.
• adding this AC to the sample card (avni-product/issues/1531)

[AchalaBelokar]

• I upload this csv on staging maha@APF

APF_Odisha_user.csv

errors-APF_Odisha_user.csv

[himeshr]

Sample Error file with validations across rows.
449159b4-9732-474c-bab0-cb39fb3243d0.csv

[1t5j0y]

Suggest replacing comma as a separator (newline or semi colon should work well) when there are multiple error messages for a row so there are no columns without headers.

[himeshr]

Suggest replacing comma as a separator (newline or semi colon should work well) when there are multiple error messages for a row so there are no columns without headers.

I did this on purpose, so that the different errors for the row show up in separate columns.. felt it was more user-friendly that way.

[AchalaBelokar]

• location is correct but comma is missing then it is showing the location spelling is mismatched ..

[himeshr]

#1214 (comment)
@AchalaBelokar we cannot perform a fuzzy search for each entry to determine if the user made a typo error while specifying the location address.. therefore, if we do not find the location based on exact search we are showing a generic error msg like "Provided Location does not exist in Avni. Please add it or check for spelling mistakes..."

As discussed in today's stand-up lets hold off testing this till discussion with developer.

[petmongrels]

we discussed this. this is not a bug.

[mahalakshme]

env: prerelease, org: JSS Singrauli UAT -> Admin -> Upload -> Select 'Daily Attendance form' -> Upload the below file

Upload this file: sample-Encounter---Daily Attendance Form---Phulwari.csv

Error file: errors-sample-Encounter---Daily Attendance Form---Phulwari (3).csv

Following issues:

• Error handling of 'Users and Catchments' file should not overflow to other CSV uploads. This case can also be added to tests.
• When multiple columns are missing, the entire error message should appear in single column. Same when multiple unknown headers present
• Track Location, Enable Beneficiary mode,Date Picker mode etc., are not mandatory headers.

[petmongrels]

first one wasn't an issue, as it was by mistake uploaded as user and ctachment file. fixed rest two.