[$500] Invisible characters are not being treated as empty

[kbecciv]

If you haven’t already, check out our contributing guidelines for onboarding and email [email protected] to request to join our Slack channel!

---

Version Number: 1.3.99.0
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:

Issue found when executing PR #31259

Action Performed:

1. Go to staging.new.expensify.com
2. Log in with any account
3. Go to Setting - Workspace
4. With workspace selected go to Settings,
5. Try invisible characters used in emojis (e.g.r \u200D), you can use this tool to copy it (paste the character in the JS/Java/C section, click Convert, and then Copy in Characters section)

Expected Result:

Invisible characters are being treated as empty.

Actual Result:

Invisible characters are not being treated as empty

Workaround:

Unknown

Platforms:

Which of our officially supported platforms is this issue occurring on?

• Android: Native
• Android: mWeb Chrome
• iOS: Native
• iOS: mWeb Safari
• MacOS: Chrome / Safari
• MacOS: Desktop

Screenshots/Videos

Add any screenshot/video evidence

Recording.5417.mp4

View all open jobs on GitHub

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~01a9d7f6443e8e5999
• Upwork Job ID: 1724829108425584640
• Last Price Increase: 2023-11-15

[melvin-bot]

Triggered auto assignment to @greg-schroeder (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~01a9d7f6443e8e5999

[melvin-bot]

Bug0 Triage Checklist (Main S/O)

• This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
• This bug is not a duplicate report (check E/App issues and #expensify-bugs)
  • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
• This bug is reproducible using the reproduction steps in the OP. S/O
  • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
  • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
• This issue is filled out as thoroughly and clearly as possible
  • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
• I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @akinwale (External)

[Juggernaut-98]

Proposal

Please re-state the problem that we are trying to solve in this issue.

Invisible characters are not being treated as empty

What is the root cause of that problem?

In side the file src/libs/StringUtils.ts we have a function to remove invisible characters removeInvisibleCharacters that seems to be missing validation against u200D.

What changes do you think we should make in order to solve the problem?

We should add validation for u200D as well.

What alternative solutions did you explore? (Optional)

[bernhardoj]

@kbecciv the app already treated \u200d as empty. You need to copy the unicode character.

[Tony-MK]

Proposal

Please re-state the problem that we are trying to solve in this issue.

Invisible characters are not being treated as empty in the edit legal name page

What is the root cause of that problem?

The root cause of this problem is that the validation of both the firstName and lastName values in LegalNamePage doesn't filter out format characters \p{Cf} because the onValidate function in the FormProvider component is used with many other components that accept emojis. Some emojis use invisible unicodes such as \u200D and are not part of the alphabet or Latin characters(CONST.ALPHABETIC_AND_LATIN_CHARS). [Examples for emojis that use \u200D](https://unicode.org/emoji/charts/emoji-zwj-sequences.html. However, LegalNamePage requires the values to match the CONST.ALPHABETIC_AND_LATIN_CHARS regex before submission. This validation step happens when the ValidationUtils.isValidLegalName function is called.Furthermore, the CONST.ALPHABETIC_AND_LATIN_CHARS regex validation stage comes before checking if the value is empty. Hence, it will fail because an invisible code key (e.g., \u200D) is present in the form value; that's why it's not treated as empty.

App/src/libs/StringUtils.ts

Lines 45 to 47 in 5e01376

	// Remove all characters from the 'Other' (C) category except for format characters (Cf)
	// because some of them are used for emojis
	result = result.replace(/[\p{Cc}\p{Cs}\p{Co}\p{Cn}]/gu, '');

App/src/pages/settings/Profile/PersonalDetails/LegalNamePage.js

Lines 54 to 64 in 5e01376

	const validate = useCallback((values) => {
	const errors = {};
	if (!ValidationUtils.isValidLegalName(values.legalFirstName)) {
	ErrorUtils.addErrorMessage(errors, 'legalFirstName', 'privatePersonalDetails.error.hasInvalidCharacter');
	} else if (_.isEmpty(values.legalFirstName)) {
	errors.legalFirstName = 'common.error.fieldRequired';
	}
	if (values.legalFirstName.length > CONST.LEGAL_NAME.MAX_LENGTH) {
	ErrorUtils.addErrorMessage(errors, 'legalFirstName', ['common.error.characterLimitExceedCounter', {length: values.legalFirstName.length, limit: CONST.LEGAL_NAME.MAX_LENGTH}]);
	}

Values in FormProvider are formatted with the ValidationUtils.prepareValues function, which passes all strings to the StringUtils.removeInvisibleCharacters function. The StringUtils.removeInvisibleCharacters function does not remove format characters \p{Cf} because they are used for emojis in other forms which accept emojis.

App/src/libs/StringUtils.ts

Lines 45 to 47 in 5e01376

	// Remove all characters from the 'Other' (C) category except for format characters (Cf)
	// because some of them are used for emojis
	result = result.replace(/[\p{Cc}\p{Cs}\p{Co}\p{Cn}]/gu, '');

What changes do you think we should make in order to solve the problem?

Besides validating the legalFirstName and legalLastName values against CONST.ALPHABETIC_AND_LATIN_CHARS, I believe additional validation in the validate function is required to ensure invisible characters don't get through for each value. Replacing the validate code before the value length validation with something similar to the code below should ignore the invisible characters.

Firstly, since emoji's are not part of CONST.ALPHABETIC_AND_LATIN_CHARS, we check if they're present in the values. We check by value.match(CONST.REGEX.EMOJIS). If present, we display the same error (privatePersonalDetails.error.hasInvalidCharacter) as if we are checking for CONST.ALPHABETIC_AND_LATIN_CHARS.

Secondly, we check if the string is empty using the ValidationUtils.isRequiredFulfilled function. If empty, display the common.error.fieldRequired error. Already, the ValidationUtils.isRequiredFulfilled will remove the invisible in the StringUtils.isEmptyString function before checking if it is empty. The StringUtils.isEmptyString function removes all invisible characters with an empty string using the CONST.REGEX.INVISIBLE_CHARACTERS_GROUPS and CONST.REGEX.OTHER_INVISIBLE_CHARACTERS regex. We replace the invisible characters in the value like the StringUtils.isEmptyString function.

App/src/libs/StringUtils.ts

Lines 17 to 23 in 5e01376

	// \p{C} matches all 'Other' characters
	// \p{Z} matches all separators (spaces etc.)
	// Source: http://www.unicode.org/reports/tr18/#General_Category_Property
	let transformed = value.replace(CONST.REGEX.INVISIBLE_CHARACTERS_GROUPS, '');
	// Remove other invisible characters that are not in the above unicode categories
	transformed = transformed.replace(CONST.REGEX.OTHER_INVISIBLE_CHARACTERS, '');

Finally, validate the value with the CONST.ALPHABETIC_AND_LATIN_CHARS regex to see if all the characters are alphabets or Latin characters. We use the ValidationUtils.isValidLegalName function.

[situchan]

This is not bug. Can be closed