1/n Adding Android support for Accessibility TtsSpan API - Support for accessibilitySpan and accessibilityLabel props in nested Text #35130

fabOnReact · 2022-10-29T12:21:32Z

Summary

Adding accessibilityUnit prop for Nested Texts and accessibilityRoles for different types of TtsSpans

An explanation of the use case is as follows:

Digits are spelled 1 0 3 0 instead of one thousand thirty
Ordinal number spelled as first instead of one
Telephone spelled as 0 1 1 1 2 8 8 7 7 instead of 11 millions 128 thousand ...etc
Adding option to specify if 10m stands for 10 meters or 10 miles, or 10 minutes
Currency 1.000.000 IDR read as one million rupias
Verbatim spells react-native as r e a c t n a t i v e
The screenreader announcement of nested text can be over-ridden with accessibilityLabel for any additional use case (previously not supported on Android and iOS)

Support for more TYPE_DATE, ELECTRONIC, FRACTION, TIME, DECIMAL, DIGITS, MONEY to be added in upcoming PRs (2/n).

Investigating the implementation of TextAttributesProps on Android and Fabric CPP.

Nested Text components are represented on Android as TextView spans. Their props are passed from the React Text component through CPP to the JAVA ReactAndroid API using TextAttributes. The accessibilityUnit is saved in Fabric attributedstring and retrieved from ReactAndroid TextLayoutManagerMapBuffer.
Background or Foreground color, custom letter spacing, and other text attributes for nested text are retrieved from TextLayoutManagerMapBuffer using the key FR_KEY_TEXT_ATTRIBUTES and represented on android using TextView spans.
The TextAttributes saved in FR_KEY_TEXT_ATTRIBUTES are sent from Javascript to ReactAndroid using the NDK Fabric CPP API, which uses Java Native Interface to share data between JavaScript, CPP, and Java without JSON serialization (bridge).

CPP configurations from fabOnReact#3

Adds the accessibilityLabel to fabric TextAttributes CPP and Java API (part of PR #35130 Adding support for Android Accessibility TtsSpan API). Implementing the above functionality requires adding a new prop to the react-native API responsible for managing attributes of nested Text.

Java and CPP TextAttributes manage the functionality.
Nested Text does not correspond to a Widget on Android but an Android TextView Span.
An example of Text attributes is accessibilityRole, font-weight, and backgroundColor.
An example of Paragraph attributes is textBreakstrategy. They are updated with different mechanisms because they may change the paragraph layout.

Decreasing fragmentation with web, using aria-label instead of accessibilityUnit (2-10th February)

Investigate HTML and react-native-web use of aria-label and role to announce tts-spans (the text of telephone numbers, digits, dates, and measures like seconds or meters). See more info in the issue-comment-1414330273.
Video test cases and code snippets of the difference in behavior between HTML, react-native-web, and react-native (issue-comment-1414330273).
Migration of telephone, measure, and other accessibilityRoles to accessibilitySpan prop.
Adding enum type AccessibilitySpan to CPP, JavaScript, TypeScript, and Java.
Migration of accessibilityUnit to accessibilityLabel for Android nested Text.
Setting AccessibilitySpan default to TYPE_TEXT.  The accessibilityLabel prop is not implemented on the iOS or Android Nested Text component (video-test-android, video-test-ios). The default implementation uses TYPE_TEXT to have aria-label override the default TalkBack screenreader announcement (videos-of-the-functionalities).
Remove roles TYPE_DATE, ELECTRONIC, FRACTION, TIME, DECIMAL, DIGITS, and MONEY.
Try to fix ReactTextTest layout tests (for ex. testTextAlignJustifyApplied).  The tests use a mocked version of UIManager and create a ReactTextView with createView. The test fails to verify the Text layout because of the missing yoga binding. The solution is replacing the original yoga implementation with a mock. More info at ReactShadowNodeImpl, YogaConfigFactory#create.
Attempt to test the implementation of ReactTextView and ReactTtsSpan using the method ReactTextViewManager#createViewInstance.
Attempt to test the implementation using ReactTextViewManager#updateState and a mocked ThemedReactContext (SimpleViewTest).

This PR fixes #30849

Additional Notes

ReactMapBufferViewManager.kt and ReactMapBufferPropSetter.kt
Text/ParagraphAttributes, TextLayoutManagerMapBuffer, ReactStyleDiffs are built using MapBuffers. SurfaceMountingManager uses by default ReactMapBufferViewManager.kt or in alternative the corresponding ReactViewManager. In the case of a controlled TextInput. The TextInput replaces the typed value (Gffd. G) with the new value (Gffd._____G). The new props are sent to the ReactEditTextViewManager using the ReactStyleDiffMap. ReactMapBufferViewManager.kt is not used.
Related PR adding accessibilitySpan PropType to DeprecatedTextPropTypes react-native-deprecated-modules#17

Changelog

[Android] [Added] - 1/n Adding Android support for Accessibility TtsSpan API - Support for accessibilitySpan and accessibilityLabel props in nested Text

Test Plan

12 January 2023 Updated Tests #35130 (comment)
7 February 2023 Comparision with VoiceOver + Safari #35130 (comment)
7 February 2023 iOS Tests #35130 (comment)
8 February 2023 Testing aria-label on nested text #35130 (comment)

Old Video Tests

#35130 (comment)

2022-10-29.14-23-10.mp4

as showcased in this example, this solution also works with nested text. The text announced in the example below is a random text just to verify that the solution works without problems.

2022-10-28.21-35-42.mp4

analysis-bot · 2022-10-29T12:46:57Z

Platform	Engine	Arch	Size (bytes)	Diff
android	hermes	arm64-v8a	8,481,907	+6,006
android	hermes	armeabi-v7a	7,803,239	+5,674
android	hermes	x86	8,958,289	+6,289
android	hermes	x86_64	8,815,357	+5,636
android	jsc	arm64-v8a	9,118,571	+4,892
android	jsc	armeabi-v7a	8,315,102	+4,555
android	jsc	x86	9,170,503	+5,159
android	jsc	x86_64	9,428,656	+4,518

Base commit: 23607ae
Branch: main

analysis-bot · 2022-10-29T13:15:03Z

Platform	Engine	Arch	Size (bytes)	Diff
ios	-	universal	n/a	--

Base commit: ac54a5b
Branch: main

pull-bot · 2022-10-29T13:26:38Z

PR build artifact for 48a01d0 is ready.
To use, download tarball from "Artifacts" tab in this CircleCI job then run yarn add <path to tarball> in your React Native project.

pull-bot · 2022-10-29T13:26:59Z

PR build artifact for 48a01d0 is ready.
To use, download tarball from "Artifacts" tab in this CircleCI job then run yarn add <path to tarball> in your React Native project.

Notes "The Screenreader stops reading the span text when the focus moves on the AccessibilityLink. The issue may be connected to the latest changes. incompatible types: CharSequence cannot be converted to SpannableString spannableDescription = text.subSequence(start, end); The start and end of the AccessibilityLink need to be updated after changing the length of the string. Complete the functionality and fix all the compile errors" "facebook#35130 (comment) https://github.com/fabriziobertoglio1987/react-native/blob/48a01d006a45c49018d8c3860f40b696b1aa7269/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java#L600"

pull-bot · 2022-10-31T15:03:21Z

PR build artifact for 08d6f43 is ready.
To use, download tarball from "Artifacts" tab in this CircleCI job then run yarn add <path to tarball> in your React Native project.

pull-bot · 2022-10-31T15:10:06Z

PR build artifact for 08d6f43 is ready.
To use, download tarball from "Artifacts" tab in this CircleCI job then run yarn add <path to tarball> in your React Native project.

pull-bot · 2022-10-31T16:00:32Z

PR build artifact for 145303e is ready.
To use, download tarball from "Artifacts" tab in this CircleCI job then run yarn add <path to tarball> in your React Native project.

pull-bot · 2022-10-31T16:04:40Z

PR build artifact for 145303e is ready.
To use, download tarball from "Artifacts" tab in this CircleCI job then run yarn add <path to tarball> in your React Native project.

pull-bot · 2022-10-31T17:36:03Z

PR build artifact for f66e914 is ready.
To use, download tarball from "Artifacts" tab in this CircleCI job then run yarn add <path to tarball> in your React Native project.

pull-bot · 2022-10-31T17:40:27Z

PR build artifact for f66e914 is ready.
To use, download tarball from "Artifacts" tab in this CircleCI job then run yarn add <path to tarball> in your React Native project.

terminating with uncaught exception of type std::length_error: vector (screenshot1, screenshot2) https://www.icloud.com/iclouddrive/0a4Urx1Av554xzHH1hh3YHy9A#first_screenshot https://stackoverflow.com/a/1183709/7295772

fabOnReact · 2022-11-01T21:00:17Z

fabOnReact@110b191

pull-bot · 2022-11-01T21:52:02Z

PR build artifact for f5a3b49 is ready.
To use, download tarball from "Artifacts" tab in this CircleCI job then run yarn add <path to tarball> in your React Native project.

pull-bot · 2022-11-01T21:55:32Z

PR build artifact for f5a3b49 is ready.
To use, download tarball from "Artifacts" tab in this CircleCI job then run yarn add <path to tarball> in your React Native project.

fabOnReact · 2023-02-07T18:58:02Z

2023-02-07.19-57-32.mp4

fabOnReact · 2023-02-07T19:15:19Z

iOS tests https://drive.google.com/file/d/1aQiP7j1qCzWQZwroliu4pqerDob5GxuN/view?usp=share_link

facebook#35130 (comment) https://github.com/facebook/react-native/compare/main...fabriziobertoglio1987:react-native:tts-span-additiona-types?expand=1

fabOnReact · 2023-02-07T21:50:45Z

fabOnReact@34ed14e

P	type	task
3	review	Read react-native-web accessibility API to announce spans
4	test	Test iOS
5	bug	Remove not supported TYPES from PR and move them to a separate branch
5.1	task	Create a separate branch.
5.2	Task	Add support for TYPE_CARDINAL
5.3	task	Add support for TYPE_FRACTION. Retrieve arg numerator and denom. by splitting on “/“
5.4	task	Add support for TYPE_TIME
5.5	task	Add support for TYPE_DECIMAL splitting the number based on TalkBack system language
5.6	task	Remove logic for TYPE_DATE, ELECTRONIC, FRACTION, TIME, DECIMAL, DIGITS, MONEY

P type task
3 review Read react-native-web accessibility API to announce spans
4 test Test iOS
5 bug Remove not supported TYPES from PR and move them to a separate branch
5.1 task Create a separate branch.
5.2 Task Add support for TYPE_CARDINAL
5.3 task Add support for TYPE_FRACTION. Retrieve arg numerator and denom. by splitting on “/“
5.4 task Add support for TYPE_TIME
5.5 task Add support for TYPE_DECIMAL splitting the number based on TalkBack system language
5.6 task Remove logic for TYPE_DATE, ELECTRONIC, FRACTION, TIME, DECIMAL, DIGITS, MONEY

Libraries/Text/Text.d.ts

fabOnReact · 2023-02-08T11:31:33Z

Main Branch - accessibilityLabel not supported on Nested Text

2023-02-08.12-28-37.mp4

PR Branch - accessibilityLabel supported on Nested Text

2023-02-08.12-46-05.mp4

fabOnReact · 2023-02-08T11:53:33Z

P	type	task
1	refactor	The check in MapBuffer on textAttributes.mAccessibilitySpan != none may not be required (link). Removing it may cause to add a TtsSpan every time accessibilitySpan is null (none). The default textAttributes.mAccessibilitySpan value is null.
1.1	task	Add a debug statement and verify if textAttributes.mAccessibilitySpan is set to none when accessibilitySpan is null.
2	refactor	The UNIT (LABEL) conversions in toMapBuffer is not required (you did not update but the functionality did not have issues).
3	refactor	Add aria-label and aria-span
3.1	task	Test accessibilityLabel and aria-label with Nested Text on main branch
3.2	task	Record and upload tests cases on main branch
3.3	task	Record and upload tests cases of aria-label on PR branch

P type task
1 refactor The check in MapBuffer on textAttributes.mAccessibilitySpan != none may not be required (link). Removing it may cause to add a TtsSpan every time accessibilitySpan is null (none). The default textAttributes.mAccessibilitySpan value is null.
1.1 task Add a debug statement and verify if textAttributes.mAccessibilitySpan is set to none when accessibilitySpan is null.
2 refactor The UNIT (LABEL) conversions in toMapBuffer is not required (you did not update but the functionality did not have issues).
3 refactor Add aria-label and aria-span
3.1 task Test accessibilityLabel and aria-label with Nested Text on main branch
3.2 task Record and upload tests cases on main branch
3.3 task Record and upload tests cases of aria-label on PR branch

fabOnReact · 2023-02-08T17:13:04Z

fabOnReact@235eda8

P	type	task
4	refactor	Review Android Studio and XCODE errors
5	refactor	Remove changes to deprecated prop types (PR#15)
5.1	task	Add changes to deprecated prop types (PR#15)
6	review	Update PR Summary
7	refactor	Update comments (link)

P type task
4 refactor Review Android Studio and XCODE errors
5 refactor Remove changes to deprecated prop types (PR#15)
5.1 task Add changes to deprecated prop types (PR#15)
6 review Update PR Summary
7 refactor Update comments (link)

fabOnReact · 2023-02-09T09:16:10Z

P	type	task
8	refactor	Unit Tests for TtsSpan (link, addRootView, ReactShadowNodeImpl, SimpleViewTest, createMockCatalystInstance, ReactInstrumentationTest)
8.1	task	Mock YogaConfigFactory
8.2	task	Read runtime stack of test (at com.facebook.yoga.YogaConfigFactory.create(YogaConfigFactory.java:12))
8.3	task	Read Test cases YogaLayoutDirection. They also use YogaConfigFactory.create()
8.4	task	Write a simple test to verify correct behaviour with specific accLabel and accSpan
8.5	task	Use reactTextViewManager createViewInstance to create instance of ReactTextView
8.6	task	Test: reactTextViewManager updateState adds a TtsSpan to text

P type task
8 refactor Unit Tests for TtsSpan (link, addRootView, ReactShadowNodeImpl, SimpleViewTest, createMockCatalystInstance, ReactInstrumentationTest)
8.1 task Mock YogaConfigFactory
8.2 task Read runtime stack of test (at com.facebook.yoga.YogaConfigFactory.create(YogaConfigFactory.java:12))
8.3 task Read Test cases YogaLayoutDirection. They also use YogaConfigFactory.create()
8.4 task Write a simple test to verify correct behaviour with specific accLabel and accSpan
8.5 task Use reactTextViewManager createViewInstance to create instance of ReactTextView
8.6 task Test: reactTextViewManager updateState adds a TtsSpan to text

fabOnReact · 2023-02-09T12:05:23Z

#35130 (comment) Thanks @NickGerleman.

Should Android accessibilityRole map 1:1 to talkback Role? I.e. expose TTSSpan as a separate API from role, like it seems to be done on Android.

I agree. I will expose TTSSpan as a separate API from role

Is it possible to avoid exposing the APIs which only apply to text spans from view props? One idea was adding a Text specific prop like accessibilitySpan to model the API.

I will add a specific prop accessibilitySpan to model the API.

How does ARIA model this today? And should React Native offer an API that looks similar for web-style usage.

I will try to build the API to look similar to the web. I include below a quick analysis. I'm happy to continue the research and futher study the Web Accessibility APIs.

I believe web uses aria-label combined with different types of roles in all of the scenarios. For this reason, the best option would be:

use accessibilityLabel (aria-label) instead of accessibilityUnit

use accessibilitySpan (aria-span) instead of accessibilityRole (role)

if accessibilitySpan is set on the Text, use the Andoid TtsSpan API to announce dates, time, units. The values are retrieved from accessibilityLabel, but they are announced using the TtsSpan API.

Phone Number
Units
Units - Other types
Units - Currencies
Date
Verbatim
Video Test of the existing TalkBack behaviour #35130 (comment)

Thanks a lot for the code review @NickGerleman. 🙏
I would add the documentation once the PR is approved.
The original comment at #35130 (comment)

fabOnReact · 2023-04-24T15:07:49Z

P type task

8 refactor Unit Tests for TtsSpan (link, addRootView, ReactShadowNodeImpl, SimpleViewTest, createMockCatalystInstance, ReactInstrumentationTest)

8.1 task Mock YogaConfigFactory

8.2 task Read runtime stack of test (at com.facebook.yoga.YogaConfigFactory.create(YogaConfigFactory.java:12))

8.3 task Read Test cases YogaLayoutDirection. They also use YogaConfigFactory.create()

8.4 task Write a simple test to verify correct behaviour with specific accLabel and accSpan

8.5 task Use reactTextViewManager createViewInstance to create instance of ReactTextView

8.6 task Test: reactTextViewManager updateState adds a TtsSpan to text

P type task
8 refactor Unit Tests for TtsSpan (link, addRootView, ReactShadowNodeImpl, SimpleViewTest, createMockCatalystInstance, ReactInstrumentationTest)
8.1 task Mock YogaConfigFactory
8.2 task Read runtime stack of test (at com.facebook.yoga.YogaConfigFactory.create(YogaConfigFactory.java:12))
8.3 task Read Test cases YogaLayoutDirection. They also use YogaConfigFactory.create()
8.4 task Write a simple test to verify correct behaviour with specific accLabel and accSpan
8.5 task Use reactTextViewManager createViewInstance to create instance of ReactTextView
8.6 task Test: reactTextViewManager updateState adds a TtsSpan to text

github-actions · 2023-12-02T05:23:30Z

This PR is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 7 days.

github-actions · 2023-12-09T05:23:39Z

This PR was closed because it has been stalled for 7 days with no activity.

fabOnReact added 2 commits October 29, 2022 13:31

draft solution partially spelling correctly

216a30a

correctly spelling a span text

48a01d0

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 29, 2022

fabOnReact changed the title ~~Adding support for Accessibility TtsSpan API~~ 🚧 Adding support for Accessibility TtsSpan API 🚧 Oct 29, 2022

remove code duplication

145303e

remove code duplication

f66e914

fabOnReact added 6 commits November 1, 2022 12:49

draft java configs to trigger setAccessibilityUnit

3d4e662

Merge branch 'main' into tts-span

1653d6b

draft cpp configs - they trigger runtime

6def7a9

Fabric cpp settings trigger runtime error

7fa58f5

terminating with uncaught exception of type std::length_error: vector (screenshot1, screenshot2) https://www.icloud.com/iclouddrive/0a4Urx1Av554xzHH1hh3YHy9A#first_screenshot https://stackoverflow.com/a/1183709/7295772

cpp mapbuffer configs before rebase

43e7c29

Merge branch 'main' into tts-span

0afdc69

fabOnReact added 2 commits November 1, 2022 22:14

avoid using key from other props

0cd0ac4

avoid using key 24 and use key 99

f5a3b49

fabOnReact added 2 commits November 2, 2022 10:24

remove log messages

08fc9ac

debug - change key number

c2adbb8

fabOnReact changed the title ~~Adding Android support for Accessibility TtsSpan API with accessibilityRole and accessibilityLabel props in nested Text~~ 1/n Adding Android support for Accessibility TtsSpan API with accessibilityRole and accessibilityLabel props in nested Text Feb 7, 2023

removing changes to ReactAccDelegate

7c80bdb

Remove not supported TYPES from PR and move them to a separate branch

34ed14e

facebook#35130 (comment) https://github.com/facebook/react-native/compare/main...fabriziobertoglio1987:react-native:tts-span-additiona-types?expand=1

fabOnReact mentioned this pull request Feb 7, 2023

removing changes to accRoles facebook/react-native-deprecated-modules#16

Merged

rename UNIT to LABEL

23241dd

fabOnReact commented Feb 7, 2023

View reviewed changes

Libraries/Text/Text.d.ts Show resolved Hide resolved

Merge branch 'main' into tts-span

112ed6f

fabOnReact changed the title ~~1/n Adding Android support for Accessibility TtsSpan API with accessibilityRole and accessibilityLabel props in nested Text~~ 1/n Adding Android support for Accessibility TtsSpan API with accessibilitySpan and accessibilityLabel props in nested Text Feb 8, 2023

fabOnReact mentioned this pull request Feb 8, 2023

adding accessibilitySpan PropType to DeprecatedTextPropTypes facebook/react-native-deprecated-modules#17

Draft

Review Android Studio and XCODE errors

235eda8

update comments

d536cf7

fabOnReact added 2 commits February 9, 2023 12:20

update comments

c154196

update check on accSpan and accLabel

76ce49a

fabOnReact marked this pull request as ready for review February 9, 2023 12:05

github-actions bot added the Stale There has been a lack of activity on this issue and it may be closed soon. label Dec 2, 2023

github-actions bot closed this Dec 9, 2023

fabOnReact mentioned this pull request Jul 23, 2024

De-duplicate building Spannable on Android #39630

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

1/n Adding Android support for Accessibility TtsSpan API - Support for accessibilitySpan and accessibilityLabel props in nested Text #35130

1/n Adding Android support for Accessibility TtsSpan API - Support for accessibilitySpan and accessibilityLabel props in nested Text #35130

fabOnReact commented Oct 29, 2022 •

edited

Loading

analysis-bot commented Oct 29, 2022 •

edited

Loading

analysis-bot commented Oct 29, 2022 •

edited

Loading

pull-bot commented Oct 29, 2022

pull-bot commented Oct 29, 2022

pull-bot commented Oct 31, 2022

pull-bot commented Oct 31, 2022

pull-bot commented Oct 31, 2022

pull-bot commented Oct 31, 2022

pull-bot commented Oct 31, 2022

pull-bot commented Oct 31, 2022

fabOnReact commented Nov 1, 2022

pull-bot commented Nov 1, 2022

pull-bot commented Nov 1, 2022

fabOnReact commented Feb 7, 2023

fabOnReact commented Feb 7, 2023

fabOnReact commented Feb 7, 2023 •

edited

Loading

fabOnReact commented Feb 8, 2023 •

edited

Loading

fabOnReact commented Feb 8, 2023

fabOnReact commented Feb 8, 2023

fabOnReact commented Feb 9, 2023

fabOnReact commented Feb 9, 2023 •

edited

Loading

fabOnReact commented Apr 24, 2023

github-actions bot commented Dec 2, 2023

github-actions bot commented Dec 9, 2023

1/n Adding Android support for Accessibility TtsSpan API - Support for accessibilitySpan and accessibilityLabel props in nested Text #35130

1/n Adding Android support for Accessibility TtsSpan API - Support for accessibilitySpan and accessibilityLabel props in nested Text #35130

Conversation

fabOnReact commented Oct 29, 2022 • edited Loading

Summary

Investigating the implementation of TextAttributesProps on Android and Fabric CPP.

CPP configurations from fabOnReact#3

Decreasing fragmentation with web, using aria-label instead of accessibilityUnit (2-10th February)

Additional Notes

Changelog

Test Plan

analysis-bot commented Oct 29, 2022 • edited Loading

analysis-bot commented Oct 29, 2022 • edited Loading

pull-bot commented Oct 29, 2022

pull-bot commented Oct 29, 2022

pull-bot commented Oct 31, 2022

pull-bot commented Oct 31, 2022

pull-bot commented Oct 31, 2022

pull-bot commented Oct 31, 2022

pull-bot commented Oct 31, 2022

pull-bot commented Oct 31, 2022

fabOnReact commented Nov 1, 2022

pull-bot commented Nov 1, 2022

pull-bot commented Nov 1, 2022

fabOnReact commented Feb 7, 2023

fabOnReact commented Feb 7, 2023

fabOnReact commented Feb 7, 2023 • edited Loading

fabOnReact commented Feb 8, 2023 • edited Loading

fabOnReact commented Feb 8, 2023

fabOnReact commented Feb 8, 2023

fabOnReact commented Feb 9, 2023

fabOnReact commented Feb 9, 2023 • edited Loading

fabOnReact commented Apr 24, 2023

github-actions bot commented Dec 2, 2023

github-actions bot commented Dec 9, 2023

fabOnReact commented Oct 29, 2022 •

edited

Loading

analysis-bot commented Oct 29, 2022 •

edited

Loading

analysis-bot commented Oct 29, 2022 •

edited

Loading

fabOnReact commented Feb 7, 2023 •

edited

Loading

fabOnReact commented Feb 8, 2023 •

edited

Loading

fabOnReact commented Feb 9, 2023 •

edited

Loading