[TIMOB-26706] Android: AccessibilityLabel is on wrong element

GitHub Issue	n/a
Type	Bug
Priority	Critical
Status	Closed
Resolution	Fixed
Resolution Date	2021-08-19T10:51:09.000+0000
Affected Version/s	Release 7.5.0
Fix Version/s	Release 10.1.0
Components	n/a
Labels	accessibility, accessibilityLabel, android, appium, emulator
Reporter	josh.mocek
Assignee	Gary Mathews
Created	2019-01-03T15:56:43.000+0000
Updated	2021-08-19T10:51:09.000+0000

Description

Here is an appium project (it is not needed, but useful): [https://github.com/joshmocekMovista/AccessibiltyLabel_Bug] Here is the titanium app: [https://github.com/joshmocekMovista/accessLabelBug](http://example.com) Use the files that I had for the titanium app. You will notice that there are accessibilityLabels on multiple elements in the XML. So far the accessibilityLabels work somewhat on iOS. On Android they are currently showing up on Labels, but you will notice that they are on the wrong element on "TextFields" and "ImageViews". You can inspect the elements with appium or with the Android Tool UIAutomatorViewer. Reference "textInputLayout-editText.png", "editText.png", "appium-TextInputLayout.png", and "appium-editText.png". Here is the titanium XML:

<TextField accessibilityLabel="textField" />

On "TextFields" instead of being on the element it is on the parent and looks like (disclaimer Android add a '.' to the accessibilityLabels:

<TextInputLayout content-desc="textField.">
    <FrameLayout>
       <EditText />
    </FrameLayout>
</TextInputLayout>

This messes up automated testing since in selenium I want to run the same code for iOS and Android (it works on iOS currently). Here is they python code I run in selenium to add text to a in iOS this works on Android it can't send the keys because it is trying to send the keys to the element instead of element:

context.driver.find_element_by_accessibility_id('textField.').send_keys('text added by automated test')

I think the Android XML should look like this:

<TextInputLayout>
    <FrameLayout>
        <EditText content-desc="textField." />
    </FrameLayout>
</TextInputLayout>

This is also a problem with the element. Reference "viewGroup-imageView.png" and "imageView.png". You will notice that the accessibilityLabel is on the element instead of the element. The Android looks like this:

<ViewGroup content-desc="textField.">
    <ImageView />
</ViewGroup>

I think it should look like this:

<ViewGroup>
    <ImageView content-desc="textField." />
</ViewGroup>

Once again this works on iOS and it doesn't on Android because it doesn't find the correct element because the "accessibilityLabel" is tied to the element instead of the element.

Attachments

File	Date	Size
appium-editText.png	2019-01-03T15:33:46.000+0000	575330
appium-TextInputLayout.png	2019-01-03T15:33:46.000+0000	510108
edtiText.png	2019-01-03T15:33:46.000+0000	545687
imageView.png	2019-01-03T15:33:46.000+0000	505821
index.js	2019-01-03T15:30:49.000+0000	796
index.tss	2019-01-03T15:30:49.000+0000	212
index.xml	2019-01-03T15:30:49.000+0000	1324
textInputLayout-editText.png	2019-01-03T15:34:18.000+0000	522468
tiapp.xml	2019-01-03T15:30:49.000+0000	4230
viewGroup-imageview.png	2019-01-03T15:34:18.000+0000	499348

Comments

Sharif AbuDarda 2019-01-04

Hello, What seems to be the issue here. Can you explain a little more? I tested the issue with the sample code provided and I got the behavior when clicked on the "Label on view" the "ID" and "Access Label" changed from "nothing" to "labelInView" and the below info on console

   [INFO]  JSON.stringify(e):
   [INFO]  {"type":"click","source":{"shadowRadius":1,"wordWrap":true,"text":"Label
    in View","ellipsize":2,"color":"#000","hiddenBehavior":4,"enabled":true,"visibl
   e":true,"touchEnabled":true,"accessibilityLabel":"labelInView","backgroundRepeat
   ":false,"height":"size","width":"size","backgroundDisabledColor":null,"keepScree
   nOn":false,"children":[],"size":{"height":21,"y":0,"width":88,"x":0},"rect":{"he
   ight":21,"absoluteX":148,"y":15,"absoluteY":256,"x":56,"width":88},"apiName":"Ti
   .UI.Label","bubbleParent":true,"id":"labelInView","horizontalWrap":true,"soundEf
   fectsEnabled":true,"_events":{"click":{}}},"force":0,"y":20.584716796875,"size":
   0,"x":95.490234375,"bubbles":true,"cancelBubble":false}

What seems to be the issue here? Thanks.

josh.mocek 2019-01-04

Yeah, technically that part works. If you go to the UIAutomatorViewer and inspect the app you can see the Android XML. Refer to the "TextInputLayout-EditText.png". The accessibilityLabel is put on the
element, but should be on the
element. The reason this is a problem is because when you are using selenium or appium and running the following code to get the element by the accessibility id it gets the outer
element so trying to 'send_keys' to that field results in an error. context.driver.find_element_by_accessibility_id('textField.').send_keys('text added by automated test') Edit: You will notice that 'content-desc' is blank on
and in the element you will see that 'content-desc' is the accessibility id. It is fine for it to be labeled 'content-desc', but that just needs to be on the element so we can actually interact with the field. The Android XML should look like: <TextInputLayout> <FrameLayout> <EditText content-desc="textField." /> </FrameLayout> </TextInputLayout>
Michael Gangolf 2019-05-12

With setText() I'm able to write into a Textfield with accessibilityLabel: driver.elementById("email.").setText("test@mail.com"); but I'm not able to read text without using xpath. So it would be nice to have the content-desc on the right element.
Gary Mathews 2021-03-10

master: https://github.com/appcelerator/titanium_mobile/pull/12538

JSON Source