Skip to content

Commit

Permalink
Add brief platform documentation for UI Automation
Browse files Browse the repository at this point in the history
This is a short document targeted at engineers working on accessibility
who aren't familiar with UIA. The intent is to identify the most
important points to consider when looking at UIA-specific test results
and to provide links to deeper documentation on MSDN.

This CL also fixes up incorrectly marked-up links I noticed on the
Android accessibility documentation page.

Bug: None
Change-Id: Iccfaf19eeae1659384244ac3f0a27e4596422c1d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2957347
Reviewed-by: Daniel Libby <[email protected]>
Reviewed-by: Dominic Mazzoni <[email protected]>
Commit-Queue: Kevin Babbitt <[email protected]>
Cr-Commit-Position: refs/heads/master@{#892172}
  • Loading branch information
kbabbitt authored and Chromium LUCI CQ committed Jun 14, 2021
1 parent 331a87c commit 805b781
Show file tree
Hide file tree
Showing 3 changed files with 66 additions and 5 deletions.
10 changes: 5 additions & 5 deletions docs/accessibility/android.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,16 @@ This document covers some of the technical details of how Chrome
implements its accessibility support on Android.

As background reading, you should be familiar with
[https://developer.android.com/guide/topics/ui/accessibility](Android Accessibility)
[Android Accessibility](https://developer.android.com/guide/topics/ui/accessibility)
and in particular
[https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo](AccessibilityNodeInfo)
[AccessibilityNodeInfo](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo)
and
[https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeProvider](AccessibilityNodeProvider).
[AccessibilityNodeProvider](https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeProvider).

## WebContentsAccessibility

The main Java class that implements the accessibility protocol in Chrome is
[https://cs.chromium.org/chromium/src/content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java](WebContentsAccessibilityImpl.java). It implements the AccessibilityNodeProvider
[WebContentsAccessibilityImpl.java](https://cs.chromium.org/chromium/src/content/public/android/java/src/org/chromium/content/browser/accessibility/WebContentsAccessibilityImpl.java). It implements the AccessibilityNodeProvider
interface, so a single Android View can be represented by an entire tree
of virtual views. Note that WebContentsAccessibilityImpl represents an
entire web page, including all frames. The ids in the java code are unique IDs,
Expand All @@ -44,7 +44,7 @@ point. That's called by the Android framework when we need to provide the
info about one virtual view (a web node).

We call into C++ code -
[https://cs.chromium.org/chromium/src/content/browser/accessibility/web_contents_accessibility_android.cc](web_contents_accessibility_android.cc) from
[web_contents_accessibility_android.cc](https://cs.chromium.org/chromium/src/content/browser/accessibility/web_contents_accessibility_android.cc) from
there, because all of the information about the accessibility tree is
using the shared C++ BrowserAccessibilityManager code.

Expand Down
8 changes: 8 additions & 0 deletions docs/accessibility/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,14 @@ Automation Inspector Chrome extension).
- macOS: Accessibility Inspector
- Windows: Inspect, AViewer, accProbe (and many others)

### Supported Platforms and APIs

* Windows: IAccessible (also known as Microsoft Active Accessibility or MSAA),
IAccessible2, [UI Automation](uiautomation.md). Chromium also supports
[mapping between IAccessible2 and UI Automation nodes](ia2_to_uia.md).
* Mac: NSAccessibility
* Linux: ATK
* Android: [AccessibilityNodeInfo and AccessibilityNodeProvider](android.md)

## Chromium's multi-process architecture

Expand Down
53 changes: 53 additions & 0 deletions docs/accessibility/uiautomation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# UI Automation

[UI Automation (UIA)](https://docs.microsoft.com/en-us/windows/win32/winauto/entry-uiauto-win32)
is the modern accessibility API on Windows.

## Key Features

### Clients and Providers

UI Automation exposes two different sets of interfaces. One is intended for
clients such as assistive technologies and automation frameworks. The other is
intended for providers such as UI widget frameworks and applications that render
their own content. Chromium implements the UI Automation provider APIs.

Clients and providers do not talk directly to one another. Instead, the
operating system gathers data from providers to present a unified tree view
across all open applications to the client.

Further reading:

* [UI Automation Provider Programmer's Guide](https://docs.microsoft.com/en-us/windows/win32/winauto/uiauto-providerportal)
* [UI Automation Client Programmer's Guide](https://docs.microsoft.com/en-us/windows/win32/winauto/uiauto-clientportal)

### Views of the UI Automation tree

Clients have the ability to filter the UI Automation tree to various subsets of
nodes. Tools such as Windows Narrator take advantage of this capability to skip
over structural details that might be interesting to an automation framework but
aren't relevant to a screen reader.

Providers can set two properties on a node to determine which views it appears
in: IsControlElement and IsContentElement. Getting the value of these properties
right is critical to ensuring assistive technologies can get the information
they need. Despite the names, there are many cases where nodes that might not be
considered a "control" should appear in the Control view of the tree. A good
litmus test is: if there's a reason a screen reader might be interested in a
node, it should appear in the control view.

Further reading:

* [UI Automation Tree Overview](https://docs.microsoft.com/en-us/windows/win32/winauto/uiauto-treeoverview)

### TextPattern

In addition to the tree view, UI Automation exposes a linear reading view
through an API known as TextPattern. This API allows a screen reader to navigate
text in more natural ways - characters, words, sentences, paragraphs, pages -
without worrying about the underlying tree structure. Windows Narrator relies
heavily on TextPattern for reading.

Further reading:

* [Text and TextRange Control Patterns](https://docs.microsoft.com/en-us/windows/win32/winauto/uiauto-implementingtextandtextrange)

0 comments on commit 805b781

Please sign in to comment.