From 7821334dc9860cdc4213e5006a821b8433cf7a6d Mon Sep 17 00:00:00 2001 From: Dominic Mazzoni Date: Wed, 1 Sep 2021 20:41:40 +0000 Subject: [PATCH] How Chrome Accessibility Works, Part 3 Follow-up to parts 1 and 2. This section covers events, actions, hit testing, relative coordinates, text bounds, and iframes - filling in many of the additional complexity that was glossed over in parts 1 and 2. It also adds a few more links to "How Chrome Accessibility Works" from other Markdown pages. Bug: None Change-Id: Ic1037d349555a981d16051526acde36247e32c0f AX-Relnotes: N/A Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3137196 Commit-Queue: Dominic Mazzoni Reviewed-by: David Tseng Cr-Commit-Position: refs/heads/main@{#917367} --- docs/accessibility.md | 17 + docs/accessibility/figures/Makefile | 3 +- docs/accessibility/figures/focus_race.gv | 37 ++ docs/accessibility/figures/focus_race.png | Bin 0 -> 25098 bytes docs/accessibility/how_a11y_works_3.md | 472 ++++++++++++++++++++++ docs/accessibility/overview.md | 4 + 6 files changed, 532 insertions(+), 1 deletion(-) create mode 100644 docs/accessibility/figures/focus_race.gv create mode 100644 docs/accessibility/figures/focus_race.png create mode 100644 docs/accessibility/how_a11y_works_3.md diff --git a/docs/accessibility.md b/docs/accessibility.md index 8f6a99b8a55f5e..a6e9d5472ad211 100644 --- a/docs/accessibility.md +++ b/docs/accessibility.md @@ -1,6 +1,7 @@ # Accessibility * [Accessibility Overview](accessibility/overview.md) +* [How Chrome Accessibility Works](accessibility/how_a11y_works.md) * [Accessibility Tests](accessibility/tests.md) ## Cross-Platform @@ -10,10 +11,26 @@ * [Performance Measurement](accessibility/perf.md) * [Reader Mode on Desktop Platforms](accessibility/reader_mode.md) +## Android + +* [Chrome accessibility on Android](accessibility/android.md) + +## Windows + +* [IAccessible2 to UI Automation](accessibility/ia2_to_uia.md) +* [UI Automation](accessibility/uiautomation.md) + ## Chrome OS +* [Autoclick](accessibility/autoclick.md) * [ChromeVox for Developers](accessibility/chromevox.md) * [ChromeVox on Desktop Linux](accessibility/chromevox_on_desktop_linux.md) * [Updating brltty braille drivers](accessibility/brltty.md) +* [Select-to-speak](accessibility/select_to_speak.md) +* [Switch Access](accessibility/switch_access.md) + +## Speech synthesis + +* [Text-to-speech in Chrome and Chrome OS](accessibility/tts.md) * [Updating the espeak speech synthesis engine](accessibility/espeak.md) * [Updating the patts speech synthesis engine](accessibility/patts.md) diff --git a/docs/accessibility/figures/Makefile b/docs/accessibility/figures/Makefile index 94bf62710b387c..7c0fbfa2a5d587 100644 --- a/docs/accessibility/figures/Makefile +++ b/docs/accessibility/figures/Makefile @@ -11,7 +11,8 @@ all: \ other_multi_process_browser.png \ proxy_approach.png \ caching_approach.png \ - multi_process_ax.png + multi_process_ax.png \ + focus_race.png %.png: %.gv Makefile dot -Tpng $< -o $@ diff --git a/docs/accessibility/figures/focus_race.gv b/docs/accessibility/figures/focus_race.gv new file mode 100644 index 00000000000000..03c62a2211137f --- /dev/null +++ b/docs/accessibility/figures/focus_race.gv @@ -0,0 +1,37 @@ +digraph graphname { + graph [fontname = "helvetica", fontsize=11, compound=true]; + node [shape="box", fontname = "helvetica", fontsize=11]; + edge [fontname = "helvetica", fontsize=11]; + rankdir="TB"; + + subgraph cluster_renderer1 { + label = "Render Process 1"; + + click1 [label="User clicks on button to open dialog"]; + dialog1 [label="Dialog opens"]; + focus1 [label="Button in dialog gets focused"]; + + click1 -> dialog1; + dialog1 -> focus1; + } + + subgraph cluster_renderer2 { + label = "Render Process 2"; + + click2 [label="User activates window 2"]; + focus2 [label="Text field in window 2 gets focus"]; + + click2 -> focus2; + } + + click1 -> click2; + focus2 -> browser_focus_2; + focus1 -> browser_focus_1; + + subgraph cluster_browser { + label = "Browser Process"; + + browser_focus_1 [label="Focus event from window 1 button"]; + browser_focus_2 [label="Focus event from window 2 text field"]; + } +} diff --git a/docs/accessibility/figures/focus_race.png b/docs/accessibility/figures/focus_race.png new file mode 100644 index 0000000000000000000000000000000000000000..b6314ed81ee95b3a82893b83ae8b9b9fa91ccde6 GIT binary patch literal 25098 zcmce;d0dX``ZoR)B8nbGgCZe>R4O#%smM^#qp`xTT3YA8Z28A@w zlcY)WsPT8)*6_aX-s`*n`MvvHdwo7@SI=Uot9*8uuJ$p!>q%AkwxRpmO$$>#4c*e*J58G1%8BPTCx=|Ptl9HORV`AYztZx0 zJ@eh?EI&kYJ$*tyDACEvcJr8WGhB7|m@rx!YCvbKf5-W4o#97`3iqk;#shPWSFcV* zvddDIQ}EBm4`nShOk^^-_8_keem1raBjKl@A_LmrfA*#){&=4A|MwqN(vv%SD`Pa$ zO+$7YZsl1s7W#I6#lqb5K}R-<@e@DkSZ|VQ|6z(*bUjK=U4(me8%zePe z&+i7c*Ih_RNWiF=A$PJ)tKs$K;H<*Lw8OSPdYhtaO-Gw8-xidWtuDO9SeWNJ{fSFA z`Sz-f`!sKenAL@ev+(dF?BJDQqy`5w|EP#sCoL^ak(CV>f266*vSP3)y|60wr1e1E zz7HiO55=Eqo;Z2(n3a`4{>-;EcKtOUTRNG@%KHRqOg(*keGQC^Vq71EhI*Hl9zP^5 zURYU4Ccg9exi}I@GkK#=wEkp&?a27JS90>cme$rZ>n^$0QFeBAHW5=6PEO8_Z{My@ z4%CN=M?INcvwHR8U8~!id$?YkH*ZWyNhvKaPjt}o@$n&%K0iE|`ZA?C)6xiw(fjqu zvInB(KJVVia7w#ttIJzh@L%{8MsxAv#fIkQZQR@>dU|>}qm`b$y_c}+gGtiz?OnTey{n9ougtbLyE<8`zG~H~cUiVQBOjO*Nu>LG&tKoCmGRhhdPpPNZuhg( z4{voAddH|Hs^J03davcO8vQ0K;WR?ppEq$fYn&sYy|Iy<#Kg?(_3C29w_^WIm)df( zDl=F%Y}mSE$BH!@_sV2iw0=$1lWxs_X^&^caO@VtwY)sZZ(UuT-@o4ol$u?2?b@~D zr%qAuIsuo7OW%8Xx;hHHF12Qh5Z8?_-zF)^70i1&L1-#EnkRRDtbAs1ZkWu;>9BQ;NwdQ+v?}wdZIiw$|@T#EiQxvh`g={>*&zG z?dO*?YCR!M+Rn#U_IqD;v4pXQ@L#VpO`pM~b#!!iVmI8&slM~*5d*0gx18m8KH`uy zEi1qF*AQDr$NB{eOUrenU3>RZY;1%cJ$ls9-=EOjp5th#asGU5y!*j}2V(|z=d8+p z^M)-vJRHfuEOvLyOZx~(*Nw?qnR220I(E(2yzTjJL608!;ej5FJzG#x5?A%huSm*i zMBi_V^yXvUYh~+GbObMc;r!YDb_xH-QLoX>9q@n&;=p z92`Eo&pnfgsU(qdb90q6G^n^t$+Hpp`T3mU^-q z0?G+#LZ3(+yL_(WX^=>>lLOv(dY>MMths&rHg@>2D_8b%a&e6gX0~%A@5kO;DG z_KZhbdJ7`#x@PjNSFc_bG&ufjxQ5Kv@$T9(;!(S}h^^fs#o*!L;WbC4#*cR8yyFNv z6Cu{QFx_5qeZ`s=NcDc34z8`NtPGR(kh-;QOTgUpaA9jJ*A|a?L2PikWHqxFQhWC7 z`Pr1dCiU&2n9D@(HT?ctgkxJlno)_J2}7-3ZS(unvdS!o@@3;cK6|}=D~&Yv^{s~o zr-Ty|k`c+%f73z3p_VM9#$(FLWS6O*w_9@@@indbKS_@(FA<6Tk}wDIS=UWg-FIan zdn#X=l=DrK$;w@E$BS0*h_G;R`Hzf@C=8sUQWK(gtwzu=ZQOWaNQqi{)Oxial}g>a zmYJJ}hv(&?Q=a&#^qT+;e&X>hT#ldq>wip`6(f?X)%Vh%Surkh_(xBgWyto1LjqJP zLqg{AUOcf?>(-^FR+EFmQGK@V-TMjKn9QzXNw2h<{CxP_dSyvvmZFxeOk~wH_5U3d z@Q>i)5nTky+10fxyAriN( zw}EKzPQ~K=Z|Bc;cV$cI8C~pK@THcjgpq*wUi?DNJzLmoHylWM<0C$dJ~rU(Xk^M_g99)yREr@Sw!#2WG^D%!w1X z3o=yf=BCU)Jvz+HC1xj6{p>WEjg8t>61Z%p{azh_QD;xjN-Zs|H#s>*UB&)%QRVns zjAeWNi&w9XMM=5YP4+!h+bu53yng-lLhp4(-DM$Gwzi2A4K<^6uNURr7W3!FPEPbz zE+eoI_C^5b;b`$TrFNHlJ<{8@EsK%!zwYI=Oj}#~h5N#+&0P6Zs@KYOhBG6bMCGqB z6=`W{85u3vQvW7`WYty7=*ND54gf(O3HgO#(Q056`iBQCQ-subqDvmsoK;b|rxN#| z|Nfrx^74~dugRf->bO9_NhPcvu&2c-~-YfY;0fZDpF&bk#~WSjjgSHH^259Zs4q7WZ~o1)@hX)_hM^J zi|Xsu=@>b-v+mcu2c%Qpl4Tov{?!E=Ac)<3eCtV7`EIU3gzO54}WaSbKM~>uGQu;urKY^tKfn9)N%m7vUF97T{G(NP9C0RE7#t09{nYQ~}h`aJRcnN3`|i@%^sa#@$qEze3D$Pxr-N7nNCLr-s}0p0|7q-nqi_ zXi7iE70}zioeG0=Esp*Q=zJZUyY#mxUCEah0%zylW;(sQBLjCvT|K*zU;EN)%i7nH*4EY`{U?tf|N4;BckY44 zaO11E7BiEi2*AG$s0hU4wpao? zUL|7p0FPyteEj&4>sCz|TT7i;e;!n1=izwdb)kAO;%PGaq}DZ z>oUkH3s|%qy1KM5?J^)-cP2!ksOUSoP7XWTnmENRj}BQ!S9LrJ55N8TwXnp(M0Ixt z*4U`v`U+}bUrw`oYsZ>Vmh0>dtK0vc>qq1w4^snbPPxria@Pp2z`0 zj7I7E9DZOW;4aZ|-ieG10D9lPV~4@}8}uV%WAEdYV@M=%ho3S4l~x_^XjnFG81b zQw~ieP~3&P`KoFtj`Ir(f^M^RfMEFdQ>3Jp<@udEciy{qPajQ))6B?L?AK>=`d_6N zXJX_A8R+S+rKawGrWh6A=SSc+qIAl1-?&eWuZ$n$iO$wuzK`ebqepvoCgxIllS1HW_SF1w_Jk$nsJB>d1*Dv!FJI2k zE?d_9Df~co7vp5I{N;+l=FA-u658nbpQ7r&G_B=JOHZ%OS>@9C{$^E%xmH?cCM!UB zBZ!QN{@Rlp_i89T*W+;273T1l82@yj>hWT14hss%F4>C z)~q>j;xL|0)#JE#)cB#;XU~o*vn1){Y+SKo#T)A`I&{4&(T`3HwKlhOf_tF{?X>X= zT7iMO$=g8`4Xj3D zYIRUJu;)8)P$0Mp@V-x6YgG%8gUO&Qkn%f2_1upj^AI~73$S_X)~&q&Q(4Eh^9Zr0 z&_s)&C!`_qsphpCKb5bEJ;|CoTD++t({hVNTkh?p`5tNDz(Su5dpdf0yis=y+j5-` zIgN0maDV3FCo3ll;3Jf0SKb9oMlPTSYqDO{qJ{eD(fGA+WzRd}NL1aZ{`y9LU>f+9E-F0=w$^Nsz$#=2C!b>O53UQZ!kv9E2t6(}lb&Jvg((Y}%ZbS=TAMC-V=CEI=tV(SaQs};-q)CnRlX>zx3ny?b5S< zUJ{?GD0|tbPao>_JQztJJ2y0CnEL{Ca`W@A1vt78aF4o|eo0I@qJxF4uR2CIZBoAR)uT|y-;3wx zceyrU1GcXxlGxV6huP8K2w^1xEn2e;O7Cpdc+Yr8T>@<;i+wt~UTp=Tj{g z?SHRf0zJVbXk{<}-xO6yFZnTU5+!>4?1iA9pjB(v{$NdCFT#W7{KA5}QrjQv?!p2z zgN#;ii*XZ5R?6JsJA=Yo1?A;y_NXPjL*n_&xq~c1_s7HaGflJ~sB`)eevw2f3E(RH z`sCPXrTmtH+S(0R_4gpqK67p*OMbyZ6N?gFRjnxPk9$~u(4ys?$Kv>wr?IgHh=AU< zJaMUC-!jzIBE;;E8XIpv6D4In#B{q? zuwXg0e|g-y_>@UhD?N_|k*ezIQGjG46qk|7$s)89zxg1Y(&7*3`+TDB&0TJKy<0*; z0u7FjnB?>yt?0LJ-%hFZoKuukP1%QFx3aUNAjp8}j9!k1l_Re3B(D)WWuNA286_pA z^~}tlxi&6%{J9eSzTkfs!F+cRTO;3XuA{e?`r?Iv-C)xO@O8V^kqMA;T^c* zsi`Rt&N4@jUaxGpCM_LR{BJZs%v{;n*w}UIyTZa_nV-S=i;9Yte)*CzWqZUPa!D=y znQO|&zM>h;{q^JF8@s;sQ$HJ*p_3q>n565}hlA}`W2V>DckzmgZ;HA)Mu18I=P~2f zDa7x$vF`FiE)%<|s;Zd4AbWa}pn|BXU-)=;$4rUz;x*t2Ly!t#qUIdL13@EoFN*3W^ONAbi01~%`!T|#w>~whE&G9p z8Nr@+y}!AJfY-H&>XnJ=`g+<_~lVSf!E5Q`}c2s{i?*UMT!%RiU`{r z>eJ|0`F^8cW8L5{3p|$Q^nSKvF%$KMaVRkWciDl0#U^M-NiRFc6|4y^ZauTgG5=4v zgcY8Envg_HJxRyG6ja*S3+4k@endtFF)arY;yW@xNRH$1g9%RiB2WmVdvj?oU#^E} z(wOV~D&>DQtm17I6^dlM~b%HQx%|~gW;g+5s@%m6yq$(!0b?XsQDR!D)T~t)m zeiHG~A`P3@tqW8zzg+#y-w#b7!8h^2fXsz zsHt4F(>V|%NRRQ9 zQKGR&uO}uZT6ka!8JL);#>}?1wD@Jj{p9IIG@!{aL_N#i|DjgB9?zf9PzWrUd0Lre zVz8MNA8^QZYOf042JA$>hzgCL|66}dq#@fMp9;_h(7fxAI`;DAO98_Vt7c|rzoGI8 zYex33B+_o|IeqQrrx{gq90o^_Lc0*MGXr`{J`jJ1Pe{$5<@*vPZ8BljVR0vmE_yuL7`4*`x08S!wsLrGc2I@5Odwr?gILx6YcI zo1d<0gD7*kieedl_wIW18hL|!CVL%_X^@__@7%dfNN9aeMU*@;EeL00D?L@A0|&sI ztpbL;(wy;G|BA|6qO1U#=FS}pya{0PE!AUT4|r~((?t%0TGjQdCu)CYe8!UpV)lGT zj~x>4^NJHZ+?D8PfrhR3%7`fYUUs^WOmWV^yIf0;PS0{zHn*h zUp=3BnqoCSW9{VP5)EzwU+B{QOTD!Lq)nQB0WEYrBOou}3Z3++5c|O$&m>yLC4 zKt^N^j-tCoqzzt~i8soLwP8XgtDz?l3aGKM@w@i+EvU-5L#@XcYgzTK(x86^P$9{@ zbzdMd1bUFIf`Vj722_9ln@Gj^3sYJ#lPi`lC*BGE$XhPvnp{l5&&F5wbA-46r2WKW zX^|iv6%UXuDwZI3bPf+&TItE+`+*g1MMZ5Pw1?J-ts>m%Iuzp}*;6$F$vQb|$TSy*1mdq< zXJ>rMy%vts8mmpEDg=f_9Sbj|LzOH+f}-mxS{lgY)lGW{Pgf(h5o~RH?vYL zWuH21p_2d52;_ccaZL0QF@}6AoUPcoqgv_j4oD;AiQj$#uYl;$s^wN}8?4ogeLYY8gplP z0=V<*iQU)oh75N2DA3V`&kt_mPS>|z9cPRyrgH4URhr2Ma9(@(a1)X}BnsO7`}fax zZCQE;k?(7a{JM#sjm(6Mr?1Zqxz}*W>URIBo-Yxj({Go!-Y!gSx_mX7rmg z%(eHPu6c(xM$>h+FD@t|!XJbHHckh&=^OV2vCR@ro`m#^&UmIfRPP1i&Zb?GQlkM* z$^ew`@e?Ns9v|_7Mnuph#9FpIAy+PyET@`&E-t16+kRkHa2P@W1auWiSy^LupqygB zdt$|6u-Ad2CfeOMz18>>X5Let5fM6)a*1yi7k36DmTzb z)EH4HjEayz?YG2VGBWbodhC{d!wUm&g-I%E=!+NUe;rp)c)w|*0%uNjnI*oL=1o*@oaG+g-)SYJ8F7^G#kN41C(3rk)8jVi*^mcJ_ zzjl_*3X;{#hyfsW$oPFC4QweA_AA4O7hDGn{ZOu6t>C(UZfSg>- z$occ<4aUBIJY>~DLf`v*dXETYxyM{{`v|gqIRkmU2>&V1(x-f9yzs)#T6H}UEMj12 z?8#7$c_~5i5dcqu;0RPEi%1bKgct%8s$rq!KD6}HCyJaL9ThczO*PjJ(wO*)6c3K4i^?b@{$ENH^Z|JD&V z?>Xi9=_h1Fe;5B#My_*npetr>-uM5IpqfaixVGM-rDZA1+0NsVORsF8V}=iQgz$PvDIs zM~=AE?_*MKz0{tcmvxNWC5a#{o}5iHgEm`Ote)Gie87FSZzmx=qL63X@Lb39eJFUA$Mh1fVx%8<6WQqol`cw= z%ja%u_UZr76kZ^Qkwj?049XHn0%X&Hi(fhtQ}cx!k5g(q!AVJ|O^V7P?H)UNlvZkK z-u6vaRsqaLFtyxJh!y(SdPpdMUTIs^^K(1Vf*2x2Wt*E=uGj*(9V&)A>f2+r#Lgc- zZUss&YFO%ITDBJ>1y?>fwi5Ix3p_O-;@?;-L6sM{@QF>>vW-(G+pYji)mPB3Ha#g8 zZol139YCIxz|E3hK}XKa&2=IAvb)B|#+caI=|S}z5)(5-_B&+XuaeaT0gMq%LPJy2 zdr*d(4qDW3ZQ+5qd5>nH9jyHWnG#A4uMuskccKit8UEw|Rh*}=HzY840@jonTw6wHV$S$p;C-3hq zkX{^J*OKeZ49>j(1{*6F+K9wS@KN{o>!CDq&ZC4Ep~#I^cq~@4C>~bcQ9P?{KWlsH z6vLNDNuo?35OTE_YO^Atg!&U939l9LT>uh3v1$lt8$ z&j=!daF7`oY~|O@rGn|02TM}Nb@9s21as{TgN?6N)~D)~P|4d8R3lAKn4;rUIHlxY z1O>eUJOE5`F^=ts-sPT`(lyCs8g2rhYKa~^s7@-4lyv=$+)gXwFx<8;EiH{!=Ipt1 z_9iD46h2f}pVReNuy4$9Ooa#PBm5h7a}^QdJ4xSjhVv8cHXBWD*sy_~^b7K-{hY6_ zZ#S^qJv^PVzeU@``agT%*3t6&`>F@aKYk>W#z41}qHWmg6J3gKj!`mM4jb(Za=dz? zSg+!5G5AteHmBa^?#YR8^5dz7~L=&yD;S^0`dY7hK*G~6dn zoNy^$%Xvr%ltP@Vw%8fSGsnT{nZR$r!vhy&Ea2=I>MOhC#UdB~-)j$*i4Sby-H3>N zJUl?_p*@kVXV7NJQj9_2f9vht1p{1nlhKtcROl!yMj&2DcL@B>g?$jZ6(k@NA>2s0 zWmmr3pgQehdg)VybbSA|kI}Qip`k0G4#H2M-ZVYZd5j2Qc$`4F+#PDA+t4$JFYx^r z>Gt@UGq-$w*MOWyh7#+Ya&>We-`1vUd8)X7kH|j>wIndH8qY0o4XwjNLqpl^CyjQa zEd)RoaG!T#;1XlOZ^3lj7!p)I-O$jWlJzC_>C@%lgh-yCPShKF6y|c%7PZCvRW=-} zA4mOva`t&gSJ%0isog%;@ff@Cqzm;H$5`i~xf57DX^{yORY zi7HU%R1>yA>&oI)^hn^SM#7726rTS=(9>~gVNM*6JPd{ z49BJ%(@m@-LWBkB2apOfLMvHUJb|58#{W2=5J);NXf0V)`Tr*3-hi)@6ta?pHU$=u zm7)9f>dcwRaKdupbKr$L0#`_uJt3RE@mR`ld*fHD&i5~h@&A&n6FLu^jm*qEND%hO zGH4*zfv`zbbu)Ix2J~Xyc}8V+oy>&`yvT1Oh=|bUl+=pwXz5Kkk+>~L-M+RxU-~3< zmGAKlxWf-p36I;Aw%qfX0@-3cNg9^+&F5PO#?9D*+ybp)W(kmy(8%iR-V zZDVs3u>l)NAXfexw7%?h=5r_+mLPgV1?)CKx0YCN(fLNz?B3&c;Ew-3%ol$0vy&B zShz}14TC~LVlzRWtV+^IqlFJ5dqq|)`?0b*v!n=-Jbo+!Ax9HFMF1Sq;?7F_F7=Hj8;|VQexn5VvIo5Nx8dRaFk;@-&2#w(V`bUC>l?{g&g0Kc zJvyZPr(O*U8qZF}$eP;3%1htFFGdD|1y{fx`~J`}pG3{`TeA%B%5xA$-~DMvh2uq1YvWCL8pR;$8o472|7zE&C>m_WXUFot@u0I=Ic7UI)Q9emgli`Db&cBE>K8rpWEY zB!X>gIix!k_mg(AKYnPk=4+U^hZ2SFlShq`ot^Oe!ooeLW0WR^@AnK7cJlUoNy15V znd2QByu|3AU_p&J;^8uY-iC%mOfA@M50sWwX4Yvi7NQw~AXl zmYgd5P4|k3Y=CngIFaza6PVbtO#)nEzmDf=n0mjW;o*RRx&+ZQ0=lvgN-Y2)SjL7| z7X#Ze+nLE^5#f1GZkO%YaM_ODHTM-_U4F)qZc}?l-;4iO=Q{rGlYWnj$eRq2dKIA- z{#bj>4?-gcP#DH2!oLF9I1Q4|gZyCx7xgCV*ke<+gBFj_&N0tU*6TU{@wW>bH*UOz zRf1RZESS1tXadUG+E-w)O&~N+TW@{ksi9V1@-{hch$rB{gdGo2w3kty#nqZVIV+gS z8@&W$*#S}q?7J5NZuBamoBs~5Yk<-MA8b5LAEpoxaGAcXDbHc-06^A4X*m(dx0uo-#1!0ohSDj^h5en!ZuNo$igFqIOGOrU?NyZ(9)h}3Nd4pjg3Yc71gchm|BBRw;>(BU?Cgx#P=kKyNp7a1+0NPErgy)WYwQ{5st6ay}FZ4goXIUYt4NyQ5@gz&IW5T@e8Mw@FnAE z*C-LBL>C7)Pl5vax1h*KR1Ij^rSAz+inuD(qR*chQM;iq{#bi+FYZ|A@R@65W!0JU zf4v(QnV}t`j03&#{>^0;a17UWV{MGhj6sY5uye zV>1TS8m7OlM(8OPm7wHSuU-vX#Cd_gKSIoLh=F{>#U(rI*w!^598U$D)2A4(l3qjQ zp#A$rny)X>z;%nC)rgBZv_g(jJQAUAZ1ML`_##^);%Ok_jTvwe7-=%LL3mJ#|Nhf% z0Un`exDLYL1jdTE4#eaiw_>EwzFKeg_E$98Xee16el`?5rS`8_dUz)NrB&zAD-ADb zK%HPBfFR^xjUw8&E$%ZkgsCPCjg>^S)jWTOcrZqQtpV+UdMFNBnAABdps92n4wdO}uS zCIl2Zk1l{jLW)7F00Z2MDzV3#C0*VX7Z-DGy?e8hg{@Wr1?0g&ON$qN3bF4Y9{b(j z7F|35qka8?F6cwlr=ygz5F}sNH1s9q889I8THV=;_{CY1tD4BF@Q5ezGvf7b|z)elEA^%X&0v;j!nsaDuzM|iE@uU8J zwaBq*eZb)`$Sds|6)CWi^VQk6Z8O9KvRAp|AfAQkZfJeM8u`NKn5gcCmX)o|;U9q9M zLPjcoeBdS+Lb{9BfZqWcdnm~FQWybeTm~*tON*HHfMHv}mUu1UUqwpJE@o?`qgMB(YW~&<=0~-kpA+n_nC+lX#JeNuE zMm$69Zv&yl*?J%29XsTAAkc*NHOybX?H>krXh1Y$Fk>Z>av%ARlJw`>Es3!e$5qBh z0}Bt0Yw*gD2%qL}KHT8WX0#$SFy*c?Gc&XGKDt48YF&8On-Qnk4zG1D5c9U)M|j8H z_#MDrsQWQ&LJ3C?F~59_(XAGEc# zy+}_lf{1XM^L*>`8O^hwv0O#%l5p#j;d*Y86IO+rhl$*h@3zS@Z*n7nY0(y|=L8VFAUcreV`H1Ajw!6h0JY&=y}0~^>}@IoyAWk|YH$xG6z>odAlP5qQF=qM!`XlS{Typo z$Y5Db?M~^$TX2f7>uomC-epg)o|8&t8gCy87+i<%K@|dw!6BbzC+q|LeSMip*=rOj+hLBeq9ORVH<%Fr^RZ!)IUX^s4E#)dwMI$9Y(C{0n-a%0)mWLi7x;K(YWp1V*vBApl%Cw5 zw;dL_U5@5brnJUCb4}(NVZBw;{hfqtM;{I^RH1bMBs3(6t20b-icxZy zV=VS#hhM)&v)*Is=lRzbtsGDsD$UQatiV^v+Sp*&^8Xm>NdJOCt>EBbVyuFo|1!Yg z7M3Y2CH4HDV;b3xM^zrZ60MU%DJBf000dtLrza+AiZqqs=q?zGtTf@6o)R`W*4n``HNi z2lHq9_Yf!&6TuoU^pE5{!az>bly85)PTa=LSFa8bqsx7Lw=v5=_&CEb)`1Qv&2`!W zeA&_1s->N=3FFw&mHI@O{W~}4Qf`$bw7I^k*ic>xBWHwbh>%hVnjgcSN0sj>bffuB zAGlKW^mf}IYDV|Q$q^$eL4?=X%$aNBAktNK{BY>g< z$<-R0R0%F|mVHkulD=A^vWm)6SI0Abf%!!PU=`sF0!=FiRTe&?z{iidF(_1eX>W1U zv*E1$CPZK*ikT-3nVop1-0JQ)J8_xxE@Y27oH!9K;+D_PFHKJB#5s?2 zT(j`d)YQxx+aRvkSn`jfB?H{E3BP7%d3NrMYl%qykFz1xRzh`h|2`X1zbP(l44;kh zS;Bvma3wK+O%r^G#JD<4eqi6dVX0MhzG~lGm*?sT#o>{-V@g$8ZJ)Smkx7!ji1WUW z!U@&G*oh+;5GSOZTd)RQ`10s^b6l$8gZ|vu{v17t=@#1=O!X1Yg{=UxgvUWZpG3qL zA=?103Zpz=AAS%>QjHq;(6O8|&M& zvHwIM(%kc4M6dVHGs`g*;b|Ga;55NlZJ4C1D3qI^sHi}+d`FViy5LNun+te?O`67= zx+DBnTK#u;qQWGcg$Onp;{O_~n=B#wa-G^xLVqqIGii2+mC#3lW`HPe;zR|KhwfX# zf{qy&w7AM$V*cq*#Ekl5ibHV7117g|8bBwMdjT=~(zG%s?ZAj8ejXQX)+323e1!K9 z^4&=!@D%O;VR(ekj~E_VO3iZ(^m+NIi?ryL^=VbK-|8p4x*ykkYgKBX$SR%KqjQ8s zq;4;xc|Akgne@n47Z2L+Iad)`_Bq(>;;Z}Rra7kZ1{!@wav$G()@XR$=_E~P#)@zQ z_c?Fx^ZbRM-Z32T8=kgZ+f$mG**-2cEaC3F|3GW@?K5%AUIkyjPeX|<1s_R?t3Cm+ zx`UTj4&&cDcnq_*zF71x`JtPbYN2U%KhJ*PKHN$Q7^(e!ZRO@WSiO_4HIo}3KGt)a zRHa|NnlON}gENBD^c;}j4$MuSws`QWMvfc==c6;GX^Kip#JQ-#33F8;=$M04b+{+c-och6ED+>55{Pk5zaCb z%*{RPgAX3e(V+y4m& zOWX%@qD?57I7G z@<8AEwqbeM&>wX9`y+&1w|O`1;5S>4Gd6v|T`2AS_lcsG^WLPNZ2KddG};`$^W z?{7NQ7 z`T#mQI;+W9?Cu-r4l@ta;0h!`>waHLJlq*Ji9fG?mlT~MF2<9F%IOpO4o47h36O%0 zjjjhj^ZV|0(cyf8KQ5c$@3%A1EwjAN%MyG+kaj!V*Jl# zHYDLH9N*z_6_s@S_2*FrrDkFnghf_x!LIi!4XlDiQA_g|)kA+Mp*~rBJp%(`ubmhS z;GK{)tnHmCc3TYa(1bYr8Dz}A!;o{0DnlXc2}$Q)(LuIrNH+^jJZT%hVUJo^(`5lR zD|8HK{cj)n`;bsZF$ zpXmyOcg^1Z1jR0jh?M8a$ty^aG&Vec%(ZPIDe34if=sG9xt}Py@P5Q6C4GQl>;;%E ziJ1;B;&2T(QLW(sC%uEkbL5-v{s(oLkY->pfhJXLcHqLNcL3k(NTsEvG8!6e(D4K> zedQ*cDYdn=B+s%k1!ABFvdz~%JJ^DWbB*@aUD%VSyN=Y04nDDoo&*lE;6R0h7YYyx z%vK!fQgh~Hx|UgH>}5)g`>3M54I3SI&o8(;eUp;*5}w6<`;rS@qf6)j7AII-VnPZ1 z2jLk+A53Hw!oCL*Bj)jA5)?fOtg@FIp{8*9M0cQoy#xNUg4WC_b(IZiiWrO_@ClqC z%;%n+DnNtc2}k}D;6)Jcg{T);DF57C3BoCj`iP0zGa3kv| zCS`G62Oz{yD6y_;N2Y_YbblP3lXx;~7?acVp1jhCsr$s}CiS?gY6$5L#y6^8gT0ws z?uo&YaNPqO{_8n7_$nl)A90F1oWjV^)7)Hb3I%rC-B^ZuZ{8dtlMh>Ucp@Y#2PVg$ z-=J->!Ql$LlNeJ{AOZfI@Tx~;+yke~%f_ZpXt|ZVrktIeN|&vZ;D#eVX!z|?e(j)y z+w5!79W*7tGAgEIfP+9>{w9$7yK4 zTeok2T5xsWG*(t`7>Qt3HN;6#VZxW!gFEmA)s90Nx&UTZTJ3Lm78e(T zVF|bl3c=R+f(s=^OzFI6K!dsDwXx{-yZ5(0)p;u_|8tpEbKl@#>ra=YT8rnc_n(oK z=ZcYSj0D(70;I#|PM>B3^9+6T)0r(|cDulV_=9f<*JUOX-UJ|Ik|*{SVI(?a*ULmO zN6@ulrUBWy0q;5r^~=lt*3FyPJdD^R$4{Rwf>=oOBIrXq;AJzf*I1L3Zi_KJ0XBw1 zjzfY-uQ1G+>N*Kj4eI%n*P~X zt>5^M7_nAA5|%LfRjZg{DtGblz)guslJM1^K42};uxFm@I52Ajh@Ox8RP65GVyw9M-g!bj84N zsSavx;fSCUDlIDjf#KZgzAaXj=SDS`Bc<_KX%Kogb?phMe#E<)`?HA_f zI|c{+i8Iuk1T2{=C1L%7590Dj6BrV!PuFs_w7;{93o$KVXJbOCX|qRdMe@P)NSc`C*|T?<`FPR|NYQE!#T_lY_IF-6 zgN}naVJnYg@V$FC9z5Xi9Y;)U_Nm3?iJG!8uLP0(RC>FIT*DP5kJK4FNdQ|8a{3y2jvjQUSW{_RcOJ4izM2^ z;qM8@_aRyE1IC(7355j)uG{STQ*fs`Fpn|(C7*Ta22h9L28_@qKgWtUn!NL*fgRl| zdKaVa^wd-_akpJvUD0KL8UTNfG0t|H_=6l~ucW2CnivB2zQlnSXMECxQE$!FDiSYS=+O4dm1#TF#kMVRKp8;hLz$j21JArKd?TPnh8*5p+ zw?;r6nD5&{_`S#lcXyoB)>%u=3dhMl25?`HJoOg()l>G}QYpBsck98OI~_@MU9*;vd->He^TUiO`BtUcL zJ@dnZ3`%I<6;7UvNIz_I4M3JSX-G4=;OGSM`G^tGoq5^ zl)-wexYN2P3Ww)D8vF*2BgZIH`ZHTE0_NeL+W+EzE@#s82FY)}7QO3&2S(XD`p265 z(S8CH#WnU3)fCaDmRN)g7ol-}0FVs~?^D(CMa*~OJ=o9IR;u^+0Y1YCeC1U&Bi%Ak zV_+7w^8AYzt|YhD-s-w(en@vd>OhGvk|Lo+Ve-Zkhkc;DC@U#_{W%|S)g$wDg?Ve7 z{hTFA&&D`$R|$*T4x2nl+=7B+o~rLX&@XN5FAo++%lZ;#B4KgNQ0RGMItwPFG@PKa zQ>{*n=Q@~{tZasd)K$ckJY<6OUl|A z>410gLXYG3xtv(HA^y1U8S7N|?*t-KEe8_yJBGjjJ}mv<(hDF`YW)yRE(b=lG8-dlD=!8}IY$ z_`vkUg!wdrTRHRSkii3Y4BipvRKU@E12+bO{!Ubs0U^Z16u80Y z?Chp1TsT^hV5x9U#*^wkCQQZxo{>l>p_oH?N;kd*hCV=5y`|aEB9j6Bh}^lM9OA$N zpc0~eCdSHtA0!zgA83dR?S)_voDDDA3&&yITnz-t2@qXr^$>q9+blAPyFC!p8cXUkO&@CI&PrC!nh|LkxqYa60uf z&JY8NmhfDPs>-NyAQ6Q<(&-yLaq1Qn-}hJ;{qfPts%(z(iVArg21Xn&2!!@(T-X6U z#0_t6Vy+9GY{6Q?L8mJ%6+dQM?@PUYy}#D8YemT#bw5@#P&cXi@E<&@2YMxH5ayeC z#~%*tJh>1V9!@W-Y&Ftxq*@Hq@5DSPP(*rs#eTi~mX7sdv4f%)%$G7w8hQgl-wq7S zysS&Q8yRUap6xN}t9Km0gNSi2%s!vp?EIbl3kOAhN1s3(M?z>a!~`r4kC62@3Xa7L z1y@>*PRO`5+rA238|01+BxpU4WovCgJcmW6k1A!U~XW2!h!L^gInW z1Q?5#7G_JG#F+t@phgeaU>X6YZ$0VRva~NSj&@@bi=9d^N zN-VF-dk*ux4g)})G&_T{;Zy$ z2tk~gzgz%py6x@ljdk*+9)~B~1R~>lR#u$Q1~UXx#F?EgdAgJbYcG9Hmjq}3F(3Qr z;N+8ZhxFI{C$BKdOisf-mQ1LR$dhJ{q?hUI_HoH)m{!nGi4Ek)wjZK8KN|Tl7(@<= zD=xUdtiy7mvQ=T2FTRYc+ZxW$a=6_C5M8u*H_64m(eUc>~ZZy&iSD;p3acF!0{wPTX zlHDc=XJU{Q`(ZPTgy1b_Kt#}xU~)eyE4uC>k~*AkzcEQ{F_ zT*(6Q@e%U-vAuDQ%{8B-Y;4u_uYrVC}dc)`oPA_woY<_34Z7+ znwfdc$xdtAerfO zRyQ(`!KMaWQb8>YH7i~gJ6a&Dm1U)L*2xN!&??DFjRR(2WVTczFp(m%^`?SLH!e27 zX3}+BY#GX`QzTTH4Z5xs_Pi`v+>ia(4}YCBO~c8@ci!)P-sf`eHP_B!)wZ`fZ6&E1vP}!sBR*3VDC+zZ)+oXukDya%eW0Fa$)|GZL%(V z%T|z}KMV=P)8GEck_H`2)YIeGlpYAOH8nMAoTq55m!i45N|u9LOU%EE-2j+=3{}}F zrBz0w&TzYiSA3N7AxKuI4gC$sZOTb?`o=DwnO^jMx}~I)mE=VOp!GGG&C&Qdag+-0 zxMWjkqf#ZD_G851fm4Ws{)(aJRl%pJ&ExMf-!3y*|6T$JOv%8Grk%_>M&s+yA+LlT zf(94=1yJR824g6O2%^UAIXTZO|CoI8g1|CZzaJPN3YJImMPOwpY^i1y$93zuOuf%z z7Z29ILi`l)cYHzu8Kp4Bq0|0c0F48(mJX)Fbc=9Q(YYZ${wX{iy`bNygM>qr@-z~Q_`;tG2c(_fsL5d5)D%vKzrkPt zCJ(R96#L4a(9WZ{>m60z8c#ommDHG)Pe2_vO5MtYGAPr%8aezRPWn)ZjS22_8RhKY zg!l?GMD(fl-=p_SgI8a&>nwPVuE-dBqaz>y%0%lxk}t}0Bd<2`fKHDZ0>?EJJ_QU) z&IxkM8BM0wV6#HPLKIpmikiVOKMOqqF@%g{qymS$rM6xgFQTA`Q4YvU01XPqWWi(~ zBj|9Pnf(}U9Yx}H0piQwF&;QBIo;RCy*GD+U|=ezc*xj5GbXUuBwH$yO5F$2TSH_ Yb(W;PymM_iUhcryk-_6$+*VNe7kgsz?f?J) literal 0 HcmV?d00001 diff --git a/docs/accessibility/how_a11y_works_3.md b/docs/accessibility/how_a11y_works_3.md new file mode 100644 index 00000000000000..1e2f7a58bcd492 --- /dev/null +++ b/docs/accessibility/how_a11y_works_3.md @@ -0,0 +1,472 @@ +# How Chrome Accessibility Works, Part 3 + +This document explains the technical details behind Chrome accessibility +code by starting at a high level and progressively adding more levels of +detail. + +See [Part 1](how_a11y_works.md) and +[Part 2](how_a11y_works_2.md) first. + +[TOC] + +## Abstracting platform-specific APIs + +In [Part 1](how_a11y_works.md) we talked about how each platform has its own +accessibility API. Chromium originally had the platform-specific accessibility +APIs scattered throughout the code, but today a large fraction of the APIs for +Windows (including IAccessible, IAccessible2, and UI Automation), Linux, and +macOS have all been isolated and abstracted in one place that makes it +relatively easy to write cross-platform accessibility code. + +These abstractions are all in the +[ui/accessibility/platform]( +https://source.chromium.org/chromium/chromium/src/+/main:ui/accessibility/platform/) +directory. + +First, gfx::NativeViewAccessible is a typedef used throughout Chromium to +represent an instance of the platform-specific accessible object on the current +platform. It's defined alongside gfx::NativeView, gfx::NativeFont, +gfx::NativeEvent, and other similar types that have equivalents on each +platform. Note that these are not wrappers or abstractions; they're just +typedefs enabling you to write a function that returns an instance of the +appropriate type on each platform. For accessibility, gfx::NativeViewAccessible +is defined to be IAccessible* on Windows, id on Mac (where 'id' is the type for +a generic Objective-C object, which has to implement the informal +NSAccessibility protocol), and AtkObject* on Linux. + +The main class in ui/accessibility/platform is AXPlatformNode. When you call +AXPlatformNode::Create, you'll get back an object that implements the +correct interfaces for the platform you're running on - currently Windows, +macOS, and desktop Linux are supported. + +For each AXPlatformNode, you need to provide an AXPlatformNodeDelegate - an +instance of a class that you implement in order to provide all of the +accessibility details about that node, in a cross-platform way. + +While AXPlatformNodeDelegate is pure virtual, a base class is provided, +AXPlatformNodeDelegateBase, with default implementations of nearly all +of the virtual functions. You can inherit from AXPlatformNodeDelegateBase +and override only a few functions in order to easily get a working object. + +As a brief sketch, if you had a custom-drawn button and you wanted to +make it accessible, you could define a subclass like this: + +```C++ +class MyButtonAXPlatformNodeDelegate + : public AXPlatformNodeDelegateBase { + MyButtonAXPlatformNodeDelegate() + : AXPlatformNodeDelegateBase() { + ... + } + + const AXNodeData& GetData() const override { + ... + } + + int GetChildCount() const override { + ... + } + + gfx::NativeViewAccessible ChildAtIndex(int index) const override { + ... + } + + gfx::NativeViewAccessible GetParent() const override { + ... + } +}; +``` + +Then to construct the accessible object, you could just write this: +```C++ +MyButtonAXPlatformNodeDelegate delegate; +AXPlatformNode* accessible = AXPlatformNode::Create(&delegate); +``` + +## Events + +In the Chromium codebase, accessibility events are notifications sent from +the browser to assistive technology that something has happened. This is +the mechanism by which assistive technology can provide real-time feedback +as the user is interacting with the browser. Some common events found on +nearly all platforms include: + +* Focus changed +* Control value changed +* Bounding box changed +* Children changed (a node added, removed, or reordered one or more children) +* Load complete (a web page finished loading) + +While many platforms share the same types of events, they're not standardized +at all, and platforms have very different names for events and different +semantics around which events are fired when, and where. As a few examples: + +* On macOS, there are separate events for expanding and collapsing a row + in a table or tree, vs expanding or collapsing a pop-up menu +* On Android there's a separate event for the checked state changing, while + on other platforms there's just a generic state changed event +* On Windows there are SHOW and HIDE events that need to be fired when + a node or subtree is created or destroyed + +In many cases, assistive technology is co-developed with initial accessibility +support in a platform's native widget toolkit - for example, TalkBack was +co-developed with the accessibility support in Android Views, +and NSAccessibility was co-developed +with AppKit's initial accessibility support. One thing that invariably seems to +happen is that event notifications get added to let assistive technology know +about changes to state of the app. + +Then when a new app comes along and needs to do some custom drawing or otherwise +implement some custom accessibility code, implementing those events ends up +being tricky. If the right events aren't fired in exactly the right order, the +assistive technology gets confused, since it was only built and tested with one +event sequence. + +This forms an implicit contract between the server and client, but it's one +that's rarely properly documented. + +For a cross-platform product like Chromium that needs to support the right +set of events to fire across so many platforms this gets very tricky. In +the early days we tried to have Blink fire the superset of all events needed +on any platform, but this often resulted in duplicate events or subtle +bugs, and a tendency for an event-related fix for one platform to accidentally +break another platform. + +Chrome's solution to this now is what we call "implicit" events. Blink, and +other parts of the codebase that build an accessibility tree simply notify that +an accessibility node is dirty, or an entire subtree is dirty. The +infrastructure crawls the dirty nodes and creates a tree mutation and propagates +it to all client interfaces. + +At the level of the client interface, we generate implicit events based on +changes to the accessibility tree as observed from that client's perspective, +using a class called AXEventGenerator. + +This allows us to keep the code that implements a particular contract in one +place and eliminate subtle differences between different types of content. + +### AXEventGenerator + +AXEventGenerator is based on the idea of applying atomic updates to an +accessibility tree. As described in +[Data structures used by the accessibility cache]( +how_a11y_works_2.md#Data-structures-used-by-the-accessibility-cache) +in part 2, an AXTree is a "live tree" that's currently being served, +and an AXTreeUpdate is a serializable data structure that represents +either a snapshot of a tree or an atomic update to apply to an existing +tree. When AXTree applies an atomic AXTreeUpdate, it allows listeners to +get callbacks for any changes that happened to the tree. In particular, +it keeps both the old and new data for each changing node temporarily +so that listeners can trigger actions based on changes. + +AXEventGenerator is thus an AXTree listener. It considers every node +that changed in the tree and figures out what events to fire. It builds +up the set of events and continues modifying it until the atomic update +is finished, enabling it to consolidate and remove duplication. + +As one example of that, a live region is a portion of a web page that +may trigger assistive technology to notify whenever an update occurs. +On some platforms, Chromium needs to fire a "live region changed" +announcement on the root of the live region whenever it changes. +AXEventGenerator keeps track of any changes that happen within a live +region and ensures that exactly one "live region changed" event is +fired on the live region root. + +There are a small number of exceptions - events that can't be fired +via AXEventGenerator. These are things that can't be inferred just +from tree changes. One such example is the "autocorrection occurred" +event. When the browser performs an autocorrection while the user is +typing, the state change just looks like any other edit. The event +ensures assistive technology can announce the autocorrection. + +### Focus events + +Focus events are one of the most important types of events, because +changing focus is often one of the most important events for assistive +technology to announce, and the focused node is the one that will be +the target of any input events. + +However, one of the challenges with focus events is that there's only +one element on the entire desktop that has focus at any one time, but +individual windows or iframes might not always be aware of the global +state of the entire desktop at the time they experience a focus change +within their scope. This can lead to a race condition. + +As an example, suppose that a user clicks a button in a web page, which after a +couple of seconds pops up a dialog and brings focus to an OK button. At the +same time, the user clicks on a different window to activate it, moving +focus to that window's active element. + +Because the windows come from different processes, the two focus events +(from the first window's dialog, and from the second window's active element) +could arrive at the browser process in either order. Here's an illustration +of this race condition: + +![This diagram illustrates a race condition where the user clicks a button +to open a dialog in one window, then before it opens activates another +window that focuses a text field. The focus events could arrive to the +browser process in either order.]( +figures/focus_race.png) + +From the standpoint of the browser, there's always only one node that has +focus. What's important here is that accessibility is completely consistent +with the browser in terms of reporting the correct node that has focus. + +The solution here is that only the browser is the source of truth when it +comes to which window has focus. Once we know which window has focus, each +accessibility tree tells us which node has focus within that tree. + +As a result, when a focus change happens in an AXTree, we can't just fire +a platform-specific focus event directly. Instead, we use that as a cue to +compute global focus and fire an update if needed. Here's an outline of the +algorithm: + +* Anytime focus changes in any accessibility tree, OR when the focused + window or iframe changes, recompute the focus. +* To compute focus, start with the focused window (or active window, + depending on the platform). If focus is in web content, + see what node is focused there. If that node is an iframe, recursively + jump into that iframe to see what's focus. +* Take the resulting deepest focused node and compare it to the last focused + node we computed. If it's different, fire a platform-specific accessibility + focus event. + +This ensures that accessibility focus events are always reliable and in sync. + +No other accessibility events have the same issue. Events like value changed, +selection changed, etc., are safe to fire even if a window is in the +background. Some assistive technology may be paying attention to background +windows. + +## Actions + +In Chromium accessibility terminology, Actions flow the opposite direction from +events. Actions are when assistive technology wants to modify or interact with +the app on behalf of the user, such as clicking a button, selecting text, or +changing a control value. + +Note that screen readers rely very heavily on events, and partially on +actions. Users often use a combination of accessibility actions along with the +keyboard to directly drive an application, or have the screen reader warp the +mouse cursor directly to an element and simulate a click on that element. + +In contrast, assistive technology such as voice control makes heavy use of +actions and relies much less on events. Voice control relies heavily on +actions that enable directly changing control values, entering text, +activating buttons and links, and scrolling the page. + +Other assistive technology such as magnifiers are in-between - they may +follow focus events a lot but make heavy use of scroll actions. + +For the most part implementing actions is relatively straightforward. +The action is received by the part of the code that implements the +platform-specific accessibility APIs. It forwards the action to the +corresponding accessibility wrapper node in Blink, and that node +calls the appropriate internal APIs to directly manipulate the underlying +element, such as clicking a button or changing the value of a control. + +One minor complication is that on many platforms, actions are supposed to +return a success/failure code. Since actions are obviously implemented +asynchronously, Chromium can't know for sure if an action succeeded, so it +has to return success if an action seems valid, even though there's a +chance it might not actually succeed. + +## Hit testing + +One specific special case of an action is a hit test. This is an API where +the assistive technology gives the x, y coordinates of a location on the +screen and asks the application (Chromium) to return which accessible +object is at that location. + +Applications of hit testing include: + +* Touch exploration on a touch-screen, or features to describe the + element as you're hovering over it with the mouse +* Using accessibility debuggers where you can click on an element and + get its accessibility properties + +Unfortunately on some platforms a hit test is a synchronous API. This is +a challenge because it's difficult to properly compute the correct element +at a location given just the accessibility tree, but blocking to wait for +a proper hit test in the render process can lead to deadlock and jankiness. +So Chromium employs the following approach: + +* The first time a hit test is received, it does an approximate hit test + based on the bounding boxes in the accessibility tree. This often returns + the correct result, but could fail in cases of complex layering or + non-rectangular objects. +* Subsequently, it makes an async call to the render process to do a proper + hit test and get the correct resulting element, and also the visible + bounding box of that element. +* The next hit test that's received, if the coordinates are within the + bounding box of the most recent proper hit test result, it returns that + result, which is correct. If the coordinates are outside of that bounding + box, go back to the first step. + +This algorithm works very well in practice when the user is moving the mouse +or dragging their finger across the screen, because we get dozens of hit +tests per second. At the edges of objects, the wrong result may be returned +for a few milliseconds, but as soon as the async result comes back, the +correct result is then returned. + +So for interactive use, it's quite seamless and reliable for users, while +still providing reasonable behavior in the less common circumstances where +a single hit test is called. + +## Relative coordinates + +Up until now we've hinted about the fact that every node in the accessibility +tree stores a bounding box, but we haven't gone into much detail as to +how that bounding box is stored. + +If we always stored the bounding box in screen coordinates, then every time +a window is dragged or scrolled, or any time any part of the page moves or +scrolls, all of the affected bounding boxes would need to be recomputed, +which would involve a lot of recomputation and sending information from +render processes to the browser process. + +To minimize that work, in Chromium accessibility nodes store relative +coordinates. + +In particular, every node stores the following fields in a struct +called AXRelativeBounds: + +```C++ +struct AX_BASE_EXPORT AXRelativeBounds final { + int offset_container_id; + Rect bounds; + Optional transform; +}; +``` + +The first field is the ID of the node's container, which can be any ancestor of +a node. That's the node that the bounds are relative to. + +The next field is the local bounding rect, relative to that container. + +The last field is an optional 4x4 transformation matrix, which can be +used to encode things like scale factors or even 3-D rotations. If this +concept is unfamiliar to you, search for tutorials on 4x4 transformation +matrices in the context of 3-D computer graphics. + +Computing the global bounding rect of a node is meant to be straightforward. +Start with the local rect. As long as the node isn't the root, keep walking +to the container node, applying the transformation matrix and adding the +bounds origin as you go. + +In addition, there are a couple of other fields relevant to the bounds +computation that are stored as sparse attributes in AXNodeData. These also +affect the bounds computation. + +* bool clips_children; +* int x_scroll_offset; +* int y_scroll_offset; + +For more information on bounding boxes, clipping, and offscreen, see +[Offscreen, Invisible and Size](offscreen.md). + +## Text bounding boxes + +Most platform-specific accessibility APIs have a number of features +specifically to deal with text. Some of those APIs allow querying the +bounding box of an arbitrary range of text - often the text caret or +selection, but not necessarily. Applications include: + +* Highlighting text as it's read aloud +* Scrolling one particular text range into view +* Drawing highlights around the caret or selection to make it easier + for users to see them + +Because these APIs are synchronous, they must be served directly out of the +accessibility cache. That means that the accessibility cache needs to have +enough information to be able to retrieve the bounding box of any arbitrary +range of text on-screen. + +It would require quite a bit of memory to store the bounding box of +every individual character. To save memory, the following representation +is used: + +In the accessibility tree, we keep track of text nodes called "inline text +boxes". This corresponds to a similar concept in Blink, which is also +sometimes called a "text run". The idea is that given a single text node, +the text can be broken down into a sequence of text runs that each have +the following properties: + +* Each text run is on a single line +* Text goes a single direction (left-to-right, for example) +* The characters in that text run are all contiguous + +In the most common scenario, a single text node contains multiple +lines of text (potentially due to automatic wrapping with soft +line breaks). In the accessibility tree that node would have multiple +inline text box children, one for each line. + +Imagine we have the following paragraph, that's very narrow so it +wraps as follows: + +``` +The quick brown fox +jumps over the +lazy dog. +``` + +In the accessibility tree, it might be represented like this: + +``` +Paragraph + Static Text "The quick brown fox jumps over the lazy dog." + Inline text box "The quick brown fox " + Inline text box "jumps over the " + Inline text box "lazy dog." +``` + +Each inline text box comes with its own bounding box and text direction. +Then, to store the bounding box of every character, all we need to +do is store the width of each character. Since we know all of the +characters are written continuously in a line going the same direction, +we can use the bounds of the inline text box and the width of each +character to compute the bounding box of any individual character. + +The AXPosition class abstracts most of this computation. + +## Iframes + +The last piece of complexity to address is that up until now we've +assumed that a single web page corresponds to a single frame, so a +web page is a single process. + +In Chromium, for security reasons iframes can also be running in +separate processes. This isn't always the case - for one thing, if +system resources are low, Chromium won't keep creating new processes, +and also, frames from the same origin (i.e. from the same website) +need to be in the same process so they can communicate synchronously +via JavaScript. But, frames from different sites can be in different +processes so accessibility code needs to deal with that. + +The essential challenge is that each frame, which may be in its own +process, needs to maintain an accessibility tree - but the end result +needs to be stitched together into a final resulting accessibility +tree in the browser process. Iframes are mostly just an implementation +detail; users and assistive technology are rarely concerned with this +detail. + +In order to stitch frames together: + +* Each accessibility tree gets a globally unique ID, we call it an AXTreeID. + For security reasons this is an UnguessableToken. +* An iframe element in an accessibility tree contains the AXTreeID of its + child frame. +* In the browser process, we keep a hash map of all of the trees, and + also cache the reverse direction (e.g. the map from the root of a + tree to its parent node). + +In order to reduce complexity, Chromium accessibility is built around the +concept that every frame is its own accessibility tree, no matter whether +the frame is in a different process or not. The advantage of this approach +is that the same codepath is used whether iframes are in the same process +or a remote process. If iframes break, they all break - that simplifies +testing and reduces the number of cases to consider. + +The concept of embedding one accessibility tree in another using an +AXTreeID is also exploited even more in Chrome OS accessibility, where +it's used to embed Android applications and more. diff --git a/docs/accessibility/overview.md b/docs/accessibility/overview.md index 73cf035e3b5374..0bbbdb8eedcaf5 100644 --- a/docs/accessibility/overview.md +++ b/docs/accessibility/overview.md @@ -518,6 +518,10 @@ which is renderer-side code, and in JavaScript by the [automation API]. The API is defined by [automation.idl], which must be kept synchronized with [ax_enums.mojom]. +## Further reading + +For more detail, read [How Chrome Accessibility Works](how_a11y_works.md). + [ax.mojom.AXActionData]: https://source.chromium.org/chromium/chromium/src/+/main:ui/accessibility/mojom/ax_action_data.mojom;l=13 [ax.mojom.RenderAccessibilityHost::HandleAXEvents()]: https://source.chromium.org/chromium/chromium/src/+/main:content/common/render_accessibility.mojom;l=47 [ax.mojom.RenderAccessibility.PerformAction()]: https://source.chromium.org/chromium/chromium/src/+/main:content/common/render_accessibility.mojom;l=86