<scribe> scribe: Wayne

<interaccess> trackbot, start meeting

<trackbot> Meeting: Accessibility Guidelines Working Group Teleconference

<trackbot> Date: 07 February 2017

<interaccess> Chair: AWK

Joining AGWG – WCAG members need to rejoin.

<AWK> +AWK

<Lisa_Seeman> yup

<Lisa_Seeman> hi

<AWK> https://www.w3.org/2004/01/pp-impl/35422/join

AWK: If you have not asked your AC rep to sign you up then ask your rep.
... You must rejoin.

<MichaelC> https://lists.w3.org/Archives/Public/w3c-wai-gl/2017JanMar/0458.html

MichaelC: I sent an email on the mailing list, there are a few boxes to check.

Testing Survey: https://www.w3.org/2002/09/wbs/35422/testing20170207/results

<Lisa_Seeman> that is a good way to chair - two screens

<Lisa_Seeman> will do that

AWK: Success criteria acceptance criteria. One big topic what types of testing? That is one of the keys. What does it mean to be testable? Automation is not required. There is human judgement. Confirm a filure automatically for language ID, you need a person to ID it as French. Is Usability testing sufficuent.
... Manual testing must incude user testing, and must not.

-1

JohnR: I have a hard time with not including users. It is weird.

Josh: I have experience with usability and users with disabilities. I have to push back on use testing on as a requirement. Availability is the problem. No I cannot require user testing.
... There is a big overhead and extreme cost.

JohnR: There are cloud based testing. When do we require user testing. If you want your product you must test with users.

<David_MacDonald> https://www.w3.org/TR/UNDERSTANDING-WCAG20/conformance.html

Josh: There may be new ways to reduce the cognitive load.

JohnR: There are cloud based testing sites.

<David_MacDonald> WCAG does this as a SHOULD, not a MUST "The content should be tested by those who understand how people with different types of disabilities use the Web."

<shwetank> I also support recommendation, but not a 'must' requirement

<Glenda> +1 user testing is very important and yet the data is very subjective

<David_MacDonald> Text says "Although content may satisfy all Success Criteria, the content may not always be usable by people with a wide variety of disabilities. Therefore, usability testing is recommended, in addition to the required functional testing. Usability testing aims to determine how well people can use the content for its intended purpose. It is recommended that users with disabilities be included in test groups when performing usability testing."

JohnF: There will be inconsistency. UT to ensure accessible. The ability to have consistent testing.

<alastairc> +1 to Josh's point on burden, and JF's point on mixed results point. WCAG can't mandate people's process.

<Joshue108> + 1 to John and thats why a good test facilitator is needed to read between the lines.

<Wilco> +1 to concerns of inconsistent results

Lisa: There are things that are not subjective like can they finish this task. They managed to access help. These are reliable. Add and A+ not just A level.

<alastairc> I like the idea of a WCAG 'plus' for public / larger companies, but that's Silver thing in my mind.

lisa: If you ask can you get a product. We can get quantitiative.

<Glenda> I agree with everything Kathy is saying.

Kathy: We do not require accessibility support method. So user testing may not control for AT. We need to include more into how to make things to work for disability. User's knowledte of the AT, or Browser bugs.
... Benchmark that user cannot do this, is not a test because of other factors.

<JohnRochford> Kathy has convinced me. I'm chaning my vote.

MikeG: We are talking about #2 be testable through manual or automated processes? Are we talking about that.
... Nobody has defined user testing. Someone is doing testing. It is should be incorporated if you say you must

<AWK> Wayne: wondering if there are things at this time that we can't really measure with existing tests

<Joshue108> WD: I'm wondering if there are things that we can't measure by the tests we originally envisioned.

<Joshue108> WD: For some esp cognitive issues - this could be the only way to capture them

<Joshue108> WD: There are things where we dont always have agreement, and testing isn't always perfect anyway.

<Glenda> ACAA - Requires carriers to test the usability of their accessible primary Web sites in consultation with individuals or organizations representing visual, auditory, tactile, and cognitive disabilities.

<Joshue108> correct

<Lisa_Seeman> the problem is cost glenda

Glenda: (see Glenda's inclusion) What if we require user testing, and later add what to do with it.

<JohnRochford> my q+ is "Why are we saying manual processes for testing *must not* include user testing?"

<Lisa_Seeman> i want my ho,me page to conform

<Zakim> Joshue, you wanted to say we have to support inclusive design methodologies and good usability

<kirkwood_> +1

<Lisa_Seeman> are there enough blind people for all the user testing of every website?

Jossh: That exciting to do this, It takes it off the dry domain. I wonder you could strongly suggest that user testing could be don regardless of the outcome.

<Glenda> Hands down, my fav part of ACAA usability testing is with people with cognitive disabilities. It has been a gold mine of data for improving the user experience for all.

<Lisa_Seeman> (there are enough coga users)

Jossh: User is a good way to get people in the room. If there is a way to get this in the room. But not require. We are trying to use techniques encourage user testing. There is a halo effect.

AWK: If we consider a user test that says if it passes the chriterion passes.

<Zakim> MichaelC, you wanted to say we want good input from users in setting the guidelines, but guidelines are too abstract and generic to mandate user testing as part of conformance

DavidM: User testing is included as a recommended. We cannot change from a should to must in 2.1. I think it is required it is not. It is strongly recommended.

MichaelC: A major purpose of guidelines is to give a start, but uses testing is strongly recommended. I don't see it can be part of a conformance model.

<KimD> +1 to Michael's comments

MichaelC: User testing is impractical for guidelines.

MichealP: We are not talking about accessibility. ... The people who might benefit might be for a well organized testing lab. I don't see it passing.

MikeE: Practicality. There are still many small companies who do not testing. It would cause people to back away. It would be an impediment. Are we trying to make 2.0 compatible with 2.1.

<Zakim> AWK, you wanted to point out that policies that adopt WCAG can require user testing

<Zakim> alastairc, you wanted to say that an SC can't require usability guidelines, or in fact testing across UAs. The trick is translation from user-requirement to content requirement.

AWK: Is certainly can be a policy requirement, like the air carriers act.

<jon_avila> CVAA also includes some requirements for consulting with people that have disabilities. Revised ICT Standards and Guidelines from Access Board dropped the requirement

<AWK> AWK: Any policy uptake process might add user testing as a requirement

<Zakim> Joshue, you wanted to say there are complex ethics issues around user testing with people with cognitive issues

Alastair: There are difficulties with testing usability and testing across user agents.

<Lisa_Seeman> we have a issue paper coming that gives direction how to deal with this josh

<alastairc> Alastair: The trick is translation from user-requirement to content requirement, it is the content that has to pass, not the interpretation of that content.

<Lisa_Seeman> it is a must for good coga inclusion

Josh: There are problems with cognitive disabilities have special requirements. The impact on the individual may be harmful.

JohnF: Testing is repeatable. Not opposed to UT.

<Mike_Elledge_> have to go...bye all

<alastairc> NB: The survey results are now 11:0

AWK: User test is strongly recommended in addition to conforming to WCAG material. We cannot have UT as a requirement. Reparable, Cost and you can do it well.

<JF> +1 to AWK

<Joshue108> +1

RESOLUTION: User testing is not a required part of a manual testing process for WCAG test criteria.

<JohnRochford> +1

+1

<Makoto> +1

<KimD> +1

<adam-lund-tr> +1

<alastairc> I'd suggest: Usability testing is not required for manual testing of WCAG test criteria.

<Greg> "User testing is not a *required* part of manual testing for WCAG Success criteria."

<Lisa_Seeman> yet

<marcjohlic> +1

<alastairc> in the 2.x series.

<Wilco> +1

<Lisa_Seeman> +1

<gowerm> user acceptance testing?

<Mike_Pluke> +1

<alastairc> wow - UAT is different from usability/user testing

<jon_avila> usability is different from user testing IMO

<alastairc> +1

<laura> +1

<Greg> +1

Jeanne: I have no objection for internal use.

LisaS: In the wording experts may not be an issue with defining expert. 8 out of 10 with a high level of content. Not 100% sure.

Alastair: The bar is that accessibility experts can do the testing and they must have a high level of confidence.

<alastairc> Alastair: we should expect current experts (e.g. this group) to be able to read the SC & description, and then have a high degree of agreement for any particular page.

AWK: This aligns with 2.0 it is subjective.

<Lisa_Seeman> it might be more then just reading

<Lisa_Seeman> one discription. you might need to follow some links etc

<David_MacDonald> I have nothing

<AWK> WG agrees that SC should have a high degree of confidence of inter-rater reliability, approximated by the expectation that 8 of 10 experts in a area would agree.

<Lisa_Seeman> +1

<Lisa_Seeman> yes but this is the expectation

<Glenda> +1 (I like 8 out of 10)

me/ 8 out of 10 doesn't work in practice: http://dl.acm.org/citation.cfm?id=2141946

<JohnRochford> 8 out of 10 is a standard measure of reliability

<JohnRochford> +1

+1

<jeanne> +1

<KimD> +1

<Kathy_> +1

<Makoto> +1

<David_MacDonald> ÍDavid: I can live with it 8 out of 10

<laura> can live with it

<shwetank> can live with it

<JF> Not happy, but will not oppose

RESOLUTION: WG agrees that SC should have a high degree of confidence of inter-rater reliability, approximated by the expectation that 8 of 10 experts in a area would agree.

<erich> +1 8 out of 10

New SC Survey: https://www.w3.org/2002/09/wbs/35422/SC_20170207/results

<Joshue108> WD: I worry about setting ourselves up for not agreeing.

<Joshue108> WD: On basic stuff.

<Joshue108> WD: I like the wording, using confidence etc hitting 80 %

<Joshue108> WD: We don't always

<Joshue108> AWK: You happy with the language?

<Joshue108> WD: On average

SC Proposal: Graphics Contrast (Issue 9)

subtopic: Graphics Contrast

AWK: I what will happen with these SCs on the list we will do a survey we can put it into the editors draft. We don't need a call for consensus for editors draft. We do need a CfC for publish. If we decides it is good then we will but it on the

JohnR: How do I revise my criterion so that we can discuss it.
... I put the text in myself and it is a link to it.

AWK: Let us talk about this off line.

LisaS: I starting if 8/10 it goes in the editors draft. What about those that do not have enough support technology, what do we do.

AWK: There are easier the WG on call says this sounds reasonable it can be done with existing tools. We will get good feedback when we put it out. We cannot agree on this wo we don't want it to be in the editors working draft. There are those that are in between. We put it in the Ed draft and put in a note and we need feedback. The group needs to agree on if it cannot be done.
... The goal is to have a chritirion that is achievable.

LisaS: There are issues.

AWK: if it does not meet the requirements. Things can be deleted.

JohnF: As we advance our FPWD we need to have SCs then add a Note with user needs.

<Lisa_Seeman> yup

<Lisa_Seeman> to wayne

<AWK> John's idea is to publish a note that lists out the user requirements that we are not able to directly meet with WCAG 2.1, as a tool to help guide future development by authors/tool makers/etc

<Lisa_Seeman> -1 to john

<AWK> we will discuss John's idea later, but it seems like a great idea since we will not be able to address everything that is a requirement today and we can update it with new requirements as they are identified.

LisaS: (I missed most of this) If it is not in the draft it won't happen. .. I don't see it happening.

AWK: There is an onus that there need to be on the proposer to support them.
... There is an expectation that there is a high quality draft. We are proposing a refinement. We are reluctant to put out something that the commenters say this is more wrong than right

<Joshue108> +1

AWK: There might want to mark for comment

+1

<gowerm> +1 onus is on task forces to improve the lanugage based on feedback

<jnurthen> +1

<Zakim> JF, you wanted to say that

<alastairc> The main issues seem to be translating from user-requirement to content requirement. It is impossible to test as a user-requirement, it needs to be something that is identifiable about the the page.

<AWK> +Rachael

<laura> bye all

trackbot, end meeting

This is scribe.perl Revision: 1.148  of Date: 2016/10/11 12:55:14  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: RRSAgent_Text_Format (score 1.00)

Succeeded: s/Avaialblity/Availability/
Succeeded: s/User testing is not part of a manual testing for WCAG test criteria./User acceptance testing is not a required part of a manual testing process for WCAG test criteria./
Succeeded: s/WCAG test criteria/WCAG Success criteria/
Succeeded: s/User acceptance testing/User testing/
Succeeded: s/one/on/
Found Scribe: Wayne
Inferring ScribeNick: Wayne
Default Present: AWK, mattg, jeanne, Greg_Lowney, David_MacDonald, JohnRochford, JF, Adam, Lund, Wayne, alastairc, marcjohlic, Lauriat, Makoto, Joshue108, MichaelC, KimD, lisa, seeman-kestenbaum, Laura, Kathy, MikeGower, kirkwood, Mike_Pluke, allanj, jon_avila, steverep, JamesNurthen, erich, Rachael
Present: AWK mattg jeanne Greg_Lowney David_MacDonald JohnRochford JF Adam Lund Wayne alastairc marcjohlic Lauriat Makoto Joshue108 MichaelC KimD lisa seeman-kestenbaum Laura Kathy MikeGower kirkwood Mike_Pluke allanj jon_avila steverep JamesNurthen erich Rachael shwetank Glenda
Regrets: Jim_Smith Srini Neil_Milliken Bruce_Bailey EA_Draffan
Found Date: 07 Feb 2017
Guessing minutes URL: http://www.w3.org/2017/02/07-ag-minutes.html
People with action items: