This document describes the formal schema of the Evaluation and Report Language (EARL) 1.0. EARL is a vocabulary, the terms of which are defined across a set of specifications and technical notes, and that is used to describe test results. The primary motivation for developing this vocabulary is to facilitate the exchange of test results between Web accessibility evaluation tools in a vendor-neutral and platform-independent format. It also provides reusable terms for generic quality assurance and validation purposes.
Status of this document
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at https://www.w3.org/TR/.
This Evaluation and Report Language (EARL) 1.0 Schema document is published as a W3C Working Group Note because the Evaluation and Repair Tools Working Group (ERT WG) reached the end of its Charter. At this time there are not sufficient implementations to complete EARL 1.0 Schema as a W3C Recommendation.
EARL 1.0 Schema completed Last Call stage before this publication. It has some implementations and may be reused in part or in whole by other W3C groups in the future, such as the Accessibility Conformance Testing (ACT) Task Force.
If you wish to make comments regarding this Evaluation and Report Language (EARL) 1.0 Schema document, please send them to [email protected] (publicly visible mailing list archive).
Publication as a Working Group Note does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
The Evaluation and Report Language (EARL) defines a vocabulary for expressing test results. It enables any person, software application, or organization to assert test results for any test subject tested against any set of criteria. The test subject might be a website, an authoring tool, a user agent, or some other entity. The set of criteria may be accessibility guidelines, formal grammars, or other types of quality assurance requirements. Thus, EARL is flexible with regard to the contexts in which it can be applied.
This document provides the core schema of EARL. Other parts of the EARL suite of specifications include:
EARL is not a comprehensive vocabulary for describing test procedures, test criteria, or test requirements but, rather, for describing the outcomes from such testing. EARL can be supplemented by test description vocabularies or other vocabularies for different aspects of the testing cycle.
The assumed audience of this specification is developers of software tools and processes who want to express test results in a machine readable format that is semantically rich. More introductory background about EARL as well as specific guidance for quality assurance tool developers, in particular for web accessibility evaluation tool developers, is provided in the Evaluation and Report Language (EARL) 1.0 Guide.
This document assumes that the reader is familiar with the Resource Description Framework (RDF) and can read its XML serialization. Readers who wish to understand more about RDF should read a general introduction or the RDF Primer [RDF-PRIMER].
The RDF representation of the vocabulary defined by this document uses the namespace http://www.w3.org/ns/earl#. The prefix earl is used throughout this document to denote this namespace. Other prefixes used throughout this document include:
This section describes the classes defined by this document. Every test result in EARL is expressed as an assertion. An EARL Assertion contains the following information:
Assertor
This can include information about who or what ran the test. For example human evaluators, automated accessibility checkers, or combinations of these.
Test Subject
This can include web content (such as web pages, videos, applets, etc.), software (such as authoring tools, user agents, etc.), or other things being tested.
Test Criterion
What are we evaluating the test subject against? This could be a specification, a set of guidelines, a test from a test suite, or some other testable statement.
Test Result
What was the outcome of the test? A test result could also include contextual information such as error messages or relevant locations within the test subject.
EARL provides flexibility to describe different types of assertions, such as those carried out by automated testing tools or by human evaluators, or those made about generic testing requirements or specific test cases.
Note: According to this example, "Cool Tool" is a resource of type foaf:Agent. According to example 5, it is also a resource of type earl:Software. These are not contradictory statements and are valid RDF representations.
Test Criterion - a testable statement, usually one that can be passed or failed. It is a super class for all types of tests including things such as validation requirements, code test cases, checkpoints from guidelines such as Web Content Accessibility Guidelines [WCAG], or others.
Example 8: Instance of a test case that is described with a title and its relationship to a test suite.
<earl:TestCase rdf:about="http://www.w3.org/TR/WCAG20-TECHS/H36">
<dct:title xml:lang="en">H36</dct:title>
<dct:description xml:lang="en">Technique H36 - Using alt attributes
on images used as submit buttons </dct:description>
<dct:isPartOf rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/"/>
<dct:hasPart rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/H36#H36-tests"/>
</earl:TestCase>
Test Result - the actual result of performing the test. It includes both machine-readable values as well as human-readable description of the results (typically error messages).
Example 9: A test result with a validity of fail and a description of the problem in English, and encoded in XHTML format.
<earl:TestResult rdf:about="#result">
<earl:outcome rdf:resource="http://www.w3.org/ns/earl#failed"/>
<dct:title xml:lang="en">Invalid Markup (code #353)</dct:title>
<dct:description rdf:parseType="Literal" xml:lang="en">
<div xmlns="http://www.w3.org/1999/xhtml">
<p>The <code>table</code> element is not allowed to appear
inside a <code>p</code> element</p>
</div>
</dct:description>
<earl:pointer rdf:resource="#pointer"/>
<earl:info rdf:parseType="Literal" xml:lang="en">
<div xmlns="http://www.w3.org/1999/xhtml">
<p>It seems the <code>p</code> element has not been closed</p>
</div>
</earl:info>
</earl:TestResult>
Test Mode - describes how a test was carried out. It reflects the information provided by the Assertor and is used to simplify some commonly used queries.
Manual - where the test was carried out by human evaluators. This includes the case where the evaluators are aided by instructions or guidance provided by software tools, but where the evaluators carried out the actual test procedure.
Semi-Automatic - where the test was partially carried out by software tools, but where human input or judgment was still required to decide or help decide the outcome of the test.
In cases where it is necessary to create further instances of earl:OutcomeValue, it is recommended that one of the following types be employed in addition:
Undetermined - the class of outcomes to denote an undetermined outcome. Usually this happens when an automated test requires human judgement to make a definite decision.
Not applicable - the class of outcomes to denote the test is not applicable. This could be due to a mismatch between the test and the subject or for any other reason.
Not tested - the class of outcomes to denote the test has not been carried out. This is useful for reporting as well as for other uses of progress monitoring.
Example 12: A test result with a non-standard outcome of "Warning", which is a type earl:Pass.
<rdf:Description rdf:about="http://example.org/my/warning#warning">
<rdf:type rdf:resource="http://www.w3.org/ns/earl#Pass"/>
<dc:title xml:lang="en">Warning</dc:title>
<dc:description xml:lang="en">the subject passed the test but there are warnings</dc:description>
</rdf:Description>
<earl:TestResult rdf:about="#result">
<earl:outcome rdf:resource="http://example.org/my/terms#warning"/>
</earl:TestResult>
A Software is any piece of software such as an authoring tool, browser, or evaluation tool. It can be used to describe an Assertor, such as a validation or other quality assurance tool, and it can be used to describe a Test Subject (for example to test compliance of an authoring tool to Authoring Tool Accessibility Guidelines [ATAG] or of a browser to User Agent Accessibility Guidelines [UAAG]).
Note:earl:Software is a sublass of doap:Project to denote the narrower meaning of executable "Software", that is an outcome of a "Project".
This section describes the properties defined by this document. EARL also uses properties from external vocabularies to provide additional information where necessary.
where the test was partially carried out by software tools, but where human input or judgment was still required to decide or help decide the outcome of the test
EARL is the result of the work of many people over the past. The editors would particularly like to thank Wendy Chisholm, Sean B Palmer, and Daniel Dardailler, whose contributions have included editing the first versions of the EARL specifications, and Leonard Kasday who set the work in motion to develop EARL. The editors apologise for any names left out of this list, and will endeavour to rectify any errors noted in comments.
Shadi Abou-Zahra, Philip Ackermann, Chrisoula Alexandraki, Shane Anderson, Myriam Arrue, Gabriele Bartolini, Giorgio Brajnik, Dan Brickley, Dan Connolly, Karl Dubost, Nick Gibbins, Al Gilman, Emmanuelle Gutiérrez y Restrepo, Dominique Hazaël-Massieux, Nadia Heninger, Sandor Herramhof, Ian Hickson, Björn Höhrmann, Carlos Iglesias, Nick Kew, Johannes Koch, Jim Ley, William Loughborough, Rui Lopes, John Lutts, Charles McCathieNevile, Libby Miller, Tom Martin, Yehya Mohamed, Daniela Ortner, Dave Pawson, Eric Prud'hommeaux, Pierre Queinnec, Chris Ridpath, Romain Roure, Christophe Strobbe, Michael Squillace, Aaron Swartz, Olivier Thoreaux, Carlos Velasco, Konstantinos Votis, and Rob Yonaitis.
