Modality Component to Host Environment DOM Requirements and Capabilities Assessment

1 Overview

This document defines the capabilities and requirements of a common DOM-based API between modality components and their host environment. The document further describes the current level of support for these capabilities in DOM 1, DOM 2, and DOM 3. This document is not a formal API definition. A formal API definition would need to address a single level of DOM and further specify common interfaces and events.

Modality components perform interface tasks pertinent to their particular interface modality (e.g voice, pen, visual display). Modality components can perform interface tasks ranging from the simple input/output such as pen strokes collection, audio playback to complex transaction like dialogs and card stacks.

The host environment allows multiple modality components to share data such as a semantic result and the confidence value associated with that result. The host environment also coordinates the activities of modality components in a multimodal application; for instance, activation, deactivation, display/prompting.

A multimodal application is comprised of many modality components interacting with the user, all coordinated by the host environment.

A common DOM-based API between modality components and their host environment allows for the creation of new modality components that will work with existing host environments. This common API would also allow for the creation of new host environments for various architectures and devices.

This DOM-based API requirements and capabilities analysis has been performed with the multimodal browser developer in mind. The objective is to enable portable multimodal content. These APIs may be (and in most cases will be) entirely hidden from the application developer. These APIs provide the foundation which allow for higher-level language constructs specific to the modality component or host environment.

2 General DOM Implementation Features

These are the general features required of a Host Environment browser implementation to support the integration of a Modality Component. Features relating to DOM support are assumed to be implemented by the DOM Implementation component of the browser.

The intent is that this DOM framework should work in a multi-document scenario. Further specification of that would be required of an interface definition standard

2.1 Component Parsed/Loaded Notification

Enables the browser to know when a component (or a proxy to a remote component) is present on the page. In response, the host enables component to register itself and latch to host environment.

2.2 Component Unloaded Notification

Enables the browser to know when a component (or a proxy to a remote component) is removed from the page. In response, the host enables component to unregister itself.

2.3 General DOM Errors

Enables the browser to be aware of DOM errors raised by the component.

2.4 URL of the Current Page

For components which specify remote resources via URLs, this enables the browser to resolve references which are relative to the current page.

2.5 Access to Current Page as Document Object

Enables access to the interfaces described in the Document section below.

3 Document Interfaces

These are the interfaces which are necessary for component integration into a document supporting the Document interface.

3.1 Component Declaration and Instantiation

This subsection describes the interfaces necessary for the declaration and instantiation of modality components.

3.1.1 Namespace Management

Document interfaces for: identification (find namespace declaration, prefix) and determine parent element tree (for scoped namespaces). Enables the host environment to identify a component by its namespace.

3.1.2 Location of Component in Document

Parent traversal interfaces. Enables the host environment to know in which elements of the document the component is placed, and to implement any location-dependent semantics accordingly. This includes scoped properties which may be inherited (e.g. xml:lang).

3.1.3 Component Self-Identification

Attribute of type ID on component node. Enables the mapping of a given component on the page to the browser's instantiation of that component.

3.2 Access to Component

This subsection describes the interfaces that a modality component may choose to enable to allow the host environment to directly manipulate the component.

3.2.1 Read Interfaces

The component exposes its public interfaces within the framework of the DOM of the current page. Access to the properties, methods and events of the modality component is then available to the flow control mechanisms of the host environment (script, SMIL, etc.). In general, these interfaces will be modality-specific, and defined for each component according to its function, however, certain common interfaces may be implemented which are standard across different modality components (e.g. Start and Stop methods, result events).

3.2.2 Property Write Interfaces

Value setting and rewrite interfaces. Enables the host environment to manipulate properties without modifying document tree structure.

3.2.3 XML Structure Modification Write Interfaces

Enables the host environment to manipulate the internal content and/or structure of the component. Examples of this may include manipulating an EMMA result before binding it into the host environment data model, and updating an inline grammar before beginning recognition.

3.3 Data Binding into Host Environment

This subsection describes the interfaces necessary for data binding from the Modality Component into the Host Environment. In the following requirements, the data to be bound called the 'result', and the location to which the data is to be bound is called the 'target'.

3.3.1 Navigation to Target Node

This enables the selection of the binding target. Since specification of target node may differ among components (for example, some may use IDs, others XPath, etc.), different means of selecting the target node may be required for different components.

3.3.2 Write Access to Target Node

Value setting/rewriting interfaces (no restructuring). Enables the writing of the result data into the target node.

3.3.3 Reading Values from Host Environment Data

This subsection describes the interfaces necessary for components to access data from the Host Environment, where the component may use host environment data in behavior, for example, where input or output presentation components must act on the basis of host environment data values. In this case the 'target' is the node in the host environment from which data is to be read.

4 Method Calling on Host Environment Flow Control

This subsection describes the interfaces necessary for components to call methods on objects in the Host Environment, for the purposes of flow control.

4.1 Navigation to Target Node

5 Events

These are the interfaces necessary for events to be passed from the modality component to the host environment.

5.1 Event Propagation

Enables modality component to take advantage of bubbling, event start/stop mechanisms

5.2 Event Creation

Enables modality component to generate events.

5.3 Event Handler Registration

Enables modality component to register handlers

6 Markup-level Abstractions

The DOM interfaces described in this document specify the low-level component interaction layer. Application authors will develop multimodal applications in markup or scripting languages which are built upon these DOM primitives. These markup or scripting languages provide at least three advantages to the application author: simplified programming constructs, for instance time-sequenced interactions could be specified through markup that is then translated by the browser into these DOM manipulations; improved sandboxing, for example preventing one component from modifying another; constraint preservation such as enforcing the data type of a particular field. The examples below possible application-layer constructs built upon DOM interfaces the that could be used to develop multimodal applications.

6.1 XML Events

XML Events 1.0 defines an XML module for authoring event bindings conformant to the DOM2 event propagation. It is a W3C Recommendation that is designed for use with XHTML and other XML-based markup languages and provides the XML author with the same level of access to attaching events and event handlers as provided by the interfaces enumerated in section 5.

6.2 XForms

The XForms 1.0 REC defines an XForms model that holds an XML instance where values collected from the user are stored and retrieved. For environments that include the XForms module, modality components can get and set values from/to this instance. Setting such values will automatically synchronize these values across different modalities. Values in the XForms instance can be manipulated using standard XML DOM calls after a reference to the XForms instance data has been first retrieved via function getInstanceDocument() ---defined in 7.3.1 of the XForms 1.0 specification.

6.3 SMIL 2.0

Synchronized Multimedia Integration Language (SMIL 2.0) is an XML-based language that allows authors to write interactive multimedia presentations. Through its timing and synchronization support, SMIL 2.0 enables the description of the temporal behavior of a multimedia or multimodal application. The DOM events and methods of a given component can be mapped to timing behaviors defined in SMIL's timing and synchronization module (beginning, duration, ending, etc.), and the application author may thereby control the activation of multimodal input and output components through SMIL markup.

6.4 Scripting modules

Many host environments support scripting for application authors, for example XHTML 1.1 Scripting Module, SVG 1.1 Scripting Module. Scripting modules typically allow access to many public interfaces of the host document and its component nodes, enabling direct programmatic control of the properties, methods and events of multimodal components through languages such as ECMAScript. Scripting hosts may provide direct language bindings of the interfaces specified in this document, and/or abstractions and programming conveniences on top of these interfaces.

Appendix A

The following table is an XSLT transformation of the sections and DOM interfaces described above. This is purely for convenience of reference.