MODALITY AND

REPRESENTATION
Whilst semiotics is often encountered
in the form of textual analysis, it also
involves philosophical theorising on
the role of signs in the construction of
reality. Semiotics involves studying
representations and the processes
involved in representational practices,
and to semioticians, 'reality' always
involves representation.
To semioticians, a defining feature of
signs is that they are treated by their
users as 'standing for' or representing
other things. Jonathan Swift's satirical
account of the fictional academicians of
Lagago highlights the inadequacies of
the commonsense notion that signs
stand directly for physical things in the
world around us.
Jonathan Swift [1726/1735]: Gulliver's Travels, Part III, 'A Voyage to
Laputa, Balnibarbi, Luggnagg, Glubbdubdrib and Japan', Chapter V
A Scheme for entirely abolishing all
Words whatsoever... was urged as a
great Advantage in Point of Health
as well as Brevity. For it is plain,
that every Word we speak is in
some Degree a Diminution of our
Lungs by Corrosion, and
consequently contributes to the
shortening of our Lives. An
Expedient was therefore offered,
that since Words are only Names
for Things, it would be more
convenient for all Men to carry
about them, such Things as were
necessary to express the particular
Business they are to discourse on.
And this Invention would certainly have taken Place, to the great
Ease as well as Health of the Subject, if the Women in conjunction
with the Vulgar and Illiterate had not threatened to raise a
Rebellion, unless they might be allowed the Liberty to speak with
their Tongues, after the manner of their Ancestors; such constant
irreconcilable Enemies to Science are the common People. However,
many of the most Learned and Wise adhere to the New Scheme of
expressing themselves by Things, which hath only this
Inconvenience attending it, that if a Man's Business be very great,
and of various kinds, he must be obliged in Proportion to carry a
greater bundle of Things upon his Back, unless he can afford one or
two strong Servants to attend him. I have often beheld two of those
Sages almost sinking under the Weight of their Packs, like Pedlars
among us; who, when they met in the Streets, would lay down their
Loads, open their Sacks, and hold Conversation for an Hour
together; then put up their Implements, help each other to resume
But for short Conversations a Man may carry
Implements in his Pockets and under his Arms,
enough to supply him, and in his House he
cannot be at a loss: Therefore the Room where
Company meet who practise this Art, is full of all
Things ready at Hand, requisite to furnish Matter
for this kind of artificial Converse.
Another great Advantage proposed by this
Invention, was that it would serve as a Universal
Language to be understood in all civilized
Nations, whose Goods and Utensils are generally
of the same kind, or nearly resembling, so that
their Uses might easily be comprehended. And
thus Embassadors would be qualified to treat
with foreign Princes or Ministers of State to
whose Tongues they were utter Strangers.
The proposal by the academicians of Lagago to substitute
objects for words highlights problems with the simplistic
notion of signs being direct substitutes for things. The
academicians adopted the philosophical stance of naive
realism in assuming that words simply mirror objects in
an external world. They believed that 'Words are only
Names for Things', a stance involving the assumption that
'things' necessarily exist independently of language prior
to them being 'labelled' with words. According to this
position (which accords with a still widespread popular
misconception of language) there is a one-to-one
correspondence between word and referent (sometimes
called language-world isomorphism), and language is
simply a nomenclature - an item-by-item naming of
Within the lexicon of a language, it is true that most of
the words are 'lexical words' (or nouns) which refer to
'things', but most of these things are abstract concepts
rather than physical objects in the world. Only 'proper
nouns' have specific referents in the everyday world, and
only some of these refer to a unique entity (e.g.
Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogo
goch - the name of a Welsh village).
As Rick Altman notes, 'A language made up entirely of
proper nouns, like the one used in horse racing forms,
offers significant representational benefits; every name
clearly corresponds to and identifies a single horse'
(Altman 1999, 87). However, the communicative function of
Whilst each leaf, cloud or smile is
different from all others, effective
communication requires general
categories or 'universals'. Anyone who
has attempted to communicate with
people who do not share their language
will be familiar with the limitations of
simply pointing to things. You can't point
to 'mind', 'culture' or 'history'; these are
not 'things' at all.
The vast majority of lexical words in a
language exist on a high level of
abstraction and refer to classes of things
(such as 'buildings') or to concepts (such
as 'construction'). Language depends on
categorization, but as soon as we group
instances into classes (tokens into types),
we lose any one-to-one correspondence
of word and thing (if by 'things' we mean
specific objects).
Furthermore, other than lexical words,
the remaining elements of the lexicon of
a language consist of 'function words' (or
grammatical words, such as 'only' and
'under') which do not refer to objects in
the world at all. The lexicon of a
language consists of many kinds of signs
other than simply nouns. Clearly,
language cannot be reduced to the
naming of things.
The less naive realists might note at this
point that words do not necessarily name
only physical things which exist in an
objective material world but may also label
imaginary things and also concepts.
Peirce's referent, for instance, is not
limited to things which exist in the
physical world but may include non-
existent objects and ideas. However, as
Saussure noted, the notion of words as
labels for concepts 'assumes that ideas
exist independently of words'
A radical response to realists is that things do not exist
independently of the sign systems which we use; 'reality'
is created by the media which seem simply to represent it.
Language does not simply name pre-existing categories;
categories do not exist in 'the world' (where are the
boundaries of a cloud; when does a smile begin?). We may
acknowledge the cautionary remarks of John Lyons that
such an emphasis on reality as invariably perceptually
seamless may be an exaggeration. Lyons speculates that
'most of the phenomenal world, as we perceive it, is not
an undifferentiated continuum' ; and our referential
categories do seem to bear some relationship to certain
features which seem to be inherently salient (
Lyons 1977, 247;
However, such observations clearly do not
demonstrate that the lexical structure of
language reflects the structure of an
external reality.
For Peirce, reality can only be known via
signs. If representations are our only access
to reality, determining their accuracy is a
critical issue. Peirce adopted from logic the
notion of 'modality' to refer to the truth
value of a sign, acknowledging three kinds:
actuality, (logical) necessity and
(hypothetical) possibility .
Furthermore, his classification of signs in
terms of the mode of relationship of the
sign vehicle to its referent reflects their
modality - their apparent transparency in
relation to 'reality' (the symbolic mode,
for instance, having low modality). Peirce
asserted that, logically, signification could
only ever offer a partial truth because it if
offered the complete truth it would
destroy itself by becoming identical with
its object.
From the perspective of social semiotics the
original Saussurean model is understandably
problematic. Whatever our philosophical
positions, in our daily behaviour we routinely act
on the basis that some representations of reality
are more reliable than others. A social semiotic
theory of truth cannot claim to establish the
absolute truth or untruth of representations. It
can only show whether a given 'proposition'
(visual, verbal or otherwise) is represented as true
or not. From the point of view of social semiotics,
truth is a construct of semiosis, and as such the
truth of a particular social group, arising from the
values and beliefs of that group.
Modality refers to the reality status accorded to
or claimed by a sign, text or genre. More
formally, Robert Hodge and Gunther Kress
declare that 'modality refers to the status,
authority and reliability of a message, to its
ontological status, or to its value as truth or fact'
(Hodge & Kress 1988, 124).
In making sense of a text, its interpreters make
'modality judgements' about it, drawing on their
knowledge of the world and of the medium. For
instance, they assign it to fact or fiction, actuality
or acting, live or recorded, and they assess the
possibility or plausibility of the events depicted
or the claims made in it.
Clearly, the extent to which a
text may be perceived as 'real'
depends in part on the medium
employed. Writing, for instance,
generally has a lower modality
than film and television.
However, no rigid ranking of
media modalities is possible.
John Kennedy showed children a simple line drawing
featuring a group of children sitting in a circle with a
gap in their midst (Kennedy 1974).
He asked them to add to this gap a drawing
of their own, and when they concentrated on
the central region of the drawing, many of
them tried to pick up the pencil which was
depicted in the top right-hand corner of the
drawing! Being absorbed in the task led
them to unconsciously accept the terms in
which reality was constructed within the
medium. This is not likely to be a
phenomenon confined to children, since
when absorbed in narrative (in many media)
we frequently fall into a 'suspension of
disbelief' without compromising our ability
to distinguish representations from reality.
Ch. Peirce reflected that 'in contemplating a
painting, there is a moment when we lose
the consciousness that it is not the thing,
the distinction of the real and the copy
disappears' (Peirce 1931-58, 3.362).
Whilst in a conscious comparison of a photographic image with a
cartoon image of the same thing the photograph is likely to be
judged as more 'realistic', the mental schemata involved in visual
recognition may be closer to the stereotypical simplicity of cartoon
images than to photographs. People can identify an image as a
hand when it is drawn as a cartoon more quickly than when they
are shown a photograph of a hand (Ryan & Schwartz 1956). This
underlines the importance of perceptual codes in constructing
reality.
Modality cues within texts include both formal features of the medium
and content features such as the following (typical high modality cues
are listed here as the first in each pair), though it is their interaction and
interpretation, of course, which is most important.

Formal features
• 3D-flat
• detailed-abstract
• colour-monochrome
• edited-unedited
• moving-still
• audible-silent

Content features
• possible-impossible
A Cabinet of Curiosities with
an Ivory Tankard (showing • plausible-implausible
reverse of cupboard door) • familiar-unfamiliar
• current-distant in time
• local-distant in space
The media which are typically judged to be the most
'realistic' are photographic - especially film and television.
James Monaco suggests that 'in film, the signifier and the
signified are almost identical... The power of language
systems is that there is a very great difference between
the signifier and the signified; the power of film is that
there is not' (Monaco 1981, 127-8).
This is an important part of what Christian Metz was
referring to when he described the cinematic signifier as
'the imaginary signifier'. In being less reliant than writing
on symbolic signs, film, television and photography
suggest less of an obvious gap between the signifier and
its signified, which make them seem to offer 'reflections
of reality' (even in that which is imaginary).
Modality judgements involve
comparisons of textual
representations with models drawn
from the everyday world and with
models based on the genre; they are
therefore obviously dependent on
relevant experience of both the world
and the medium.
Robert Hodge and Gunther Kress argue that:

Different genres, whether classified by medium

(e.g. comic, cartoon, film, TV, painting) or by
content (e.g. Western, Science Fiction, Romance,
news) establish sets of modality markers, and
an overall value which acts as a baseline for the
genre. This baseline can be different for
different kinds of viewer/reader, and for
different texts or moments within texts.
(Hodge & Kress 1988, 142)
Piaget illustrates the 'nominal realism' of young children
in an interview with a child aged nine-and-a-half:

"Could the sun have been called 'moon' and the moon
'sun'? - 'No.' 'Why not?' - 'Because the sun shines
brighter than the moon...' 'But if everyone had called the
sun 'moon', and the moon 'sun', would we have known it
was wrong?' - 'Yes, because the sun is always bigger, it
always stays like it is and so does the moon.' 'Yes, but
the sun isn't changed, only its name. Could it have been
called... etc.?' - 'No... Because the moon rises in the
evening, and the sun in the day.'
(Piaget 1929: 81-2)
Thus for the child, words do not
seem at all arbitrary. Similarly,
Sylvia Scribner and Michael Cole
found that unschooled Vai
people in Liberia felt that the
names of sun and moon could
not be changed, one of them
expressing the view that these
were God-given names.
The anthropologist Claude Levy-Bruhl
claimed that people in 'primitive' cultures
had difficulty in distinguishing between
names and the things to which they
referred, regarding such signifiers as an
intrinsic part of their signifieds (cited in
Olson 1994, 28). The fear of 'graven
images' within the Judeo-Christian
tradition and also magical practices and
beliefs such as Voodoo are clearly related
to such a phenomenon.
Emphasizing the epistemological significance
of writing, David Olson argues that the
invention (around 4000 years ago) of 'syntactic
scripts' (which superceded the use of tokens)
enabled referential words to be distinguished
more easily from their referents, language to
be seen as more than purely referential, and
words to be seen as (linguistic) entities in their
own right. He suggests that such scripts
marked the end of 'word magic' since
referential words came to be seen as
representations rather than as instrinsic
properties or parts of their referents.
However, in the Middle Ages words and
images were still seen as having a natural
connection to things (which had 'true
names' given by Adam at the Creation).
Words were seen as the names of things
rather than as representations. As Michel
Foucault (1926-84) has shown, only in the
early modern period did scholars come to
see words and other signifiers as
representations which were subject to
conventions rather than as copies
(Foucault 1970).
By the 17th century clear distinctions were
being made between representations
(signifiers), ideas (signifieds) and things
(referents). Scholars now regarded
signifiers as referring to ideas rather than
directly to things. Representations were
conventionalized constructions which
were relatively independent both of what
they represented and of their authors;
knowledge involved manipulating such
Whilst the seventeenth century shift
in attitudes towards signs was part
of a search for 'neutrality',
'objectivity' and 'truth', in more
recent times, of course, we have
come to recognize that 'there is no
representation without intention and
interpretation' (Olson 1994, 197).
The literary theorist Catherine Belsey argues that

Language is experienced as a nomenclature because its

existence precedes our "understanding" of the world.
Words seem to be symbols for things because things are
inconceivable outside the system of differences which
constitutes the language. Similarly, these very things
seem to be represented in the mind, in an autonomous
realm of thought, because thought is in essence
symbolic, dependent on the differences brought about by
the symbolic order. And so language is 'overlooked',
suppressed in favour of a quest for meaning in
experience and/or in the mind. The world of things and
subjectivity then become the twin guarantors of truth. (
Hamlet refers to: 'the purpose of
playing, whose end, both at the first
and now, was and is, to hold, as
'twere, the mirror up to nature'
(Shakespeare, Hamlet, III, ii), and
being 'true-to-life' is probably still a
key criterion in judgements of
literary worth.
However, Belsey comments:
The claim that a literary form reflects the world
is simply tautological. If by 'the world' we
understand the world we experience, the world
differentiated by language, then the claim that
realism reflects the world means that realism
reflects the world constructed in language. This
is a tautology. If discourses articulate concepts
through a system of signs which signify by
means of their relationship to each other rather
than to entities in the world, and if literature is a
signifying practice, all it can reflect is the order
The medium of language comes to acquire the
illusion of 'transparency': this feature of the
medium tends to blind its users to the part it
plays in constructing their experiential worlds.
'Realistic' texts reflect a mimetic purpose in
representation - seeking to imitate so closely
that which they depict that they may be
experienced as virtually identical (and thus
unmediated). Obviously, purely verbal signifiers
cannot be mistaken for their real world referents.
Whilst it is relatively easy for us to regard words
as conventional symbols, it is more difficult to
recognize the conventionality of images which
resemble their signifieds.
The notion of reality as
degenerative is found in the
Romantic mythology of a primal
state of unmediatedness
(referring to children before
language or human beings
before The Fall)
(Chandler 1995, 31-2). In his
book The Image, Daniel
Boorstin charted the rise of
what he called 'pseudo-events'
- events which are staged for
the mass media to report (
Boorstin 1961). However, any
'event' is a social construction -
bounded 'events' have no
We might posit three key historical shifts in
representational paradigms in relation to
Peirce's differential framing of the referential
status of signs:

• an indexical phase - the signifier and the

referent are regarded as directly connected;
• an iconic phase - the signifier is not regarded
as part of the referent but as depicting it
transparently;
• a symbolic phase - the signifier is regarded
as arbitrary and as referring only to other signs.
Such a schematization bears
some similarity to that of the
postmodernist Jean Baudrillard.
Baudrillard interprets many
representations as a means of
concealing the absence of reality;
he calls such representations
'simulacra' (or copies without
originals) (Baudrillard 1984).
He sees a degenerative evolution in modes of
representation in which signs are increasingly
empty of meaning:

These would be the successive phases of the

image:
1. It is the reflection of a basic reality.
2. It masks and perverts a basic reality.
3. It masks the absence of a basic reality.
4. It bears no relation to any reality whatever:
it is its own pure simulacrum.
(Baudrillard 1988, 170)
Baudrillard argues that when speech and writing
were created, signs were invented to point to
material or social reality, but the bond between
signifier and signified became eroded. As
advertising, propaganda and commodification
set in, the sign began to hide 'basic reality'. In
the postmodern age of 'hyper-reality' in which
what are only illusions in the media of
communication seem very real, signs hide the
absence of reality and only pretend to mean
something.
For Baudrillard, simulacra - the signs which
characterize late capitalism - come in three
forms: counterfeit (imitation) - when there was
still a direct link between signifiers and their
signifieds; production (illusion) - when there
was an indirect link between signifier and
signified; and simulation (fake) - when
signifiers came to stand in relation only to other
signifiers and not in relation to any fixed
external reality.
Baudrillard's claim that the Gulf War never
happened is certainly provocative (Baudrillard
1995).
Such perspectives, of course, beg the
fundamental question, 'What is "real"?' The
semiotic stance which problematizes
'reality' and emphasizes mediation and
convention is sometimes criticized as
extreme 'cultural relativism' by those who
veer towards realism - such critics often
object to an apparent sidelining of
referential concerns such as 'accuracy‘.
However, even philosophical realists would accept that
much of our knowledge of the world is indirect; we
experience many things primarily (or even solely) as
they are represented to us within our media and
communication technologies. Since representations
cannot be identical copies of what they represent, they
can never be neutral and transparent but are instead
constitutive of reality. As Judith Butler puts it, we need to
ask, 'What does transparency keep obscure?'
(Butler 1999).
Semiotics helps us to not to take representations for
granted as 'reflections of reality', enabling us to take
them apart and consider whose realities they represent.