Syntactic Theory of Visual Communication, Part One
Paul Martin Lester
There can be no words without images.
Before we are four years old, most of us have learned "The Alphabet Song." Sung to the same tune of "Twinkle Twinkle Little Star," it is unlike any other song because no pictures come to mind when singing it. With Twinkle, we can look up in the night's sky an imagine a little star out of the billions shining just for us. But a song about the letters in the alphabet do not carry any visual equivalents. Children soon match up, however, concrete nouns with images for each letter in the song. Children's books help to solve the mystery. "A is for apple ...." Each letter of the alphabet becomes a picture that corresponds with a complex set of direct and mediated images. We no longer have to think of an actual red, juicy apple. We can simply see the letter 'A' and know that it stands for that fruit.
Before children learn to read and write, they do not know the difference between a line drawing and a letter. When an adult writes an 'A,' it is simply another drawing. It is a picture, different than a face or a house, but it is still just another image drawn with a colored pencil on while paper. Soon children learn that combinations of these letter-pictures mean more complicated things. When the drawings 'A-P-P-L-E' are combined, they form another picture which we learn stands for the name of the fruit. Now the letter-pictures become word-pictures that can spark other images in our minds of the thing they stand for. We further learn that these word-pictures can be combined with other word-pictures to form sentence-pictures. To a child, there is no difference between words and pictures—they are one and the same.
Early on however, we are taught to make distinctions between words and pictures and to not think of them in the same way. We are taught that although we can gain meaning from each, reading words is valued more than reading pictures. We are taught that pictures play a separate and subservient role to the words. And although we are taught how to make pictures with our colored pencils and our watercolor paints, we get much more instruction on how to form, with our large lead pencils, the lines and curves that make letters and words. We get one class where we make pictures—art. The other classes are devoted to writing
or reading stories whether in a grammar or in a geography class. We are taught to read stories, but we are never taught how to read images.
In the Disney classic, Beauty and the Beast, the macho Gaston satirizes Belle's reading habits. "How
can you read this," he asks, "there are no pictures?" She answers with a condescending, "Well, some people use their imagination." And yet, when the viewer of the animated movie is shown a close-up of a page in her book, she points to a picture of a castle that illustrates the story.
There are strong indications that the status of images is improving. We live in a mediated blitz of images. They fill our newspapers, magazines, books, clothing, billboards, computer monitors and television screens as never before in the history of mass communications. Something is happening. We are becoming a visually mediated society. For many, understanding of the world is being accomplished, not through reading words, but by reading images. Philosopher Hanno Hardt warns that the television culture is replacing words as the important factor in social communication. Shortly, words will be reserved for only bureaucratic transactions through business forms and in books that will only be read by a few individuals. Reading is losing to watching because viewing requires little mental processing.
Critics blame everything from the rise in the crime rate to the deterioration of educational institutions on the concurrent rise in the number of mediated images that can be seen daily. Rebellious youth cling to visual symbols because words are associated with old ways of communicating and old ways of establishing social order. Words are repressive while pictures are fascinating, easily understood within a particular culture and can be made personal mediums of expression. One of the first acts in 1917 by the new Russian government was to transform churches into motion picture theaters in order to show propaganda films. One of the first tasks of a rebellious force is to capture the television station. All rebellions understand that to control a country, the pictures must also be controlled.
Several years ago, a headline alarmed many around the country: Why Johnny Can't Read. The question is often asked by educators who worry about the thinking and writing capabilities of their students. The answer is often simplistic: Too many pictures and not enough words. But there are other answers. Maybe there is little written that persons want to read. Maybe persons feel there is no point in reading when Instructor's Manual for Harris and Lester's Visual Journalism A Guide for New Media Professionals 1
there are no jobs that require reading. Maybe parents do not read and do not support that habit in their children. Or maybe individuals are reading, but what they read is not understood by those from other cultures.
Visual messages, with their own rules of syntax, are being read, but this language means nothing to those who can only read words.
The wall space and signs in many cities often are coated with multi-colored spray painted messages. Termed vandalism, graffiti or tagging depending on the speaker, these visual messages are actually a complex written form of communication. Graffiti messages may mean the mark of a territorial border by a gang, a plea for understanding and hope for the future, grief for a killed loved one, anger toward an enemy, a show of playfulness and humor as part of a national fad, an act of criminal vandalism or simply an individual expression that signals the writer's existence. As with any symbolic communicative system, if you do not know the language, you will have trouble deciphering the message.
A brave new (visual) world
Although it is unclear what may be the social, religious and educational affects this visual culture will have upon the world's society, the use of images may foster a return of the word's importance. Or rather, a communication medium in which words and pictures have equal status may be a result of the recent explosion in pictures. Because television images cross all international borders, they become more easily understood by almost everyone. Words are easily forgotten, but pictures stay in our minds. We may not remember many of the facts that led to the brief student uprising in China's Tiananmen Square in 1989, but you can never forget the image of the lone protester standing defiantly in front of a line of menacing, green Chinese tanks. If you have seen the picture, you remember it not only because it is a highly emotional image, but because you have thought about the image in your mind with words. Words and pictures become one powerfully effective communicative medium inside your own mind.
John Sculley, the former CEO of Apple Computers, Inc. is not afraid of the world's dependence on images for communication. "We live in a visually intensive society," he asserts. Look at the most common medium for visual expression—television. Programs can be watched from direct broadcast, from cable and fiber optics, from satellites, from VCRs and from laser disks. On some television sets you can even watch more than one program at a time. In a television commercial for Kodak's Photo CD technology in which still images can be viewed on the screen by a CD disk player, the announcer reports that "Pictures have never been so powerful." When televisions and computers are linked, viewers will actually be able to alter the content and technical considerations of programs to suit their individual interests. The combination will indeed be a powerful medium for pictures.
Computers make the production and distribution of images available to almost anyone at an incredible speed. More than any other technological innovation, computers are responsible for the explosion in images. Like the 'Big Bang' that many scientists think sent the universe on its course, computers will serve to expand the use of pictures. Today, 20 percent of the U.S. population can use a computer. But 80 percent of school-age children have learned to become computer literate. By the turn of the century, Sculley predicts that 98 percent of all the words and pictures created in the world will be computer mediated. By that time, virtual reality—the ultimate fusion of computer and television technologies in which viewers become active users of the medium—will be inexpensive and accessible.
Educational psychologist Jerome Bruner of New York University cites studies that show persons only remember ten percent of what they hear, 30 percent of what they read, but about 80 percent of what they see and do. When all members of society whether at home, in school and on the job learn to use computers for word and picture processing, the switch will be made from passive watching to active using. There will no longer be the barrier between the two symbolic structures. Words and pictures will become one, powerful and memorable mode of communication.
The syntax of symbols
Linguistic theorists categorically assert that since pictures are presentational and not discursive, they have no formal grammar. Without a grammar, images cannot be considered a language. Without a language, pictures cannot be read. Traditionally, syntax and grammar refer to the system of rules that are used to turn words into sentences. The syntactic rules applied for proper sentence structure are culturally based—they
change from language to language and over time. Once learned, the rules make it easy for an individual within a culture to write and read stories on an infinite variety of subjects about the myths that shape a Instructor's Manual for Harris and Lester's Visual Journalism A Guide for New Media Professionals 2
culture. Words are the basis for human understanding because of their narrative structure. Philosopher Susan Sontag writes, "Only that which narrates can make us understand."
There are two major stumbling blocks that linguistic philosophers impose as to why images cannot be considered a language:
1) Images do not have common elements similar to a written language's alphabet and
2) Images have no recognized syntax.
Linguistic philosopher, Noam Chomsky writes that "a language is defined by its alphabet (i.e. the finite set symbols out of which its sentences are constructed) and its grammatical sentences."
American philosopher Suzanne Langer wrote "Photography ... has no vocabulary .... There are no items that might be called, metaphorically, the 'words' of portraiture." For Langer, the problem with admitting that there is a visible language is not that there are no words -- pictorial elements -- in still or motion picture photography, but that there is no universally acceptable language of visual description. However, some have devised such a language for visuals with well-defined alphabets.
French semiotician, Fernande Saint-Martin attempts to create an alphabet for images in her book, Semiotics of Visual Language. For Saint-Martin, color is the basic visual element. Color, as a form of light, gives shape and substance to the visible world. Her basic visual alphabetic letter is called the "coloreme." A coloreme is the smallest element within a direct or mediated image that can be focused within the foveal field of the retinas. A coloreme can be composed of a picture's actual color, texture, size, boundaries, direction or position in the frame of view. These physical attributes of the image, once noticed and identified, find meaning through successive viewings, similar to the ideas expressed by the constructivists.
In 1987 Irving Biederman published his theory of visual perception in which he outlined the alphabet of objects. Any written alphabet is simply a collection of symbols that correspond to the sounds made during speech. A basic unit of speech is called a phoneme. For all the languages in the world, only 55 phonemes are needed. The 44 phonemes for the English language are simplified into 26 characters—the alphabet.
Webster actually lists 50 alphabetic symbols that are used to describe all the sounds needed to pronounce the more than 50,000 words in the Dictionary, but 26 letters suffice. Biederman recognized that every object is composed of primitive shapes or parts. He called these basic components "geons," short for geometrical ions. Through his research, he discovered that only 36 geons are needed to make all objects. But because of previous experience with an object, most persons can recognize it by viewing only three geons. An object that is unusual, obscured, seen at an odd angle or out of context will require more geons. It seems that the mind may store images symbolically within the brain in the form of geons in order to make recognition quick and simple. Biederman's work may be an important link between the way words and images are thought to be viewed. Once a written language is learned, a reader usually does not have to consciously analyze every letter within a word. Whole words become a part of a "verbal geon."
The problem with the theories of Saint-Martin and Biederman is that their schemes could never fully describe all of the elements that make up a detailed image. Coloremes and geons offer indications of a symbolic storage of images, but the processing of coloremes and geons become so individualized to the point that no two persons would identify the same pictorial elements. A visual 'alphabet' should be easily recognized by all those using it every time. Coloremes and geons are far too abstract for such a purpose. There can probably never be an alphabet for images. Nevertheless, the brain does respond to an image's basic visual elements of color, form, depth and movement. Although these elements may not have the simple symbolic meaning of alphabetic characters, a difficulty in finding an alphabet for images indicates the richness of communication possibilities of pictures over words.
The other major problem linguists have over regarding images as a language is that there is no recognized syntax or order in which to view the elements within a picture.
Noam Chomsky writes that languages contain surface and deep structures. Surface structure refers to the rules of grammar while deep structure involves knowing the meaning of each word within a sentence. A sentence is considered to have proper grammar if it follows specific rules and if the words are understood. A chief rule of syntax is the arrangement of the words in their proper order. Wilson Hicks, an early picture editor for Life magazine writes that "the verbal medium is discursive." Words must follow a linear, horizontal order between two points. Langer asserts that a visual structure is taken in all at once by the viewer without any specific rules of order.
The sentence, "I like green or red apples" obeys the rules of proper syntax. The words are in the proper order and convey meaning. Now consider the sentence, "I like said or verb apples." It is grammatically incorrect. The word choices are confusing. The sentence makes no sense. But within this context, the Instructor's Manual for Harris and Lester's Visual Journalism A Guide for New Media Professionals 3
sentence, "I like said or verb apples" has meaning because it is used as an example of improper syntax. Context, then, is a key factor in determining meaning. Words do not always obey the standard rules of syntax, yet they communicate. Poetry is an excellent example of words playing with syntactic rules.
The sentence, "Smoc moccoa crylie schemee" may or may not be of proper syntactic orientation because the phrase is foreign—the words have an unknown meaning. The sentence, "My quoin gave me
pneumoconiosis" may or may not mean something to you depending on your knowledge of the English language. Still, you would recognize the two groups of 'words' as sentences.
Words do not always need to be known in order to form sentences. Author David Lodge once wrote that "Language is a net that holds thought trapped within a particular culture." Imagine all that is lost to you because you do not know the meaning of the individual words.
A basic rule of syntax is that words that make up a sentence follow a linear, discursive pattern. The sentence, "Apples like or green I red" violates that rule. And yet, like poetry or a foreign language, and helped by a previous experience with the sentence in its proper grammatical order, the sentence does have meaning. (Try reciting the mixed up sentence to friends and see if they can figure it out.) Words within a sentence do not necessarily have to follow the rule of discursiveness in order for there to be meaning. Grammar is simply a set of rules invented to make life easier for those who use them.
The great documentary photographer, Lewis Hine, who often used words to accompany his photographs once said, "If I could tell the story in words, I wouldn't need to lug a camera." It is beyond question that words and pictures are different animals. But each possess a language that some can interpret better than others. Photographic historian Helmut Gernsheim wrote that "Photography is the only 'language' understood in all parts of the world, and bridging all nations and cultures." On the other hand, photography philosopher John Berger admits that "photographs supply information without having a language of their own. Photographs quote rather than translate from reality." Sol Worth, an expert of visual communication wrote of a compromise between the two points. "Pictures are not a language in the verbal sense. Pictures have no lexicon nor syntax in a formal grammarian's sense. But they do have form, structure, convention and rules."
The semiotic approach to visual communication stresses the idea that images are a collection of signs that are linked together in some way by the viewer. The study of semiotics, as detailed by Morris, divides itself into three areas: pragmatics, semantics and syntactics. Pragmatics is the study of the origin, common uses and communicative effects of signs. Semantics is an area of semiotics in which the researchers attempt to determine the significance of signs within and throughout various cultures. Syntactics is the study of the ways signs are combined with each other to form complex messages.
Since semiotics grew out of linguistic theory, pragmatics, semantics and syntactics are terms used in analyzing written or spoken communications. However, since a direct or mediated image is nothing more than a collection of signs, the area of syntactics is of most interest to visual communicators because an image is a collection of graphic elements that convey meaning for the viewer. Peirce wrote that a sign always refers to another sign and never to the actual object "in itself." Agreeing with him, de Saussure noted that the meaning communicated by a sign depends not on actual objects but on other signs. For him, a sign by itself means nothing.
Roland Barthes described the "chain of associations" or signs that make up a picture's narrative. In verbal language the narrative is linear. One word follows the next in a specific rule-based order. In that regard, verbal communication is considered discursive. Pictures, on the other hand, are presentational. Signs within an image are presented in various ways, many times depending on the style of the image-maker. The chain of signs is more tightly controlled with text than with images. One exception might be poetry in which the order of the words can have non-linear, presentation qualities. In fact, the Greek poet Simonides around 500 BC wrote that paintings were "silent poetry and poetry painting that speaks."
Individual symbols within a picture have no precise alphabetic relationship, but when used in combination, meaning is found for an image through a traditional, discursive method. That method happens inside the mind of the viewer. Suzanne Langer wrote that "grammatical structure is not a symbol of itself, but it ties together several symbols, ... to make one complex term." In a picture, that term or idea, as Langer writes, is expressed through "visual forms -- lines, colors, proportions, etc. and are just as capable of articulation, i.e. of complex combinations, as words." Some might argue that words express ideas through "visual forms—lines, colors, proportions, etc. and are just as capable of articulation, i.e. of complex combinations," as pictures. The syntactic theory of visual communication
Instructor's Manual for Harris and Lester's Visual Journalism A Guide for New Media Professionals 4
The writer William Saroyan once remarked that "One picture is worth a thousand words. Yes, but only if you look at the picture and say or think the thousand words." Suzanne Langer writes that because language names relationships rather than illustrating them, "one word can take care of a situation that would require a whole sheet of drawings to depict it." For her, one word is often worth a thousand pictures. The photographer Minor White wrote a little poem about the two forms of communication:
I delight in photographs. I delight in words. I delight in mixing both To see what happens if they blend.
Words and pictures are intricately linked in journalism, advertising and educational contexts. Words printed under a photograph, the caption, signal the importance of the common sense representations portrayed in the image. Words beside a picture in an advertisement explain a product and its attributes clearly to a potential customer. Words spoken by an instructor give explanations and life to slides projected from the back of a darkened classroom. Words and images work together for most successful communications. As Hicks wrote in his book Words and Pictures, "It is not correct to say that either medium
supplements the other. The right verb is complements. Even pride of authorship are mingled."
The words that are associated with a picture and the pictures that accompany words should never be chosen casually. Both symbolic systems have enormous impact upon the reader. Roland Barthes noted that "The text loads the image, burdening it with a culture, a moral, and imagination."
Perception of the importance attached to words or pictures in publications is often communicated by the size, position and proximity of the words to the visuals. On the front page of a newspaper, the most important story and photograph of the day often takes up the most space and is above the horizontal fold. The headline, story text and caption are near one another. Since the caption and picture credit are nearest to the image and often duplicate the picture's content, Barthes asserts that they become part of the picture's denotative message. The headline and story, because they are printed farther away, have a slightly different connotation than that of the picture. Newspaper reading research has found that readers look at a photograph first, scan the caption, read the headline and if they are still interested, read the story.
For most publications, a photojournalist is strictly responsible for the image, a reporter writes the story, a copy editor makes up the headline, another editor writes the picture's caption and a layout artist puts the words and pictures together on the page. It is a wonder that there are not more connotative problems between words and images with so many different individuals involved in the process. If a caption, for example, does not adequately explain the content of a picture, confusion among the readers can easily be the result.
Despite occasional problems in discerning the meaning from pictures and w