Way too many moons ago, I came across Paul Ryan and his unique approach to videography. I spent an intense cerebral day with Paul in his apartment near Columbia University in New York City. We were exploring ways to generate new products in the video capture and story telling arena at Digital Equipment Corporation (DEC). My research group was enamored with the video editing tools that were showing up on the Macintosh and wondered if we could find something unique that we could do on the DEC VAXstations.

Paul walked me through his journey of discovery on the capabilities of this new medium of video in the McLuhan sense. He showed me several clips from his Nature in New York City series of explorations of ecology surrounding a large city. What captured my imagination was his “automatic composition” technique for creating educational videos. I particularly liked that he was able to teach it to junior high school students and was mesmerized by some of the student compositions. On his website, Paul shares his process through shooting the video in the categories of firstness, secondness and thirdness and then doing the automatic composition by traversing his relational circuit:

The Categories of Firstness, Secondness, and Thirdness

Because I wanted a notational system for video that was responsive to the totality of the environment, I was attracted by the comprehensiveness of the categories of firstness, secondness, and thirdness as developed by the American philosopher Charles Peirce (1839-1914). Following Kant, Peirce subscribed to the architectonic theory of philosophy (Apel: 1981). By architectonic, he meant the art of constructing systems, i.e., uniting manifold ways of knowing under one idea. The idea or concept of a formal whole determines a priori both the scope of the manifold content and the positions that the parts occupy relative to each other. This unity makes it possible to determine, from our knowledge of some parts, what other parts are missing, and to prevent arbitrary additions. Knowledge can grow organically, like the body of an animal.

For Peirce, knowledge corresponds to three modes of being: firstness or positive quality, secondness or actual fact, and thirdness or laws that will govern facts in the future. Peirce held that these categories of being are phenomenologically evident to anyone who pays attention to what happens in the mind. Direct observation will produce these categories of knowledge.

Firstness is positive quality. The taste of banana, warmth, redness, feeling gloomy: these are examples of firstness. Firstness is the realm of spontaneity, freshness, possibility, and freedom. Firstness is being “as is” without regard for any other.

Secondness is a two-sided consciousness of effort and resistance engendered by being up against brute facts. The “facticity” or “thisness” of something, as it exists, here and now, without rhyme or reason constitutes secondness. To convey the pure actuality of secondness, Peirce often used the example of pushing against an unlocked door and meeting silent, unseen resistance.

Thirdness mediates between secondness and firstness, between fact and possibility. Thirdness is the realm of habit, of laws that will govern facts in the future. With a knowledge of thirdness we can predict how certain future events will turn out. It is an ‘if…then’ sort of knowledge. Thirdness consists in the reality that future facts of secondness will conform to general laws.

When we attempt to interpret a natural site with a video camera, we are confronted with “everything.” We need to make selections. If those selections are arbitrary, the final tape can leave out significant aspects of the ecosystem. Significant omissions can make the interpretation of the site faulty. Peirce’s categories of firstness, secondness, and thirdness are, in effect, a theory of everything. Using these comprehensive categories, it is possible to make selections that are responsible to “everything” at the site. The way in which Peirce’s categories can be used to organize video perception of ecological sites is evident in my videotape titled Nature in New York City (Ryan: 1989a). Consider the following list of the four sites in the tape and how my interpretation of the sites was guided by using Peirce’s categories.

1. Horseshoe crabs laying eggs, Jamaica Bay, Gateway National Recreation Area, Brooklyn and Queens. Firstness: eggs, signifying the possibility of new crabs; secondness: predator birds and meddlesome boys; thirdness: pattern of crabs mating and context of urban habitat.
2. Clay Pit Pond, Clay Pit Pond State Preserve, Staten Island. (Five phenomena selected: deciduous trees, evergreen trees, abandoned cars, grass, reeds.) Firstness: quality of five phenomena plus pond surface; secondness: facticity of five selected phenomena; thirdness: patterns of phenomena in the context of the pond.
3. Stand of trees, forest in Inwood Hill Park, Manhattan. Firstness: melting snow, bark surfaces and sprigs of green; secondness: burnt wood and litter; thirdness: children swinging on rope, pattern of tree crowns.
4. Waterfall, Bronx River, New York Botanical Garden, The Bronx. Firstness: quality of surface water and texture of turbulence; secondness; water turbulence; thirdness: explicit water patterns, geological context of the falls.

This twenty-seven-minute tape was edited in six-second passages set up in 4/4 time for musical interpretation. Each passage corresponds to firstness (F), secondness (S), or thirdness (T) and the passages fade into each other. A given sequence might run FSFT, SFST, TSFS, TFSF.

The Relational Circuit

 The Nature in New York City tape was composed using what I call “the relational circuit.” The relational circuit organizes the categories of firstness, secondness, and thirdness in unambiguous, relative positions. The circuit is to the Earthscore System what the staff and bars are to classical music notation. I originated the relational circuit based on my own video experimentation and a study of Peirce’s failed attempt to develop a logic of relationships.

Peirce thought that the organization of knowledge according to his three fundamental categories required a new kind of formal logic. During his lifetime he made four major attempts to construct a philosophic system, each attempt guided by new discoveries he had made in logic, but none succeeded.

The major difficulty Peirce had was with continuity. Peirce believed all things were continuous and that the concept of the continuum was the master key to philosophy. However, he was never able to organize his categories in a logical continuum (Murphey: 1961).

Working with video, I was able to construct a topological continuum that, I believe, supplies the formal logic necessary to realize Peirce’s architectural dream. My purpose here is to present the circuit for the reader’s inspection and show how it organizes Peirce’s three categories.

The relational circuit is a self-penetrating, tubular continuum with six unambiguous positions. The circuit below is depicted in three dimensions. There is a part contained by two parts, the position of firstness. There is a part contained by another part and containing a part, the position of secondness. There is a part that contains two parts, the position of thirdness. The circuit organizes differences in terms of these three positions and the three “in-between” positions that connect them in the continuum. Peirce’s three categories map onto this continuum. The positions are named to correspond to the categories.

The relational circuit provides the core of a notational system which can regulate composing with firstness, secondness, and thirdness in a way that is analogous to the way the painter Cezanne composed with what he fondly called his little blues, little browns, and little whites (Lacan 1978: 110). Nature in New York City is an example of such a composition. However, since this tape was produced by me as a solo artist, it falls short of the ideal in which cooperating videographers interpret the ecology. 

Paul shared that he was present during the time frame when this new medium of video was at a crossroads of evolution – would the medium be primarily used for education or for entertainment? Through the hindsight of history the path was clear – go for entertainment and for decades no one looked back. With the advent of the flipped classroom, MOOCs and mobile video capturing smart phones, we are slowly coming back to video as a key component of education and lifelong learning.

We weren’t able to make the business case for DEC going into the video products business at the time. My next encounter with the power of video was the acquisition of AfterEffects while I was at Aldus. The brilliant software engineers of CoSA developed a product that is still popular twenty years after the initial development. They brought the concepts of Adobe Photoshop to moving images. I left Aldus when we merged with Adobe and was not able to bring Paul Ryan’s composition techniques to video software editing tools.

Then on January 27th, 2010, Steve Jobs and Apple rocked my world with the introduction of the iPad (I’m now on my fourth upgrade of the iPad to the iPad Air). During the launch demo, Jobs alluded to a new vision for what traditional media (books, movies, photos) could be if you combined them in a powerful tablet computer. I envisioned that we could finally escape the tyranny of the black box of video. I was reminded of Marvin Minsky‘s Society of Mind interactive CD where each page of the CD had a page from his book and Marvin “walked” around the “page” explaining the key concepts. Peter Meyers further explored these ideas in his Breaking the Page. What if we could really integrate and synthesize video and text and then generate automatic compositions like Paul Ryan had in mind.

While the vision was compelling, I wasn’t motivated to go build it. I keep hoping that someone will come along to fulfill the vision. I keep buying and trying apps that purport to make it easier to shoot and compose “interactive video” both for consumer use or for my professional video ethnography work in studying entrepreneurs.

So with some hope and a lot of cynicism I was pointed to Touchcast which exclaimed “TouchCast is a platform for a new class of video apps or vApps that combine the look and feel of TV graphics with the interactivity of the web.” Sitting on my deck overlooking Puget Sound with my trusty iPad, I looked at the demos and got interested. So I generated a quick version of a test app. Oh my, it actually works.  In just a few minutes, I had an interactive video with pointers to my blog and a PDF that illustrated the concept I just did a riff on.

Then I noticed that they had a developer capability. TouchCast ships with several vApps.

And amazingly you can add your own vApps.

I think I’ve found the platform for developing my book content (Emails to a Young Entrepreneur) into a mobile app that can combine presentation videos with the book content and with the exercises.

Of course, I always want to push the envelope so I am recruiting a couple of developers to see how far they can push the vApps. I am hoping I can take the Vidbolt and CommentBubble annotating capability and get them added to TouchCast. Maybe I can even talk my colleague, Kelly Franznick of BlinkUX to add TouchCast to his Feedback Panel for live streaming and commenting on user research video ethnography client projects.

Stay tuned for some early TouchCast prototypes.