Many years ago I came across a diagram of the four boxes of knowing:

I was reminded of the box about “what you know that is wrong” while listening to Dana Chisnell’s talk on “Rethinking User Research and Usability Testing for the Social Web” at the ConveyUX conference sponsored by BlinkUX. Dana is the author of Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests which is a required textbook for UW HCDE 517 on usability testing.

Dana started her talk with a story about Shelly.  In Dana’s 20+ years of doing usability research, she’d only had two subjects cry during the testing. As Shelly broke into tears while testing a simplified financial planning tool, Dana was at a loss for what to say.  Finally, she asked “what is going on and how may I help?”

Shelly replied “Whenever I do anything with financial planning, I always do it with my father.”

Dana suggested “Well, let’s give him a call.”

While Shelly thought that might be a good idea, she knew her dad wasn’t available for calls at work that day.

As Dana continued with the research study, she paid more attention to what many of the subjects had to share – that they always did financial planning in conjunction with someone else. It suddenly dawned on Dana that in our always connected world that almost every task that we do has some component of checking with a friend or colleague or expert. Yet, the field of usability testing has a single person sitting in front of a single computer. In that moment, Dana started rethinking the field of user experience research.

The observation that captured Dana was “people don’t live in the world doing one task with one device out of context.”

Dana then went on to describe five characteristics of users today:

• The nature of being online is social
• Scale is a game changer
• Tasks aren’t what you think (activities – goals that emerge and change)
• Satisfaction is correlated with task completion
• Users continuously design your UI in real time

In thinking about UX social research design, Dana now asks questions like:

• Where are you with planning?
• Who do you ask when you don’t know?

The answers to the above would have told us:

• Who is part of planning and deciding
• Who is trusted
• Why these people are important
• Why they are trusted
• How planning happens

In summary, the “rethinking user research” provided these insights on the challenges with the current UX paradigm:

• We’re not getting the answers we need
• Experimenting is limited because we’re pressured to go to market
• We’re looking for things we know about, using old fashioned tools
• We’re missing things we don’t know about

As Dana finished up, I looked around the room and noticed that several of the HCDE students in attendance were in shock. As we gathered at the back of the room, they looked at me and said “Now what do we do?  We are using her textbook for our usability class and now the AUTHOR has just told us her work is crap. How can we learn anything if the MASTERS keep changing their mind?”

And isn’t this the crux of what it means to achieve mastery – the Malcolm Gladwell 10,000 hours rule (Outliers: The Story of Success) – the ability to keep learning and revising one’s theory in practice.

As I rode home on the ferry that evening, I was delighted to have another example of a MASTER who realizes that what she knew was wrong and then evolved her theory in practice.  Chris Alexander went through this major upheaval in his work several times.  The most recent led to the revision of the work chronicled in Timeless Way of Building to a complete reworking of his theories in The Nature of Order.

I picked up Stephen Grabow’s Christopher Alexander: The Search for a New Paradigm in Architecture to re-read Alexander’s “Ah Hah” moment and see if there were any parallels with the transform in user research that Dana Chisnell discovered.

Here are a few excerpts from Grabow starting on page 170:

“There was one fact above everything else that I was aware of, and that was that the buildings were still a bit more funky than I would have liked.  That is, there are just a few little things that we built down there that truly have that sort of limpid beauty of things that have been around for ages and that, actually, are just dead right.  That’s rare; and it occurred in only a few places.  Generally, speaking, the project is very delightful – different of course from what is generally being built, not just in the way of low cost housing – but it doesn’t quite come to the place where I believe it must.

“This is very important because here we have the question of the quality without a name again.  That is always my reference point, and I ask myself, when all is said and done, after a lot of us have spent a year building these buildings, to what extent has this timeless quality actually manifested itself?

“Part of the reason for the whole of the work of phase two was that I noticed that when other architects were using the pattern language they were still making the same old architecture and saying that it had the pattern language in it.  And that of course led to the whole analysis of phase two – all of those fundamental and sweeping economic, political, and procedural changes that are needed to make this happen correctly.  But what I am saying now is that, given all of that work (or at least insofar as it came together in the Mexican situation) and even with us doing it (so there is no excuse that someone who doesn’t understand it is doing it), it only works partially.  Although the pattern language worked beautifully – in the sense that the families designed very nice houses with lovely spaces and which are completely out of the rubric of modern architecture, so there is no problem on that level anymore – this very magical quality is only faintly showing through here and there.  So of course I began to think about this more and more.

“There is something which in one instance tells you to be simple and which in another tells you to be more complicated.  It’s the same thing which is telling you those two things.  And I became aware of this over and over again in that building project – that there were things like this going on and that I had a reasonably clear intuition about it but that it was very difficult to make explicit and absolutely beyond my capacity to explain to anyone. In spite of all my efforts at trying to explain it over the past ten or fifteen years, I just could not explain this matter – although I knew what I was feeling about it and knew that it had nothing to do with me personally. . .

“I gradually began to realize that I was not taking the problem seriously enough and that I had a very casual attitude to construction.  I thought if one followed a set of operations which were defined by the patterns – actually they were more specific than patterns because they refer to particular versions of patterns – then that was enough. But what I realized was that the craft element was crucial.

“I began to be aware that the actual craft of building in itself was so gigantic and fascinating  in its own way, as the whole of the pattern language. . . These observations – that the actual decision making process in the resolution of the details of construction had not been made explicit, and that the process itself required a level of mastery not accounted for by the previous work – would lead Alexander into the last and perhaps most fascinating series of experiments of the second phase of his research: the precise identification of the geometrical properties of the unity of space.  It would also bring into focus his little-known research in cognitive psychology at Harvard under Bruner and bring him full circle back ot the original questions he posed as a student at Cambridge twenty-five years earlier.  In the meantime, however, his experiments in construction would prove to be the decisive ingredient in forcing a confrontation with the existing paradigm.

“Kuhn points out that the differences between paradigms constitute more than a disagreement about the behavior of the world.  They are directed not only out towards the nature of reality but also back upon the field that produced them.  And since paradigms are the source of the methods, problem definitions, and standards of solution accepted by the field’s practitioners at any given moment, the reception of a new paradigm will often necessitate a redefinition of the corresponding field.

“For Alexander, the attempt to discover new processes by which buildings are made now places him in a radically different posture vis-a-vis the current paradigm [see paradigm shift from The Structure of Scientific Revolutions].  For the first time since Notes on the Synthesis of Form his work can no longer be mistaken as paradigm extension or modification, but rather, as a completely different conception of building.  It entails the realization, for example, that to produce the particular geometry called for, the building itself needs to be in a constant state of creation. [NOTE: Stewart Brand captured this concept in loving detail in  How Buildings Learn: What Happens After They’re Built.]

“It involves a dynamic process where you need to be constantly modifying the design while you are building it because it is only while the building is going up that you appreciate exactly how to make certain details.  It includes the idea of working without drawings, by and large, and it includes the idea that a set of construction processes are specified rather than drawings and that these processes take off where the pattern language stops.  In other words, you use the pattern language up to a certain point, you lay out the building, and then you initiate a set of processes, you carry these processes out, and when you finish carrying them out you have a building.  Its a completely dynamic conception of the translation of a design into an actuality as opposed to having a set of working drawings which are complete and finished and describe a building which somebody then figures out how to construct.  This is a completely different idea.  It is more like sculpture in the sense that it is constantly reacting to the last operation they just performed and deciding if it is correct.

“As a consequence of this conception, fundamental changes in the methods and practice of building are necessary. The way contractors work and the way architects relate to contractors needs to be altered.  An amalgamation of contracting and designing needs to occur in which the architect is also a builder.  And because the building is to be constantly built and rebuilt during its lifetime, the relationship between the architect-builder and the environment needs to be on a relatively permanent as opposed to temporary basis – resembling the traditional family physician’s relationship to his patients.”

With these deep realizations, Alexander had to step back from his Pattern Language work and completely redefine his theories.  He then spent 27 years of research and writing to arrive at the four volume Nature of Order series.  From the Nature of Order website an overview of this work and Alexander’s transformation:

“Alexander has advanced a new theory of architecture, matter, and organization, that has attracted thousands of readers and practical followers throughout the world. His grasp of the fundamental truths of traditional ways of building, and his understanding of what gives life and beauty and true functionality to towns and buildings, is put forth in a context that sheds light on the character of order in all phenomena. Taken even further, hundreds of examples are given to show how the theory has been put to use in his many projects around the world.

“The four books of The Nature of Order redefine architecture for the 21st century as a field, as a profession, as practice and as social philosophy. Each of the books deals with one facet of the discipline. This worldview provides architecture with a new underpinning, describing procedures of planning, design, and building, as well as attitudes to style, to the shapes of buildings, and to the forms of urbanization and construction. Here is an entirely new way of thinking about the world. As one writer has expressed it, “The books provide the language for the construction and transition to a new kind of society, rooted in the nature of human beings.”

“The four books, each one an essay on the topic of living structure, are connected and interdependent. Each sheds light on one facet of living structure: first, the definition; second, the process of generating living structure; third, the practical vision of an architecture guided by the concept of living structure; and fourth, the cosmological underpinnings and implications brought into being by the idea of living structure.

“The books offer a view of a human-centered universe, a view of order, in which the soul, or human feeling and the soul, play a central role. Here, experiments are not only conceivable in the abstract Cartesian mode, but a new class of experiments relying on human feeling as a form of measurement, show us definitively the foundation of all architecture as something which resides in human beings. Whether this “something,” which is demonstrated and discussed throughout the four books, is to be regarded as a new entity underlying matter, or what used to be called the “soul,” is left for the reader to decide.

“Taken as a whole the four books create a sweeping new conception of the nature of things which is both objective and structural (hence part of science) – and also personal (in that it shows how and why things have the power to touch the human heart). A step has been taken, through which these two domains – the domain of geometrical structure and the feeling it creates – kept separate during four centuries of scientific thought, have finally been united

“The four volumes can be read separately, independently, and in any order. However, it is together as a whole that they have their greatest impact. For each book explores comprehensively different aspects of the coherence of our universe, and brings us at last to being at one with it.

“These concepts reach far beyond the field of architecture. Scholars and practitioners in many fields are finding the relevance of these ideas to their own areas of study and practice – physics, biology, philosophy, cosmology, anthropology, computer science, and religious studies, to name a few.”

Some twenty years later, Frank Gehry had to transform himself from an architect to a builder and then operator of the buildings that he designed as he was unable to get his designs built.  From “Is Designing Software Different from Designing Other Things?“, we catch a glimpse of how Gehry had to change his theories of design:

“In a more complex example, Frank Gehry in a video, at a Technology, Education and Design (TED) Conference put on by Richard Saul Wurman, described his challenges in creating the kind of public building designs such as the Guggenheim Museum in Bilbao, Spain, the Experience Music Project in Seattle, and the Disney Concert Hall in Los Angeles. When he first started exploring complex curved shapes for the exterior of buildings he was startled to discover that when he put his designs out to construction bid, the contractors quoted him five times the normal fees. He realized that no one knew how to build his creations. So he had to form a company to first adapt Computer Aided Design (CAD) tools to design the complex metal shapes, and then develop the software that would connect his CAD tools with CNC equipment to cut and mill the complex metal shapes. The end result was that he was able to build his distinctive creations for the same cost as traditional construction methods. During his presentation he reflected on whether he was now a building architect or a software designer.

“These changes are causing the field of architecture to look more like the field of software design. Lindsey details the extent to which computer systems and particularly the Dassault CATIA CAD system have entered Gehry’s practice of architecture. The computer is used for simulations of the digital and physical models, direct detailing, computer aided manufacturing, coordination of the electrical, mechanical and plumbing systems, and as a framework for the operation of the building after construction. Gehry describes how his evolving process is changing the craft of building design and construction:

“This technology provides a way for me to get closer to the craft. In the past, there were many layers between my rough sketch and the final building, and the feeling of the design could get lost before it reached the craftsman. It feels like I’ve been speaking a foreign language, and now, all of a sudden, the craftsman understands me. In this case, the computer is not dehumanizing; it’s an interpreter.”

The significance of the changes that Gehry has made in his fluent design process shows up in the organizational interventions that the software is bringing to the building industry as described in Digital Gehry:

“Ultimately, allowing for all communications to involve only digital information, the model could signal a significant reduction in drawing sets, shop drawings, and specifications. This is already reflected in the office’s current practices where the CATIA model generally takes precedence (legal as well as in practice) over the construction document set. This is a significant change in standard practice where specifications take precedence over drawings and specified dimensions are subject to site verification. . . . . Glymph states that `both time and money can be eliminated from the construction process by shifting the design responsibility forward’. Along with this responsibility comes increased liability. When the architect supplies a model that is shared, and becomes the single source of information, the distributed liability of current architectural practice is changed.”

“Building on the experience of Gehry, we see that this combined hard and soft design can shift forward into the area of operating a building as well. One software system can act as a shared repository and information refinery for the design, build, distribute, intervene and, now, the operate phase knowledge base.”

On a much smaller scale, Chris Anderson captures these same ideas in his Makers: The New Industrial Revolution. Chris describes the challenges his grandfather faced after inventing an automated lawn sprinkling system but he did not have the ability to do the capital formation to bring his invention to market.  Chris replicated his grandfather’s journey in a matter of months with “maker” technology and was able to not only prototype the automated system but manufacture, market and distribute his creation in a matter of months for very little investment.

At about the same time that Alexander and Gehry were transforming their theories, Eli Goldratt went through the gut wrenching reformulation of his Theory of Constraints (TOC).  Most of the experiences that led to TOC were generated by starting with the manufacturing plant.  It wasn’t until consulting with a Brazilian retail department store that he realized that he had most of the TOC backwards. His revised theory can be found in his Socratic Method story form in the book Isn’t It Obvious? 

Goldratt gives some insight into his transformational experience in his book The Choice:

What Choice Do We Have?

“My name is Efrat. I’m accustomed to reading my father’s writing out loud to him. He claims that my comments, and more so my body language, help him to spot when his arguments are unclear.

Once I asked him, “Why me?”

“Because, unlike so many other people, you don’t fool yourself that you know everything about organizations, let alone about human behavior.”

I like his answer. I worked hard to get a PhD in organizational psychology. I invested many years to learn how much we don’t know. No wonder the title of this report, which I am about to read aloud to him, is of particular interest to me. It’s called “Freedom of Choice.”

“Father, what is the choice you made that impacted your life the most?” I ask.

Decisively he answers, “I wanted to live a full life. The most important decision that led directly to it was my decision to constantly devote time to understanding, really understanding, each one of my areas of interest: family, friends and work.”

Knowing that when Father says “really understand,” he means spending endless hours in the attempt to decipher the causalities that govern a situation, I sigh, “That’s not easy.”

“Who is talking about easy? Do you want an easy life?” he asks.

Being the daughter of my father I have heard this question more than once. “I know, I know. If you want an easy life just grab a hammer, a big one, and hit yourself on the head, hard. You’ll have a very easy life. They’ll even bring the food to your bed.”

I definitely want to live a meaningful life—a full life—and so does everyone I know.

I’m also aware that even though people want to live a full life, most people don’t achieve it.

“Why is it so difficult for people to admit that they don’t want an easy life?” he asks.

“Because they do want an easier life, and living a meaningful life is so difficult to reach.”

He gestures impatiently with his hand. “There are ways to make it more attainable. It only requires one to think. To think clearly. To think like a true scientist.”

“In other words,” I say cynically, “you just have to be born a genius.”

He immediately responds, “No, you don’t. I was not born with any exceptional brainpower, and I have my IQ results from my youth to confirm it. I am a bodybuilder. Practice, practice, practice. Efrat, when will you realize that you, like every other person, have enough intuition and brainpower to think like a true scientist?”

I don’t buy it. But there is something else in Father’s decisiveness that bothers me even more. “Father, in what way does thinking like a true scientist enable a person to live a full life?”

He grins, and true to his Socratic approach, rather than answering, he asks, “Maybe you can deduce the answer from the report we intended to start reading fifteen minutes ago?”

I start to read.

The report is about what has been taking up almost all his time over the past few weeks. It all started with a coincidence—two retail chains expressed interest in implementing his theory. Within two weeks it had grown into a major opportunity involving five of the largest retail chains in Brazil. Then, exactly when he was starting to steer his group into this new and incredibly promising segment, it crumbled into nothing.

When I finish reading, he asks, “Well?”

“What a disappointment it must be for you,” I say.

“Why are you talking about disappointment?” he asks in a surprised voice.

I firmly say, “Everybody feels a sense of disappointment when an initiative doesn’t work. The more important the initiative, the bigger the disappointment. Even when a person makes the right choice, even when a person is optimistic and chooses to look on the bright side, even when the person is made out of iron, he or she will be disappointed. The fact that you repress these feelings doesn’t mean that they don’t exist.”

He smiles. “A typical argument from a psychologist. Now that you claim that I just repress my feelings of disappointment, how can I be taken seriously when I say that I don’t feel any?”

I dismiss his remark; I know that I’m right.

“Let’s view it from a different angle,” he suggests. “Let’s suppose that you are a scientist and you are trying to build an instrument that is based on a new approach. Of course, being experienced, you will first do an experiment; you will build a prototype. What would you expect from the prototype?”

I choose my words carefully. “Only a fool expects a prototype to work perfectly the first time. What one expects is to find out what does work according to expectations, and what doesn’t.”

“Well put,” he encourages me. “Now, suppose that the prototype verifies a few new things that do work, and reveals one thing that doesn’t work. Since one thing was not working, the prototype, as an instrument, did not work well or didn’t work at all. Do you suppose, my dear daughter, that you, the scientist who built the prototype, would feel disappointed?”

I see where he is leading. It’s interesting. “Just a little bit,” I say.

“And once you figure out how to fix the things that were not working? How would you feel then?”

“I would feel energized,” I admit.

For my father, every situation is an opportunity to learn, every new initiative is an exploration. I glance over the document I have just read. It’s evident that he was constructing and experimenting as he moved along. The analogy of a prototype is appropriate.

“What’s the difference between the scientist who designs a prototype of an instrument and any other person who just uses the instruments?” he asks.

That’s an easy question. Confidently I answer, “Most people don’t know much about the inner workings of an instrument; for them the instrument is just a box. So if it doesn’t work, they will feel disappointment. If they need it to work, then they won’t just be disappointed, they will be frustrated.”

He nods in agreement.

“As for the scientist,” I continue, “he knows how and why the instrument works; he is familiar with the components of the instrument; he understands the cause-and-effect relationships that make it work. Therefore, even if the prototype, as an instrument, didn’t work, as long as it provided new knowledge of which cause-and-effects are valid and which are not, the satisfaction of making progress compensates for the disappointment.”

Father leans forward and says, “When a proto-type—a new initiative—doesn’t work, we face two alternatives: one is to bitch about reality and the other is to harvest the gift it just gave us, the knowledge of what has to be corrected. That is the reason I titled the report Freedom of Choice.”

Before I have time to digest what he said, he continues, “But enough talking about instruments and prototypes. Let’s talk about reality—the reality you just read about. Do you still think that I was disappointed?”

After a short silence he repeats his question. “Efrat, do you still think that I was disappointed?”

Finally I answer, “You were probably fine, but I’m sure that the people around you were deeply disappointed.”

“You’re right,” he admits.

“And I bet it wasn’t easy for you to help them overcome their disappointment. I have no doubt that it drained their energy and that you had to work hard to revive their enthusiasm and determination. ‘Freedom of choice’ you call it. Well, for you it might be easy, but for most people it is quite difficult to make the productive choice.”

After a while he asks, “Why?”

“Why is it difficult for everybody or why is it easy for you?”

“Why is it different for me?”

Hesitantly I answer, “You’re always the scientist. You are constantly figuring out how the world is ticking, trying to verbalize the cause-and-effect connections—on any subject, in any situation.” I continue more confidently, “For you, everything is like a prototype. No wonder situations that trigger disappointment and frustration for others are, for you, a source of energy.”

This is a new realization for me. It is evident that the approach of a scientist gives a substantial advantage. But what is that approach?

On the one hand, one has to be humble to assume that one doesn’t know. Actually, to avoid feeling disappointed, one has to expect that things will, most probably, not work the first time.

On the other hand, one must be arrogant—have the confidence that one is capable of figuring out how to make things work.

Put these two requirements together and you have a nice oxymoron: humble arrogance.

Looking at Father I say, “This is the first time that I’ve become aware of how helpful the scientist’s approach can be in maintaining the stamina needed to go after new initiatives.”

“It is also helpful in generating the initiatives to start with,” he comments.

“Probably,” I say.

Father doesn’t like such a noncommittal response.

“Do you agree with what Seneca said two thousand years ago, that ‘good luck is preparation meets opportunity’?” he asks.

I slowly say, ‘And knowing the causes and effects that govern a situation is the best preparation.”

Father continues to guide me. “What happens if someone is not prepared—if he is blind to the stream of opportunities that reality is presenting to him?”

It’s not too difficult to predict the outcome. “If someone isn’t prepared, then he won’t see most of the opportunities. Such a person will be waiting for good luck to provide everything on a silver platter.” Taking it further, I continue, “And if a person doesn’t have enough opportunities, he will feel that life didn’t give him a fair chance, that he is constrained by circumstances, that he is powerless.”

How many of my friends have I just described?

Isn’t this line of thought too simplistic? I have to think more about it.

“Bad luck happens when reality meets lack of preparation,” he concludes. “Approaching reality like a scientist, if done well, also provides the needed preparation.”

Then he adds, “If someone is not prepared, what freedom of choice does he have?”

Now I start to realize that it isn’t just the freedom to choose the bright side. Freedom of choice is also connected to the ability to recognize situations that can be turned into real opportunities.

Father interrupts my thoughts with a big sigh, “Unfortunately, in spite of all my efforts, too often I find myself unprepared.”

Goldratt, Eliyahu M.; Goldratt-Ashlag, Efrat (2012-02-08). The Choice, Revised Edition (Kindle Locations 114-214). The North River Press Publishing Corporation. Kindle Edition.

The challenge with leading a Reflective Practitioner life is taking the time to revisit and revise one’s own theory of our professional discipline(s). In a recent blog post on “Seeing Organizations” I shared my major transformational moment when I realized that I am always taking into account organizational design and organizational interventions when I design software products. It is with great joy that I am also learning how to teach organizational seeing.

Where are you in developing AND/OR revising your theory of your profession?

What do we think we know that might be wrong? Or at the very least incomplete?