Day 200 of Self Quarantine             Covid 19 Deaths in U.S.:  205,000

Rest in Peace Justice Ruth Bader Ginsburg                                   VOTE!!

Professor David Socha asked me a second question after “where do you observe different parts of the software product development life cycle – what is the diagram that shows where to observe?”

I am continuously sketching the answer to David’s question.  As usual, I start with the complexified version of the diagram.  Over the last week, David and I discussed, debated, added and subtracted a lot from this diagram.  This blog post is the first attempt to come up with a diagram of an Agile OODA (Observe – Orient – Decide – Ask) HCD (Human Centered Design) Loop (virtuous cycle) for software product development.

As I stare at this diagram, I realize it is more of a macro diagram and is heavy with framework systems that require a lot of description and unpacking.  And this diagram is “simple” because I would really mess it up more if I added all the loops of iteration throughout the diagram.

David did a nice job in “Observe, Don’t Ask Part 2” of enumerating all of the activities in the software product development life cycle where you either need to Observe or Show.  The more interesting question is where you wouldn’t do Observe or Show?  We both realized the challenge of trying to document what we mean versus showing a more complete context of what we mean.

As I tweaked the above diagram, I wondered if there was a diagram that showed the feedback and feed forward loops  or maybe a Venn diagram to illustrate relationships in a clearer way.  As I doodled with different loops, I realized that I had not referenced the OODA Loop that John Boyd developed as a fighter pilot.

Boyd’s work in developing the Observe-Orient-Decide-Act loop was an important step forward in military tactics.  This strategy was the basis for our winning so quickly in Gulf War I and was adapted for business and commercial use in the books Winning in Fast Time and Certain to Win.  While Boyd did not write any books, he was famous for putting his materials into an extensive slide deck and giving the presentation to anyone who would listen.

Boyd’s original OODA loop diagram looked like this:

I love that there are both Feed Forward arcs and feedback arcs.

The folks at IMARC AI provide a simpler update of the original OODA loop.

As I play with this simplified diagram, I realize it fits nicely underneath all of the steps and frameworks in the complex diagram.  The “observe” step is obviously the “observe, don’t ask” step.  The “orient” step fits nicely with what I mean by “show, don’t tell.”  The “act” step needs a reminder of don’t dilly dally with “Act, don’t delay” which is the fundamental speed insight of Boyd.  The decide step fits a mantra of “prototype, don’t guess.”  So many times in the product development process, the software engineers are forced to guess what is meant by a user story or a requirements document or a short discussion during a daily stand-up.  What human centered design practitioners learn early on is to do rapid prototyping even if it means doing paper prototyping.

I can’t help myself and realize that I have to add to the simple OODA diagram.  While I love the original icons, I didn’t like the ACT step as it implied thinking to me.  Acting is about moving and I wanted to have something that had hand movement.  I also felt that there needed to be something in the middle that was an indicator of the “so what.”  Where does the OODA loop for software product development lead – to value.

For each step in the complex diagram (and the many sub-steps), the work is about figuring out what to observe, and then what insights arise from the observation, that need to be carried to the next step to “show” the developers.   At each step there are one or more decisions.  Some of those decisions can be made based on “patterns that repeat.”  Others require some rapid and iterative prototyping.  The prototyping takes the guess work out of deciding what needs to be built – the “act” step.  Then once something is built, that prototype needs to be observed in use “in the wild” of an actual user’s environment.

Then, the OODA loop repeats until there is a measure of good enough as described in the Part 2 blog post.

“To claim that any given thing is Good Enough is to agree with all of the following propositions:

• • • • • • • It has sufficient benefits.
            • It has no critical problems.
            • The benefits sufficiently outweigh the problems.
            • In the present situation, and all things considered, further improvement would be more harmful than helpful.

The micro level OODA loop steps for software product development are:

Now that we have an idea of the core process underlying the more complex model, let us look at the components of the larger model.  The descriptions that follow assume that we are in an early stage startup where the product and the company are mostly the same thing.  Inside a large company the equivalent is intrapreneuring where a new product and new business are created together.

The macro OODA loop for software product development has several internal and external components.  The Observation step in the macro OODA loop is:

Over the years I’ve learned to start any new product idea by trying to fill out the nine boxes of a lean canvas.  The lean canvas below is an early draft I did for the Know Now book I am writing:

The lean canvas is a paper prototype of your product and business idea.  Filling in each box gives you a quick way to see what you know and what you think you know (The Four Boxes of Knowing).  The work is to figure out which box of content you know the least about.  That box becomes the priority to learn more about.  Most early stage entrepreneurs leap right into the Solution Box and have a wide range of ideas of how to solve for some problem.  Over the years I’ve found that I rarely understand the problem or the existing alternatives and who the early adopters might be.

For the Know Now book project, my hypothesis was that the early adopters would be product managers.  While I have done product management off and on for 20 years, I realize how little I remember about how I educated myself about product management.  So the first thing I needed to learn about is product managers and how they learned to become a product manager.

The key components of the Observe step come from the Human Centered Design process that I learned about at the Institute of Design.  Ideo offers a range of online courses and extensive consulting services that help companies implement a robust human centered design process.  A starting point for learning about Human Centered Design is Vijay Kumar’s book on 101: Design Methods.

As I interviewed several product managers, I was also coaching several early stage startup CEOs.  I realized that one of the challenges in an early stage startup is that the CEO acts as the product manager due to lack of funds.  In an early stage startup, the product and the company are one and the same.

In parallel with identifying users to observe, the CEO as product manager is also developing a business plan for the product and trying to understand the external environment as depicted in the micro OODA loop.  Useful frameworks for observing the external world are captured in the Scenario Planning Process and in collections of useful frameworks.  The kinds of information that the CEO is looking for are social dynamics, economic issues, political issues, technological issues, culture trends, legislative and regulatory trends, and ecological issues.

Along with observing external factors, the CEO is always trying to estimate TAM, SAM and SOM for the potential product.

Now that you have identified the users to observe and performed the observations (where possible using video ethnography), then it is time to ORIENT.  The Macro OODA Loop for Orientation is:

As you make sense of your observations there are several frameworks that guide what to look for.  The observations should include the full flow of work.  Within the flow of work you are looking for the hassles that a user encounters.  Adrian Slywotzky describes hassle maps in his book Demand: Creating What People Love Before They Know They Want It and in a short monograph on The Art of Hassle Map Thinking.

An example of a hassle map is the one that Zipcar created:

No matter how manual or automated a given work flow is there are still hassles that a user must overcome.  This step of the Orient process is to identify all of the hassles and to start looking for solutions to eliminate the hassles.

Service Science emerged from IBM as their business transitioned from hardware to software to services.  Jim Spohrer in “The Emergence of Service Science: Toward Systematic Service Innovations to Accelerate Co-Creation of Value” provides a diagram of the evolution of automation of a manual task.

After identifying the hassles of a user, the next step is to understand where the user’s work environment is in the sequence of work evolution.  Each step has a different set of software development activities and degree of difficulty.  For most of my professional life, I focused on human systems that are mostly manual and then looked for ways to augment that system.  Spohrer points out that is just one step along the path.  Over the last 20 years of software product development in the eDiscovery space, we’ve seen the progression to Augment with our Attenex Patterns product, and then to outsourcing with companies like FTI Consulting and Lighthouse, and now with full automation with the AI/ML capabilities of NUIX Discover.

The next step in orienting to what you observe is understanding whether there are particular steps that have the most costs or that lead to the most revenue.  In the eDiscovery market the EDRM model is the generic view of the work flow.  When we did our early studies, we realized that most of the costs are incurred in the four boxes of processing, review, analysis and production.  We could make a huge difference on the cost structure of a matter if we focused our augmenting efforts just on those four boxes.  While all of our competitors tried to solve for each of the boxes, many of which weren’t generating costs, we focused on where the costs were.

Usually when finding out where the costs are in any workflow, you will also identify the bottlenecks or constraints to the work process.  Eli Goldratt with his Theory of Constraints asserts that in any give work process there are only 2-3 bottlenecks or constraints.  He recommends that the automation work focus on the bottlenecks.

Now that you have gone through the orientation steps of the Macro OODA model for software, it is time to build a prototype solution to test your understanding of what you have observed and oriented.

These prototypes can be as simple as a paper prototype as described in Vijay Kumar’s 101 Design Methods or as complicated as a software prototype.  Fortunately, our tools for building software prototypes have gotten so good that even complicated prototypes can be produced quickly.  Each week that goes by show cases another low code or no code tool for building sophisticated applications.

The Lean Canvas is a paper prototype of a business.

With the first pass of the Orientation loop producing a prototype, it is time to test the prototype with a real user in order to DECIDE what to do next.  The Macro DECISION Agile OODA loop is:

There are four possible actions that you need to decide on to move forward:

• 1. Go on to the next step
  2. Recycle within this step
  3. Pivot (take another approach)
  4. Abandon the project

When moving in fast time, the most likely decision is that you need to recycle within this step.  If you have done good work in the Observe and Orient steps, the step 4 of abandoning is unlikely.   If you have recycled several times and don’t seem to be making progress, it is often advisable to go back to the Observe and Orient steps to see if you missed something.

The first step within DECIDE is to commit to doing daily demos rather than agile daily stand up meetings.  The purpose of the daily demo is to SEE the actual progress being made.  It is a combination of understanding what was accomplished the previous day and designing the work for the next day.  The meeting should be rapid fire and last for just 15 minutes.  Once or twice a week, enough problems with the daily design may become evident and a longer stand-up meeting is scheduled to explore in more detail the execution problems with the prototype.  These longer meetings typically use the micro OODA process to go back to basics.

The participants in the daily demo should include the appropriate developers (usually a front end UI developer and a back end data manipulation and storage developer), a senior architect, a product owner or product manager, and a business owner.  An ideal room for a Daily Demo should include a large screen that can be seen by all and a white board for design discussions.  The meetings should be recorded and an audio transcript is generated.  Photos of the white board are also taken and included in the records of the meeting.  The meeting artifacts become the fodder for the weekly and monthly retrospective analysis.

The second step of the DECIDE step is to do a limited release to target users and have them use the prototype in an actual work environment.  The software should be instrumented so that actions the users take can be analyzed.  Video ethnography techniques to record what happens around the software usage are important to capture as well.  The product owner and UX representative check for additional insights through analysis of the telemetry and the video ethnography that are brought to the next day’s daily demo.

A key part of the DECIDE step is evaluating the prototype against the Outcomes and Impact that the product will have for the customer’s business.  An Outcomes focused product development effort is described in a “Product Produces Outcomes.”  As part of the Observe and Orient steps the product team develops a North Start Metric to guide the development.  The North Star Metric should be closely related to the desired outcomes for the product.

After DECIDING what to do next, it is time to ACT and launch a version of your product to a larger customer audience.  The Act Macro OODA loop is:

Now it is time to take ACTION.  As mentioned above, there are relatively few types of decisions to make during the DECIDE step.  “How to decide whether to adopt, adapt or abandon” describes a process for acting on the decisions.

As you progress the first time through the cycle, you are deciding whether it is time to launch an Minimum Viable Product to gather information from a larger sample size of users.

Through telemetry data you quickly identify who is using the product on a regular basis and these users are good candidates to perform more in depth observation.  Opening up your product to a larger audience, particularly in cloud SaaS (software as a service) environment, helps you capture key metrics for product success:

• • How many users invited to participate actually sign up and log in?
  • How regularly do the participants log into the product (ideally multiple times per day)?
  • How much do the participants use the product?

The answers from your telemetry data and your qualitative research should help you answer the key criteria for getting ready to launch your V1 product and your business.  The launch criteria are;

Bringing all the pieces of the macro agile OODA loop together, we have:

The first pass through the macro agile OODA loop gets us to a V1 of a product that becomes the test for a product solution fit as described by Ash Maurya at Leanstack.  The meta agile OODA loop cycles through the macro OODA process as the product evolves from V1 to Vn and the product becomes more of a Whole Product as described by Geoffrey Moore and Philip Kotler.

The Leanstack model identifies the key criteria that it takes to scale from an idea to a product solution fit to a product market fit to a product at scale.  Maurya’s books Running Lean and Scaling Lean describe the journey to scaling.  At Leanstack, Maurya provides tools and coaching services to aid early stage startup teams to navigate the journey to scaling.

For enterprise software products, there is another key element of software product design which Geoff Moore describes as the Whole Product concept to avoid the startup disaster of the chasm.  Moore identified that most startups don’t understand that the earliest customers are not representative of the Early Majority customers.

Moore in his books Crossing the Chasm and Zone to Win illustrates that the Whole Product concept is the method for avoiding the chasm and scaling.

Most enterprise software product development teams only focus on the core and generic product layers.  Yet, in order for a customer to use a product to produce an outcome that will have a business impact the expected and augmented product layers are needed.  These two layers usually involve creating partnerships with other suppliers of related products.  From a technical perspective this means creating application programming interfaces (APIs) within your own product and using APIs from those products you are partnering with.

Throughout this whole process of Agile OODA HCD software product development, you are trying to co-create value for the customer.  The service science researchers have nicely formalized the value co-creation process represented by this diagram from Vargo and Lusch in “Service-dominant logic 2025.”

The diagram provides both a virtuous cycle of value co-creation and a narrative of how the pieces flow together:

“Actors involved in Resource Integration and Service Exchange enabled and constrained by endogenously generated Institutions and Institutional Arrangements  establishing nested and interlocking Service Ecosystems of Actors …”

That is certainly a mouthful of complex concepts from a pair of academic marketing professors.  What is means to me is that in order for any company that produces a product to receive value it must first help co-create value with its customers.

The diagram below summarizes the keys to “Observe, Don’t Ask” in order to reduce the uncertainty of software product development by using the Agile OODA Loop at a micro, macro and meta level in order to reliably and quickly generate successful products.

Observe, Don’t Ask.  Show, Don’t Tell.  Prototype, Don’t Guess.  Act, Don’t Delay.

• • Part 1   Observe, Don’t Ask.  Show, Don’t Tell
  • Part 2   Where does “Observe, Don’t Ask” show up in software product development?
  • Part 3   The OODA Loop
  • Part 4   Orient, Evaluate and Prototype
  • Part 5   Video Highlights for Show, Don’t Tell
  • Part 6:  Show the software, don’t try to describe it

Day 194 of Self Quarantine             Covid 19 Deaths in U.S.:  201,000

Rest in Peace Justice Ruth Bader Ginsburg                                   VOTE!!

After reading the first post on “Observe, Don’t Ask,” Professor David Socha asserted “this post still begs the question of where do you observe.  Can you create a diagram for where to observe?”

A longtime colleague and collaborator, Eric Robinson shared:

“I like Kelly Franznick’s story.  The implications of that story go all the way back to product architecture and designing for the purpose of observability.  How do you make observability a core component of the product?  What if it had been a client-installed, on premises, monolithic application?  The engineer couldn’t have the quick turn around and put changes in front of the users for the next day’s ethnography cycle.

“And the ‘action logging’ was critical (not just ‘system logging’). The ability for the software engineers to see the problem directly from the logs and prove the existence of the problem is powerful and immediate.  How often have we seen stories come into the product backlog from anecdotal information and have an engineer say ‘that’s not a problem’ and the agile user story gets dropped?  The metric should be ‘how long does it take us to find out the client had a problem?’  The unit of measurement should be seconds or minutes, not days, weeks or months.

“Continuous Integration, Testing and Deployment (CI, CT, CD) are prerequisites for Continuous Observation.  The Daily Demo is a software engineer and product-centered approach — which is probably the best thing you can get quickly without daily access to a real user.  However, these questions are always present:

• • • Are we helping the customers (improving their lives, adding value, etc)?
    • How do we know?
    • How do we get the feedback?
    • How soon can we know?

“The opposite of these questions also need asking — did we decrease value, or create a problem, etc?”

After I prodded Professor Socha to tell me more, he went on to explain what he meant about “where to observe?”

“My understanding of waterfall is that is framed around the use of documents (mostly text, sometimes diagrams) to communicate what is important for the people who do the actual work.  Marketing and business analysts would work with customers, business executives, and other stakeholders to figure out what is most important to provide to customers and the business, and then record that in a document. The document is the product of “requirements elicitation”. It seems that business folks often use a slide deck to communicate such things. While there inevitably is some collaboration involved, and some back and forth discussion, the product of this phase is all about “sage on the stage,” broadcast, and “unambiguous” descriptions.

“Part of the frustration that provoked Agile methods is the myth that such documents communicate well. Few people  actually read a hundred page specification document, unless they are embedded in systems like Boeing aircraft design where those documents are taken to a much higher level of rigor and analysis. Bill Erdly talks about how he was involved in using ALL-IN-1 to automate the production of their Boeing proposals so that they could keep making changes up to the last hour before pushing the button and printing a huge document (many thousands of pages) that they would then ship off to DC with one copy on a plane (to get there in time) and the rest on several pallets sent by rail. Their system did all sorts of computation on the data, creating summaries from the raw data, doing cross-checks, etc. It was document automation in the large. And given the regulatory environment and the physics of the domain they were working in, such analyses captured in the documents could represent some goodness.

“This is different when it comes to products that are about delivering adaptive human experiences, the stuff of software-as-a-service etc. Especially different are products that have to scale and adapt to the emergent needs of many varied contexts. In those cases, no document will be correct. But a document represents a hard, reified “truth” that is harder to modify. Or to understand. Which then requires the creation of all of these change-control mechanisms to ensure that any document changes are “correct” and “good”.

“Okay, I’m blabbing about the maladies of documents.  Let me return to your question.

“Where to observe?

• • • • in naturalistic contexts where customers or customers’ customers are doing their thing. That’s the whole decades-old message about taking your software engineers to the field with the customer support team so they can observe and listen to what the actual customer does, and how the customer service representatives address the issues / opportunities / things that come up. But software engineers tend to want to stay in their offices and “be productive”. And there’s always “too much work to do” to justify the day it takes to send them into the field. Besides “it’s too dangerous” sending a software engineer, cause they will say the wrong thing or act “too nerdy” and put off the customer.
      • in the user research lab (vs reading Blink’s report a month later).
      • walking the hallway as a manager to hear the conversations, or to bump into people to have spontaneous conversations (was it Cutler at Microsoft who talked about planning to take hallway walks at just the right time to meet so and so so that he could have such and such a “serendipitous” conversation about an issue at hand?)
      • being in the room with your cross-functional team, both as a team member and as the team manager. To be more creative (see this interview by James Shore about how distributed software development work requires 30% more software developers to get the same level of creativity as when collocated). Software development is a creative industry. Creativity is as important as productivity. Bjorn also highlighted the need for junior developers to be in the same room in order to learn. Remember, Vygotsky notes that all higher-order concepts are social first, so we need to build in the social if we want to get to higher-order concepts. (There, do you feel better now that I’ve mentioned Vygotsky?)
      • when prioritizing features, make sure you can show the current state of a feature, what it actually looks like. That can help the conversation focus on what part of the thing really needs change now, and what parts can be left as they are for the moment.

“Jointly seeing and discussing helps satisficing, aiming for Good Enough. See Bach’s seminal article on A Framework for Good Enough Quality: Beyond the Buzzword (available here along with related articles). That framework generalizes to any context of satisficing; one just has to have the hard conversations about what is critical, etc. Here is the good enough definition from that article:

“To claim that any given thing is Good Enough is to agree with all of the following propositions:

• • • • • • • It has sufficient benefits.
            • It has no critical problems.
            • The benefits sufficiently outweigh the problems.
            • In the present situation, and all things considered, further improvement would be more harmful than helpful.

After absorbing David’s thoughts on the “where” of user research, I Google searched for more on user observation.  I came across Jim Ross’s “The Role of Observation in User Research.”  He asserts:

“User research consists of two core activities: observing and interviewing. Since we’re most interested in people’s behavior, observing is the most important of these activities because it provides the most accurate information about people, their tasks, and their needs.

“While interviewing is also very important, the information people provide during interviews isn’t always accurate or reliable. Often, research participants don’t know why they do things, what they really need, what they might do in the future, or how a design could be improved. To really understand what people do, you can’t just ask them, you have to observe them.”

Ross goes on to list the types of observation in user research:

• • • Usability Testing
    • Contextual Inquiry
    • Naturalistic Observation
    • Shadowing
    • Covert Observation
    • Participant Observation

“A naturalistic observation lets you to see what happens over a longer period of time, whether you’re observing one person or a group of people. You can see how a normal day unfolds without introducing your own interruptions or influencing participants. For example, while you might hear about particular problems during a contextual inquiry, observing participants over a longer period of time provides a better understanding of how often such problems occur and what causes them.”

At the end of Ross article, there is a pointer to David Travis and his book Think Like a UX Researcher:  How to Observe Users, Influence Design, and Shape Business Strategy.

In the first couple of pages, Travis likens UX research to the work of Sherlock Holmes in detecting:

“Opinions are not facts and speculation is not evidence.  Instead, his primary method of collecting facts was careful observation.  ‘You know my method, Watson.  It is founded upon the observation of trifles.'” p. 12

While I absorbed the fullness of the Holmes quote, I thumbed back a few pages to re-read why we do observation – to gain insights.

“The best way of gaining actionable and testable insights is not to ask, but to observe. Your aim is to observe for long enough that you can make a decent guess about what’s going on. Asking direct questions will encourage people to make things up, not tell you what is actually going on.

“There are two ways to observe. We can observe how people solve the problem now. or we can teleport people to a possible future and get them using our solution (a prototype) to see where the issues will arise.

“The key point is: What people say is not as useful as what people do, because people are unreliable witnesses.” p. 4

Travis goes on to provide excellent definitions of the problems of poor user research and the lists of what a user researcher should do.

The Seven Deadly sins of UX:

“The problem isn’t with the quantity of UX research. It’s with the quality: organizations struggle to distinguish good UX research from bad UX research. Here are seven examples of poor UX research practice that we’ve come across in our work with clients—along with some ideas on how to fix them.

• Credulity.

“The dictionary defines credulity as a state of willingness to believe something without proper proof. The form this takes in UX research is asking users what they want (and believing the answer).

The best way of gaining actionable and testable insights is not to ask, but to observe. Your aim is to observe for long enough that you can make a decent guess about what’s going on. Asking direct questions will encourage people to make things up, not tell you what is actually going on. There are two ways to observe. We can observe how people solve the problem now. or we can teleport people to a possible future and get them using our solution (a prototype) to see where the issues will arise. The key point is: What people say is not as useful as what people do, because people are unreliable witnesses.

• Dogmatism.

“Dogmatism is the tendency to lay down principles as undeniably true, without consideration of evidence or the opinions of others. The form this takes in UX research is believing there is one “right” way to do research.

• Bias.

“Bias means a special influence that sways one’s thinking, especially in a way considered to be unfair. UX research is a continual fight against bias. There are a handful of different kinds of bias that matter in UX research, but it’s response bias we want to discuss here. This is caused by the way in which you collect data.

• Obscurantism.

“Obscurantism is the practice of deliberately preventing the full details of something from becoming known. The form this sin takes in UX research is keeping the findings in the head of one person. UX research is often assigned to a single person on a team. That person becomes the spokesperson for user needs, the team’s “expert” on users.

• Laziness.

“Laziness is the state of being unwilling to exert oneself. The form this takes in UX research is in recycling old research data as if it’s boilerplate that can be cut and pasted into a new project. our favorite example of this comes from the world of personas. We find that clients often approach the process of developing personas as a one-time activity.

• Vagueness.

“Vagueness means not clearly or explicitly stated or expressed. In terms of UX research, we see it when a team fails to focus on a single key research question and instead tries to answer several questions at once. This sin is partly caused by the sin of laziness.

• Hubris.

“Last but not least we have Hubris. Hubris means extreme pride or self-confidence. In UX research, it takes the form of taking undue pride in your reports. All UX researchers suffer from this to some extent, but those with PhDs are the worst. And we say that as proud recipients of a PhD.5 UX researchers love data. And when you love something, you want to share it with people. So you create detailed reports packed with graphs and quotations and screenshots and callouts. Look at my data! Look at how beautiful it is!

“Overly detailed reports delay the design process. You don’t need to do extensive analyses in a spreadsheet to find the top problems. That analysis is useful later, when you want to dig into the details, but the critical findings need to be fed back quickly. This is so the design can be modified and so the build-measure-learn cycle can continue. Instead, you need to create information radiators (like usability dashboards and one-page test plans) to get teams understanding the data so they can take action on it. Information radiators are essentially advertising billboards that gradually permeate the team’s awareness of your results. As a general rule, if people need to turn the page, your report is too long. So ask yourself: how can we capture the results in a single glance? This could be a concise visual way of presenting research data, like a user journey map, a persona, or a usability testing results dashboard.

Information Radiator

Once more I feel like I have been asleep.  I have never heard of Information Radiator before.  I love it.

Think Like a Detective

“So what can we learn about doing UX research from the greatest detective of them all—Sherlock Holmes? Holmes was an investigator par excellence, but he was not a super hero (he did not have super powers). Instead, he had well-honed skills and specialist knowledge about a few things. And he was nothing if not methodical. His method comprised these five steps:

• • • • Understand the problem to be solved.
      • Collect the facts.
      • Develop hypotheses to explain the facts.
      • Eliminate the least likely hypotheses to arrive at the solution.
      • Act on the solution.

“Here are some things we can learn from Holmes’s approach that can help our UX research thinking:

• • • • Focus on the problem not the solution.
      • Create an explicit research question (actually write it down with a question mark at the end).
      • Don’t start doing any research until you have this question.
      • Don’t assume the question has never been asked before.
      • Find out what your colleagues and your company already knows.
      • Do an archival search—start by reading any prior research reports.
      • Interview team members and stakeholders.
      • Use a checklist to collect background information in a systematic manner.
      • Leave nothing to guesswork.

As I go through these definitions and lists, Travis answers my question of whether to have a hypothesis before hand.  He answers “No.”  However, you need to have a research question.  One good question is better than hundreds of small questions.  He suggests that it is more important to know a lot about a little, rather than a little about a lot.

“You may not get to wear a disguise or crawl about on the carpet with a magnifying glass, but here are some things we can learn from Holmes to improve our observation skills:

• • • • Watch people actually doing their work—don’t just get a demonstration.
      • Remember that your participants are the experts, you are the “novice.”
      • Focus on the most typical tasks, busiest days, typical days, and critical incidents.
      • Find out what activities precede and follow the task you are observing.
      • Look for inconveniences, delays, and frustrations.
      • Shadow people; follow them wherever they go.
      • Point to things and find out what they are for.
      • Get copies or photos of artifacts, samples, forms, and documents.
      • Make diagrams of the workspace.
      • List the tools people are using.
      • Note people dynamics and interactions.
      • Be alert to things happening simultaneously.
      • Record anything unusual about the scene you are looking at.
      • Ask yourself if anything is missing.
      • Observe behavior at a low level of detail—watch what people touch and what they look at.
      • Pay attention to the sequences and timing of events and actions.
      • Don’t get in the way.
      • Pay attention to trifles.”

• • “Our models, personas, scenarios and stories should include:
  • • • The primary goals that people have.
      • The workflow of tasks people carry out.
      • The mental models people build.
      • The tools people use.
      • The environments people work in.
      • The terminology people use to describe what they do.”

“Fundamentally, all UX research answers one of two questions:

(a) “Who are our users and what are they trying to do?

(b) “Can people use the thing we’ve designed to solve their problem?”

You answer the first question with a field visit and you answer the second question with a usability test.

I think you get the idea.  Just paying attention to these bullet points helps get to Outcomes and behavior changes that are needed to support Value Co-Creation.

David Socha then followed up with these thoughts and questions:

“Your notes point to the WHERE of user research, but do not extend to the need to use observe, don’t ask and show don’t tell in other aspects of the software development process. Such as in the daily demos, or when deciding if a work item is complete: after a developer makes a change, have the developer sit a customer / product owner / business analyst / etc. in front of the computer with the new system running (the system with the change) and ask that “customer” to give the system a try. Don’t tell them how the new thing works. Don’t walk them through it. Watch them using the new system. See where they do what you expect and, even more importantly, where they do something you had not expected. Ask them how well that works for them. Be curious. It’s not about whether the thing “works” or the bug is “fixed” – it is about whether the new system is the right new system. It may be that you fixed the bug, but the way the system now works actually isn’t what is needed … for whatever reason.

“In other words, I think that the observe+show mindset should be considered and embraced across many more aspects of the software development process. I don’t have a good sense or visualization of what that means yet. I still need to figure that out. My feeling is that it is something about creativity: anywhere there is a need for creativity, or perhaps any discussion about user experience (which can only be understood by experiencing it), one should try to avoid trying to use words to describe the thing. Something like that.”

Observe, Don’t Ask.  Show, Don’t Tell.  Prototype, Don’t Guess.  Act, Don’t Delay.

• • Part 1   Observe, Don’t Ask.  Show, Don’t Tell
  • Part 2   Where does “Observe, Don’t Ask” show up in software product development?
  • Part 3   The OODA Loop
  • Part 4   Orient, Evaluate and Prototype
  • Part 5   Video Highlights for Show, Don’t Tell
  • Part 6:  Show the software, don’t try to describe it

Day 190 of Self Quarantine             Covid 19 Deaths in U.S.:  200,000

Rest in Peace Justice Ruth Bader Ginsburg                                   VOTE!!

Ever since I became acquainted with the human centered design methods of the Institute of Design at Illinois Tech, I preach the gospel of the importance of user observation.  I shared my first exposure to these methods in Observing Users for Software Development.

Then in 1992 while visiting the Institute of Design of the Illinois Institute of Technology, my views on design were transformed by a five minute video from a student class project.  This video was my first introduction to the power of user observation.  Sitting in a miserable concrete walled classroom on the 13^th floor of a non-descript research building over looking some of the worst slums of South Chicago, I could barely hear the nervous student introducing his project.  It had something to do with improving the ability of the business traveler to work in a hotel room.  As someone who usually travels 150,000 air miles a year and spends >50 nights per year in hotel rooms, he had my attention, if not my expectation that he could shed any light on a frustrating environment.

The student created a relatively simple task for a male and female pair of business colleagues.  The pair had to create a business report in a hotel room, and then type the results into their laptop PC.  In the process they had to confer with other employees over the phone to get information for the report.  The student would videotape their activities in the hotel room for later analysis.  The first several minutes of the videotape showed the awkward dance of the professional colleagues trying to find a work surface that would accommodate their needs, while avoiding the cultural taboos associated with the only work surface available – the bed.

The pair searched in vain for something that would work and yet the bed remained the only place that is large enough, was convenient to the phone, the power outlets and the available light.  The pair finally concludes that the bed is the only viable place and they start to lay their papers and computers on the bed.  They then realize that there is no comfortable place to sit.  The single chair in the room is too high for the bed surface.  Yet, it hurts to kneel on the floor and it is awkward to sit on the bed without disturbing the papers and computers.  Throughout all of this trial and error, the male and female are trying not to invade each other’s personal space so that they don’t cross the line into intimacy.

After five minutes of trying to work, the pair throws their hands up and quits the exercise.  They cannot get work done in that environment.  I was amazed at how completely the five minute video transformed my experience as a business traveler from unnamed frustration with a hotel room as a work environment to being able to clearly articulate my frustrations.  And in that moment, a solution space opened up for tens of ways to transform the business traveler’s hotel working experience.  No interviews were needed.  No audio was even present on the videotape.  Just watching the interactions said it all.  The student also showed some interviews with business travelers that provided no insights on either the problems or the solutions as a counterpoint to the power of user observation.  Even though we might be experienced business travelers, we are not usually conscious about what bothers us to be specific about the problems.

This short experience of video ethnography led me to the mantra of “Observe, Don’t Ask!”

While mentoring a Brandon Fleming, CEO of Chimerocyte, a biotech startup, I encouraged him to use video ethnography while he observed how the lab test that he is commercializing is performed today.  He did the video work and is starting to analyze the video through an extension of the AEIOU method of qualitative observational research.

In our coaching session, Brandon asked “now that I have this video, how do I translate it into a good user story?  How do I write a powerful agile user story?”  In my “bad Skip” coaching style I immediately started yelling and swearing.  Not at Brandon, but at the realization that I had never talked about or written about the next step after video ethnography “showing the video to the software engineers.”  I thought it was obvious.

I need another mantra “Show, Don’t Tell!”

While collaborating on “Wide-Field Ethnography: Studying Software Engineering in 2025 and Beyond” for the 38th International Conference on Software Engineering “Visions of 2025 and Beyond,” Kelly Franznick shared a recent story about one of their clients acting on information from the user observation research in real time.

I asked Kelly how many research clients were watching the observation streams live versus coming back to watch the videos at a later “more convenient” time.

Kelly shared that they are surprised that over 80% of the use of Feedback Panel is for real time viewing and there is little use of the stored videos.  They find that many of their clients are doing lab studies at BlinkUX in their observation conference rooms. Their clients facilities don’t have the ease of being able to watch a session as a shared group and collaborate around a white board simultaneously.

He shared an example.

“One of our clients is a software company. When we have a subject using their online Software as a Service (SaaS) product, the key members of the development team (product manager, marketing manager, development manager, and lead software architect) come into one of our conference rooms. During a recent sessions, the team watched the subject encounter a serious problem with their software.

“The team almost in unison exclaimed that they had never seen the problem before. They all realized that it was a serious issue. The developers were asked if they had any evidence of how big an issue this problem was for users. The developers acknowledged that they had no instrumentation (log file captures of the problem) in their application.

“So the lead developer immediately went online into their code and installed some instructions to log whenever the problem occurred and alert him to the problem. Within minutes the alerts started flowing to the developer. After the software ran for a few hours on the live system which had thousands of users accessing the application, they found that 10-15% of users were encountering this problem.

“While users were still being studied in the BlinkUX lab, the development team designed a solution for the problem. The software engineer then implemented the solution and called back to the rest of the team to test the solution. That evening the software changes were made to the operational system.

“When the team came in the next day, the next set of subjects were using the just changed software and with a big sigh of relief the team realized that they had indeed fixed the problem.

“By spending the time to watch the user research in real time, the developers could see the problem at least 30 days earlier than the report that BlinkUX would provide as their final research artifact.

“From a business standpoint, this was a big deal as the problem was in the initial landing page access to their product and they were finding that 10-20% of users were abandoning their investigation into the product. They were losing potentially 20% of their revenue from a problem they didn’t even know existed.”

Kelly’s example illustrates the second mantra “Show, Don’t Tell!”

As I talked abut the above example with Brandon, I explained that instead of trying to explain very poorly in writing in a natural language like English the precision you need for a software application, you should show the software engineer what is needed.  You need to show the software engineer the video that you just collected or the subsets of the video that have the insights that you identify.  Natural language is a poor communicator of your insights, but the video both shows needs and opportunities and allows you to re-view the user experience many times.

The most powerful form of communication would be for all the software engineers and product managers to observe real users in their natural work environment but that is usually very difficult, expensive and intrusive.  So video ethnography is the best choice.  Where video ethnography is not possible like in observing lawyers and not wanting to break privacy and client confidentiality or in health care patient settings, then action oriented research where the software engineers do similar work is the next best method.

“So how does this relate to the Daily Demo (versus the Agile Daily Stand-up) that you advocate we use?” Brandon asked.

“It’s the same thing.  The purpose of the Daily Demo is to SHOW your previous day’s work instead of giving a status of green light, yellow light, red light of an agile daily stand-up,” I impolitely bellowed.  Again, I realized that I have never written about the linkage between “Observe, Don’t Ask and Show, Don’t Tell.”

I went on to describe that this is part of the virtuous cycle of observing the need during user research, designing and building a prototype utilizing the daily demo, and then using video ethnography when doing user research with the prototype in the user’s actual workplace.  Then you bring the video ethnography back to the daily demo to help guide the design using your North Star Metric or better yet have the software engineers observe in real time as Kelly Franznick suggested above.

Observe, Don’t Ask.  Show, Don’t Tell.

I discovered the power of the daily demo out of desperation (my usual form of innovation motivation) while I was Chief Product Officer at FTI Consulting.  We were spending a lot of time doing things right in the continuous integration (CI), continuous testing (CT), and continuous deployment (CD) processes of a modern DEVOPS and Agile shop.  Yet, it was clear that we weren’t doing the right things when it came to winning new eDiscovery business or getting customers to do “continuous adoption (CA)” of the features we were building.  We were a feature factory, not a customer outcomes provider.

I started an experiment with one of our ten agile teams to move to a Daily Demo approach to development rather than the standard agile daily stand-up.

At a daily stand-up, the task board is displayed and each person talks about their tasks for the next day (green light, yellow light, red light).  It is more administrative than substantive.

A daily demo is about demonstrating the work from the previous day.  One person demonstrates the current work on the large display.  One or more people take notes.  Somebody drives the discussion. Then a conversation occurs about what seems to be working, what questions arise about how the work evolves, what work then needs to be done (UI and back-end database) for the next day.  What is really helpful for me is that by having a separation of work in two or more software engineers I get a much better feel for the data issues and then the UI, workflow, and use case issues.

The ideal participants for a daily demo are the key software engineers (at least one front end UI developer and one back-end data storage and network “micros” services engineer), a software architect, a product manager (product owner), and a business owner (someone at an executive level who is regularly involved in sales activities).

A daily demo is about a conversation, not administrative status reporting.

A daily demo is group experiential learning.  We are learning together around a customer need and a business goal.  The experiential learning often spans multiple logic levels of understanding.

A daily demo is about designing together in the context of your North Star Metric.

A daily demo is about seeing if we all understand in the same way a user’s real needs.

A daily demo should be about 15 minutes evenly divided between showing the previous days work and designing the next days work.

By having me in the room we also had a different set of expertise from a Product Owner.  A product owner typically acts as the Subject Matter Expert (SME) and should have regular contact with real users.

What I brought to the daily demo is the combination of roles:  Business owner (how do we monetize this and thinking through which features will appeal to influencers, purchasers and users), Product Manager, SME, and UX practitioner.  My twenty years of legal customer experience brought forward usage contexts not just from eDiscovery in the FTI example but also from the perspective of forensic investigations, Early Case Assessment, contract intelligence, intellectual property intelligence, and professional services project management (BCG, McKinsey, other FTI Segments).

We recorded all of our daily demo sessions so that we could put the audio, video and transcripts into our semantic network analytics tool (FTI Ringtail, now NUIX Discover).  These recordings and analytics aided us to prepare for our weekly retrospectives as well as providing context and training examples for our product marketing group, our professional services groups, and our early adopter customers.  They could see and hear how their needs were injected into the software product design and development.  Show, not just tell.

Brandon’s two questions helped me see that the implications of using video ethnography and their potential impact are not at all obvious.

I reached out to Professor David Socha, a long time collaborator, who teaches software development management and evidence based design at the University of Washington at Bothell to see if he connected the dots of video ethnography replacing the need for user stories and the “show, not tell” of the daily demo replacing the administrative daily stand-up.  He was as stunned as I was that he had not made these connections before.

As we talked we realized the extent of the virtuous cycle of Observe AND Show that starts with observing user needs and showing that continues throughout the software development life cycle and the value co-creation frameworks of Service Science.  This virtuous cycle goes from idea to Minimum Viable Product (MVP) to launch of the V1 product to Product Market Fit and beyond as Ash Maurya describes in Scaling Lean.

Agile processes were invented and evolved to deal with this language problem of natural languages that are not precise enough and don’t convey enough to the software engineer to design what a customer actually needs.

Observe, Don’t Ask.  Show, Don’t Tell.

Of course, my colleague David Robinson would just laugh and say “Great, Skip.  You once again discovered my mantra of ‘Experience First, Make Meaning Second.’  How many times are you going to have to learn this lesson?”

Thanks Mr. Robinson.  Evidently I am going to continually learn this lesson.

Observe, Don’t Ask.  Show, Don’t Tell.  Prototype, Don’t Guess.  Act, Don’t Delay.

• • Part 1   Observe, Don’t Ask.  Show, Don’t Tell
  • Part 2   Where does “Observe, Don’t Ask” show up in software product development?
  • Part 3   The OODA Loop
  • Part 4   Orient, Evaluate and Prototype
  • Part 5   Video Highlights for Show, Don’t Tell
  • Part 6:  Show the software, don’t try to describe it