Day 190 of Self Quarantine Covid 19 Deaths in U.S.: 200,000
Rest in Peace Justice Ruth Bader Ginsburg VOTE!!
Ever since I became acquainted with the human centered design methods of the Institute of Design at Illinois Tech, I preach the gospel of the importance of user observation. I shared my first exposure to these methods in Observing Users for Software Development.
Then in 1992 while visiting the Institute of Design of the Illinois Institute of Technology, my views on design were transformed by a five minute video from a student class project. This video was my first introduction to the power of user observation. Sitting in a miserable concrete walled classroom on the 13th floor of a non-descript research building over looking some of the worst slums of South Chicago, I could barely hear the nervous student introducing his project. It had something to do with improving the ability of the business traveler to work in a hotel room. As someone who usually travels 150,000 air miles a year and spends >50 nights per year in hotel rooms, he had my attention, if not my expectation that he could shed any light on a frustrating environment.
The student created a relatively simple task for a male and female pair of business colleagues. The pair had to create a business report in a hotel room, and then type the results into their laptop PC. In the process they had to confer with other employees over the phone to get information for the report. The student would videotape their activities in the hotel room for later analysis. The first several minutes of the videotape showed the awkward dance of the professional colleagues trying to find a work surface that would accommodate their needs, while avoiding the cultural taboos associated with the only work surface available – the bed.
The pair searched in vain for something that would work and yet the bed remained the only place that is large enough, was convenient to the phone, the power outlets and the available light. The pair finally concludes that the bed is the only viable place and they start to lay their papers and computers on the bed. They then realize that there is no comfortable place to sit. The single chair in the room is too high for the bed surface. Yet, it hurts to kneel on the floor and it is awkward to sit on the bed without disturbing the papers and computers. Throughout all of this trial and error, the male and female are trying not to invade each other’s personal space so that they don’t cross the line into intimacy.
After five minutes of trying to work, the pair throws their hands up and quits the exercise. They cannot get work done in that environment. I was amazed at how completely the five minute video transformed my experience as a business traveler from unnamed frustration with a hotel room as a work environment to being able to clearly articulate my frustrations. And in that moment, a solution space opened up for tens of ways to transform the business traveler’s hotel working experience. No interviews were needed. No audio was even present on the videotape. Just watching the interactions said it all. The student also showed some interviews with business travelers that provided no insights on either the problems or the solutions as a counterpoint to the power of user observation. Even though we might be experienced business travelers, we are not usually conscious about what bothers us to be specific about the problems.
This short experience of video ethnography led me to the mantra of “Observe, Don’t Ask!”
While mentoring a Brandon Fleming, CEO of Chimerocyte, a biotech startup, I encouraged him to use video ethnography while he observed how the lab test that he is commercializing is performed today. He did the video work and is starting to analyze the video through an extension of the AEIOU method of qualitative observational research.
In our coaching session, Brandon asked “now that I have this video, how do I translate it into a good user story? How do I write a powerful agile user story?” In my “bad Skip” coaching style I immediately started yelling and swearing. Not at Brandon, but at the realization that I had never talked about or written about the next step after video ethnography “showing the video to the software engineers.” I thought it was obvious.
I need another mantra “Show, Don’t Tell!”
While collaborating on “Wide-Field Ethnography: Studying Software Engineering in 2025 and Beyond” for the 38th International Conference on Software Engineering “Visions of 2025 and Beyond,” Kelly Franznick shared a recent story about one of their clients acting on information from the user observation research in real time.
I asked Kelly how many research clients were watching the observation streams live versus coming back to watch the videos at a later “more convenient” time.
Kelly shared that they are surprised that over 80% of the use of Feedback Panel is for real time viewing and there is little use of the stored videos. They find that many of their clients are doing lab studies at BlinkUX in their observation conference rooms. Their clients facilities don’t have the ease of being able to watch a session as a shared group and collaborate around a white board simultaneously.
He shared an example.
“One of our clients is a software company. When we have a subject using their online Software as a Service (SaaS) product, the key members of the development team (product manager, marketing manager, development manager, and lead software architect) come into one of our conference rooms. During a recent sessions, the team watched the subject encounter a serious problem with their software.
“The team almost in unison exclaimed that they had never seen the problem before. They all realized that it was a serious issue. The developers were asked if they had any evidence of how big an issue this problem was for users. The developers acknowledged that they had no instrumentation (log file captures of the problem) in their application.
“So the lead developer immediately went online into their code and installed some instructions to log whenever the problem occurred and alert him to the problem. Within minutes the alerts started flowing to the developer. After the software ran for a few hours on the live system which had thousands of users accessing the application, they found that 10-15% of users were encountering this problem.
“While users were still being studied in the BlinkUX lab, the development team designed a solution for the problem. The software engineer then implemented the solution and called back to the rest of the team to test the solution. That evening the software changes were made to the operational system.
“When the team came in the next day, the next set of subjects were using the just changed software and with a big sigh of relief the team realized that they had indeed fixed the problem.
“By spending the time to watch the user research in real time, the developers could see the problem at least 30 days earlier than the report that BlinkUX would provide as their final research artifact.
“From a business standpoint, this was a big deal as the problem was in the initial landing page access to their product and they were finding that 10-20% of users were abandoning their investigation into the product. They were losing potentially 20% of their revenue from a problem they didn’t even know existed.”
Kelly’s example illustrates the second mantra “Show, Don’t Tell!”
As I talked abut the above example with Brandon, I explained that instead of trying to explain very poorly in writing in a natural language like English the precision you need for a software application, you should show the software engineer what is needed. You need to show the software engineer the video that you just collected or the subsets of the video that have the insights that you identify. Natural language is a poor communicator of your insights, but the video both shows needs and opportunities and allows you to re-view the user experience many times.
The most powerful form of communication would be for all the software engineers and product managers to observe real users in their natural work environment but that is usually very difficult, expensive and intrusive. So video ethnography is the best choice. Where video ethnography is not possible like in observing lawyers and not wanting to break privacy and client confidentiality or in health care patient settings, then action oriented research where the software engineers do similar work is the next best method.
“So how does this relate to the Daily Demo (versus the Agile Daily Stand-up) that you advocate we use?” Brandon asked.
“It’s the same thing. The purpose of the Daily Demo is to SHOW your previous day’s work instead of giving a status of green light, yellow light, red light of an agile daily stand-up,” I impolitely bellowed. Again, I realized that I have never written about the linkage between “Observe, Don’t Ask and Show, Don’t Tell.”
I went on to describe that this is part of the virtuous cycle of observing the need during user research, designing and building a prototype utilizing the daily demo, and then using video ethnography when doing user research with the prototype in the user’s actual workplace. Then you bring the video ethnography back to the daily demo to help guide the design using your North Star Metric or better yet have the software engineers observe in real time as Kelly Franznick suggested above.
Observe, Don’t Ask. Show, Don’t Tell.
I discovered the power of the daily demo out of desperation (my usual form of innovation motivation) while I was Chief Product Officer at FTI Consulting. We were spending a lot of time doing things right in the continuous integration (CI), continuous testing (CT), and continuous deployment (CD) processes of a modern DEVOPS and Agile shop. Yet, it was clear that we weren’t doing the right things when it came to winning new eDiscovery business or getting customers to do “continuous adoption (CA)” of the features we were building. We were a feature factory, not a customer outcomes provider.
I started an experiment with one of our ten agile teams to move to a Daily Demo approach to development rather than the standard agile daily stand-up.
At a daily stand-up, the task board is displayed and each person talks about their tasks for the next day (green light, yellow light, red light). It is more administrative than substantive.
A daily demo is about demonstrating the work from the previous day. One person demonstrates the current work on the large display. One or more people take notes. Somebody drives the discussion. Then a conversation occurs about what seems to be working, what questions arise about how the work evolves, what work then needs to be done (UI and back-end database) for the next day. What is really helpful for me is that by having a separation of work in two or more software engineers I get a much better feel for the data issues and then the UI, workflow, and use case issues.
The ideal participants for a daily demo are the key software engineers (at least one front end UI developer and one back-end data storage and network “micros” services engineer), a software architect, a product manager (product owner), and a business owner (someone at an executive level who is regularly involved in sales activities).
A daily demo is about a conversation, not administrative status reporting.
A daily demo is group experiential learning. We are learning together around a customer need and a business goal. The experiential learning often spans multiple logic levels of understanding.
A daily demo is about designing together in the context of your North Star Metric.
A daily demo is about seeing if we all understand in the same way a user’s real needs.
A daily demo should be about 15 minutes evenly divided between showing the previous days work and designing the next days work.
By having me in the room we also had a different set of expertise from a Product Owner. A product owner typically acts as the Subject Matter Expert (SME) and should have regular contact with real users.
What I brought to the daily demo is the combination of roles: Business owner (how do we monetize this and thinking through which features will appeal to influencers, purchasers and users), Product Manager, SME, and UX practitioner. My twenty years of legal customer experience brought forward usage contexts not just from eDiscovery in the FTI example but also from the perspective of forensic investigations, Early Case Assessment, contract intelligence, intellectual property intelligence, and professional services project management (BCG, McKinsey, other FTI Segments).
We recorded all of our daily demo sessions so that we could put the audio, video and transcripts into our semantic network analytics tool (FTI Ringtail, now NUIX Discover). These recordings and analytics aided us to prepare for our weekly retrospectives as well as providing context and training examples for our product marketing group, our professional services groups, and our early adopter customers. They could see and hear how their needs were injected into the software product design and development. Show, not just tell.
Brandon’s two questions helped me see that the implications of using video ethnography and their potential impact are not at all obvious.
I reached out to Professor David Socha, a long time collaborator, who teaches software development management and evidence based design at the University of Washington at Bothell to see if he connected the dots of video ethnography replacing the need for user stories and the “show, not tell” of the daily demo replacing the administrative daily stand-up. He was as stunned as I was that he had not made these connections before.
As we talked we realized the extent of the virtuous cycle of Observe AND Show that starts with observing user needs and showing that continues throughout the software development life cycle and the value co-creation frameworks of Service Science. This virtuous cycle goes from idea to Minimum Viable Product (MVP) to launch of the V1 product to Product Market Fit and beyond as Ash Maurya describes in Scaling Lean.
Agile processes were invented and evolved to deal with this language problem of natural languages that are not precise enough and don’t convey enough to the software engineer to design what a customer actually needs.
Observe, Don’t Ask. Show, Don’t Tell.
Of course, my colleague David Robinson would just laugh and say “Great, Skip. You once again discovered my mantra of ‘Experience First, Make Meaning Second.’ How many times are you going to have to learn this lesson?”
Thanks Mr. Robinson. Evidently I am going to continually learn this lesson.
Observe, Don’t Ask. Show, Don’t Tell. Prototype, Don’t Guess. Act, Don’t Delay.
On the Way to Somewhere Else 