Empirical approaches to Product Development are easy to talk about but hard to do  (deterministic approaches die hard)

After Napster was acquired by BestBuy, I spent a lot of time thinking about the meaning of ‘product development’. BestBuy outsources the majority of its software development efforts to Accenture – a large consulting organization-  and, as a result, a contracting based metaphor permeates how many people at BestBuy think and talk about product development. “The business” and its needs are treated as separate from the people who are building the products. Cost, time to market, and technical feasibility dominated most early stage product ideation discussions that I was part of. “Should we do it?” was usually superseded by “Can we do it?” and that was very quickly followed by “How quickly can we do it?”  I have spent most of my professional career arguing that the best, most successful products occur when singular teams are responsible for marrying business value, customer delight, and technical excellence and that separating those concerns with rigid organizational boundaries results in sub-optimal solutions. So, I spent the next couple of months clarifying my understanding of product development and, with many of my colleagues at Napster, communicating our perspectives to our new partner.

Definition

“Product Development” is the process of

–          identifying market problems or opportunities

–          coming up with ideas about how to address those problems/opportunities

–          converting those ideas into functional products

–          Delivering those products to market, successfully.

I take a broad view of the product development process. In many organizations, product development is viewed as the activity of implementing requirements or “converting ideas into functional products”.  Leveraging available technologies to realize ideas is a critical piece of Product Development. But it is not the only piece.

Goal

Value creation is the primary goal of a product development process. Success is realizing a market opportunity. This kind of success is usually described with terms like “more revenue”, “increased conversion”, “increased cash-flow”, “enterprise-value”, and “customer loyalty”. When product development is viewed as purely a technical activity and is not explicitly tethered to market outcomes, success is usually described using different terms like “functional correctness”, “extensibility”, “predictability”, “efficiency”, and “quality”.  These objectives are definitely respectable, but are they more important than near-term or long-term market success?

Clusters

Product development activities can be broken into clusters that are punctuated by two critical questions: “Are we building the right product and are we building the product right”.

Clusters:

–          Product/Market Fit Cluster – “Are we building the right product”?

• Will an existing market or market segment adopt it?
• Can we create a new market?
• Are we solving a material problem for consumers?
• Will they part with dollars or attention to have that problem solved?
• Who are the early evangelists and how do we reach them?
• How do we cross the chasm to widespread adoption?

–          Technical Cluster  – “Are we building the product right”

• Are we building a well designed software products?
• Are we building cost effectively
• Can we easily modify, add, and remove features?
• Do we have a flat cost of change curve?
• Is our platform scalable?
• How quickly do we eliminate defects?

These two questions form the yin and yang of product development.  Find a great customer problem to solve but offer a poor or buggy solution; no customers will use or buy the product. Architect the perfect software design for a problem that does not exist; no customers will use or buy the product.

The tendency to focus product development process improvement efforts on the technical cluster is rooted in Taylorist theories and early approaches to manufacturing.  In this model, success comes from producing many products and reducing production costs through division of labor, aggressive use of technology for automation, and the application of methods to ensure consistent outcomes.  Here the focus is on increasing efficiency and reducing variation. Unfortunately, these approaches work only when you are building the same thing over and over.  Most product development efforts are for products that have never existed before or are materially different from their predecessors.

Challenge

Product development is filled with risk. 75% of all New Product Development programs fail commercially.

Risk categories include

–          technical risk: can the opportunity be realized given existing technologies; how much will it cost

–          product/market fit risk: is there a significant consumer problem to solve; is there appetite for the solution

–          Business model risk: is there a business model associated with the product that will support a viable business; are there cost-effective distribution channels that will support customer and sales growth.

–          organizational risk: Is there sufficient organizational will, resource availability, and focus required to support the ongoing effort that committing to a product entails

Though success can transform markets, societies, and economies, failure is the common outcome.  Many product development schools of thought have emerged over the last couple of decades. Some have focused only on the technical issues, others on product/market fit issues, and others have painted broad strokes across both. The amount of literature available boggles the mind.

Approaches:

That being said, there are two fundamental approaches to seeking the promise of bringing a successful product to market.  What distinguishes them is their respective

–          historical frame of reference and world view

–          basic ideas about what is most valuable

–          Attitudes about risk management.

Deterministic

In mathematics, philosophy, and computer science, a deterministic system is one that is completely predictable; no randomness exists. Given the same set of inputs, an identical output will emerge every time.  Very few systems or processes are deterministic. Most are stochastic and have outcomes that vary, even given the same set of inputs. Even so, many modern manufacturing techniques take processes that have somewhat random outputs and apply statistical techniques to bring them as *close* to deterministic as possible; identify the causes of variation and control them. These approaches are really the foundation of deterministic product development. The elimination of risk and uncertainty is viewed as both (i) possible and (ii) a primary goal.  Deterministic approaches want to treat product investments like cash in terms of safety, but like equity in terms of return.

Historically deterministic approaches have focused more on the technical cluster and on execution than on the product/market fit cluster.  RUP, Waterfall, and up-front based detailed specification approaches fall into this category.  Deterministic approaches in software product development are the most popular today, though over the last 10 years, with the explosion of internet and the need to rapidly develop high quality software given a significant amount of uncertainty, they have been severely criticized.

Deterministic process approaches make significant investments in detailing work plans so as to ensure certain and repeatable outcomes. “If the plan is well articulated and sufficiently detailed, success will be assured”. Change of plan is viewed as failure and as something to be avoided. In the waterfall approach gates are used to ensure that specific plans have been prepared and reviewed and forward progress is blocked if insufficient planning has occurred. These approaches have yielded positive results in industries that

• • must deliver mission critical systems (where their failure results in catastrophe)
  • are very mature and the products well understood

Empirical

The empirical approaches have been around for as long as deterministic approaches, but, in the software world, have only risen in popularity over the last decade. The Toyota Production System, Deming, and earlier Training in Workforce programs during WW II are really the basis of the empirical school of thought. Empirical defined means “Information gained by means of observation, experience, or experiment – refers to the use of working hypotheses that are testable using observation or experiment”. Empirical processes view beliefs about markets, products, and technologies as hypothesis that must be continually tested and validated in the face of feedback.

• • Product specifications become working hypothesis. Ultimately customer adoption validates their correctness.
  • Technical architecture’s become working hypothesis. Ultimately  ease of evolution and scalability tests their correctness

It is here that Empirical approaches differ from their Deterministic brethren. Empirical approaches seek continual feedback throughout the development process. They believe that Product Development is a fundamentally uncertain activity. Instead of trying to make the process more deterministic, and therefore predictable, through identifying and managing all factors that impact variation, Empirical approaches swing to the other side of the fence and say “embrace change –  You will not achieve predictability, but you can achieve positive market outcomes  if you invest everywhere in feedback mechanisms.”. Empirical approaches make investment in

• • transparency
  • feedback mechanisms (ongoing review of the product by customers, engineers, analysts, and exeuctives as it is being developed)
  • eliminating significant up-front investment
  • speed of evolution

Through consistent and frequent evaluation of the product development process, empirical methods seek optimal outcomes though they do not try to predict them.

On the Technical cluster side of the house, Scrum, Lean, & XP are some of the dominant empirical approaches being employed today. On the Product/Market fit cluster side of the house, Customer Development, Database Marketing, Goal-oriented design constitute the main empirical approaches being used.

Napster & Best Buy

When BestBuy acquired Napster, we had just finished re-structuring our approach to product development. We had broken the organization into a collection of cross-functional teams that were responsible for different product lines or streams of value production. On the technical cluster side we were had adopted a time-boxed, incremental approach with short iterations, continuous integration, and frequent releases. On the product/market fit side we were experimenting with a/b testing frameworks, crowd-sourced user feedback, and goal-directed design approaches. As a result, we were releasing far more products, far more frequently and had dramatically improved morale on the product side of the house. To be honest, we had made more headway in the technical arena than the Product/Market Fit arena, though I believe we were headed in the right direction there.  We were struggling to get teams more attached to market trends and indicators.

At a theoretical level, BestBuy liked the idea of “empowered” semi-autonomous cross- functional teams; especially at the executive level. They were definitely very interested in being “agile” and in how it worked when building software. We spent a fair amount of time talking about how “agile” and empirical approaches to product development extended beyond just converting ideas into products and extended into fundamental organizational and operational philosophy. However, it became evident that many of the folks at BestBuy were primarily interested in empirical methods as ways to increase developer productivity. They liked the idea of a broad definition of product development and pushing more frequent customer attitude and behavior feedback mechanisms into the process, but eyes lit up when we talked about moving away from waterfall and building stuff in parallel. I think that this was a direct consequence of their working experience with Accenture.  As I understand it, Accenture adopts a very formalized stage-gated, waterfall approach to building software with lengthy development cycles, and from the perspective of many people at BestBuy this is far from ideal.

Post Merger Experiences

When we started working together on projects, the differences in our approaches became very clear, at a tactical level. We both agreed on the value of adapting to feedback, team ownership, and agile in theory but in practice

–          BBY wanted to create specifications through consensus with little feedback from marketplace feedback mechanisms

–          BBY wanted to plan products and the resource allocation plans to realize them in great detail for lengthy time frames. Small investment approaches to validate hypothesis were not part of the culture

–          BBY focused on issues in the technical cluster (cost, technical feasibility, resource issues) more than items in the product/market fit cluster

I do want to point out that BestBuy is a large organization with many very brilliant employees.  Many of the people with whom I dealt with definitely understood the value of adopting empirical methods when tackling risky market problems. The challenge was the fact that BestBuy’s existing business flourished and became wildly successful through the application of deterministic approaches to development. BestBuy has many organizational and policy structures built around large investments in planning, predictability, and variance reduction. It reminded me that becoming “agile” or “consumer-driven” in a real and tangible way, is a herculean effort.  Grasping concepts and approaches is the tip of the iceberg.