“On the surface, most companies agree that quality and testing are important: however, when we dig deeper and start to listen… it’s interesting to hear who treats [it] as an investment to be maximized versus a cost to be minimizes.” Ronald Cummings - John and Owais Peer22
The roles of quality assessment and testing (QA) can be critical in delivering software products that satisfy customers and produce positive business outcomes. Unfortunately, QA is often viewed and run as a cost center rather than a strategic partner, a view sometimes held by quality professionals themselves. In this article I'll explore the effect our views on quality have on how we position QA in an organization.
Before I jump in, I want to be clear that my purpose here is not to criticize the authors I mention. I find their work incredibly valuable. I’m not criticizing “shifting-left”, identifying problems, or even setting standards. I reference these authors and practice to illustrate a subtle pattern of thinking that frames the topic of quality and the impact it has on the QA position within the organization.
The Influence of Plato
“For the pattern is existent for all eternity; but the copy has been and is and shall be throughout all time continually.” - Plato 19
In my opinion, the root cause of QA being seen as a cost center is an ongoing debate over what quality means. This debate has its roots in ancient Greece and Plato's concept of perfection.
According to Plato, everything in this world; the tree outside, your coffee, and your table are imperfect. If they were perfect, the argument goes, they would never change. Your table would never fade, scuff, move or need replacing. The fact that things do change means they “...are only imperfect copies”2 of a real Form, which is an immutable, intrinsically perfect object in true reality.4 To have knowledge of true reality, Plato believed that our souls have contact with these Forms, and through “a process of recollection”18 we rediscover our innate understanding of their goodness.
This view of the good has led to two perspectives within the product and QA industry. The first perspective, which I call "Quality as Intrinsic Excellence," closely resembles Plato's intrinsic viewpoint. The second perspective, which I call "Quality as Value," emerged as a response to the limitations of the first. However, the second perspective has not ended the debate or had the intended cultural effect on QA practices. This is because it has not fully departed from Plato's influence.
Quality as Intrinsic Excellence
“Quality is neither mind nor matter, but a third entity independent of the two . . . even though Quality cannot be defined, you know what it is." - Robert M. Pirsig20
The Quality as Intrinsic Excellence perspective can be observed in the viewpoint presented by Tuchman, who described quality as achieving "the highest standard" and avoiding "compromise with the second best".3 According to this perspective, quality is something we strive for, which is separate from the product itself. It's something we recognize when contemplating greatness and it remains constant over time. Therefore, we should capture this standard in our specifications to ensure we consistently produce quality products. If we can do this, then the question is answered: "Quality is conformance with specifications"7. In their study of the history of defining quality, Reeves and Bender captured the intrinsic perspective nicely: "A romance novel may provide good entertainment value… but can it be claimed to be a quality product when compared to a great work of literature that endures throughout the ages?".5
The primary objection to this perspective is that it “leaves little practical guidance” to anyone trying to decide what quality is, how to determine when excellence is achieved, and who gets to set the standard.8
These points were primarily driven by the service industry, which was uneasy with the abstract concept of excellence without any focus on who was buying.5 As Software as a Service gained prominence, the buyer’s needs and focus on customers were further elevated and the Quality as Value perspective solidified.
Quality as Value
“Quality is value to some person.” - Jerry Weinberg17
The Quality as Value perspective places a clear emphasis on a product's “capacity to satisfy wants” and value judgments of customers or stakeholders.5 This acknowledges that even if a product meets a high standard, it will be considered low quality if customers aren’t happy. Ultimately, the individuals or groups that the company deems significant determine the level of quality.
At first glance, this definition appears to be completely different from the Quality as Intrinsic Excellence perspective. However, In a recent article, Janet Gregory and Lisa Crispin point out, just like excellence, a value based definition “doesn’t help us evaluate our product quality”.14 What is value? Which person gets to decide? What if they say, value is achieving the highest standard and not compromising for second best?
They seem different, one deals with a vague standard, while the other emphasizes the importance of people. However, recall Plato and Pirsig's ideas regarding intrinsic good and quality. They suggest we all possess an innate understanding of quality, which we simply need to remember.21 What better way for a company to rediscover the intrinsic good and sell a product than by engaging with their stakeholders?11 This reveals that, in practice, both the intrinsic and value definitions of quality are the same, both ultimately redound on someone to choose what counts as the good. To really drive this point home, one of Janet and Lisa’s readers comments that discovering quality is “only as good as the underlying model” and warns of the danger in using anything other than “transcendent perceptions” to validate it.14
Indeed, there are many studies that hypothesize what transcendent quality might be, then survey and analyze customers, managers and even user stories to discover quality.6,9 Not surprisingly, the findings of such studies are inconclusive on what quality or its attributes are. Despite that, the idea that quality is something with attributes that should be identified and built into a product, is popular in QA. Anne-Marie Charrett puts it this way: “Quality morphs and molds itself to the context it's living in. In reality, we never will achieve quality. Quality is too fast for us and horribly intangible to be able to emphatically state “we have quality”. But we can go on a journey to improved quality, and we can monitor how we are progressing.” 1
Melisaa Fisher in her Medium post titled “Testing Principles 2.0” writes: “One of our goals is to find risks that impact the business, operations, project, product and quality attributes.”15
James Bach, in his article critiquing the idea of building quality into a product asserts that quality is separate from the product: “Instead, quality is a relationship. Excellent quality is a wonderful sort of relationship. Instead of “building” quality, it’s more coherent to say we arrange for it.”13
A relationship between what and what?
James concludes the article with a “new myth” of quality in the form of a garden. A story that tells us that quality “is out of our hands”. Why? We aren’t “all powerful”, “all knowing”, and the “product is subject to decay”;13 just like your table…
Michael Bolton points out that these ideas of quality are an example of the reification fallacy and in a 2023 article specifies what he means by quality: “Quality is not a property of a product; it’s a set of many-to-many-to-many relationships between elements of the product, a variety of customers, and their different needs, desires, and preferences.” 17
Unfortunately, Michael also falls into the reification fallacy in order to create the many object model: many, many, many. He turns elements, needs, desires and preferences, into things that have a relationship with people. This is not far off from conceptualizing quality as a third thing with a relationship with people and product. I’m going to address this topic further in my next post, but I want to preview some thoughts to provide a non-reified perspective. Elements, capabilities, features are aspects of products; needs, desires, preferences are aspects of people. In this context, only people and products exist as things and the relationships of note are between people and people, people and product, and product and product.
From this reading, it’s clear to me that the industry is still struggling with what to make of quality. Is it a separate thing in the product or a separate thing apart from the product? What if both are wrong?
QA as a cost center
“[IT] Projects should be on time, within budget, and at a high level of “quality,” whatever that word might mean… We always want to increase quality attributes if we can without increasing cost” - Mark Schwartz16
Why is this debate about quality a problem, and how does it cause QA to be a cost center? Despite the differences in definitions, they all imply quality is a separate thing and standard of good. They only differ in who defines the good. QAs who hold these views, implicitly or explicitly, know their job is related to quality and they need a standard to evaluate the product. Whether that standard is defined by their own ideals, the CEO's sense of excellence, or the customer's transcendent perception of goodness, it doesn't matter. They feel a need for a standard, and enforcing that standard increases the cost of production making them a cost center.
To overcome the lack of guidance, many QA organizations demand specifications from the product organization to be used as the standard. After all, these QAs think, quality is conformance with specifications.
However, this doesn’t solve their problem. The product organization doesn’t have a perfect product they use to construct requirements. At most, they have existing products that aren’t perfect. Simply saying quality is value to customers doesn't help. PMs talk to customers, but those conversations are for understanding the problem and validating solutions, which are also not perfect. Customers aren’t experts on what solves their problem or what quality is. This means that requirements are necessarily best guesses, incomplete, and subject to change. In desperation, the QA organization may attempt to use an external standard like the ISO 9126 model. 12
From this, an us-vs-them environment develops, where PMs are frustrated with QAs because they’re inflexible, asking for complete and stable standards before it’s known if the product solves a problem. QAs get frustrated with their PMs because they don’t produce specs or definitions of quality from customers. Ultimately, QA's concerns will be overruled, the product shipped, and they will explain this result as their struggle to get their team to care about quality. The tension between shipment and perfection, confirms to the company that QA is a cost center that needs to be minimized. Likely with automation.
This frustration and fear of being automated away, understandably, motivates QAs to abandon quality. If no one will define quality, why should they be on the hook and seen as the roadblock? The engineers, managers, or whole team become responsible to ensure the product meets quality standards. QAs, in turn, "shift-left" to identify problems early in the development process. What standard do they use to identify a problem early? Who knows.
Even with these tactics to smooth out the process, QAs are still a cost center, they have just been moved upstream because it's cheaper to have delays early. In this mode, QAs have to show that every dollar spent identifying problems early reduces the cost of rework and that the reduction is greater than the opportunity cost of any delay. Garvin puts it this way: “Firms are…performing suboptimally: were they only to increase their expenditures on prevention… [and not] find their rework… falling by an even greater amount.” 8
Accounting for the benefits of things not happening is challenging, if not impossible, for QA organizations, compounding the problem of feeling the need for a standard to justify delay. In order to show their worth to the company, QAs fall into the trap of producing output and vanity metrics like; test case counts, automation coverage, and bug tickets rather than metrics that demonstrate forward looking business impact.
In this reality, QAs find themselves in an impossible position, dealing with a vague, ever-changing standard of quality and no path to fix a core organizational problem stagnating the profession.
For the future of QA, it's critical we resolve this fundamental debate in the industry, so that we can construct a positive business enhancing narrative of what we do. To do that we need to throw out the debate between quality being in the product or apart from the product and change our conception of what quality actually means. We need to understand and embrace the fact that quality is our knowledge of the product. A perspective I will elaborate on in my next post.
Sources
[1] Anne-Marie Charrett, “Quality is a journey - but do you know your destination?” Anne-Marie Charrett, March 23, 2018, https://www.annemariecharrett.com/quality-engineering/.
[2] Arthur Herman, The Cave and the Light: Plato Versus Aristotle, and the Struggle for the Soul of Western Civilization, Illustrated edition (Random House Trade Paperbacks, 2014), https://www.amazon.ca/Cave-Light-Aristotle-Struggle-Civilization/dp/0553385666
[3] Barbara W. Tuchman, “THE DECLINE; OF QUALITY,” The New York Times, November 2, 1980, sec. Archives, https://www.nytimes.com/1980/11/02/archives/the-decline-of-quality-quality.html
[4] Brian Duignan, “Plato and Aristotle: How Do They Differ? | Britannica,” accessed March 20, 2023, https://www.britannica.com/story/plato-and-aristotle-how-do-they-differ
[5] Carol A. Reeves and David A. Bednar, “Defining Quality: Alternatives and Implications,” The Academy of Management Review 19, no. 3 (1994): 419–45, https://doi.org/10.2307/258934.
[6] David Ameller et al., “A Survey on Quality Attributes in Service-Based Systems,” Software Quality Journal 24 (February 8, 2015), https://doi.org/10.1007/s11219-015-9268-4.
[7] David A. Garvin, “Product Quality: An Important Strategic Weapon,” Business Horizons 27, no. 3 (May 1, 1984): 40–43, https://doi.org/10.1016/0007-6813(84)90024-7
[8] David A. Garvin, “Competing on the Eight Dimensions of Quality,” Harvard Business Review, November 1, 1987, https://hbr.org/1987/11/competing-on-the-eight-dimensions-of-quality.
[9] Fabian Gilson, Matthias Galster, and Francois Georis, “Extracting Quality Attributes from User Stories for Early Architecture Decision Making,” 2019, https://doi.org/10.1109/ICSA-C.2019.00031.
[10] Gerald Smith, “The Meaning of Quality,” Total Quality Management 4 (January 1, 1993): 235–44, https://doi.org/10.1080/09544129300000038.
[11] Ipek Ozkaya et al., “Making Practical Use of Quality Attribute Information,” IEEE Software 25, no. 2 (March 2008): 25–33, https://doi.org/10.1109/MS.2008.39.
[12] “ISO/IEC 9126,” in Wikipedia, March 14, 2023, https://en.wikipedia.org/w/index.php?title=ISO/IEC_9126&oldid=1144500024.
[13] James Bach, “Quality is Dead #2: The Quality Creation Myth,” Satisfice, March 13, 2009, https://www.satisfice.com/blog/archives/5200
[14] Janet Gregory and Lisa Crispin, “Quality - Different Perspectives,” Agile Fellowship, January 3, 2023, https://agiletestingfellow.com/blog/post/quality-different-perspectives
[15] Melissa Fisher, “Testing Principles 2.0” fishouthebox, November 19, 2022, https://fishouthebox.medium.com/testing-principles-2-0-6f8077a03096.
[16] Mark Schwartz, A Seat at the Table: IT Leadership in the Age of Agility, 1st edition (Portland, OR: IT Revolution Press, 2017), https://www.amazon.ca/Seat-Table-Leadership-Age-Agility/dp/1942788118
[17] Michael Bolton, “Quality: Not Merely The Absence Of Bugs,” DevelopSense, February 2, 2009, https://developsense.com/blog/2009/02/quality-not-merely-absence-of-bugs
[18] Plato, Meno (Cambridge University Press, 2011).
[19] Plato, The Timaeus of Plato (Macmillan, 1888).
[20] Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance: An Inquiry Into Values, 1st edition (Mariner Books, 2005), https://www.amazon.ca/Zen-Art-Motorcycle-Maintenance-Inquiry/dp/0060839872
[21] Robert M. Pirsig and Wendy K. Pirsig, On Quality: An Inquiry into Excellence: Unpublished and Selected Writings (Mariner Books, 2022), https://www.amazon.ca/Quality-Excellence-Unpublished-Selected-Writings/dp/0063084643.
[22] Ronald Cummings - John and Owais Peer, Leading Quality: How Great Leaders Deliver High Quality Software and Accelerate Growth (Roi Press, 2019) https://www.amazon.ca/Leading-Quality-Leaders-Software-Accelerate/dp/1916185800