I did a talk at TestBash Germany last week that sparked lots of positive response, but also some critique. Critique is fair: It was a 30 minute inspirational talk in which I wanted to explain why Immanuel Kant’s work “Critique of Pure Reason” matters to testers. Quite a few people found me afterwards, asked me questions, and commented: Critique. Job well done (I’m padding my own shoulder here).

But, as it sometimes happens in these times of SoMe, a tweet showing one of my slides, followed up by a tweet of another, both included to provoke thinking about the state of software testing, sparked a discussion which has taken a direction I feel I need to relate explicitly to. Click the tweet to read the thread:

I was laughing on stage, and it came from my heart as what I intend to say with the slide in the talk is that all current ways to think about testing are bad, but I have the the answer. An arrogant message? Well, I’m on stage for 30 minutes, honest, but performing the gig. I mean what I say, but I also asked people to judge for themselves (I’m sure they did), and it was in the context of a talk intended to entertain and get people thinking at the same time.

I think criticizing the status quo is central if you propose something new. “If it aint broke, don’t fix it,” and I think there are broken things in testing. I dot it publicly regularly when I blog on social issues in Denmark on DenOffentlige.DK, where it’s welcomed. Danish prime minister Lars Løkke Rasmussen thanked me for critique on one occasion when I complained about a reform he proposed to help young people out of the educational system. I made him think, he said.

So I can’t claim ignorance. I feel I hadn’t thought enough about this. It was a pair of good slides in the context of the conference, the stage and the talk, but definitely bad on twitter just so.

When the whole talk is published in a few weeks, I hope people will go and listen to both context and intention, and perhaps even offer me feedback on the whole thing.

However, the cat is out of the box now, so I need to explain my critique more thoroughly. I criticized the three testing “philosophies” I see out there: ISTQB, Context Driven Testing, and Agile testing. Here we go:

ISTQB

I haven’t seen anything yet from these people, but that doesn’t surprise me. ISTQB is pretty robust to critique. That’s actually my problem with it: It’s a castle. As long as it stays inside the castle, that’s ok, I have no problem with it. Problem is, these people don’t stay in the castle: They dominate the market for software testing.

Context Driven Testing

James Bach cares about critique. I love him for that! But, James, please don’t take it personally. You’re one of the greatest thinkers about testing. I hope you’ll dig your way though Kant’s first Critique as I think you’ll find it enlightening. Kant is a genius with his own pedantic style, but his philosophy is consistent.

With that in place, here’s my answer to the thread above: I have a philosophical (logical) problem with the principles of context driven testing, that – when I read them literally–confuse me about whether testing can critique quality. Here’s the argument:

Testing is a thing that critiques things, including things about things, e.g. quality. Critique is not just critique, it involves judgement. CDT has principles and they can be used to self-evaluate whether I’m context driven, so I can judge my testing, but the principles are so crafted that when read literally, they don’t allow me to critique and judge my testing. Here’s why: The first and second principles says this: “(1) The value of any practice depends on its context. (2) There are good practices in context, but there are no best practices.” The first one exclude the possibility that a practice can be valued before it’s practiced in context. Since testing in context is practice, it means the value of testing can’t be known a priori. The second says there are no best practices, which implies there are no worst ones either. No best or worst, even in theory? Well, if that principle is to be taken literally, then no, there are no best or worst practices, not even in theory. But if there is no best and worst, not even in theory, it means that value is not a thing: Value is, at best, a point, or an empty void, perhaps even everything. The implication of this is that every practice, even type of practice, is of unknown value (everything is a thing, but it’s a thing we cannot know), even when attempted valued in context, that is a posteori. So the snake seems to be eating its own tail here: No testing in CDT can have value. This is paradoxical because testing must have value to critique as things without value have no ontology and only things with ontology can create bodies of critique, i.e. critique with ontology. The reason for this has to do with the philosophical concept of ontology, which things must have in order to critique. This gives the context driven tester a problem: When context driven (according to the principles), I can’t assign a value to my work, and if I can’t do that, my meaningful existence as a working tester ends. Second, it gives my stake holder a problem: If my testing is literally valueless, then the outcomes of my testing must be valueless too as the critique is without body, ontology, value. Valueless things can exist, and quantum physics shows that things happen out of nothing, but things that happen out of nothing cannot represent knowledge as knowledge depends on ontology and value. So, what is that which is something, but not knowledge? I think data is a good word for that, potentially not noise, but still not knowledge, probably not even information. In a provocative tone I’d say data is that which discriminates no testing from context driven testing. Mind blown? I tell you, I was when I realized.

Of course, I’m overly pedantic and attacking a single principle on purely logical grounds, which is in many ways unfair since there’s much more to CDT than it’s first two principles, e.g. great people doing awesome work training important skills to testers with the aim om maximizing the outcomes of testing (my interpretation), thereby actually working to maximize the value of testing, making testing a sustainable activity, and saving the IT-world from the politics of ISTQB), CDT will survive the paradox, I’m sure. But the principles are still around and read, and even though the argument above above is long and annoying (I would probably get lost in it myself, if I was reading it from someone else) it is my experience that people actually read the principles, and that the paradox ‘gets’ them intuitively, even if they can’t express it.

Agile Testing

I actually have only noticed a single reaction to my slides from agilists, but I’d like to share my problem with agile, as it is actually much, much bigger than any of the two above, and doesn’t only extend to testing. My problem with ISTQB is mostly political, whereas that of CDT is philosophical. Agile’s problem is that it doesn’t appreciate testing because it doesn’t fundamentally desire to create value. It’s secret desire is power for the developers. That’s my own diagnosis, you are welcome to disagree, but here’s why: There This time my problem starts with the values, for example the one that values working software over comprehensive documentation. The old rule, “no work is done before the paperwork is done” still applies, but this principle in combination with Agile’s idea of continuous delivery combined with budgetary constraints (resources are always scarce in the real world) turns the principle into a choice between documentation OR software. In a resource constrained reality, the net result is therefore that there’s always more important work to do delivering working software than to document what was done, and there is therefore never good documentation coming out of Agile. This is just one example of why Agile projects are never done, as the other value statements tend to collapse in the same way as soon as they are subjected to resource constraints. The result is developers working long hours, economics out of control, technical debt growth, and lots of wasted work. Agile is on steroids creating legacy, I think. And I’m not done yet: Who really knows quality? At the core of Agile is not quality for customers, but for developers since they are the ones writing the software and that has been forgotten by the ones writing the principles, that contains no model for judging the core values of quality, innovation, collaboration, truth. There’s a fundamentally unfair situation: Developers own the power to make the software, and the users’ only power is reduced to influence in the daily collaboration. Oh yes, there’s potentially also the possibility of rejecting deliverables to production – but then always at the cost of precious and expensive time, delays and hence suffering in production because of still unsolved problems: At the cost of quality. Notice, I’m not looking for a silver bullet to solve this forever, because one such does not exist, but for a fairer relationship between developers and customers. Market mechanisms will not fix this, not even over iterations, retrospectives, and team learning. Why? Well Adam Smith wasn’t wrong about market dynamics, but his ideas have proven themselves insufficient by far. Agile is far from demonstrating the value of self-organizing organizations. I think it is probably taking us further away from just that.

dsc_0837_12190551833358608724.jpg

There. Those are my professional opinions about the three. Meanwhile, I work in Agile, performing CDT and traditional testing. Is there a solution? I think so. For now, I work by the principle that testing is by its very intention and nature practical/empirical and rational, rooted in knowledge a priori and drives human experience (knowledge) a posteori. But that’s old news, the Stoics knew that already some 2000 years ago.

And there are great people in Agile and CDT actually trying to make things valuable for those that matter, and I listen to them a lot.

But I think its time look for new models for value than those that are implied in the above testing philosophies. If you disagree, and find there’s no problem, that’s ok with me, but please accept at least an honest attempt looking for a new philosophy in testing by consulting one of history’s greatest and most acknowledged philosophers, and that such work should nok be without a critique of the status quo.

Feedback welcome

I’m living my life by principles like this: We don’t know who we are until we see ourselves in the face of the other.

Even though I had the stage light in my face, I felt connected to the audience. Performing a talk starts in the speaker between the gut, heart and the intellect: Patos, ethos, and logos. But without connection to the audience, the talk may be perfect, but without effect. From my gut feeling and the feedback I received, what I said had effect.

Feel free to use the comments below. I’ll try to be quick approving comments, but please accept my apologies up front that replies may take a little while as I’m facing a busy time right now both privately and professionally.

dsc_0862_14561996357472331978.jpg

#Testfest done.

Reklamer

6 thoughts on “A Stage Performance

  1. Well, I have to say that Kant is still on my ‘to be read’ pile. But:

    “Judgement” is a very loaded word. It can either imply JUDGEMENT – “Choose A or B, but choose well, for your decision is for all time and your very life hangs in the balance!” Or, it can imply a process of using all your skills, knowledge and experience to arrive at a wise choice depending on the prevailing circumstances at the time. This second definition is not so absolute; it implies a nuanced decision according to conditions and circumstances.

    You say that if there are no best practices, then that implies that there are no worst practices either. I think that’s a fallacy. There are always going to be some practices that are the worst possible, things that no person in their right mind would do, such as not testing at all, or concentrating on only one type of bug, or critiquing the UI design without looking at functionality (or vice versa)..

    When James Bach says “there are no best practices”, then I interpret that to mean that there are a range of practices; but the choice of an approach to use must be a decision, a judgement (if you like) based on what is the best fit to the particular circumstances at the time. On that basis, all the practices that deliver good outcomes are equally valid, and amongst that sub-set of all possible practices, then there are no worst practices. The worst ones have already been rejected; what is left are all those available that best fit the environment.

    You mentioned quantum physics. Quantum theory did not replace Newtonian physics for a range of practical purposes; as Klaatu says in the original (1956) film “The Day the Earth stood still”, “It serves to get me from one planet to another”. Newtonian physics works best on the macro scale and delivers workable, useful results for that sort of problem. Quantum physics explains observed phenomena at the micro scale, even if the implications seem outlandish if translated to the macro. Thus it is with philosophies. Most people manage everyday life without deep introspection and applying philosophical principles to their actions, both in and out of testing. Whilst it may be true that “the unexamined life is not worth living”, most of the time we have to deal with life as it is on a minute-by-minute basis. Any philosophy runs the risk of getting too deeply involved in over-thinking issues.

    Though my mind may change once I’ve read my Kant…..

    1. Thanks for your good reflections. I appreciate it. I agree judgement can, for sake of the argument, imply both of the things you mention here. As for the fallacy, I actually think nothing could sometimes be best, even in terms of testing. What important is that value (to someone in context, it’s concrete) implies a best and worst. Nothing or something would be special cases of best and worst (not necessarily in that order). What if there is only one possible case? Thats the same as saying there’s nothing, logically, as there is nothing to choose, measure or value.

      I agree in your interpretation of James saying it, but here were talking about a principle.

      Niels Bohr points out in his 1954 Columbia speech that our senses has biological amplification mechanisms that mean that we can sense things on quantum level. This leads him to think that it happens elsewhere in the body, including the brain. I think the random (something coming out of nothing) is something we need to embrace in philosophy.

      Overthinking is certainly a risk here.

  2. You take on Agile was absolutely fascinating. I would love to see you flesh that out more. When you talked about real world constraints and them implications that have for putting agile into practice, it got my mind whirling. Maybe you flesh some of that out in your talk as well? please share when it is available!

    1. Thank you very much for that. I actually did not flesh it out much as my talk was about a philosophy that I think is needed to move testing onwards beyond the current philosophies of testing. The above is so far my best attempt to describe a risk with agile most is missing and therefore regularly getting trapped by, namely that while Agile is fun for devs, and fun is good for many reasons, once the fun stops delivering value (and sometimes it does), the situation becomes difficult and there are no mechanisms to save you, except learning about value for those who matter.

      A problem is that the value of code seems unrelated to the value of products (this seems proven when I think about the amount of code in a new car, I’m told there are ten times as much in a car as in an F35 fighter, a development project costing many times more than new cars), yet working software still equals working code. But developers can’t know the value of a complex software product, they’re walking a forest, so to speak, and can’t see the big picture of the forest, and neither can the testers who are put in the team’s.

      Agiliats suggest experimentation and learning to solve this, and I think that’s good, but they often forget the caveats that it requires that all those who are learning ‘get’ the big picture, which will never happen in a large, budget constrained project where stake holders don’t want to pay for everyone knowing everything. Why should they?

      I’m courious to hear your thoughts on the subject?

      1. I think I will have to process this a bit more before I can articulate my thoughts on this. This is an interesting line of reasoning and I”ll have to think it through a bit more. Thanks for getting the wheels of thought going 🙂

Skriv et svar

Please log in using one of these methods to post your comment:

WordPress.com Logo

Du kommenterer med din WordPress.com konto. Log Out /  Skift )

Google+ photo

Du kommenterer med din Google+ konto. Log Out /  Skift )

Twitter picture

Du kommenterer med din Twitter konto. Log Out /  Skift )

Facebook photo

Du kommenterer med din Facebook konto. Log Out /  Skift )

Connecting to %s