Sprint Demos can be a Scam

The idea is great. With proper User Stories and Acceptance Tests there should be no surprises during the Sprint Demo. The Product can be released to the User after a successful Sprint Demo.

Reality can be that after a successful Sprint Demo there are still several issues found in production. These issues can be related to usability, but also to real defects in the Product.

Sprint Demos can give a false sense of progress and only show that part of the System is working (the subset of features planned for the Sprint).

How is that possible when all the Sprint Demos went perfectly well? What are we missing here?

Do your Users find more issues in production than anyone expected?

There are two main issues I have seen with Sprint Demos:

  1. It is easy to show a demo that gives a false sense of progress when using mocking and stubbing. What we are demonstrating that bits and pieces seem to be working, but we are not really demonstrating an integrated system.
  2. Acceptance Tests have a different purpose than System Integration Tests.  Developer sometimes confuse these to be the same. Acceptance Tests cannot be used to prove proper System Integration.

The purpose of Acceptance Tests is proving to the User that a feature is present and working as expected. The purpose of System Integration Tests is proving to the Programmer that the System Components work together properly.

Joe Rainsberger has written about how Integrated Tests are a scam. These are not the same as Integration Tests. I recommend reading his blogs on the topic (see below).

Integrated Tests rely on the correct implementation of more than a single piece of nontrivial behavior of the System.

Integration Tests focus on checking integration points between subsystems, systems, or any other nontrivial client/supplier relationship.

Unless the Team writes proper micro-tests, it will be hard to pinpoint the root-cause of a failing Integrated Test. A combination of micro-tests and System Integration Tests are be able to find root-causes of system issues.

Here is some recommended reading. After reading I suggest thinking about what this means for how you write tests.

My recommendations:

  • Learn to write proper micro-tests
  • Learn to write contract tests and collaboration tests
  • Learn to write integration tests at integration points in your system
  • Unlearn writing integrated tests

Blindly following the mechanics of the Scrum framework does not guarantee that a Team develops a quality system.

Technical development and testing practices are needed to reap the benefits of Scrum and eliminate a false sense of progress and quality.