If you want to release software faster, it’s a given that you need automated deployment capabilities, and an incremental product change process like SCRUM – but you also need confidence in the code base. This confidence comes from having agreement on what the application currently does, and being able to verify quickly at any time that it still works as users expect.
Acceptance defines how you demonstrate it is working
Even in an agile development process such as SCRUM it is of the utmost importance that you define acceptance criteria that can be quickly verified. When the business and the team agree to implement something in a sprint, coding shouldn’t be started until a description exists in English of how to demonstrate that it works properly. This description provides both a script for the developer to demo the functionality to the business, and a set of steps to guide the developer or a QA resource to implement an automated test that verifies it works correctly without requiring manual testing.
It is important that the acceptance criteria leave as little room for interpretation or the need to get more info from the author of the criteria as possible.
Bad acceptance criteria
Go to the products page. Click on a product and enter in purchase details on the next page. Submit the order and make sure there aren’t any errors.
Good acceptance criteria
- Go to http://mydevserver/somepage
- Click the product “My Test Product”
- In the page that appears, enter the following values:
- Name: Test
- Quantity: 6
- Click the “Submit” button
- On the page that appears, the message “Product ‘test’ was added to your cart’ must appear on the page.
- An email must have been sent that includes the subject ‘About your order from XYZ corporation’. The body should contain the message in the attachment to this user story where ‘Product name’ in the attachment is replaced with ‘Test product’ and ‘User’ is replaced with the account you were signed in as when you made the purchase.
Quality products are not an accident
When you haven’t defined acceptance criteria that is either written up or contained in an existing automated test that is self documenting, there is functionality in your application that is subject to change at any time. It can change in that if refactoring is done, the existing functionality can break without catching it with a manual test – but even worse, there is nowhere the team can go to understand its original intent and the reason for its existence in the first place. This leads to incorrect assumptions when enhancements to it are made, disagreement between team members in communicating its operation and intent, and a deceptive view on its completeness.
In a green field project, it’s easier to enforce automated acceptance testing being done as you can hold back kicking off the next sprint until all acceptance tests can demonstrate passing and the features are demoed and accepted by the business. The business and the team get used to the velocity needed to do so, and the rate at which new features are available.
In a brown field (existing) project, chances are there will be significant functionality that already exists for which no automated tests exist, verifying it must be done through manual testing, and in a worse scenario; it may not be tested regularly at all and the business has lost clarity on its behavior and operation. To automate verification of it working properly will cause the velocity of the team to slow down.
When this happens, a team gets into what I like to call “quality debt”. The more features that exist in the application that are unverified, or the longer it takes to verify those features, the more investment it takes to ensure quality of the application – and in turn the slower you can release. This situation is in direct opposition to continuous delivery, and to get your product ready for quicker cycle times a strategy for paying down this debt is crucial. The higher up this scale you are, the lower your debt.
Evaluating acceptance readiness for continuous delivery in existing functionality
If you are on a brown field project and want to set a goal to reduce your release cycle time, the team needs to evaluate each function of the software to determine how far along this progression of acceptance readiness it is.
- Implemented – the software does something in the code, and it was demonstrated at some point. No tests or documents exist that describe it however.
- Partially verifiable – a document or test exists somewhere that describes how the function should work, but it has grown out of sync with changes over time of the feature, or is incomplete and does not cover all cases.
- Acceptance testable – a document exists somewhere that describes how the function should work, but it takes manual effort to verify it each time a release grows near.
- Automated acceptance – Acceptance tests exist that can be run automatically on demand and report their success or failure. They cover all implemented aspects of the feature as of the current sprint.
Pay off existing debt
If your project’s features are not at phase 4 (Automated acceptance), you will need to schedule time in coming sprints to help them get there. I would caution against jumping from implementation to automated acceptance without establishing the acceptance criteria again. If there is no document or test that says what a feature should do, automating it without making sure it meets the needs of the business (and that what it does is understood) is dangerous because your are investing in automating something that is more likely to change. Revisit each existing feature with the business and come up with acceptance criteria first, then automate.
Don’t add new debt
In addition to scheduling a portion of the workload of the sprint to pay down the existing debt, avoid adding to it by defining better acceptance criteria in user stories scheduled for completion that relate to new work. Also require that automated acceptance tests be built before the conclusion of the sprint that prove proper operation of those stories. You will find that you can get less done in the sprint, but at each one’s conclusion you have features that can be released to users with confidence as they can be verified automatically at any time.
The pain only lasts as long as you want it to
The harsh truth is that the more your existing application does that is not automated for acceptance, the more painful it is to get out of debt. An organization can continue to go about business as usual without addressing it, but doing so leaves them at a competitive disadvantage as they take longer to get to market with new functionality. Organizations with better discipline around managing their workload and establishing automatable acceptance processes will respond more quickly to changes in the market, deliver features that meet the needs of their customers faster, and deliver them with more consistent quality.
Also published on Medium.