Acceptance Testing: what is it good for?

Overview

Acceptance Testing is an approach to providing fast feedback based on business scenarios.  It helps teams avoid the brittleness and long delays associated with automated GUI testing.  I'll also look at a specific tool, Gherkin, and explore in detail how its approach allows us to build automation steps that are business-readable, very reusable, and potentially even allow non-technical people to author tests with no development involvement.

What puts the "Acceptance" in "Acceptance Testing?"

First, let's get the context right: the word Acceptance here is the same as the Acceptance in Acceptance Criteria, the "back of the card" additional details on on a User Story that the team needs to estimate the work.  It's unrelated to the Acceptance in User Acceptance Testing, except maybe at some overarching goal of making people happy.

The Acceptance in Acceptance Criteria does relate to the ceremony at the end of the work where the Product Owner decides whether to accept the work or not.   Our Acceptance Criteria then, are the Criteria that the Product Owner will use to make that decision.  (Or at least some of the criteria -- not intended to be a contract).

Acceptance Testing is expressing those criteria as tests.  At this level, it could be anything that the Product Owner can dream up.  All of that should be tested, but not all of it can be automated.  This puts it squarely in Q2 of the testing quadrants that are at the heart of Agile Testing by Lisa Crispin and Janet Gregory:

Behavior Driven Development with Gherkin

With all this in mind, let's look at a User Story, some Acceptance Criteria, and a Gherkin test.  We'll use the User Story format of "As a .. I want to ... so that ..", and add Acceptance Criteria as bullet points afterward.

As a nurse,
I want to scan a patient's id and get a list of prescribed medications
so that I give the correct medication to the correct patient

• Patient's name and current room number is displayed
• List of all currently prescribed medication list is displayed, ordered alphabetically
• Medications already administered are indicated in red.
• If the patient is not found, a message is displayed with the info from the scanned id

Let's look at another format, the Gherkin BDD format.

Given precondition
When actor + action
Then observable result

This standard format, like the standard User Story format, is a lot like the tool pegboard that you have in your garage:  It tells you where to put things so you can find them quickly, and tells you when something is missing.

Extending the garage metaphor, Gherkin itself is only a tool -- it can be used for many purposes including Unit Testing, GUI testing, load testing, etc.  Let's look at how to use Gherkin to create  Acceptance Tests from our Acceptance Criteria.

Our first Criteria is that the patient's current room and floor is displayed.  Let's first re-word that in the Gherkin format, called a Scenario:

Given a patient is in a particular room
When the nurse scans the patient's id
Then the patient's room and floor is displayed.

Okay, big whoop.  This isn't very helpful -- it's just wordier.  In this case, when the actor and action are the same as the user story, it's not very interesting.  But let's add some specific examples to it:

Given a patient with id 1234 named Kevin Klinemeier is in room 305B
When the nurse scans patient id 1234
Then Kevin Klinemeier and 305B is displayed

Now that we have specifics in place, this is an Acceptance Test.  This looks like the kind of thing we could actually automate, and it begins to suggest some other scenarios:  What happens if the wrong patient id is scanned?  Can there be two patients with the same ID?  Can there be two patients in the same room?  But first, this is just English-- how does it get automated?

Each step in the BDD Scenario is implemented by a developer.  If we were working with a user interface, it might look like this:

First choose the variable parts, described here by underlines:

Given a patient with id 1234 named Kevin Klinemeier is in room 305B

Then write some "wiring and glue" code to put the variables into place, described here as a UI-automation pseudocode:

 public void givenAPatient(patientID, patientName, roomNumber) {  
   window.open(PATIENT_ENTRY_SCREEN)  
   writeIntoTextBox("patient_id", patientID)  
   writeIntoTextBox("patient_name",patientName)  
   writeIntoTextBox("room_number", roomNumber)  
 }  

Great, that makes sense.  Except it stinks!  Our goal as Agile Testers is to provide fast feedback, right?  But now we can't run our test until the GUI is finished.  Furthermore, automated GUI testing is itself notoriously brittle.  Instead of a simple writeIntoTextBox method, we're more likely to have something that looks like this:

 writeIntoTextBox("xpath:=/html/body/div/div[2]/div/div/form/label/input[1]",patientID);  

That's xpath in there, and it's specifying an input box on a particular place on the page.  If the page changes the order of the fields, the test breaks.  If it changes the location of the fields, the test breaks, if it removes something that is above the fields, the test breaks.  You get the idea.

Instead, we avoid the problems of GUI automation by testing in the "middle".  The unit test level is too early, and as a result is too technical and doesn't speak to the business.  The GUI layer is too late, it doesn't allow the team feedback on both behavior and design at a time when it can be resolved.  The sweet spot is to test services, and hence design services for this kind of testing.  This is best described by Mike Cohn's testing pyramid:

Part of the feedback we are providing is on whether a given software design is Testable.  There are many roadblocks to testability in GUI, but much less at the service level.  Furthermore, when we provide these BDD Scenarios as the starting place for testing at the service level, our services as a result are not only testable, but based on the underlying business concepts, and that's just what we want out of a good service layer.

To complete the example, let's look at a service-layer implementation for the step we have above:

Given a patient with id 1234 namedKevin Klinemeier is in room 305B

 public void givenAPatient(patientID, patientName, roomNumber) {  
   patientEntryService.createPatient(patientID,patientName,roomNumber);  
 }  

When the nurse scans patient id 1234

 public void scanPatientId(patientID) {  
    result = patientScanService.scan(patientID);  
 }  

Then Kevin Klinemeier and 305B is displayed

 public void shouldContainPatientNameAndRomNumber(patientName, roomNumber) {  
    assert.that(result.getName).isEqualTo(patientName);  
    assert.that(result.getRoomNumber).isEqualTo(roomNumber);  
 }  

BDD and Building Blocks

We now have one working example: the patient scan.  It looks like maybe a lot of work.  The exciting part of BDD, and the thing that makes this all a scalable part of long-term software development processes, is what happens when we move on to the next Scenario.  Let's pick one of the other Scenarios we came up with before: scanning a patient that doesn't exist.  First let's write it with Given .. When .. Then:

Given a patient with id 5678 named Joe Justice is in room 205A

When the nurse scans patient id 0001

Then an error is displayed

 I've highlighted the first two steps in green: they're free!  Well, we already paid for them, but they're free to re-use in this new scenario.  The only new wiring to write is looking at the error that is returned.

Once we have a few of these basic building blocks, we can really take that development effort and amplify it through everyone else on the team who can create these near-english Scenarios:  QA, BA, Product Owner, perhaps even Customers and Users.  As an example, with just the four different steps we've completed so far, we can create and automatically run all the following test Scenarios without further development help:

Scenario:  Duplicate patient ID
Given a patient with id 2838 named Bill Bates is in room 203D
And a patient with id 2838 named Clay Cummings is in room 405F
When the nurse scans patient id 2838
Then an error is displayed 

Scenario:  Multiple patients in room
Given a patient with id 8432 named Amelia Anderson is in room 403B
And a patient with id 9392 named Nelly Newborn is in room 403B
When the nurse scans patient id 2838
Then Amelia Anderson and 403B is displayed
And Nelly Newborn and 403B is displayed

These scenarios might also drive discussion:  should we be able to tell the difference between an error for a patient id that is missing vs an error for a patient id with multiple entries?  And that's just the kind of business-level input we are looking to provide feedback for, delivered before the GUI is even started.  

Summary

Acceptance tests that express business concepts are powerful, but automation at the GUI level is problematic and waiting for the GUI layer to be built introduces long delays for testing.  Instead, provide faster feedback by automating these tests at the "middle" layer (service layer) where the business concepts can be expressed, but the GUI details are not yet in the way.

Using the BDD Gherkin format to write tests creates tests that are not only automatable, but also creates small building blocks that reduces the overall cost of automation and allows non-technical team members to extend the automated test suite.