Test harness built on eXecutable Specification

Vlad Mettler   2015/05/07   Comments Off on Test harness built on eXecutable Specification

THE HARNESS

Synopsis: Post describes implementation of ports and adapters architecture while creating a BDD-like test harness around a legacy system.

The post and code is available at: https://bitbucket.org/vladekm/xs_harness

eXecutable Specification (XS) is an expression of the desired system capabilities.

We will use XS to build a harness of tests around an existing system. This harness would allow us to conduct a refactoring and re-architecting with a fairly high level of confidence.

The situation

  1. Legacy system needs refactoring
  2. Legacy system needs extension
  3. Legacy system needs decoupling and modularization
  4. Often, legacy system does not have a well defined interface

The proposal

Executable specification should drive the development of a test harness around the legacy system.

Harness of automated tests around the legacy system

In order to start refactoring we must prove that the changes we make to the code, do not change its functionality. Similarly, before we extend the system with new functionality, we must ensure that we do not break the existing one.

As far as the official interface to the system is concerned – if we find ourselves in need of a test harness then we probably do not have a clearly defined and tested interface. We may take into consideration some of the documentation provided with the system, but this leads to a dangerous path of trusting that the documentation is in sync with the reality.

The type of testing that is about to be proposed is very close to functional testing. We are not interested in the internal modularization of the existing software. We want to make sure that it works as a whole. This will open a path for modularization or even split of the tested system into discrete services.

It is of crucial importance to be able to isolate tests from each other and from the possibility of random data getting into the system under test. We need to be in total, deterministic control over the system in order to reliably assert and analyze the outcomes of system changes during the tests.

Ports and adapters introduce seams between the tested system and its dependencies

It may be difficult to provide all the services necessary for a system to function. The tested system may rely on a 3rd party REST interface that cannot operate in a sandbox mode, hence preventing us from execution of calls without interfering with a live system. We can address this situation by applying a ports and adapters pattern to our communication with that 3rd party REST API and inject a mock REST service providing a test double for the adapter.

Supporting such a mock may seem time consuming but is important for our understanding of how our system communicates with other systems.

Ports and adapters introduce seams between the testing and tested systems

The harness tests are independent from the system they are testing. They do not know anything about the internals of the system under test. They are oblivious to the persistence layer or architectural choices. They see only the outer shell of the system.

Sometimes the only outer shell that will be visible to us is a web page. Sometimes we will have access to a REST or RPC service. All these entry points are valid entry points for our tests.

Additionally we must make sure that the test cases we are building can operate as individual units. Initially we want to provide a test suite for the whole system but one of our goals is the extension and amendment of the system. We may be forced to dramatically change or even remove some of the tests we write. Hence, we must ensure that our test units do not depend on each other.

We will tap into the capabilities of our system with adapters specific to each existing interface. We may have an adapter talking to a web page and another one to the REST.

We need to organize these adapters so that the functionality they expose is unified. This will be achieved by the use of contracts. Each contract is effectively a port into which an adapter capable of implementing that port can be plugged. These ports lay the foundation for the future API facade of our system.

Evolution of an Interim SDK providing a programmatic interface to the tested system

It would be a standard procedure to start the testing exercise with writing a test suite around an official API and to base the tests on documentation available. This is usually a luxury which is not available. Hence, there is a need to evolve an official API to a system. It is possible to make it a part of the process providing the harness of tests.

We do not know the outline of the system we are about to test. So we must embark on a journey to discover it. There usually will be an obvious entry point to the system. It may be a ‘log in’ page or some kind of registration form. Usually this is a good place to start with the test harness.

Starting with that we will discover that we need some other entry points in order to set up our tests. E.g. we might need to setup a user before we can test the login or password reminder functionality. This is a good thing. Our plans (at least initially) need to be very flexible. We must find the end of the spaghetti of functionality dependency so that we start testing at the very core fucntionality of the system.

Eventually we will end up with a series of adapters connecting us to existing implementation. These adapters, when fed to the SDK’s ports, will allow us to provide a unified, programmatic interface to the system. A first step to the redefinition of implementations and the introduction of new APIs.

Use of a ubiquitous language (XS)

We could define our harness as a series of programs executed against the existing entry points. It would be adequate but we can do better. We can base these programs on a business readable domain specific language. This language would be a basis for exchange of information between:

  • the requirements owners
  • the programmers
  • the testers
  • the eXecutable Specification

A move towards this way of specifying requirements signals to the business the importance of successful communication during the software development process.

Participation of all the stakeholders in this exercise is essential for the test harness to suceed. In order to facilitate this we must provide a language which is easy to understand by all the stakeholders.

The diagram

We are aiming to deliver a system implementing the following architecture:

Diagram of the harness

Example of a test harness built using eXecutable Specification

In order to illustrate the ideas mentioned in the preceding paragraphs, we are going to build a small proof of concept test harness around the search engine DuckDuckGo. We will be testing that the results of arithmetical operations which can be entered into DuckDuckGo’s search actually return correct answers.

Problem definition using Gherkin as syntax for XS

Gherkin example – simple

Feature: DuckDuckGo can do arithmetic

Scenario: DuckDuckGo can add
GIVEN that I search DuckDuckGo with term: "2 + 2"
WHEN I parse the arithmetic result
THEN the result is "4"

Scenario: DuckDuckGo can substract
GIVEN that I search DuckDuckGo with term: "6 - 2"
WHEN I parse the arithmetic result
THEN the result is "4"

Scenario: DuckDuckGo can multiply
GIVEN that I search DuckDuckGo with term: "3 * 2"
WHEN I parse the arithmetic result
THEN the result is "6"

Scenario: DuckDuckGo can divide
GIVEN that I search DuckDuckGo with term: "8 / 2"
WHEN I parse the arithmetic result
THEN the result is "4"

The DSL we used in that example is called Gherkin. Thanks to its similarity to English, it is easily understood by non-technical team members.

We encourage the reader to dig into Gherkin’s syntax at a later date. For now we need to understand that feature files are divided into scenarios representing a different aspect of the feature. Each scenario comprises a number of steps following a Given/When/Then pattern. The pattern matches the following actions:

  1. Set up of the initial system condition
  2. Action carried out on the system
  3. Analysis of the change of the system’s state or values returned by it.

The feature file we defined is quite naive and we could expand it with more examples to make sure that we are testing more possibilities. This is usually done to provide examples that can be used for triangulation of the implementation.

Gherkin example – triangulation values

Feature: DuckDuckGo can do arithmetic

Scenario Outline: DuckDuckGo can add
    GIVEN that I search DuckDuckGo with term: "<term1> + <term2>"
    WHEN I parse the arithmetic result
    THEN the result is "<result>"
        Examples:
        |term1|term2|result|
        |2    |2    |4     |
        |3    |8    |11    |
        |123  |341  |464   |

Scenario Outline: DuckDuckGo can subtract
    GIVEN that I search DuckDuckGo with term: "<term1> - <term2>"
    WHEN I parse the arithmetic result
    THEN the result is "<result>"
        Examples:
        |term1|term2|result|
        |7    |2    |5     |
        |12   |3    |9     |
        |464  |341  |123   |

Scenario Outline: DuckDuckGo can multiply
    GIVEN that I search DuckDuckGo with term: "<term1> * <term2>"
    WHEN I parse the arithmetic result
    THEN the result is "<result>"
        Examples:
        |term1|term2|result|
        |3    |2    |6     |
        |8    |3    |24    |
        |8    |13   |104   |

Scenario Outline: DuckDuckGo can divide
    GIVEN that I search DuckDuckGo with term: "<term1> / <term2>"
    WHEN I parse the arithmetic result
    THEN the result is "<result>"
        Examples:
        |term1|term2|result|
        |12   |4    |3     |
        |8    |4    |2     |
        |104  |13   |8     |

We have expanded the list of examples that we are covering. We have also introduced another feature of Gherkin called Scenario Outline; these are templates for scenarios which will be then parse the values from the Examples section.

It is possible to further complicate the Gherkin file by elimination of the 4 scenario outlines. The gains of concise expression of intent are slightly outweighed by increased syntactical complexity. Some of the intended audience members, especially those non-technically inclined, may be feeling a bit overwhelmed with a programmatic like look of the resulting specification.

Gherkin example – overly complex

Feature: DuckDuckGo can do arithmetic

Scenario Outline: DuckDuckGo can add, subtract, multiple and divide
    GIVEN that I search DuckDuckGo with term: "<term1> <operator> <term2>"
    WHEN I parse the arithmetic result
    THEN the result is "<result>"
        Examples:
        |term1|operator|term2|result|
        |2    |+       |2    |4     |
        |3    |+       |8    |11    |
        |123  |+       |341  |464   |
        |6    |-       |2    |5     |
        |8    |-       |3    |9     |
        |464  |-       |341  |123   |
        |3    |*       |2    |6     |
        |8    |*       |3    |24    |
        |8    |*       |13   |104   |
        |8    |/       |2    |4     |
        |8    |/       |4    |2     |
        |104  |/       |13   |8     |

Framework setup

We now need to choose a technology to translate these specifications into something executable. Gherkin was originally used in the Cucumber package written in Ruby. Our example will be written in Python and will be using the behave package (http://pythonhosted.org/behave/) in order to parse the specification.

The initial structure of our test harness files:

duckduckgo_harness/
    functionality/
        duckduckgo_can_do_arithmetic/
            features/
                duckduckgo_can_do_arithmetic.feature
            steps/
                steps.py
            environment.py
    requirements.txt
  • functionality directory holds all the features that we are testing. Each of those functionalities is named according to a pattern which allows us to quickly identify the nature of the functionality. In our case we have a duckduckgo_can_do_arithmetic
  • features directory contains our actual feature file as per behave’s specification. The chosen convention is to give the feature file the same name as its functionality directory. We place the text constituting our executable specification in functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature file.
  • steps directory is responsible for translation of each step present in the .feature files into a series of executable python commands
  • environment.py file present in the root of the harness is responsible for setting the stage for the behave process. We’ll keep it empty for now
  • requirements.txt traditionally holds a list of non-standard library Python packages needed for our project to operate

This is a milestone 1 in the repository. We will be adding these milestones for easy reference of code changes.

We will need to initialize our environment and make sure that it contains all the needed packages (pip install -r requirements.txt).

We can run our first behave command:

behave duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/

Which is going to result in the following output:

Feature: DuckDuckGo can do arithmetic # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:1

  Scenario Outline: DuckDuckGo can add                # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:3
    Given that I search DuckDuckGo with term: "2 + 2" # None
    When I parse the arithmetic result                # None
    Then the result is "4"                    # None

  Scenario Outline: DuckDuckGo can add                # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:3
    Given that I search DuckDuckGo with term: "3 + 8" # None
    When I parse the arithmetic result                # None
    Then the result's value is "11"                   # None

<more failed scenarios>

Interestingly the end of the output is going to contain a suggested steps implementation skeleton. We are going to ignore these as the suggestions would result in a lot of repetition or very similar steps. We are going to use a feature of behave allowing us to specify generic steps capable of accepting parameters.

We are creating our own steps.py file as:

from behave import given, when, then


@given(u'that I search DuckDuckGo with term: "{search_term}"')
def search_duckduckgo(context, search_term):
    assert False


@when(u'I parse the arithmetic result')
def fetch_result(context):
    assert False


@then(u'the result is "{expected_result}"')
def check_result(context, expected_result):
    assert False

Running the test suite again we will see a bunch of failing tests.

Feature: DuckDuckGo can do arithmetic # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:1

  Scenario Outline: DuckDuckGo can add                # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:3
    Given that I search DuckDuckGo with term: "2 + 2" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:5 0.000s
      Traceback (most recent call last):
        File "/Users/vlad/.virtualenvs/harness/lib/python2.7/site-packages/behave/model.py", line 1173, in run
          match.run(runner.context)
        File "/Users/vlad/.virtualenvs/harness/lib/python2.7/site-packages/behave/model.py", line 1589, in run
          self.func(context, *args, **kwargs)
        File "/Users/vlad/projects/harness/duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py", line 7, in step_impl
          assert False
      AssertionError

    When I parse the arithmetic result'               # None
    Then the result is "4"                            # None

<more failing tests>

Failing scenarios:
  duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:3  DuckDuckGo can add
  duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:3  DuckDuckGo can add

<more failing tests summary>

0 features passed, 1 failed, 0 skipped
0 scenarios passed, 12 failed, 0 skipped
0 steps passed, 12 failed, 24 skipped, 0 undefined
Took 0m0.002s

This is a good thing. It means that our gherkin is syntactically correct but the tests are not passing. Now we need to modify the steps to do something useful.

We need a way of communicating with DuckDuckGo in order to execute the command and fetch the result. In classical BDD approach this would be done directly in the steps. Unfortunately this introduces britleness to the steps as they do not operate on functionality of the given web page but rather on the HTML details of that page.

Pages and page components

In order to avoid the tight-coupling of steps and the construction of the HTML page serving as the interface, we will introduce a layer of indirection betwen the steps and the HTML elements. We will use pages and we will populate them with page_components.

  • Pages will be responsible for navigation and holding together the page_components.
  • page components will be providing the API to the functionality hidden behind the DOM.

Our new structure is now marked as milestone 2 in the repository

duckduckgo_harness/
    functionality/
        duckduckgo_can_do_arithmetic/
            features/
                duckduckgo_can_do_arithmetic.feature
            steps/
                steps.py
        environment.py
    page_components/
        __init__.py
        answer_widget.py
        search_box.py
    pages/
        __init__.py
        base.py
        home_page.py
        results_page.py
    __init__.py
    requirements.txt

We now have a BasePage object responsible for providing the shared functionality for other Pages:

class BasePage(object):

    """Shared functionality for the Pages"""

    def __init__(self, browser, url, **kwargs):
        self.url = url
        self.browser = browser
        self.page_components = Placeholder()
        for k, v in kwargs.items():
            setattr(self.page_components, k, v)

    def visit(self):
        self.browser.get(self.url)

The pages are initialised in the environment.py and the page objects are passed on to them using dependency injection:

def initialize_pages(context):
    context.pages = Placeholder()
    context.pages.search_page = HomePage(
        context.browser, URL, search_box=SearchBox(context.browser)
    )
    context.pages.results_page = ResultsPage(
        context.browser, URL, answer_widget=AnswerWidget(context.browser)
    )

We then update the steps to use the pages which are now available on the context:

@given(u'that I search DuckDuckGo with term: "{search_term}"')
def search_duckduckgo(context, search_term):
    context.pages.search_page.search(search_term)


@when(u'I parse the arithmetic result')
def fetch_result(context):
    context.data['result'] = context.pages.results_page.read_the_result()


@then(u'the result is "{expected_result}"')
def check_result(context, expected_result):
    assert expected_result == context.data['result']

The tests are now passing.

behave duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/
Feature: DuckDuckGo can do arithmetic # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:1

  Scenario Outline: DuckDuckGo can add -- @1.1        # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:9
    Given that I search DuckDuckGo with term: "2 + 2" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 2.220s
    When I parse the arithmetic result                # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.044s
    Then the result is "4"


<scenarios removed for brevity>


1 feature passed, 0 failed, 0 skipped
12 scenarios passed, 0 failed, 0 skipped
36 steps passed, 0 failed, 0 skipped, 0 undefined
Took 0m22.317s

For a simple implementation of BDD style testing we could stop here. But we are aiming for something more by using switchable adapters!

Adapter

We have already implemented all the logic necessary to communicate with a browser using Selenium/webdriver. We now have to refactor our code to hide this logic in an adapter. We really want to end up in a situation where our code in steps does not know much about how we implement the communication with the system under test. This will open a possibility of replacing the adapters with equivalents working with different endpoints.

Since we have decided that the pages and page objects are now implementation details hidden in an adapter we will postulate the following structure:

duckduckgo_harness/
    adapters/
        selenium/
            page_components/
                __init__.py
                answer_widget.py
                search_box.py
            pages/
                __init__.py
                base.py
                home_page.py
                results_page.py
            __init__.py
            selenium_adapter.py
        __init__.py
    functionality/
        duckduckgo_can_do_arithmetic/
            features/
                duckduckgo_can_do_arithmetic.feature
            steps/
                steps.py
        environment.py
    __init__.py
    requirements.txt

The new file (adapters/selenium/selenium_adapter.py) is now playing a role of configuration previously held within the environment.py. It also exposes two public methods. A SeleniumAdapter needs a browser to operate. This is still handled in the environment.py file but should be probably delegated a bit higher in order to be able to run the test suite in different browsers. This is outside of the scope of this tutorial.

We implement the adapter:

from .pages import HomePage
from .pages import ResultsPage
from .page_components import SearchBox
from .page_components import AnswerWidget


class Placeholder(object):
    pass


class SeleniumAdapter(object):
    """WebDriver based adapter

    Provides functionality necessary to communicate with DuckDuckGo
    via Web Interface using Webdriver.
    """
    URL = 'http://www.duckduckgo.com'

    def __init__(self, browser):
        self.pages = Placeholder()
        self._initialize_pages(browser)

    def _initialize_pages(self, browser):
        self.pages.search_page = HomePage(
            browser, self.URL, search_box=SearchBox(browser)
        )
        self.pages.results_page = ResultsPage(
            browser, self.URL, answer_widget=AnswerWidget(browser)
        )

    def get_search_result(self, search_term):
        self.pages.search_page.search(search_term)
        return self.pages.results_page.read_the_result()

    def parse_arithmetic_result(self, search_result):
        return int(search_result)

and we use those functions in steps:

from behave import given, when, then


@given(u'that I search DuckDuckGo with term: "{search_term}"')
def search_duckduckgo(context, search_term):
    context.data['raw_result'] = context.adapter.get_search_result(search_term)


@when(u'I parse the arithmetic result')
def fetch_result(context):
    context.data['result'] = context.adapter.parse_arithmetic_result(
        context.data['raw_result']
    )


@then(u'the result is "{expected_result}"')
def check_result(context, expected_result):
    assert int(expected_result) == context.data['result']

We point the environment.py to the new adapter:

from selenium import webdriver

from duckduckgo_harness.adapters import SeleniumAdapter


class Placeholder(object):
    pass


def before_all(context):
    profile = webdriver.FirefoxProfile()
    context.browser = webdriver.Firefox(firefox_profile=profile)
    context.adapter = SeleniumAdapter(context.browser)
    context.data = {}


def after_all(context):
    if getattr(context, 'browser', None):
        context.browser.close()

and we have a functioning test suite again:

(xsharness) vlad@hex ~/projects/priv/xs_harness $ behave duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/
Feature: DuckDuckGo can do arithmetic # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:1

  Scenario Outline: DuckDuckGo can add -- @1.1        # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:9
    Given that I search DuckDuckGo with term: "2 + 2" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 2.291s
    When I parse the arithmetic result                # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "4"                            # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can add -- @1.2        # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:10
    Given that I search DuckDuckGo with term: "3 + 8" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 1.660s
    When I parse the arithmetic result                # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "11"                           # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can add -- @1.3            # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:11
    Given that I search DuckDuckGo with term: "123 + 341" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 2.072s
    When I parse the arithmetic result                    # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "464"                              # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can subtract -- @1.1   # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:19
    Given that I search DuckDuckGo with term: "7 - 2" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 1.695s
    When I parse the arithmetic result                # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "5"                            # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can subtract -- @1.2    # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:20
    Given that I search DuckDuckGo with term: "12 - 3" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 1.783s
    When I parse the arithmetic result                 # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "9"                             # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can subtract -- @1.3       # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:21
    Given that I search DuckDuckGo with term: "464 - 341" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 1.785s
    When I parse the arithmetic result                    # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "123"                              # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can multiply -- @1.1   # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:29
    Given that I search DuckDuckGo with term: "3 * 2" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 1.833s
    When I parse the arithmetic result                # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "6"                            # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can multiply -- @1.2   # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:30
    Given that I search DuckDuckGo with term: "8 * 3" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 1.701s
    When I parse the arithmetic result                # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "24"                           # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can multiply -- @1.3    # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:31
    Given that I search DuckDuckGo with term: "8 * 13" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 1.741s
    When I parse the arithmetic result                 # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "104"                           # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can divide -- @1.1      # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:39
    Given that I search DuckDuckGo with term: "12 / 4" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 1.819s
    When I parse the arithmetic result                 # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "3"                             # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can divide -- @1.2     # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:40
    Given that I search DuckDuckGo with term: "8 / 4" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 1.789s
    When I parse the arithmetic result                # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "2"                            # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can divide -- @1.3        # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:41
    Given that I search DuckDuckGo with term: "104 / 13" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 1.923s
    When I parse the arithmetic result                   # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "8"                               # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

1 feature passed, 0 failed, 0 skipped
12 scenarios passed, 0 failed, 0 skipped
36 steps passed, 0 failed, 0 skipped, 0 undefined
Took 0m22.095s

We have reached milestone 3 in the repository

Our test suite is implemented following the tenets we have defined in the first part of the article:

  1. the steps use an adapter to communicate with the system under test.
  2. the adapter isolates the steps from any unnecessary knowledge and/or technology.
  3. the adapter uses indirection via pages and page components in order to interact with the underlying technology.

The SDK

At this point we have a working test suite and we have built an adapter which exposes the functionality of the system we are testing. Interestingly, This adapter sounds a lot like an official interface to the system. It does contain some ‘private’ and ‘interface-centric’ stuff.

We might want to try to standardize this interface and make sure that it is properly documented. We also want to provide a level of indirection for our future API adapter.

We are going to create an SDK for that purpose. An important idea that we have to keep in mind is that the creation of this SDK is driven by a necessity to use its methods in tests we are writing. No method can appear here unless proven necessary by one of the ‘given’ or ‘then’ steps in our test suite.

We are going to put the sdk at the same level as the adapters. Our new file structure:

duckduckgo_harness/
    adapters/
    functionality/
        duckduckgo_can_do_arithmetic/
            features/
                duckduckgo_can_do_arithmetic.feature
            steps/
                steps.py
        environment.py
    sdk/
        __init__.py
        duck_duck_go.py
    __init__.py
    requirements.txt

and the content of the new file:

import zope.interface
from zope.interface.verify import verifyclass


class iduckduckgoport(zope.interface.interface):
    def get_search_result(search_term):
        """return search results"""

    def parse_arithmetic_result(search_result):
        """extracts the arithmetic result from the search results"""


class duckduckgosdk(object):

    def __init__(self, adapter):
        verifyclass(iduckduckgoport, adapter.__class__)
        self.adapter = adapter

    def get_search_result(self, search_term):
        return self.adapter.get_search_result(search_term)

As a rule of thumb we are only elevating to the sdk the adapter method used in the ‘given’ step. The reason for this is that this method would be most likely to be used in other tests. Multiple reuse suggests a public interface.

It is also worth stopping here to explain some of the implementation choices. In order to be able to use the ports and adapters pattern we must provide a definition of the communication (interface) so that adapters know what to implement. The important bits here are the definition of the communication and the possibility of switching the adapters (more on that in the next section).

We have chosen to use zope.interfaces to define and enforce the interface.

import zope.interface
from zope.interface.verify import verifyClass


class IDuckDuckGoPort(zope.interface.Interface):

    """Define the shape of communication for the adapters.

    All the adapters must implement this interface.
    """

    def get_search_result(search_term):
        """Return search results"""

    def parse_arithmetic_result(search_result):
        """Extracts the arithmetic result from the search results"""


class DuckDuckGoSDK(object):

    def __init__(self, adapter):
        verifyClass(IDuckDuckGoPort, adapter.__class__)
        self.adapter = adapter

    def get_search_result(self, search_term):
        return self.adapter.get_search_result(search_term)

    def parse_arithmetic_result(self, search_result):
        return self.adapter.parse_arithmetic_result(search_result)

We also need to refactor the steps, the adapter and the environment build script.

from behave import given, when, then


@given(u'that I search DuckDuckGo with term: "{search_term}"')
def search_duckduckgo(context, search_term):
    context.data['raw_result'] = context.sdk.get_search_result(search_term)


@when(u'I parse the arithmetic result')
def fetch_result(context):
    context.data['result'] = context.sdk.parse_arithmetic_result(
        context.data['raw_result']
    )


@then(u'the result is "{expected_result}"')
def check_result(context, expected_result):
    assert int(expected_result) == context.data['result']

We have reached milestone 4 in the repository. We have introduced another abstraction layer so that we can start switching between adapters. Time to write a ‘fast’ adapter by using a direct access to a RESTful API instead of all that Selenium stuff.

The Adapter Switch

Once we have the sdk in place it is time to demonstrate the flexibility of the system. We are going to provide another adapter following our sdk contract. It turns out that there is a duckduckgo module that allows communication with the duckduckgo api. We will install it and try using it in a separate adapter. Hopefuly, using the api instead of a browser and the html interface will speed the things up.

Our new structure:

duckduckgo_harness/
    adapters/
        api/
            __init__.py
            rest_adapter.py
        selenium/
            page_components/
            pages/
            __init__.py
            selenium_adapter.py
    functionality/
        duckduckgo_can_do_arithmetic/
            features/
                duckduckgo_can_do_arithmetic.feature
            steps/
                steps.py
        environment.py
    sdk/
        __init__.py
        duck_duck_go_arithmetic.py
    __init__.py
    requirements.txt

The new adapter implementation:

from HTMLParser import HTMLParser
import zope.interface
import duckduckgo

from duckduckgo_harness.sdk import IDuckDuckGoPort


class DuckDuckGoAPI(object):
    zope.interface.implements(IDuckDuckGoPort)

    def get_search_result(self, search_term):
        return duckduckgo.query(search_term)

    def parse_arithmetic_result(self, search_result):
        return int(strip_tags(search_result.answer.text))


class MLStripper(HTMLParser):
    def __init__(self):
        self.reset()
        self.fed = []

    def handle_data(self, d):
        self.fed.append(d)

    def get_data(self):
        return ''.join(self.fed)


def strip_tags(html):
    s = MLStripper()
    s.feed(html)
    return s.get_data()

The only change required in the previous code is the amendment of the context namespace creation to use the new adapter rather than the old one.

from selenium import webdriver

from duckduckgo_harness.adapters import SeleniumAdapter
from duckduckgo_harness.adapters import DuckDuckGoAPI
from duckduckgo_harness.sdk import DuckDuckGoSDK


class Placeholder(object):
    pass


def before_all(context):
    profile = webdriver.FirefoxProfile()
    context.browser = webdriver.Firefox(firefox_profile=profile)
    selenium_adapter = SeleniumAdapter(context.browser)
    rest_adapter = DuckDuckGoAPI()
    context.sdk = DuckDuckGoSDK(rest_adapter)
    context.data = {}


def after_all(context):
    if getattr(context, 'browser', None):
        context.browser.close()

We run the test suite again:

(xsharness) vlad@hex ~/projects/priv/xs_harness $ behave duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/ --no-capture
Feature: DuckDuckGo can do arithmetic # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:1

  Scenario Outline: DuckDuckGo can add -- @1.1        # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:9
    Given that I search DuckDuckGo with term: "2 + 2" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 0.097s
    When I parse the arithmetic result                # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "4"                            # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can add -- @1.2        # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:10
    Given that I search DuckDuckGo with term: "3 + 8" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 0.098s
    When I parse the arithmetic result                # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "11"                           # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can add -- @1.3            # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:11
    Given that I search DuckDuckGo with term: "123 + 341" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 0.091s
    When I parse the arithmetic result                    # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "464"                              # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can subtract -- @1.1   # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:19
    Given that I search DuckDuckGo with term: "7 - 2" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 0.103s
    When I parse the arithmetic result                # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "5"                            # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can subtract -- @1.2    # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:20
    Given that I search DuckDuckGo with term: "12 - 3" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 0.095s
    When I parse the arithmetic result                 # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "9"                             # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can subtract -- @1.3       # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:21
    Given that I search DuckDuckGo with term: "464 - 341" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 0.094s
    When I parse the arithmetic result                    # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "123"                              # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can multiply -- @1.1   # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:29
    Given that I search DuckDuckGo with term: "3 * 2" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 0.098s
    When I parse the arithmetic result                # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "6"                            # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can multiply -- @1.2   # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:30
    Given that I search DuckDuckGo with term: "8 * 3" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 0.092s
    When I parse the arithmetic result                # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "24"                           # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can multiply -- @1.3    # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:31
    Given that I search DuckDuckGo with term: "8 * 13" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 0.094s
    When I parse the arithmetic result                 # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "104"                           # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can divide -- @1.1      # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:39
    Given that I search DuckDuckGo with term: "12 / 4" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 0.093s
    When I parse the arithmetic result                 # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "3"                             # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can divide -- @1.2     # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:40
    Given that I search DuckDuckGo with term: "8 / 4" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 0.094s
    When I parse the arithmetic result                # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "2"                            # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

  Scenario Outline: DuckDuckGo can divide -- @1.3        # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/features/duckduckgo_can_do_arithmetic.feature:41
    Given that I search DuckDuckGo with term: "104 / 13" # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:4 0.111s
    When I parse the arithmetic result                   # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:9 0.000s
    Then the result is "8"                               # duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/steps/steps.py:16 0.000s

1 feature passed, 0 failed, 0 skipped
12 scenarios passed, 0 failed, 0 skipped
36 steps passed, 0 failed, 0 skipped, 0 undefined
Took 0m1.164s

The result is that the test executes now in under a second as opposed to previous 20-30 seconds. We have managed to leverage a more effective way of accessing the system once we have discovered/created it.

We have reached milestone 5 in the repository.

It would be nice to be able to choose between the adapters. Let’s get the SDK to do that for us. Also, we will get rid of the browser if we are running the rest adapter.

It is possible to test both adapters by providing a correct environment variable (adapter=www or adapter=api with api being the default one):

$ ADAPTER=api behave duckduckgo_harness/functionality/duckduckgo_can_do_arithmetic/

The conclusion

This example may appear a bit trivial but it has allowed us to demonstrate all the main tenets of this testing approach.

We have driven our testing approach from an existing untested www/html interface. By sticking to a few simple rules we ended up with a system that provided us with:

  • domain specific language
  • test coverage allowing us to start refactoring the system under test
  • an interim sdk allowing us to interface with our system programmatically
  • two adapters exposing different access points to our system under tes