Spice your tests with a hint of randomness (WIP)

Why should Continuous Integration-tests have randomness?

Randomness will allow you to have more test coverage after you have executed test more than once. Test coverage meaning coverage over input space (allowed input values).

Randomness gives your tests possibility to catch annoying input data (if input data is randomized) specific bugs that are unlikely to be covered without randomized data.

There is no point executing same unit test against same unchanged code -- and this is what is usually happening when running all your tests.

And then the math:

• lets assume that there is a bug that shows up in 10% of the input space (for example when the tested functions integer argument is dividable by 10.. TODO: think a better example)
  
• this means that a test executions possibility to not detect this bug is 90%
• if the test uses randomized inputs and it can choose any input then the possibility that the test misses the bug in N executions is 0.9^N (the test will find the bug in 100 executions with a possibility grater than 99.99%)
• if the test hasn't randomized input it doesn't matter how many times it will be executed - it will either find the bug first time or not find it at all (the test will find the bug in 100 -- or million executions wit a 10% possibility)
  

Problems with randomness

• Repeatability


• Readability


• More work than with a simple nonrandom data

How to handle these problems

• Repeatability - log the seed of the random number generator


• Readability


• More work than with a simple nonrandom data - one randomized test can cover in multiple executions many nonrandom test cases, data generators can be used in many tests if they are organized nicely

Implementing asynchronous Robot Framework keywords

Before tests can be run I have to start all the necessary systems (processes, servers, stuff and things). Let's imagine that there are multiple systems that need to be started. I would like to start them all at the same time and wait until they are all ready to rock 'n' roll before starting my tests (assume that it would take a lot longer if I started all the systems in sequence).

So how to do this with Robot? Basic idea is to have re-usable keywords for starting each system and a keyword for waiting until the systems are ready so that testing can begin (and we don't have to use ugly and unreliable sleeps). Doing this on Robot keyword level makes it possible to have different combinations of systems in different tests.

*** Settings ***
Documentation   Example of using parallel things
Suite Setup     StartSystems
Library         SystemStarterLibrary.py

*** Test Cases ***
... Here should be my tests

*** Keywords ***
StartSystems
   ${SYSTEM1_STARTED}=  Asynchronously Start System 1
   ${SYSTEM2_STARTED}=  Asynchronously Start System 2
   Wait until  ${SYSTEM1_STARTED}  ${SYSTEM2_STARTED}

The way I'm going to implement this is by using python decorator that executes the function that it decorates in a separate thread. The decorated function will return the thread object so that it can be used to implement the waiting functionality.

I'm using this little code for the decorator.

After I've imported that to my SystemStarterLibrary.py I can implement system starter functions as normal functions.

@run_async
def asynchronously_start_system_1():
  .. do stuff to start system 1

@run_async
def asynchronously_start_system_2():
  .. do stuff to start system 2

Now all I need to do is to implement Wait until.

def wait_until(*stuff):
  for something in stuff :
     something.join()

This is kind of OK but it will wait forever if starting of some system will take forever. So it is better to have some timeout that will tricker setup failure after the timeout.

*** Keywords ***
StartSystems
    [Timeout]  5 minutes
    ${SYSTEM1_STARTED}=  Asynch Start System 1
    ${SYSTEM2_STARTED}=  Asynch Start System 2
    Wait until  ${SYSTEM1_STARTED}  ${SYSTEM2_STARTED}

And that should do it.

Hello World Robot Framework library

This is the way I did my first Robot Framework keyword library. It should show the basic steps to add your own python keywords.

Let's do it in a test-driven way!

Failing test case

Add a file called HelloWorld.txt. This is our robot test suite file.

Add following text to the file:

*** Test Cases ***
HelloWorld
   Hello World

After this run pybot HelloWorld.txt - this will execute your Hello World test case.
Output should be something like:

==============================================================================
HelloWorld                                                                   
==============================================================================
HelloWorld                                                            | FAIL |
No keyword with name 'Hello World' found.
------------------------------------------------------------------------------
HelloWorld                                                            | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================

Now we have a failing test case! So we can begin to implement our super cool Hello World keyword.

Keyword file

Add a file called HelloWorld.py to the same directory as our HelloWorld.txt test suite.

Add following text to the file:

def hello_world():
   print "HELLO WORLD!"

Now we have implemented our fine keyword that prints "HELLO WORLD!". Although our test still fails..

Passing test case

We have to import our super cool library to our test suite. Add following lines to the HelloWorld.txt (before test cases):

*** Settings ***
Library         HelloWorld.py

After this run pybot HelloWorld.txt - and watch it PASS:

==============================================================================
HelloWorld                                                                    
==============================================================================
HelloWorld.HelloWorld                                                         
==============================================================================
HelloWorld                                                            | PASS |
------------------------------------------------------------------------------
HelloWorld.HelloWorld                                                 | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
HelloWorld                                                            | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================

Thats it.

Installing Robot Framework on Ubuntu

Lately I've been learning to use the robot framework. These are my notes on how to install it.

Installing pybot - normal robot thing

First install python if it's not already installed.


sudo apt-get install python


Then install easy_install and robotframework.


sudo apt-get install python-setuptools
sudo easy_install robotframework


After this you should have pybot (normal robot thing) installed.


pybot --version
== Should output something like ==>
Robot Framework 2.5.4 (Python 2.6.5 on linux2)


Installing jybot - jython version of robot

I'll assume you have done all the things to install pybot so far (it also installed jybot and all you have to do now is to install correct version of jython).

First a word of warning: Ubuntu is still (you should check is this still valid point if your reading this in the future) using old version of Jython that doesn't work with current jybot.

So we have to first download Jython 2.5 (or later) from jython webpage. Follow jythons installation instructions.

Add JYTHON_HOME to PATH so that you can use it.

After this jybot should work.


jybot --version
== Should output something like ==>
Robot Framework 2.5.4 (Jython 2.5.1 on java1.6.0_20)