BeakerLib Newsletter #02: Phases

BeakerLib is built on the concept of phases: separated test segments with different purposes. A failure in setup has different implications than failure in test itself or cleanup.

Concept

A phase is a sequence of commands with a single-high level purpose. Here are some  examples of phase purposes:

  • Configuration and launch of a daemon for testing
  • Test case compilation
  • Test case execution and result analysis
  • Cleaning up all temporary files created by a test

A phase can hold any number of assertions: declared conditions that the test writer believes should hold if everything goes well. If all assertions in a phase hold during execution, the phase is successful. If any single assertion does not hold, the phase fails.

Types of phases

There are two types of phases, based on the implication of phase failure:

  • FAIL phase: If this phase fails, it means the test just reproduced a bug. The test works as expected, and the tested program does not.
  • WARN phase: If this phase fails, a simple warning is emitted. This is used whenever anything else than the tested behavior fails: setup, cleanup or any other auxiliary code.

The type of the phase is stated at the beginning of the phase. The phase is started with one of the following commands:

  • rlPhaseStart type [name]
  • rlPhaseStartSetup [name] (shortcut for rlPhaseStart WARN [name])
  • rlPhaseStartTest [name] (shortcut for rlPhaseStart FAIL [name])
  • rlPhaseStartCleanup [name] (shortcut for rlPhaseStart WARN [name])

The name parameter is an optional string summarizing the phase purpose. If it is not provided, one will be generated.

Phase end

The phase is ended with the following command:

  • rlPhaseEnd

Whenever a phase is ended, its result is evaluated and reported. If any of the asserts failed, either FAIL or WARN is emitted, depending on a phase type.

BeakerLib detects if it is running in an automated Beaker job, and if so, it reports the results to Beaker with a summary log of progress done so far. If the test is not run in Beaker automated job, just a textual information is emitted.

There are no implicit phase ends, if you fail to call the rlPhaseEnd command, no result will be reported for a phase.

Phase state

BeakerLib provides a function to determine the state of the current phase:

  • rlGetPhaseState

This function returns a count of asserts in current phase that failed so far.

Further notes on phases

The phases have limited nesting capability: the test won’t fail and the journal XML will capture the nested structure, but we lack reporting capabilities which can express the nested nature of the phases (Beaker treats a job as a sequence of reported results). Therefore, nesting of phases is discouraged unless you really know what you are doing and you can interpret the results.

Assert and logs performed without a started phase are reported (an artificial phase is created for them), but we discourage from doing this.

The result reported to Beaker is based on the phase type, and cannot be changed – there is no way how to emit a WARN result from FAIL phase, or vice versa.

Next time: Journal

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.