XQSuite - Annotation-based Test Framework for XQuery

(3Q22)


XQSuite is a test framework for XQuery modules based on XQuery function annotations.

Introduction

The XQsuite test framework allows tests to be added to functions using function annotations. Since annotations are ignored during normal execution, this does not interfere. The framework has the following properties:

  • Tests can be defined within the actual application code. A function can be tested where implemented.

  • Tests are written as ordinary XQuery functions.

  • More complex integration tests can be combined into separate modules.

  • To run a suite of tests, all you need is a small, main XQuery which imports the modules to be tested.

  • XQSuite is itself implemented in XQuery and can therefore be extended.

XQSuite has two main components:

  1. A number of XQuery function annotations to be used within the code module under test

  2. A test runner, which takes a sequence of function items and interprets the test annotations

To use test annotations in an XQuery module, the XQsuite namespace must be declared:

declare namespace test="http://exist-db.org/xquery/xqsuite";

A function will be processed by the test runner if it has at least one test:assert* annotation (assertEquals, assertEmpty, etc.). Functions without such an annotation will be ignored. For example:

declare %test:assertEquals("Hello world") function local:hello() {
    "Hello world"
};

When the test runner encounters this function, it will evaluate it and compare its return value to the assertion.

eXist-db previously used a descriptive XML format to define XQuery tests. XQSuite replaces this, though the old format and test runner is still supported.

Supported Test Annotations

The following annotations are supported:

%test:arg($varName, $value1, $value2, ...)

Set the function parameter with variable name $varName to the sequence constructed by converting the remaining annotation parameters to the sequenceType as declared by the function parameter. See Passing parameters to tests.

%test:args($arg1, $arg2, ...)

Run the function, using the supplied literal arguments. There must be one annotation argument for each parameter the function takes. See Passing parameters to tests.

%test:assertEmpty, %test:assertExists

Expects the function to return an empty sequence (assertEmpty) or a non-empty sequence (assertExists).

%test:assertTrue, %test:assertFalse

Checks if the effective boolean value of the returned result is true or false.

%test:assertEquals($value1, $value2, ...)

Tests if the return value equals the specified argument.

  • If the function returns more than one item, each item is compared to the given annotation values in turn. The number of returned items has to correspond to the number of annotation values.

  • If the function returns an atomic type, the assertion argument is cast into the same type and the two are compared using the eq operator.

  • If the sequence returned by the function contains one or more XML elements, they will be normalized (ignorable whitespace is stripped). The assertion argument is then parsed into XML and the two node trees are compared using the deep-equals function.

%test:assertEqualsPermutation($value1, $value2, ...)

Iterate over the items in the return sequence, of e.g., a for loop to check if the sequence contains the correct items in the correct order.

%test:assertError($err)

Evaluating the tested function should result in a dynamic error. If a value is given for $err, it should either be: 1) the error code of the dynamic error, or 2) a regular expression that matches the error description.

%test:assertXPath($path-as-string)

Tests if the return value of the tested function using the given XPath expression.

The annotation value is executed as an XPath expression. The assert passes if the XPath expression returns a non-empty sequence or a single atomic item whose effective boolean value is true. Within the XPath expression, the variable $result contains a reference to the result sequence returned by the tested function.

%test:assumeIntenetAccess($uri)

Marks that a test assumes that Internet access is available. If the $uri can be reached over HTTP via a HEAD request, then the test will be executed as part of the test suite. If the connection to the $uri cannot be made, then the test has the same status as a test marked with %test:pending. This can be useful when you have a test which relies on an external resource.

%test:name

Provide a more descriptive name for your test than the XQuery function name, which is the default.

%test:pending($reason)

Marks a test as pending, which means that it will not be executed as part of the test suite. This can be useful when you want to write a test for a problem or new feature but you have not had time to fix the problem or implement the feature yet. The number of pending tests is shown in the test suite report as a reminder.

%test:stats

Collects statistics using system:trace() and prepends them as element <stats:calls xmlns:stats="http://exist-db.org/xquery/profiling"> to the returned sequence of the test function.

%test:setUp, %test:tearDown

Special functions which will be called once before/after any other tests in the same XQuery module are run. Use these functions to upload data, create indexes, users or prepare anything else needed for the tests to run.

%test:user($username, $password)

Executes the test function as a specific user.

Passing parameters to tests

To test a function which takes parameters, use the %test:arg or %test:args annotation in combination with one or more %test:assert* assertions. For example:

declare
    %test:arg("n", 1) %test:assertEquals(1)
    %test:arg("n", 5) %test:assertEquals(120)
function m:factorial($n as xs:int) as xs:int {
    if ($n = 1) then
        1
    else
        $n * m:factorial($n - 1)
};

  • The %test:arg annotation is used to set the parameters for the next test run triggered by the assertion.

  • The tested function will be called once for every sequence of %test:arg annotations followed by one or more %test:assert*. The order of the annotations is therefore important!

  • The first argument to %test:arg denotes the name of the function parameter variable to set, the remaining arguments are used to create the sequence of values passed. So to test a function which takes a sequence with more than one item for a parameter, just append additional values to %test:arg. The test runner will convert these values into a sequence.

  • There must be as many %test:arg annotations for every test run as the function takes parameters.

  • The result returned by the function call is passed to each assertion in turn, which may either pass or fail. In the example above, we assert that the function returns 1 if the input parameter is 1, and 120 if the input is 5.

  • If all function parameters expect exactly one value, you can also use %test:args. Instead of specifying one %test:arg annotation for every parameter, use a single %test:args annotation to define all parameter values at once. %test:args takes a list of values. Each value is mapped to exactly one function parameter. Example:

    declare
    %test:args("Hello", "world")
    %test:assertEquals("Hello world")
function local:hello($greet as xs:string, $user as xs:string) {
    $greet || " " || $user
};

Automatic Type Conversion

XQuery annotation parameters need to be literal values, so only strings and numbers are allowed. XQSuite therefore applies type conversion to every argument and to values used in assertions.

For example, the following function expects a parameter of type xs:date:

declare
    %test:args("2012-06-26")
    %test:assertEquals("26.6.2012")
    %test:args("2012-06-01")
    %test:assertEquals("1.6.2012")
function fd:simple-date($date as xs:date) as xs:string {
    format-date($date, "[D].[M].[Y]")
};

The string value passed to the %test:args annotation is automatically converted to the xs:date type declared for the function parameter. The same applies to the assertion values.

Type conversion works for atomic types as well as XML nodes.

If automatic type conversion fails the test case will fail as well.

Custom Assertions

Annotation based assertions are limited in so that you can

  • a) can only use (sequences of) string and numeric literals

  • b) the value must be static, meaning you cannot use variables in annotations

  • c) there is no way to add assertions that are specific to your domain

This is why the function test:fail was introduced allowing you to create custom assertion functions.

xquery version "3.1";

module namespace t="http://exist-db.org/xquery/test/xqsuite";

declare namespace test="http://exist-db.org/xquery/xqsuite";

declare %test:assertTrue function t:test-type() as xs:boolean? {
    t:assert-is-foo(())
};

(: will only return true  :)
declare function t:assert-is-foo($actual as item()*) as xs:boolean? {
    $actual eq "foo" or
        test:fail("Is not foo", $actual, "foo")
};

These functions can also be in a module to re-use them.

xquery version "3.1";

module namespace assert="http://line-o.de/xq/assert";

import module namespace test="http://exist-db.org/xquery/xqsuite"
    at "resource:org/exist/xquery/lib/xqsuite/xqsuite.xql";

declare function assert:map ($e as map(*), $a as item()*) as xs:boolean? {
    if (exists($a) and count($a) eq 1  and $a instance of map(*))
    then (
        for-each(map:keys($e), function ($key as xs:anyAtomicType) {
            if (not(map:contains($a, $key)))
            then test:fail("Key " || $key || " is missing", $e, $a, "map-assertion")
            else if ($e($key) ne $a($key))
            then test:fail("Value mismatch for key '" || $key || "'", $e, $a, "map-assertion")
            else ()
        })
        ,
        true()
    )
    else test:fail("Type mismatch", $e, $a, "map-assertion")
};

Here is an example using the above assertion library

xquery version "3.1";

module namespace assert="http://line-o.de/xq/assert";

import module namespace test="http://exist-db.org/xquery/xqsuite"
    at "resource:org/exist/xquery/lib/xqsuite/xqsuite.xql";

declare function assert:map ($e as map(*), $a as item()*) as xs:boolean? {
    if (exists($a) and count($a) eq 1  and $a instance of map(*))
    then (
        for-each(map:keys($e), function ($key as xs:anyAtomicType) {
            if (not(map:contains($a, $key)))
            then test:fail("Key " || $key || " is missing", $e, $a, "map-assertion")
            else if ($e($key) ne $a($key))
            then test:fail("Value mismatch for key '" || $key || "'", $e, $a, "map-assertion")
            else ()
        })
        ,
        true()
    )
    else test:fail("Type mismatch", $e, $a, "map-assertion")
};

The output will look like this

?listings/custom-assertion-output.xml?

Fallback for backwards compatibility

If you want to run custom assertions on versions of eXist-db that do not have test:fail there is a fallback that you can use. This could be the tests for a library package, for instance

xquery version "3.1";

module namespace assert="http://line-o.de/xq/assert";

import module namespace test="http://exist-db.org/xquery/xqsuite"
    at "resource:org/exist/xquery/lib/xqsuite/xqsuite.xql";

declare variable $assert:fail :=
    let $builtin := function-lookup(xs:QName("test:fail"), 4)
    return
        if (exists($builtin))
        then
            $builtin
        else (: fallback :)
            function ($message, $expected, $actual, $type) {
                error(xs:QName("assert:" || $type), $message, map {
                    "expected": $expected,
                    "actual": $actual
                })
            }

declare function assert:map ($e as map(*), $a as item()*) as xs:boolean? {
    if (exists($a) and count($a) eq 1  and $a instance of map(*))
    then true()
    else $assert:fail("Type mismatch", $e, $a, "map-assertion")
};

Running a Test Suite

To run a suite of tests, create a main XQuery which:

  1. Imports the xqsuite.xql test framework

  2. Calls test:suite with the sequence of function items to test

Here is an example of a module for test, stored as math.xql:

xquery version "3.0";

module namespace m="http://foo.org/xquery/math";

declare namespace test="http://exist-db.org/xquery/xqsuite";

declare
    %test:arg("n", 1) %test:assertEquals(1)
    %test:arg("n", 5) %test:assertEquals(120)
function m:factorial($n as xs:int) as xs:int {
    if ($n = 1) then
        1
    else
        $n * m:factorial($n - 1)
};

To run the tests in this module, create a main XQuery, for example called suite.xql, and store it in the same collection:

xquery version "3.0";

import module namespace test="http://exist-db.org/xquery/xqsuite" 
at "resource:org/exist/xquery/lib/xqsuite/xqsuite.xql";

test:suite(
    inspect:module-functions(xs:anyURI("math.xql"))
)

Executing the main XQuery, for instance within eXide, will return the test results as an XML fragment (see Test Suite Output).

In the main XQuery, the function: inspect:module-functions returns a function item for every public function defined in the module loaded from the location URI argument.

The location URI is resolved in the same way as in a normal module import: a relative path is interpreted relative to the location of the main XQuery.

Test Suite Output

A call to test:suite returns the results of the tests as an XML fragment, using the schema defined by the xUnit test tool. The xUnit format is supported by many systems, for instance Jenkins.

For every XQuery module tested, test:suite creates a single <testsuite> element. The result of each test run is output as a <testcase> element. It will be empty if the test passed. If the test failed (meaning that the test did not meet its assertions, or that it raised an fn:error), there will be a <failure> element containing the expected result of the function, and an <output> element with the actual result. In the case that the test resulted in an unexpected error that is not covered by fn:error, there will be an <error> element with the error type and message. The <testsuite> element will also contain the timestamp when the tests began, the duration of the tests, and a count of the number of tests, failed tests, pending tests, and tests that raised unexpected errors not covered by fn:error. Here is a sample showing a failed test:

<testsuites>
  <testsuite package="http://exist-db.org/xquery/test/bang" timestamp="2012-10-16T10:30:12.966+02:00" failures="1" pending="0" errors="0" tests="19" time="PT0.046S">
    <testcase name="constructor" class="bang:constructor"/>
    <testcase name="functions1" class="bang:functions1"/>
    <testcase name="functions2" class="bang:functions2"/>
    <testcase name="functions3" class="bang:functions3">
      <failure message="assertEquals failed." type="failure-error-code-1">
        RED BLACK GREEN
      </failure>
      <output>
        RED BLUE GREEN
      </output>
    </testcase>
    <testcase name="functions4" class="bang:functions4"/>
    <testcase name="nodepath" class="bang:nodepath"/>
    <testcase name="nodepath-reverse" class="bang:nodepath-reverse"/>
    <testcase name="nodes1" class="bang:nodes1"/>
    <testcase name="nodes2" class="bang:nodes2"/>
    <testcase name="numbers1" class="bang:numbers1"/>
    <testcase name="position1" class="bang:position1"/>
    <testcase name="position2" class="bang:position2"/>
    <testcase name="position3" class="bang:position3"/>
    <testcase name="position4" class="bang:position4"/>
    <testcase name="position5" class="bang:position5"/>
    <testcase name="precedence1" class="bang:precedence1"/>
    <testcase name="precedence2" class="bang:precedence2"/>
    <testcase name="precedence3" class="bang:precedence3"/>
    <testcase name="sequence" class="bang:sequence"/>
  </testsuite>
</testsuites>