Testing

Overview

Different kinds of tests play an essential role in maintaining a high quality code base both for eXist-db itself, and for applications that interact with eXist-db.

While the terminology varies between different sources dealing with software testing, the following categories are widely acknowledged to apply irrespective of terminological differences.

1. Validation: XML comes with different means of validating and thus testing your data structures. When implementing a test suite, you should first consider how your code can leverage these native features via strong typing and schema-based validation.

2. Unit testing: These tests typically operate at the level of individual functions. eXist-db has its own unit testing framework for XQuery named XQSuite. It is prominently used in our bug reports and source-code repo. Most applications will need to run multiple unit test tools to test, e.g. Java, JavaScript, and XQuery code.

3. Integration testing: Here we look at your program as a whole, similar to how a user would interact with it, by simulating user input and running your program not in isolation but in concert with different applications.

4. (WIP) Performance and Stress testing: Performance testing ensures that your code keeps running under heavy load, or with very little resources. Stress tests go one step further by purposefully crashing your application to ensure that it recovers gracefully. Practically speaking, the difference between the two is fluid.

When running these different tests in concert, we usually speak of end-to-end testing (e2e). There will never be a one-size-fits-all solution to take care of all your testing needs. Writing good test suites takes time and planning. You can find out more about available options and their use cases in the articles linked above. The remainder of this article will discuss some general considerations for how to design your test suite, without referencing a specific implementation. The listings use pseudo-code to focus on relevant lines. Pseudo-code is not intended to be executable, so it won't run if you copy-paste it into, e.g., eXide. For working code, please consult the articles links throughout.

Picking a Type

Generally speaking, you should follow the sequence provided above. Whatever can be tested as part of your validation tests, is best tested there. When validation testing isn't a good fit, go for unit tests, followed by integration and lastly performance tests. The reasons for this are twofold: on the one hand, validation tests tend to be faster than unit tests, which in turn are slower then integration tests and performance tests. On the other hand, each type tends to be easier to configure, with a higher chance of providing you with the information that you need when a test fails.

Selection Example

To illustrate this, let's take a theoretical example: imagine your code includes an online form where users submit a date that needs to be stored in the database. For the purpose of this example it doesn't matter if this is text-input form, a calendar selector, or other form. Provided that you are using eXist-db, we assume that the date will be stored in some kind of XML file.

a simple form:

that will create something like this:

<date>
  $USER_INPUT
</date>

By using a schema for validating this XML file, you can ensure the correct datatype xs:date of the provided input, and prevent users from storing something in the database that is not a date.

declare function local:check-input($input as item()*) as xs:date {
 $input cast as xs:date
};

To check that your local:check-input function is working as expected, you can define a few unit tests. Typically, such tests would include corner cases like BCE dates, leap years, or dates in foreign date formats, as well as a basic test date that should simply store in the db.

declare
    %test:arg("input", '11.11.2010') %test:assertEquals('11.11.2010')
    %test:arg("input", '5') %test:assertError()
    function local:check-input($input as item()*) as xs:date {
 $input cast as xs:date
};

For in-depth examples of unit testing in XQuery please see the examples in the XQSuite article

Integration testing is most effective when it can rely on existing unit tests and focus on how a user typically interacts with the UI components. Does the browser display the input form correctly at different resolutions, can the form be selected with a mouse, or on touch screens, etc.?

Lastly, performance and stress tests would simply assume all of the above to be in place. They would check for problems that might occur, e.g., when large numbers of users select the same or invalid dates, or access the same form simultaneously.

This quick example should give you an idea on how to think about testing your code. Each type of test has its own reason for being and requires your attention. A good performance test cannot substitute unit or integration tests. If you feel that your tests are not working well, because they frequently fail to show problems in your code, and when they do they don't make identifying the source of the problem easier, chances are that you are trying to test something with one type of test that would be better suited for another type.

Test Coverage

Ideally, we would always achieve full test coverage for our code. So that every meaningful unit of code has a corresponding set of tests. While other programming languages offer automated means of analysing your code to identify areas that aren't tested, such tools are unfortunately not very common for XQuery. The task of determining how much testing is necessary for you to ship with confidence rests on each developer. For apps published under the eXist-db namespace, we have outlined a set of minimum requirements (more is obviously desirable) which are featured in the section on integration testing.