eXist-db v3.0 available

# Release Notes

## v3.0 - February 9, 2017 It is our great pleasure to announce the public stable release of eXist-db v3.0.

We encourage everyone to upgrade to this new stable release. eXist-db v3.0 is the culmination of almost 1,500 changes made in the last two years bringing numerous new features and improvements. During this time eXist-db went through extensive stress testing demonstrating substantial performance gains confirmed in production settings.

### New Features - Support for XQuery v3.1, including the array and map data types, serialisation and JSON parsing - Support for Braced URI Literals from XQuery v3.0 - Facility to boost attributes in the Lucene full text index - eXist-db version detection for EXPath packages. Packages should explicitly specify which versions of eXist-db they are compatible with; eXist-db v2.2 is assumed by default. - Prototype support for Portable EXPath XQuery Extension Functions written in Haxe - New service wrapper, Yet another java service wrapper (yajsw) - New Jetty version (HTTP/2 etcetera)

### Improved Performance

  • Sequence type checking on recursive function parameters has been drastically sped up
  • Lucene full-text and range indexes have been switched to "near realtime" behaviour. This improves query performance on frequently updated documents.
  • Performance boost for new range index.
  • Improved optimisation of wildcard steps in path expressions, e.g. prefix: and :name
  • Better performance for util:eval Optimisation of fn:fold-left and fn:fold-right

### Mission Critical Bug Fixes

There have been numerous bug fixes and enhancements since eXist-db 2.2, the most critical are:

  • Solved a potential deadlock which manifested when storing XQuery files into the database under certain conditions
  • Fixed a memory leak when storing query results into the HTTP session; Web applications making use of the HTTP session should now consume less memory and scale further
  • Fixed an occasional deadlock when shutting down the database
  • Fixes to match highlighting with the Lucene full text index
  • Lucene range index now correctly handles != comparisons
  • Substantial improvements in whitespace handling in mixed content
  • Windows compatibility in service wrapper
  • Fixes in RESTXQ
  • Security fixes for privilege escalation

### Clean up and Refactoring

  • Rewritten HTML5 Serializer
  • Rewritten XML:DB and XML-RPC APIs
  • Updated to the latest version of RESTXQ
  • Improved Java Admin Client document viewing and editing
  • Clean up of eXist-db's Test suite
  • Extensive internal refactoring to exploit new Java 8 features
    • lambda expressions where possible
    • use of Optional
    • Java 7 auto closables
    • NIO.2
  • Upgrades of 3rd party libraries:
    • Saxon HE 9.6.0-7
    • Jetty 9.3.9.v2016051
    • Lucene 4.10.4
    • Quartz 2.2.3
    • Log4j 2.7
    • FOP 2.1
    • Tika 1.12 (Content Extraction module)
    • betterForm 5.1-SNAPSHOT-20160615
    • and many more
  • Removed legacy code:
    • SOAP API and SOAP Server
    • XACML Security feature
    • eXist's own Full-Text Index (replaced by eXist Lucene Full-Text Index)
    • Removed the Versioning extension; Now available as a separate EXPath Package

### Backwards Compatibility issues

  • eXist-db v3.0 is not binary compatible with previous versions of eXist-db; the on-disk database file format has been updated, users should perform a full backup and restore to migrate their data.
  • eXist.db v3.0 and subsequent versions now require Java 8; Users must update to Java 8!
  • Due to the legacy Full Text Index being removed, the text (http://exist-db.org/xquery/text) XQuery module has also been removed. Users should now look toward fn:analyze-string, e.g.
    • instead of using text:groups() use analyze-string()//fn:group,
    • instead of text:filter("apparat", "([pr])") use analyze-string("apparat", "([pr])")//fn:match/string())
  • There have been changes to some of the internal APIs. e.g. XQueryService has been moved from DBBroker to BrokerPool.
  • EXPath packages that incorporate Java libraries may no longer work with eXist-db v3.0 and may need to be recompiled for our API changes; packages should now explicitly specify the eXist-db versions that they are compatible with.

eXist-db v3.0 is available for download from GitHub. The older Sourceforge download page is no longer updated. Maven artifacts for eXist-db v3.0 are available from our Maven Repository.

## Development

For all details see our commits

eXistdb Meetup in Vienna

During this year's TEI conference in Vienna we're organising an informal meetup. Take the opportunity to meet a few of the eXistdb developers on Tuesday, Sep 27th, 7pm at WerkzeugH, Schönbrunnerstr. 61, Wien 5.

As, since its inception in 2001, eXistdb development has always been driven by the needs of a large user community, we believe that the TEI conference is a great occasion for scholars to meet eXistdb contributors and practitioners, and to exchange ideas how the DH community could both engage and benefit further from eXistdb development.

We would be grateful if anyone planning to come could fill a very short form, nevertheless don't let it stop you from coming anyway! As for the agenda this is going to be a completely informal do, with main goal to provide a space for all of us interested in eXistdb in one way or another to actually meet and have a chat over a drink in a relaxed atmosphere of an alternative bar in Vienna.

The agenda so far:

  1. beers!
  2. hear about any new features being developed and opportunity to request new features
  3. food and drinks

While this setup is specifically for Travis-CI you will possibly find useful information to integrate existDB in other CI platforms as well.

It is language independent but was tested only with Java and nodeJS. Both setups are provided.

You can test applications, that are served from the db or consume one of its APIs.

## tl;dr

CI? You should have it.

The complete setup to test anything against one or more versions of eXistDB on TravisCI with caching enabled

## Preface

I was recently working on node-exist. It is a node package, that consumes eXist's RPC API. In order to run the tests the database needed to be up.

Mocking of the database responses could have solved that problem, but…

Now you got 2 problems

  1. how to test against another version or multiple ones
  2. validating mock responses (of multiple versions)

A much better solution is for the tests to run on a continous integration platform. Here, every commit can be tested against different versions of the database in parallel. And, there is no need to have them runnning on the development machine. Plus, everyone with access can verify if a certain build is running and which database versions are supported by your application.

There are more arguments pro continuous integration, which you can easily find online.

TravisCI seemed to be a reasonable choice, not only because eXistDB itself runs its automated tests here.

## Which version to test against?

As travis automatically starts one build per entry in env your tests will be run against the 3.0.RC1 and the 2.2 release of eXist with

```yaml env: - EXISTDBVERSION=eXist-3.0.RC1 - EXISTDBVERSION=eXist-2.2 ```

Add or remove versions as you need them.

## before install

It proved to be handy to store the installation folder of the current DB version in an environment variable `EXISTDBFOLDER`. It will be used by following scripts and commands.

```yaml before install: - export EXISTDBFOLDER=$HOME/exist/${EXISTDBVERSION} ```

## installation and setup

To install the database we download the source from github, extract it and then call its build routine.

```sh export TARBALLURL=https://github.com/eXist-db/exist/archive/${EXISTDB_VERSION}.tar.gz

mkdir -p ${EXISTDBFOLDER}
curl -L $TARBALL_URLtar xz -C ${EXISTDBFOLDER} --strip-components=1
cd ${EXISTDBFOLDER}
./build.sh
```

All of the above can be nicely packed into a setup script.

`yaml install: - ci/setup-db.sh `

Note: `EXISTDBVERSION` is the environment variable we defined at the very beginning. This can be a tag or branchname or even a commit hash.

## Can't we start already?

Yes, but in order to do that we have to start eXist in the background and wait for it to listen to requests. Last, ensure that by doing a very simple one.

```sh cd ${EXISTDBFOLDER} nohup bin/startup.sh & sleep 30 curl http://127.0.0.1:8080/exist ```

Yes, you guessed it. There is a database start-up script.

`yaml before_script: - ci/start-db.sh `

Note: Without changing to its installation folder a bunch of exceptions will end up in nohup.out complaining about log files that cannot be found and opened.

## That's it

Run your tests!

```yaml script: - do test / ```

Now it is time to clean up the closet:

```yaml afterscript: - cd ${EXISTDB_FOLDER} - bin/shutdown.sh ```

## Tweaks

### Caching

Downloading and building exist from source can take up to 3 minutes. So, you may want to speed up your tests by caching the built database. The archived cache has still to be loaded from s3 but you will gain an extra minute or so - YMMV.

#### Is this version of existDB already cached?

In our case that boils down to: does the folder exist?

```sh if [ -d "$EXISTDBFOLDER" ]; then echo "Using cached eXist DB instance: ${EXISTDBVERSION}." exit 0 fi ```

#### Teardown this Database

Remove any data that is or might be left behind by your tests and remove logs, too. To make sure that you will always get the latest version for branches or refs like HEAD, anything but releases should be excluded from caching.

```sh if [[ "${EXISTDBVERSION}" == eXist ]]; then echo "reset data and logfiles for ${EXISTDBVERSION}" cd ${EXISTDBFOLDER} ./build.sh clean-default-data-dir rm webapp/WEB-INF/logs/.log exit 0 # fi

echo "exclude ${EXISTDBVERSION} from cache" rm -rf ${EXISTDBFOLDER} ```

Put together:

```yaml before_cache: - ci/teardown-db.sh cache: directories: - $HOME/exist ```

## What if Java is not you first language?

If you are not testing a java-application, as I was, you need to install java 1.8 into to the testing container with:

`yaml addons: apt: packages: - oracle-java8-installer `

and make this version the default by adding

`yaml - export JAVA_HOME=/usr/lib/jvm/java-8-oracle `

to the before_install step.

## The Gist of the Story

(Integration) testing is necessary, so if you're application depends on eXistDB then you should definitely have a look at this project.

examples on GitHub

results on Travis

### contents

  • .travis.yml (Java setup)
  • node.travis.yml (nodeJS setup)
  • ci/* (helper scripts described above)
  • utility/* (ant, Java setup utility functions)
  • project/* (sample project)

XML Prague 2016

# XML Prague 2016

This year the XML Prague conference is taking place from 11th to 13th of February.

Topics will be:

  • Markup and the Extensible Web
  • Semantic visions and the reality
  • Publishing for the 21st century
  • XML databases and Big Data
  • State of the XML Union

As each year members of the community and other interested parties are invited to join the 'eXist-db Unlike Preconference' to share and exchange knowledge around eXistdb.

eXist 3.0.RC1 available for download

It is our great pleasure to announce the first release candidate for eXist 3.0.

As always, the purpose of this release candidate is to collect feedback on performance and compatibility. Whilst it is considered feature complete, we would not yet recommend it for production environments.

eXist 3.0.RC1 is the culmination of over 550 changes made in the last six months. The main focus has been on fixing bugs, migrating to Java 8 and improving the performance of eXist. The move to Java 8 alone has brought many internal improvements in eXist, enabling us to work with a leaner and safer code base; resulting in better resource and lock management, and improved performance for our users.

## New Features * Support for No link definition found for id 1, including the array and map data types, serialisation and JSON parsing Support for Braced URI Literals from XQuery 3.0 Facility to boost attributes in the Lucene full text index eXist version detection for EXPath packages. Packages should explicitly specify which versions of eXist they are compatible with; eXist 2.2 is assumed by default. Prototype support for No link definition found for id 3 written in Haxe
## Improved Performance Sequence type checking on recursive function parameters has been drastically sped up Lucene full-text and range indexes have been switched to "near realtime" behaviour. This improves query performance on frequently updated documents Improved optimization of wildcard steps in path expressions, e.g. prefix: and :name Better performance for util:eval * Optimisation of fn:fold-left and fn:fold-right

## Mission Critical Bug Fixes

There have been numerous bug fixes and enhancements since eXist 2.2, the most critical are:

  • Patched a memory leak in the Java service wrapper that occurred on certain Linux systems
  • Solved a potential deadlock which manifested when storing XQuery files into the database under certain conditions
  • Fixed a memory leak when storing query results into the HTTP session; Web applications making use of the HTTP session should now consume less memory and scale further
  • Fixed an occasional deadlock when shutting down the database
  • Fixes to match highlighting with the Lucene full text index
  • Lucene range index now correctly handles != comparisons
## Clean up and Refactoring Rewritten HTML5 Serializer Removed the legacy SOAP API and SOAP Server Removed the legacy Full Text Index Removed the Versioning extension; will be made available as a separate app package. Rewritten XML:DB and XML-RPC APIs Updated to the latest version of RESTXQ Improved Java Admin Client document viewing and editing Clean up of eXist's Test suite * Extensive internal refactoring to exploit new Java 8 features
## Backwards Compatibility issues eXist-3.0.RC1 is not binary compatible with previous version of eXist; the on-disk database file format has been updated, users should perform a full backup and restore to migrate their data. eXist 3.0.RC1 and subsequent versions now require Java 8; Users must update to Java 8! Due to the legacy Full Text Index being removed, the text (http://exist-db.org/xquery/text) XQuery module has also been removed. Users should now look toward fn:analyze-string. There have been some small changes to some of the internal APIs. e.g. XQueryService has been moved from DBBroker to BrokerPool. * EXPath packages that incorporate Java libraries may no longer work with eXist 3.0 and may need to be recompiled for our API changes; packages should now explicitly specify the eXist versions that they are compatible with.

eXist-3.0.RC1 is available for download from No link definition found for id 2. The older Sourceforge download page is no longer updated. Maven artifacts for eXist-3.0.RC1 are available from our mvn-repo.

[1]: http://exist-db.org/exist/apps/wiki/blogs/eXist/XQuery31

[2]: https://github.com/eXist-db/exist/releases/tag/eXist-3.0.RC1

[3]: http://www.adamretter.org.uk/presentations/implementation-of-portable-expath-extensionsxml-london20150607.pdf