Database Deployment

(2Q19)


This article explains how to install eXist-db as a stand-alone or embedded server.

Overview

There are three ways to deploy the eXist-db database:

In a Servlet Context

The database is deployed as part of a web application. eXist will happily live together with other servlets. This is the default mode established when the database is deployed according to instructions provided in the Basic Installation Guide.

In this mode all resources used by eXist-db will have paths relative to the web application's current context. For example, eXist will store all its database files in the WEB-INF/data directory of the web application (unless specifically configured otherwise).

Stand-alone Server Process

In stand-alone mode, eXist-db runs in its own Java virtual machine (JVM). Clients have to access the database through the network, either using the XML-RPC, WebDAV protocol or the REST-style HTTP API.

Embedded in an Application

In embedded mode, the database is basically used as a Java library, controlled by the client application. It runs in the same JVM as the client, no network connection is needed. The client has full access to the database.

All three deployments are thread-safe and allow concurrent operations by multiple users. Servlets running in the same web application context will have direct access to the database. External client applications may still use the supplied network interfaces.

Detailed instructions on how to set up eXist-db for use with a servlet-engine are provided in the Basic Installation Guide. The sections in this document address the other two deployment options.

  • Section 2 introduces the XML:DB URI and explain how different servers are addressed by Java clients.

  • Section 3 deals with running eXist as a stand-alone server

  • Section 4 addresses the required steps to directly embed eXist into an application, including how to embed it in the XMLdbGUI.

Addressing Servers using the XML:DB URI

One way to access eXist-db from Java applications is to use the XML:DB API. The XML:DB API uses a specific URI scheme to locate a collection of XML resources on the server. You will encounter XML:DB URIs when working with the Java client, the backup tools and sometimes in XQuery functions. It is therefore important to understand how the URI scheme addresses servers and resources.

eXist's XML:DB API implementation supports transparent access to remote as well as embedded database servers. This means the database on the server is available on the client as though it were locally connected to the client. The user should not have to be aware of where a resource is physically located. Applications need not be affected by how the database has been deployed.

The XML:DB URI identifies the database implementation, the name of the collection, and optionally the location of the database server on the network.

In a Servlet Context

Use the following URI to points to the Shakespeare collection on a remote server which is running in a servlet engine:

xmldb:exist://localhost:8080/exist/xmlrpc/db/shakespeare

The host part: localhost:8080/exist/xmlrpc describes the path to the XML-RPC listener, which is running as a servlet. The collection part (/db/shakespeare) defines the collection to retrieve.

Stand-alone Server Process

If eXist had started as a stand-alone server, the URI would change its port to 8088:

xmldb:exist://localhost:8088/xmlrpc/db/shakespeare
Embedded in an Application

To access an embedded instance of the database, we simply drop the host part altogether and use three forward slashes (///) in front of the collection path:

xmldb:exist:///db/shakespeare

You can use the Java Admin Client, which utilizes the XML:DB API to access the database, to experiment with these settings. The Login dialog offers a text field where you can specify an XML:DB URI to connect to.

By default, the client tries to access the database engine located at the base URI:

xmldb:exist://localhost:8080/exist/xmlrpc/

Change the entry in the GUI to try another URI or pass it using the -ouri option on the command line. For example:

bin/client.sh -ouri=xmldb:exist://

This will cause a local database instance to run in the same Java virtual machine as the client.

There is also a short form for this command. The -l option causes the client to launch a local database instance. For instance:

bin/client.sh -l

Running as a Stand-alone Server

There are many cases in which it is preferable to have the database engine running in its own Java virtual machine. In stand-alone mode, eXist will launch its own, embedded web server to provide XML-RPC, WebDAV and REST services to the outside world. The embedded server is based on a stripped-down Jetty. It uses a limited configuration, excluding all the additional services available in a full-blown servlet environment.

The stand-alone deployment is more reliable and performs better than the web application setup, since no other threads (simultaneous tasks) are running. The stand-alone database server offers XML-RPC, WebDAV and REST-style HTTP interfaces for external client access. Please note that it does not support SOAP.

By default, the stand-alone server listens on port 8088, though this can be changed (see configuration below). The Java class for launching the server is org.exist.jetty.StandaloneServer.

Starting the Stand-alone Server

To start the server use either the bin/server.sh (Unix) or the bin\server.bat (Windows) command.

The server is multi-threaded, a server thread is assigned to each client request. If the specified maximum number of threads is reached, the server will block the client until one of the server-threads is available to respond. By default, the maximum number of threads is 5. To change this, use the -t option, e.g.:

bin\server.bat -t 20

To access the stand-alone server using the Java Admin Client (or by your own Java classes), start the XML:DB server URI to xmldb:exist://localhost:8088/xmlrpc

Standalone Server Configuration

The tools/jetty/etc/standalone/WEB-INF/web.xml configuration file is used when the server operates in standalone mode. An instance of the Jetty web server is configured using its settings. The services offered with eXist have their own configuration servlet element in this file.

Currently, the servlet API alternatives are "webdav, "xmlrpc", and "rest". Any of these servlets can be disabled by setting a init-param enabled attribute to "no".

Controlling the Binding Address (<indexer>)

You can control the binding address by changing the addConnector <Call> element in tools/jetty/etc/standalone.xml:

port

The port on which the server will listen. The Jetty web server will bind to port 8088 by default.

host

The host-name on which the server will respond.

address

The IP address on which the server should bind. This may be useful when the server has multiple addresses that serve the same host-name.

WebDAV Servlet

The WebDAV servlet provides services at the context address specified. This servlet is represented by the <webdav> element in the server.xml configuration file. For this element, the context attribute controls the web server context at which the WebDAV services are provided. If the context attribute is not specified, it defaults to /webdav/.

XML-RPC Servlet

The XML-RPC servlet provides database API services to clients, like the Admin GUI client. This servlet is represented by the <xmlrpc> element in the server.xml configuration file. For this element, the context attribute controls the web server context at which the XMLRPC services are provided. If the context attribute is not specified, it defaults to /xmlrpc/.

Warning:

If you disable this servlet, you disable the use of the Java Admin Client.

REST Servlet

The REST servlet provides HTTP/REST-style interactions with the database. It is configured by a <servlet> element with <servlet-class> org.exist.http.servlets.EXistServlet in the tools/jetty/etc/standalone/WEB-INF/web.xml configuration file. It depends on the XQueryURLRewrite filter for context, default is /.

The REST Servlet has a number of other parameters that can be set by child elements:

<servlet>
        <servlet-name>EXistServlet</servlet-name>
        <servlet-class>org.exist.http.servlets.EXistServlet</servlet-class>
      <init-param>
            <param-name>use-default-user</param-name>
            <param-value>true</param-value>
        </init-param>
...
 </servlet>
form-encoding

The default encoding of form POSTs.

container-encoding

The default encoding of the servlet container for all HTTP interactions except POSTs.

use-default-user

A boolean value (true/false) that indicates whether the effective user must be used for non-authenticated interactions. If any of these fails a simple AUTH interaction is undertaken.

user

The username of the default user.

password

The password of the default user.

Custom Servlets

Any servlet can be configured to run in the Jetty server by adding a <servlet> element to the server.xml configuration file. The <servlet> element has the standard attributes of enabled and context, as well as the class attribute to specify the servlet implementation class.

This element can have any number of <param> child-elements to set parameters for the servlet. For example:

<servlet enabled="yes" context="/myservlet/*" class="com.example.MyServlet">
      <param name="auth" value="true"/>
      <param name="demo" value="false"/>
 </servlet>

Forwarding Requests

The forwarding-request settings allow you to map incoming URL requests to specific resources on the server (e.g. queries). This is done with the <forwarding> element. Inside this element is a single <root> element and any number of <forward> elements.

Each <forward> element specifies a specific primary URL that is forwarded to secondary URL. This allows you to map a "clean" URL to an XQuery or some other resource within the server. The target is always specified by a destination attribute. For instance:

<forward path="/admin" destination="/db/admin/admin.xql"/>
<forward path="/docs" destination="/db/products/docs.xml"/>

Here the first <forward> element maps the URI path /admin to the XQuery resource /db/admin/admin.xql, the second one maps the URI path /docs to the document /db/products/docs.xml.

The <root> element maps requests to the root directory of the server (for instance /) to a specified resource. For example:

<root destination="/db/admin/admin.xql" />

This maps a requests for the (default) server root to the XQuery resource /db/admin/admin.xql.

Shutting Down the Database

Use the shutdown.bat (Windows/DOS) and shutdown.sh (Unix) scripts to shutdown the database. These commands connect to the default server URI, usually xmldb:exist://localhost:8080/exist/xmlrpc.

If your database is running in stand-alone mode, you must specify a different server URI. For instance, to stop eXist when running in stand-alone mode, listening on port 8088, use:

$EXIST_HOME/bin/shutdown.sh --uri=xmldb:exist://localhost:8088/xmlrpc

Embedding eXist in an Application

In the embedded mode, the database runs in the same Java virtual machine as the client application. The database will not be accessible by any outside application and no network listeners are started. You can embed eXist-db into any Java application using the XML:DB API.

To prepare the environment for using an embedded eXist, follow the steps below:

  1. Copy conf.xml and log4j.xml to the target directory.

  2. Create a subdirectory data in the target directory.

  3. Edit the files attribute in the <db-connection> section of conf.xml to point to this data directory.

  4. Do the same for the journal-dir in the <recovery> element.

  5. To see logging output, edit log4j.xml. The simplest way is to change the appender-ref in the <root> category to console. This will result in most log messages being written to the console.

  6. Create a lib directory below the target directory and copy the following .jar files from eXist-db into it:

    • exist.jar

    • lib/extensions/exist-modules.jar

    • lib/core/antlr-X.X.X.jar

    • lib/core/commons-pool-X.X.jar

    • lib/core/commons-collections-X.X.jar

    • lib/core/commons-logging-X.X.X.jar

    • lib/core/log4j-X.X.X.jar

    • lib/optional/slf4j-api-X.X.X.jar

    • lib/optional/slf4j-log4j12-X.X.X.jar

    • lib/core/quartz-X.X.X.jar

    • lib/core/sunxacml.jar

    • lib/core/xmldb.jar

    • lib/core/xmlrpc-client-X.X.X.jar

    • lib/core/xmlrpc-common-X.X.X.jar

    • lib/core/xmlrpc-server-X.X.X.jar

    • lib/core/jta.jar

    • lib/core/pkg-repo.jar

    If you plan to use extension modules (like the n-gram index), copy the corresponding jars from lib/extensions. The default eXist configuration needs:

    • exist-ngram-module.jar

    • exist-lucene-module.jar

    • exist-versioning.jar

    For the Lucene module copy the jars from extensions/indexes/lib:

    • lucene-core-X.X.X.jar

    • lucene-regex-X.X.X.jar

    The lib/endorsed directory furthermore plays a special role: the Java releases come with their own XML support libraries, including Xalan (for XSLT processing), an XML parser, and the standard Java interfaces for SAX and DOM. Some features of eXist will not work properly with a wrong version of Xerces or the resolver .jar (schema validation, catalog loading, ...). To ensure the correct versions are available, we have included these versions of Xerces, Xalan and Saxon, plus the standard interfaces used by both of them. You can use Java's endorsed library loading mechanism to ensure the correct XML support libraries are loaded.

    • Create a directory lib/endorsed

    • Copy all .jar files from eXist's lib/endorsed directory into the newly created directory

    Specifying the -Djava.endorsed.dirs=lib/endorsed system property on the Java command line will force the JVM to prefer any library it finds in the endorsed directory over its own system libraries (copying the jars into $JAVA_HOME/jre/lib/endorsed will do the same thing). Note that the batch and shell scripts included with eXist all set the java.endorsed.dirs system property to point to lib/endorsed.

  7. Make sure your CLASSPATH includes these .jar files.

Internally, eXist has two different XML:DB driver implementations: the first communicates with a remote database engine using XML-RPC calls; the second has direct access to a local instance of eXist. Which implementation is selected depends on the XML:DB URI as described above.

To access an embedded database, simply drop the host portion (for instance localhost:8088/xmlrpc/) from the URI. For instance: xmldb:exist:///db

To start an embedded database instance, simply set the system property exist.initdb to true. This will notify the XML:DB driver to read the configuration settings when starting the database if none has been previously started. For example, to launch your own Java application with an embedded instance, you may enter the command:

java -Dexist.initdb=true MyApp

The driver will try to read the eXist configuration file, create the required database files (if absent) and launch the database.

The driver must be able to read the configuration file conf.xml. It seraches for conf.xml in the directory specified by the exist.home system property. You should therefore ensure a copy of conf.xml is present in the correct directory. For example, to launch your own application, while setting exist.home:

java -Dexist.initdb=true -Dexist.home=/home/exist/eXist MyApp

Important:

The paths to the data and log directories in conf.xml need to point to writable directories.

Instead of using the -Dexist.initdb property, you can also tell the database driver to create a local database instance during the initialization of the XML:DB database manager within your Java code. Set the create-database property on the created Database object to true. For example:

Class cl = Class.forName("org.exist.xmldb.DatabaseImpl");
Database database = (Database) cl.newInstance();
database.setProperty("create-database", "true");
DatabaseManager.registerDatabase(database);

When running eXist in embedded mode, ensure proper shutdown the database before your application exits. The main reason is to flush all of the unwritten data buffers to disk. The database uses a background thread to periodically synchronize its buffers with the data files on disk, and this thread will keep running unless you shutdown the database.

There is a special XML:DB service, DatabaseInstanceManager, which contains a single method: shutdown. To properly shutdown the database instance, retrieve the service from the /db-collection and call shutdown(). For example:

DatabaseInstanceManager 
manager = (DatabaseInstanceManager) 
    collection.getService("DatabaseInstanceManager", "1.0"); 
manager.shutdown();

To summarize, a minimal working class with a single static main to start/stop the db may look like this:

import org.xmldb.api.DatabaseManager;
import org.xmldb.api.base.Collection;
import org.xmldb.api.base.Database;
import org.exist.xmldb.DatabaseInstanceManager;

public class TestDB {		
	public static void main(String args[]) throws Exception {
		// initialize driver
		Class cl = Class.forName("org.exist.xmldb.DatabaseImpl");
		Database database = (Database)cl.newInstance();
		database.setProperty("create-database", "true");
		DatabaseManager.registerDatabase(database);
		
		// try to read collection
		Collection col = 
			DatabaseManager.getCollection("xmldb:exist:///db", "admin", "");
		String resources[] = col.listResources();
		System.out.println("Resources:");
		for (int i = 0; i < resources.length; i++) {
			System.out.println(resources[i]);
		}
		
		// shut down the database
		DatabaseInstanceManager manager = (DatabaseInstanceManager) 
			col.getService("DatabaseInstanceManager", "1.0"); 
		manager.shutdown();
	}
}

Put this code into a Java file TestDB.java, store it into the target directory (which we prepared above), and compile and run it with:

javac TestDB.java java -Dexist.initdb=true -Dexist.home=. TestDB