URL Rewriting

Introduction

Good web applications provide meaningful and consistent URLs to the user. eXist-db's URL Rewriting facility is eXist's oldest internal mechanism for providing short, human-readable URLs to XQuery web applications, conveniently masking the often complex hierarchy of XQuery modules, HTML files, data, and other resources. (A newer facility for achieving these goals is eXist-db's implementation of RESTXQ. A third method is to place a reverse proxy between eXist-db and the end-user. Many applications combine two or even all three of these methods.)

For a brief overview of how eXist-db's URL Rewriting facility works, consider what happens when eXist-db receives an HTTP request:

1. eXist-db's Jetty web server receives an HTTP request. In the default configuration this is handled by the XQueryUrlRewrite servlet for any URL starting with the path /exist.

2. The XQueryUrlRewrite servlet first checks the URL against a series of URL patterns defined in an eXist-db configuration file, called controller-config.xml and described in the section below on Base Mappings. If eXist-db finds a matching root pattern, which is /apps by default, eXist-db will look within the associated controller root path, which is the /db/apps database collection by default, for controller scripts: special XQuery files named controller.xq (or in older applications controller.xql). A controller script applies to the collection it is stored in and its descendants. They form a collection hierarchy—a URL space within which URL rewriting can be flexibly applied.

3. The controller.xq script examines the URL using the provided Controller Variables, and produces an XML-formatted directive in the Controller XML Format.

4. The XQueryURLRewrite servlet interprets the directive as a series of instructions on what to do next. These instructions may be as simple as forwarding the request to a resource on the server (or elsewhere), or as complex as a pipeline using the Model-View-Controller pattern (see MVC and Pipelines) and other servlets such as eXist-db's XQueryServlet or XSLTServlet. The controller script isn't limited to returning these types of instructions; like any XQuery main module, they can return any data you may wish. But URL Rewriting instructions are the special capability of controller scripts.

xquery version "3.1";

declare namespace exist="http://exist.sourceforge.net/NS/exist";

declare variable $exist:path external;
declare variable $exist:resource external;
declare variable $exist:controller external;
declare variable $exist:prefix external;
declare variable $exist:root external;

<dispatch xmlns="http://exist.sourceforge.net/NS/exist">
  <forward url="{$exist:controller}/modules/transform.xq">
    <add-parameter name="doc" value="{$exist:resource}.xml"/>
  </forward>
</dispatch>

xquery version "3.1";

declare namespace exist="http://exist.sourceforge.net/NS/exist";

declare variable $exist:path external;
declare variable $exist:resource external;
declare variable $exist:controller external;
declare variable $exist:prefix external;
declare variable $exist:root external;

<dispatch xmlns="http://exist.sourceforge.net/NS/exist">
  <forward url="data/{$exist:resource}.xml"/>
  <view>
    <forward url="{$exist:controller}/modules/transform.xq"/>
  </view>
</dispatch>

Controller XML Format

As we have seen, the controller.xq script is expected to return a single XML element: a <dispatch> or <ignore> element. The script can actually return any data, as you wish, but the <dispatch> or <ignore> elements have special properties in the context of a controller.xq script.

Note that all of the elements described in this section must be in the http://exist.sourceforge.net/NS/exist namespace.

<dispatch xmlns="http://exist.sourceforge.net/NS/exist">
  <redirect url="..."/>
</dispatch>

<ignore xmlns="http://exist.sourceforge.net/NS/exist"/>

<dispatch xmlns="http://exist.sourceforge.net/NS/exist">
  <redirect url="..."/>
</dispatch>

HTTP/1.1 302 Found
Location: http://localhost:8080/exist/apps/doc/
Content-Length: 0

<dispatch xmlns="http://exist.sourceforge.net/NS/exist">
  <forward url="{$exist:controller}/modules/transform.xq">
    <add-parameter name="doc" value="{$exist:resource}.xml"/>
  </forward>
</dispatch>

<add-parameter name="xxx" value="yyy"/>

<set-attribute name="xxx" value="yyy"/>

<clear-attribute name="xxx"/>

<set-header name="xxx" value="yyy"/>

<dispatch xmlns="http://exist.sourceforge.net/NS/exist">
  <redirect url="..."/>
  <cache-control cache="yes"/>
</dispatch>

Controller Variables

Five URL Rewriting-related variables are pre-bound and made available to the controller.xq script for convenience. For example, if the request path is /exist/apps/sandbox/get-examples.xq these variables will be bound to the following values:

These variables are pre-bound for simplicity and convenience. You do not need to explicitly declare the variables or the exist namespace. However, doing so is considered best practice, by adding this block to the prolog of your controller.xq script. For instance:

declare namespace exist="http://exist.sourceforge.net/NS/exist";

declare variable $exist:root external;
declare variable $exist:prefix external;
declare variable $exist:controller external;
declare variable $exist:path external;
declare variable $exist:resource external;

These variables can also be accessed by any query within the controller hierarchy via the request:get-attribute() function, e.g., request:get-attribute("$exist:prefix").

Some further remarks about each variable:

$exist:root

Is bound to the root of the current controller hierarchy. This may either point to the file system or to a collection in the database. You can use this variable to locate resources relative to the root of the application.

For example, assume that you want to process a request using stylesheet db2xhtml.xsl, which could either be stored in the /stylesheets directory in the root of the webapp or, if the app is running from within the database, the corresponding /stylesheets collection. You want your app to be able to run from either location. The solution is to incorporate the value of the $exist:root variable into your logic:

<forward servlet="XSLTServlet">
  <set-attribute name="xslt.stylesheet" value="{$exist:root}/stylesheets/db2xhtml.xsl"/>
</forward>

$exist:prefix

If the current controller hierarchy is mapped to a certain path prefix, then the $exist:prefix variable will be bound to that prefix.

For example: the default configuration maps the path /apps to a collection in the database (see below). In this case, the $exist:prefix variable would be bound to the value /apps.

$exist:controller

Is bound to the part of the URL leading to the current controller.xq.

For example, if the request path is /sandbox/test.xq and the controller is in the xquery collection, the $exist:controller variable will be bound to the value /sandbox.

$exist:path

Is bound to the last part of the request URL, i.e., after the section leading to the collection containing the controller.xq.

For instance, if the resource example.xml resides within the same collection as the controller query, the $exist:path variable would be bound to the value /example.xml.

$exist:resource

Is bound to the part of the URL after the last /; this is usually pointing to a resource.

For instance: example.xml.

In addition, within a controller.xq file, you also have access to the entire XQuery function library, including the functions in the HTTP request, response, and session modules.

Accessing Resources Not Stored in the Database

If your controller.xq is stored in a database collection, all relative and/or absolute URLs processed by the controller will be resolved against the database, not the file system. This can be a problem if you need to access common resources, which should be shared with other applications residing on the file system or in the database.

The forward directive accepts an optional attribute absolute="yes|no" to handle this. If one sets absolute="yes", an absolute path (starting with a /) in the url attribute will resolve relative to the current servlet context, not the controller context.

For example, to forward all requests starting with a path /libs/ to a directory within the webapp folder of eXist-db, you can build upon the following example snippet:

if (starts-with($exist:path, "/libs/"))
then
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
        <forward url="/{substring-after($exist:path, '/libs/')}" absolute="yes"/>
    </dispatch>

This simply removes the /libs/ prefix and sets the attribute absolute="yes", so that the path will be resolved relative to the main context of the servlet engine, typically /exist/. In your HTML, you can now write paths such as:

<script type="text/javascript" src="/libs/scripts/jquery/jquery-1.7.1.min.js"/>

This will locate the jquery file in webapp/scripts/jquery/..., even if the rest of your application is stored in the db and not on the file system.

Locating Controller Scripts and Configuring Base Mappings

By convention, the controller script must be named controller.xq (or in older applications controller.xql; if both controller.xq and controller.xql are placed in the same collection, the former takes precedence, and the latter is ignored).

If you wish to allow anonymous users to access resources using the URL Rewriting framework then you need to ensure that the controller.xq script is world-executable, e.g., sm:chmod(xs:anyURI("/path/to/controller.xq")), "o+x").

By default, the URL Rewriting framework will try to guess the path to the controller script by looking at the request path, starting with the deepest, most specific collection, and then examining each parent collection until a controller script is found or the root of the database has been reached.

This can be configured and overridden via the controller-config.xml configuration file in $EXIST_HOME/etc/webapp/WEB-INF, which defines the base mappings used.

In fact, one web application may have more than one controller hierarchy. For example, you may want to keep the main webapp within the file system, while some tools and scripts should be served from a database collection. This can be done by configuring two roots within the controller-config.xml file.

The configuration file has two components:

• <forward> actions which map URL patterns to servlets

• <root> elements that define the root for a file system or database collection hierarchy

The <forward> elements specify path mappings for common servlets, similar to the servlet mapping in $EXIST_HOME/etc/webapp/WEB-INF/web.xml. The advantage to specifying them in controller-config.xml is that the XQueryURLRewrite servlet (which implements the URL Rewriting framework) becomes a single point of entry for the entire web application, and no additional handling of the servlet paths is necessary in the controller script.

For example, if we had registered a servlet mapping for /rest in web.xml, we would have needed to make sure that this path is ignored in our main controller.xq scripts. However, since the mapping is done via controller-config.xml, XQueryURLRewrite handles the path, which then doesn't need to be accounted for in our controller.

The <root> elements define the roots of a directory or database collection hierarchy, mapped to a certain base path. For example, the default controller-config.xml uses two roots:

<!-- HTTP requests to /apps are mapped onto the database path /db/apps -->
<root pattern="/apps" path="xmldb:exist:///db/apps"/>

<!--
  ++ The default fallback web application is served from the
  ++ /etc/webapp directory on the filesystem. 
-->
<root pattern=".*" path="/"/>

This means that paths starting with /apps will be mapped to the collection hierarchy below /db/apps. All relative or absolute URLs within the URL space managed by the controller.xq script will be resolved against the database, not the file system. Everything else is handled by the catch all pattern pointing to the root directory of the webapp (which, by default, corresponds to $EXIST_HOME/etc/webapp). However, there's a possibility to escape this path interpretation as described below.

MVC and Pipelines

The XQueryURLRewrite servlet does more than just forward or redirect requests: the response can be processed by passing it to a pipeline of views. "Views" are again just plain Java servlets. The most common use of a view would be to post-processes the XML returned from the primary URL, either through another XQuery or an XSLT stylesheet (XSLTServlet). XQueryURLRewrite passes the HTTP response stream of the previous servlet to the HTTP request received by the next servlet. It is fully possible to extend eXist-db by adding in your custom own servlets.

Views may also directly exchange information through the use of request attributes (more on that below).

You define a view pipeline by adding a <view> element to the <dispatch> element returned by the controller. The <view> element is a wrapper around another sequence of <forward> or <rewrite> actions.

For example, assume we have some XML documents written in DocBook format, and that we wish to render this as HTML by transforming the XML to HTML through an XSLT stylesheet webapp/stylesheets/db2html.xsl. This can be done by returning the following <dispatch> element from a controller.xq:

<dispatch xmlns="http://exist.sourceforge.net/NS/exist">
  <view>
    <forward servlet="XSLTServlet">
      <set-attribute name="xslt.stylesheet" value="stylesheets/db2html.xsl"/>
    </forward>
  </view>
  <cache-control cache="no"/>
</dispatch>

In this example there are no forwarding actions except for the view, so the request will be handled by the servlet engine in the default way. The response is then passed to the XSLTServlet. A new HTTP POST request is created whose body is set to the response data of the previous step. The XSLTServlet gets the path to the stylesheet from the request attribute xslt.stylesheet and applies it to the provided data.

If any step in the pipeline generates an error or returns an HTTP status code >= 400, the pipeline processing stops and the response is send back to the client immediately. The same happens if the first step returns with an HTTP status 304 (NOT MODIFIED), which indicates that the client can use the version it has cached.

We can also pass a request through more than one view. The following document applies two stylesheets in sequence:

<dispatch xmlns="http://exist.sourceforge.net/NS/exist">
	<!-- NOTE: query results are passed to XSLT servlet via a request attribute -->
	<set-attribute name="xquery.attribute" value="model"/>
	<view>
		<forward servlet="XSLTServlet">
			<set-attribute name="xslt.input" value="model"/>
			<set-attribute name="xslt.stylesheet" value="xquery/stylesheets/acronyms.xsl"/>
		</forward>
		<forward servlet="XSLTServlet">
			<clear-attribute name="xslt.input"/>
			<set-attribute name="xslt.stylesheet" value="stylesheets/db2html.xsl"/>
		</forward>
	</view>
</dispatch>

The example also demonstrates how information can be passed between actions. XQueryServlet (which is called implicitly because the URL ends with .xq) can save the results of the called XQuery to a request attribute instead of writing them to the HTTP output stream. It does so if it finds a request attribute named xquery.attribute, which should in turn contain the name of the request attribute that the output should be saved to.

In the example above, xquery.attribute is set to model. This causes XQueryServlet to fill the request attribute model with the results of the XQuery it executes. The query result will not be written to the HTTP response as you might expect, instead at this point the HTTP response body remains empty as the data is inside the request attribute.

Likewise, XSLTServlet can take its input from a request attribute instead of parsing the HTTP request body. The name of the request attribute should be given in attribute xslt.model. XSLTServlet discards the current request content (which is empty anyway) and uses the data in the attribute's value as input for the transformation process.

XSLTServlet will always write to the HTTP response. The second invocation of XSLTServlet therefore needs to read its input from the HTTP request body which contains the response of the first servlet. Since request attributes are preserved throughout the entire pipeline, we need to clear the xslt.input with an explicit call to clear-attribute.

The benefit of exchanging data through request attributes is that we save one serialization step: XQueryServlet directly passes the node tree of its output as a valid XQuery value, so XSLTServlet does not need to parse it again.

The advantages become more obvious if you have two or more XQueries which need to exchange information: XQuery 1 can use the XQuery extension function request:set-attribute() to save an arbitrary XQuery sequence to an attribute. XQuery 2 then subsequently calls request:get-attribute() to retrieve this value. As it can directly access the data passed in from XQuery 1, no time is lost serializing and deserializing the data.

Let's have a look at a more complex example: eXist-db's eXide web application needs to execute a user-supplied XQuery fragment. The results should be retrieved in an asynchronous way, so the user doesn't need to wait and the web interface remains usable.

Older versions of eXide's ancestor sandbox application used the util:eval() function to evaluate the query. However, this had side-effects because util:eval() executes the query within the context of the parent query. Some features like module imports could not work properly. To avoid util:eval(), the controller code below passes the user-supplied query to XQueryServlet first, then post-processes each returned result and stores it into a session for later use by the AJAX frontend:

if (starts-with($path, "/eXide/execute"))
then
  let $query := request:get-parameter("qu", ())
  let $startTime := util:system-time()
  return
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <!-- Query is executed by XQueryServlet -->
      <forward servlet="XQueryServlet">
        <!-- Query is passed via the attribute "xquery.source" -->
        <set-attribute name="xquery.source" value="{$query}"/>
        <!-- Results should be written into attribute "results" -->
        <set-attribute name="xquery.attribute" value="results"/>
        <!-- Errors should be passed through instead of terminating the request -->
        <set-attribute name="xquery.report-errors" value="yes"/>
      </forward>
      <view>
        <!-- Post process the result: store it into the HTTP session and return the number of hits only. -->
        <forward url="session.xq">
          <clear-attribute name="xquery.source"/>
          <clear-attribute name="xquery.attribute"/>
          <set-attribute name="elapsed" value="{string(seconds-from-duration(util:system-time() - $startTime))}"/>
        </forward>
      </view>
    </dispatch>
else
  if (starts-with($path, "/sandbox/results/"))
  then
    (: Retrieve an item from the query results stored in the HTTP session. The
      format of the URL will be /sandbox/results/X, where X is the number of the
      item in the result set :)
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
      <forward url="../session.xq">
        <add-parameter name="num" value="{$name}"/>
      </forward>
    </dispatch>

The client passes the user-supplied query string in a request parameter, so the controller has to forward this to XQueryServlet somehow. XQueryServlet has an option to read the XQuery source from a request attribute, xquery.source. The query result will be saved to the attribute results. The second XQuery, session.xq, takes the result and stores it into an HTTP session, returning only the number of hits and the elapsed time.

When called through retrieve, session.xq looks at parameter num and returns the item at the corresponding position from the query results stored in the HTTP session.

Special Attributes Accepted by eXist-db Servlets

eXist-db's XQueryServlet as well as the XSLTServlet expect a few predefined request attributes. The names of these attributes are listed below, and are reserved, that is to say that they should not be used for other purposes.