Jetty5 to Jetty6 migration

Recently I started to work on the upgrade of the webserver that is used internally by eXist-db: Jetty.

The migration from Jetty5 to Jetty6 turns out not to be straight forward, quite some things have changed there. Progress on this project will be posted here, together with links to important documentation.

  • Upgrade of libraries done (done)
  • ... added servlet-api-2.5-6.x.jar jetty-util-6.x.jar jetty-6.x.jar
  • ... removed jasper jars ; we don't need them I think because we have no JSPs?
  • Need to check Tomcat jar; can this file be deleted?
  • We should move the servlet-api-jar in lib/core to /lib/optional ?

Links

Update on JSR-225

At this moment we are working on the XQJ a.k.a. jsr-225 drivers for the eXist database. We are quite close to integration the code into 1.3/trunk, small steps still need to be taken.

For those who cannot wait:

The interface is tested in the junit test suite, but Oracle supplies a vendor independent testsuite as well. Due to licensing limitations this code cannot be included in the eXist-db distribution, but these can be downloaded easily.

  • Download the Oracle files with ./build.sh prepare-jsr225

remote:

  • For remote tests: start database ./bin/startup.sh
  • Start the test suite: ./build.sh test-prepare test-jsr225-remote test-wrapup
  • Check the results in ./test/junit/html/index.html

local:

  • Start the test suite: ./build.sh test-prepare test-jsr225-local test-wrapup

Some Module Issues

As most of you (should) know by know, eXist does support xquery modules. With this it is possible to efficiently re-use xquery code. A small xquery example:

import module namespace mod1 = "urn:module1" at "module1.xqm"; <a> <b>{mod1:showMe()}</b> </a>

and a not too complex module:

module namespace mod1 = "urn:module1"; declare function mod1:showMe() as xs:string { "hi from module 1" };

In this example the module is located in the same collection as the query; This rather simple query-module relation works perfectly and is used in many applications. Things get more difficult when a module itself has a relation with one ore more modules.

It turns out that modules are always resolved relative to the 'first executed' xquery. That means that if your module imports another module (e.g. module3.xqm in collection sub) special things happen:

Layout: /db/query.xq /db/collection/module2.xqm /db/collection/sub/module3.xqm

In this example query.xq imports module2.xqm and module2.xqm imports module3.xqm ; One would expect that the following code would be sufficient for module2.xqm:

import module namespace mod3 = "urn:module3" at "sub/module3.xqm";

What actually happens is that the query engine tries to resolve module3.xqm from /db/sub/module3.xqm. The only solution to have the module loaded correctly, is to specify a full path in the database:

import module namespace mod3 = "urn:module3" at "xmldb:///db/collection/sub/module3.xqm";

The bad thing about this construct is that it makes your application less portable/movable.

I think this limitation does not make sense. I'll investigate this issue and 'll (try to) make a fix for it (unless Perig disagrees with the solution :-) )

Java Webstart and Servlet containers

This week there was an issue reported concerning the code that enables the download of the InteractiveClient via Java Webstart. According to the report the code was not able to find the relevant jar files.

In short some context: JnlpHelper.java retrieves information of the location of eXist and the jar files by calling ConfigurationHelper.getExistHome() which -somehow- does not always return the right value. Since we had some issues before with this class, I wanted to try something else.

According to the servlet specification the best way of retrieving the location of files in a servlet context (on server extracted from war-file) is calling getServletContext().getResource("/") ; this should return a styled URL to the toplevel directory of exist.

Unfortunately this does not work for tomcat. Tomcat does actually return a type URL which cannot be resolved to a 'real' directory. There are several reports on this subject and is for us a dead end.

The other servlets of eXist-db use another construction to get the application directory information: getServletContext().getRealPath("/") but (quote: javadoc)

The real path returned will be in a form appropriate to the computer and operating system on which the servlet container is running, including the proper path separators. This method returns null if the servlet container cannot translate the virtual path to a real path for any reason (such as when the content is being made available from a .war archive).

There were some discussions on this a few years ago on the ML, but in the end it turns out that we do not have a real alternative than using getRealPath().

The consequence of this decision is that eXist-db cannot run in webcontainers that

  • do not expand the war files when installed
  • block access to the local filesystem. (This does not make sense anyhow, because exist does actually need to write data to 'some' file location.)

The webstart fix is now part of 1.3/trunk and 1.2.5.

ChangeLog 1.2

  • Fixed in-memory DOM tree construction bug: in some very specific cases, constructed nodes were skipped during serialization and did not show up in the output.
  • Bug fix: cached password for remote XMLDB API connection is not properly reset after a user's password was changed.
  • On db shutdown, properly unregister all JMX MBeans.
  • XInclude filter: properly implement
  • Fixed XPath bug: for some descendant-axis queries on nested elements, duplicate nodes were not filtered out properly. The query engine returned duplicate results.
  • Prevented NPE's in ant tasks.
  • reading collection.xconf is now fail safe. Errors in the configuration document will not cause an exception.
  • Temporary workaround for bug "Invalid collection.xconf causes a non startable database"
  • New functions gives you the xpath of a node, attributes in the hierarchy are converted to predicates
  • Fix NPE in bad-formed module URI
  • Fixed NPE upon update delete of root element
  • Fixed namespace issue: in-scope namespace declarations were not properly visible to functions located in other modules.
  • Enhanced rendering of XQuery Function Documentation.
  • Clean up SAXSerializer instances before returning them to the pool.
  • Fixed concurrency issue: built-in XQuery modules
  • Fixed base-uri() call on document node causing a class cast exception
  • Fixed locking issue where permission exception caused lock on collection not to be released.
  • Build jmx extensions into extra jar lib/extensions/exist-jmx.jar.
  • Fixed the HistoryTrigger
  • Applied patch 1775400 supplied by Michael Westbay
  • Fixed deficiencies in computed document constructors : they should allow nested document constructors
  • Fixed chmod/chown implementation
  • Added function
  • Applied Get collection cache size for Cache stats; patch by Gary Larsen
  • Applied patch Fix adminClient's dialog gridBag resizing ; Kindly contributed by Avagvwctopoulos Koctis (sperxios)
  • Use launch4j to generate a native windows .exe wrapper around the generated installer. Without this, eXist will not install properly on windows vista.
  • Applied patch #1824991, webdav lock operation does not meet RFC4918 requirements, supplied by Brad Clements.
  • Updated log4j to 2.1.15
  • Catch NPE in NativeBroker.getTextSearchEngine and log a warning.
  • Fixed parser bug (ignoring whitspace)
  • Fixed another positional predicate bug
  • Fixed NPEs if request/response/session object is not in scope and a function is called
  • Fixed a bug in
  • Fixed: &#x3c;&#x3e; tags lost during ancestor or parent path steps.
  • Fixes for various issues which improved XQTS results a lot.
  • Updated uses the user in the current xquery context
  • Added RedirectorServlet as a different, maybe more powerful mechanism to redirect HTTP requests.
  • Added some missing HTTP request functions:
  • Removed passing a Java Collection Object
  • Libraries from lib/extensions need to be copied into the war distribution as well.
  • Split of modules into seperate jars, and moved all modules to lib/extensions
  • Fixed bugs in
  • Fixed XQTS test base-URI-18 regression
  • Fixed construction bug.
  • Fixed: Another NGram bug fix: matches were not tagged if the query result was included into a constructed node.
  • Fix for NPE in
  • Added xsl-fo extension module
  • Applied Patch to fix NPE in CollectionCache on startup
  • Applied NPE in CollectionCache on startup
  • Don't approximate in seconds-from-duration()
  • Fix for Double default namespace
  • Added function , which encodes a string into a value of type
  • Clean up of loged SAXParseExceptions.
  • Validation is now configurable per collection, just like range indexes.
  • Validation: Added namespace of root element of parsed document to validation report.
  • Fixed a BTree bug which led to stack overflows. It seems this could have caused db corruptions in some cases.
  • Added new function , which uses the neko HTML parser to parse an HTML string into well-formed XML.
  • Small modifications in function to avoid another NullPointerException.
  • Fixed NullPointerException in DocumentImpl.
  • Added function :string?) to parse a string value into an XML fragment.
  • webDAV: extended DAV header: webdav version 1 and 2 are now reported, OPTIONS: added method DELETE, COPY, MOVE
  • Fixed ConcurrentModificationException in Collection, which was leading to a deadlock.
  • Extended logging regarding webdav.
  • Fixed NPE during shutdown.
  • Misc validation improvements
  • will have extra attribute 'repeat' when error message is reported more than once
  • Fixed: executing query the first time included all documents in the database, though it was executed in the context of a single collection.
  • Fixed bug 1691177 leading to "attribute X has no parent" error.
  • Fixed bug 1691177 leading to "attribute X has no parent" error.
  • Introduced FODC0002 error in
  • Introduced a new function that will be needed by the XQTS app.
  • Deprecated in favour of
  • Deprecated in favour of
  • Improve performance of CollectionConfigurationManager by replacing XmldbURI with CollectionURI.
  • Fixed Wrong illegal type error message
  • Fixed locking issue: collection was modified without being locked.
  • Major redesign with respect to collection configurations (to fix an existing memory leak)
  • Fixed remaining bugs in in-scope-prefixes() and namespace-uri()
  • Fixed 'string' cast as
  • Improved which now passes all the XQTS tests.
  • Experimental: added a "" pragma which can be used to force eXist to not use the structural index to select nodes.
  • Applying SF Patch 1768921 - fix for SQL Extension module for properly closing JDBC connections, recordsets and statements.
  • More deadlock issues fixed. Better deadlock detection: identifies circular wait dependencies and throws a DeadlockException if automatic resolution is not possible.
  • extended to append fragments of different node types.
  • Update for webdav inferface: fixed NPE for UNLOCK method, MacOsX fix: allow storage of 0 byte XML documents (xml place holder)
  • Added NGram functions and
  • Corrected free memory computation: reserve at least 20% of the max. memory for emergency.
  • Applied New function: Deep-Copy ; Scott Warren - ocomsoft
  • Partly implemented propose borrowing Ant's patch.xml and check.xml
  • Applied LDAP passwords patch ; Submitted By andrewuwe
  • Applied patch Control number of backup files ; Submitted By: Gary Larsen
  • Applied Fix for validation against Schemas using a namespace
  • Fixed potential ArrayIndexOutOfBounds exception.
  • Permit storage of binary resources using
  • Webdav-html page: binary file size are rounded
  • Fix for webdav ; collections now browseable again for webbrowsers (
  • Another deadlock fix: detect and handle a circular wait on different resources.
  • Fixed URL Handler shows ERROR in log, but works OK
  • Multiple grammar catalogs should be allowed for validation
  • Fix for WebDAV: webdav interface can now be mounted my MacOsX 'Finder'.
  • Added compression module :
  • Redesigned exist startup scripts (.sh .bat)
  • Added JMX agent to eXist.
  • Another deadlock fix: XPathQueryService.beginProtected/endProtected need to reserve a DBBroker between the two calls.
  • Validation of XML using DTD's fail when using the XMLgrammarPool. Seem to be a xerces issue. added workaround
  • Fixed bug 1687079: "Extension module namespaces missing".
  • Fixed bug 1683636: " fails on function in module". Imported modules are now visible within the eval() function.
  • Applied patch 1739796: REST server patch for text/html mimetype
  • Fixed bug 1740891: looses its context. ModuleContext needs to overwrite XQueryContext.copyContext.
  • Fixed bug 1740880: Element construction with namespace
  • Fixed bug 1740885: ne operator does not work.
  • Validation: implicit/explicit validation share grammar caching
  • Fixed deadlock issues
  • Fixed: NodePool grew without bounds, thus causing memory issues after a while.
  • Made validation report more usable
  • Further updates to the SOAPServer to provide better interoperability
  • Added new extension package org.exist.fluent, an API for databases running in embedded mode that takes advantage of Java 5.0 features.
  • Committing patches by José María Fernández, add's RPC encoded style alongside document literal style.
  • Added indexes: N-gram and Spatial
  • Fixed NPE in serializer.
  • functions now accept instead of less convenient, butter better to the specs
  • Deadlock conditions found and fixed.
  • Revision of validation function. A year of work.... promised in Prague (2006)
  • Fixed long standing issue regarding and
  • Fixed doc(""))
  • Do not enforce XPath 1.0 compatibility.
  • Fixed an NPE on the resolver.
  • Fixed two major flaws in our implementation of document locks.
  • Fixed: min() and max() can not operate on QNames
  • Fixed "&#x3c;" and "&#x3e;" are escaped in comments
  • Fixed NPE in client index tool
  • Fixed Incorrect filename displayed in status ; patch by Gary Larsen
  • Added implementation of custom protocol handler for ;
  • Added: Classes that implement a ring buffer. To be used by upcoming xml validation features and exist protocol handler.
  • webdav: Fixed locking issue in webdav/MKCOL
  • Fix Webstart: missing sTax jar
  • New HTTPModule.
  • Added "add-exist-id=element|all|none" to the available serialization options. T
  • Repaired unwanted dependency with org.apache.commons.pool.PoolableObjectFactory (caused NoClassDefFoundError for OxygenXml).
  • now uses ValueComparison as per the specs
  • Fixed XQuery bug: optimized path step on descendant axis sometimes ignored child nodes! Example: //a where b is a child of a.
  • Upgrade to ant 1.7.0, 4.x and xmlunit 1.1
  • Added
  • Implement according to the specs.
  • Fixed ClassCastExceptions.
  • Fixed possible memory leak in optimizable expressions.
  • Added an option to disable/enable optimizations at runtime: declare option "enable=yes|no";
  • The pragma now also understands an optional "enable" parameter: //((# enable=no#) { foo })
  • Fixed issue with globally declared variables.
  • Small change in XmldbURI to avoid creating URIs too frequently.
  • Installer: ask for admin Password
  • Fixed performance leak: expressions like /nodeA//nodeB were extremely slow.
  • Changed XInclude to be compliant with the specs.
  • Fixed bug in XQuery lexer
  • Handle these corner test cases : ,"") ,"") ,"")
  • Fixed return type of String-derived atomic values.
  • Binary storage format changed!!!!
  • Low-level changes in the B+tree code: use prefix compression for keys stored in leaf pages.
  • Better processing of return type in avg() and sum()
  • Missing EXIST_HOME causes some tests to fail.
  • Fixed: range indexes were not properly updated after removing a node.
  • Image: added
  • Fixed lack of timezone in gMonth string value.
  • Fixed: XQuery parser complained about "in" as element name: &#x3c;root&#x3e;&#x3c;in&#x3e;foo&#x3c;/in&#x3e;&#x3c;/root&#x3e;.
  • Made base-uri() consistent between in-memory and persistent nodes.
  • Fixed client to work in headless mode again
  • XUpdate performance improvments.
  • New: Modularized indexing architecture.
  • Added new function,
  • When executing a query via the REST interface, the base URI property of the query should be set to the collection path specified in the query.
  • Register stax-api-1.0.1.jar which is required by all Java versions &#x3c; 1.6
  • Implemented java.xml.stream.XMLStreamReader for fast DOM traversals in embedded mode.
  • Fixed Startup scripts fail under solaris
  • Applied patch by Christian Salm: NPE in NativeBroker when recovery is turned off.
  • Fixed left hand operand of a general comparison involves nodes that do not belong to the context sequence's document set.
  • Optimize now works on multi-predicate expressions
  • Fixed: the SOAP interface refused to accept any query containing an ampersand, e.g. //foo
  • Optimize
  • Applied patch users' groups eXtensions by Massimo Martinelli, Marco Tampucci, and Ovidio Salvetti
  • Updated Java service wrapper.jar to version 3.2.3.
  • Set HTTP status if XQuery source file could not be found.
  • New combined range index on named element or attribute.
  • Added generator, Updated webdav and atom code to use new UUIDgen classes.
  • Fixed: resolver.jar not found by Cocoon-block build.
  • We only need one Base64 encoder/decoder in eXist. Favour org.exist.util.Base64Encoder
  • Fixed function without argument (used to throw ArrayIndexOutOfBounds).
  • During backup, ignore already existing contents.xml files in the database.
  • Fixed NPE in fulltext query.
  • Fixed: the xmlrpc interface (via methods upload/parseLocal) changed the mime-type of a resource to application/octet-stream if it could not determine the mime-type from the file name.
  • Image: Added a scale() function.
  • Applied patch: Search classpath for eXist's home
  • Applied patches (16289750/1628975) Ststaus generator; by Gary Larsen - Cache usage info for StatusGenerator Displays stats and graph of cache size usage. Also modified some graph colors so that buffer hits are green (green is good) and a glance can spot problems.
  • Added file format version id to symbols.dbx to inform user of changes in the binary storage format.
  • Fixed QName serialization bug.
  • Atom: added support for getting atom entries by id
  • Webstart: Updated asocat/pack200 compression library.
  • Fixed: fulltext lookups with wildcards (e.g. para &#x26;= "sinnlich") were partially broken.
  • Fixed slow queries on //
  • Fixed match-count behaviour
  • Fixed: support for multiple unary operators, e.g. 7, +-9
  • Fixed nested document construction bug:
  • Fixed string representation of in-memory attributes : let attr := attribute d { "xxx" } return string(attr)
  • Fix for HTML: cardinality issue in
  • webDAV improved error messages on 'PUT'
  • Fixed class cast exception in : let x := 1.1 return x div
  • should return false if the document exists but is not a binary resource.
  • Added dedicated performance test suite. Call with "build.sh benchmark", output goes into test/benchmark.
  • Fixed NPE in CollectionCache.
  • Fix for decoding resources names in webadmin interface
  • Improved string representation of decimal values (XQTS)
  • Changed to accept an empty sequence as start string
  • Added some tests for fulltext index by qname.
  • Added commons daemon code
  • Fixed: binary resource could not be removed via the web interface
  • , and should work with binary resources as well
  • Bug fix: acquired lock not properly released in functions and
  • Applied fix for HTML admin: size of binary docs too accurate
  • Applied add getServletContext to HttpSessionWrapper
  • Log any exceptions whilst sending SMTP mail and fix the truncated subject encoding problem
  • Fix NPE in NativeValueIndex.
  • xmlrpc: parseLocal should return 'true' when document is successfully inserted
  • Updated to XForms Filter 1.2 a lighter distribution, and updated to the latest version of Chiba
  • Improve performance of self axis queries in
  • The collection cache is now adaptive, which means it can grow under load. The limit is set in conf.xml.
  • Smaller performance improvements in node sets.
  • Bug 1589927: Fixed NPEs in quantified expressions with specified sequence type.
  • Webstart: should not throw exception/500 for incorrect filename extension
  • Webstart: Make webstart jar-download more robust
  • Webstart: Add DateHeader to jnlp download servlet
  • Make (# work with other axes than child.
  • Fixed a critical bug leading to failures during crash recovery, usually followed by a data corruption.
  • XMLRPC: getDocumentData/getNextChunk could not handle binary documents.
  • Small performance improvement (to speed up document removal).
  • Updated commons httpclient to 3.0.1
  • New optimization pragma (# #). Can currently be used in combination with a named fulltext index (see above) and the fulltext operators/functions &#x26;=, |= and near().
  • now returns an empty sequence instead of an empty string. Not compliant with the specs but compliant with the XQTS and Saxon.
  • Allowed user to switch authenticated user for the session if they re-send the Basic auth header
  • Bug: Nodes not properly update in fulltext index after deletion. Wrong node id comparison.
  • Added gev's profile pragma.
  • Moved jars to lib/user.
  • Added back service daemon class for commons daemon startup
  • Fixed: When an HTML file is inserted into eXist, it is converted to XML first using NekoHTML
  • Make sure the Scheduler is shutdown when shutting down eXist
  • Implemented
  • Added new standard function : normalize-unicode() (relies on the ICU4J library)
  • Fixed "he",2) which should return "2", not an empty string.
  • Webstart: added support for Pack200/gzip shrinked jar files.
  • deprecated doctype() in the fn namespace. Moved it to the util NS...
  • and are deprecated. Use instead.
  • Fixed: in an enclosed expression, an element was sometimes added as text if the previous node was a text node.
  • Changed the type of maxAge to (instead of int seconds) for
  • and failed to work with a relative URL for the collection, e.g. "/db", "admin", ""). Fixed.
  • Applied patch provided by Robert Koberg - Rest Authentication ; Also added code to store an authenticated user in the session
  • Patch 1588840 applied to the SOAP Server. Allow's XML Fragments. Fixes SOAP-ENV prefix. Fixes NPE when requested XQWS doesnt exist. Thank you José María Fernández
  • Added
  • Implemented codepoint-equal() function
  • Addressed range-indexed attributes.
  • Improved error message
  • Reviewed the way arithmetic operations (division) are handled.
  • Fixed memory leak: byte buffers for caching index data would never shrink. Limit those buffers to 64K max.
  • Improved the design of some fulltext related functions
  • Cleaned code in GeneralComparison
  • Added "group by" patch from Boris Verhaegen
  • Made code more robust in min()/max()
  • Enforce param check in
  • Fixed a few local namespace-uri issues that were reported by the XQTS.
  • Added pragma
  • Added more genericity in code.
  • Fixed: inserting a CDATA node via an XQuery update command led to an exception
  • Added ability to schedule with Cron timings and added an UserXQueryJob for scheduling XQuery jobs.
  • Added an extension module for performing scheduler related functions
  • Bug fix: overwriting a binary resource with an XML document led to a database corruption.
  • Introduced new Scheduler based on Quartz
  • Added function
  • Move jUnit tests to separate code tree ; additional changes will follow
  • Fixed: In-memory CDATA were not serialized at all
  • Added a new method inside xmlrpc isInServiceMode
  • Updated Javadocs.
  • Added new "service mode": a dba admin user can now switch the database into a protected service mode.
  • Fixed a few bugs in poisition().
  • Added BackupSystemTask: running the backup as a system task guarantees a consistent backup.
  • Added XQuery function that can be used to trigger arbitrary system tasks.
  • Backup can now write to a .zip file instead of the file system.
  • Interactice client: Added Bokmål translation by Trond Albinussen

Started on 4-11-2006 (rev4787)