The Python bindings for Xapian are packaged in the xapian
module,
and largely follow the C++ API, with the following differences and
additions. Python strings and lists, etc., are converted automatically
in the bindings, so generally it should just work as expected.
The examples
subdirectory contains examples showing how to use the
Python bindings based on the simple examples from xapian-examples
:
simpleindex.py,
simplesearch.py,
simpleexpand.py.
There's also
simplematchdecider.py
which shows how to define a MatchDecider in Python.
The Python bindings come with a test suite, consisting of two test files:
smoketest.py
and pythontest.py
. These are run by the
"make check
" command, or may be run manually. By default, they
will display the names of any tests which failed, and then display a count of
tests which run and which failed. The verbosity may be increased by setting
the "VERBOSE
" environment variable: a value of 1 will display
detailed information about failures, and a value of 2 will display further
information about the progress of tests.
Xapian exceptions are translated into Python exceptions with the same names
and inheritance hierarchy as the C++ exception classes. The base class of
all Xapian exceptions is the xapian.Error
class, and this in
turn is a child of the standard python exceptions.Exception
class.
This means that programs can trap all xapian exceptions using "except
xapian.Error
", and can trap all exceptions which don't indicate that
the program should terminate using "except Exception
".
The xapian Python bindings accept unicode strings as well as simple strings (ie, "str" type strings) at all places in the API which accept string data. Any unicode strings supplied will automatically be translated into UTF-8 simple strings before being passed to the Xapian core. The Xapian core is largely agnostic about character set, but in those places where it does process data in a character set dependent way it assumes that the data is in UTF-8. The Xapian Python bindings always return string data as simple strings.
Therefore, in order to avoid issues with character sets, you should always
pass text data to Xapian as unicode strings, or UTF-8 encoded simple
strings. There is, however, no requirement for simple strings passed into
Xapian to be valid UTF-8 encoded strings, unless they are being passed to a
text processing routine (such as the query parser, or the stemming
algorithms). For example, it is perfectly valid to pass arbitrary binary
data in a simple string to the xapian.Document.set_data()
method.
It is often useful to normalise unicode data before passing it to Xapian -
Xapian currently has no built-in support for normalising unicode
representations of data. The standard python module
"unicodedata
" provides support for normalising unicode: you
probably want the "NFKC
" normalisation scheme: in other words,
use something like
unicodedata.normalize('NFKC', u'foo')
to normalise the string "foo" before passing it to Xapian.
All iterators support next()
and equals()
methods
to move through and test iterators (as for all language bindings).
MSetIterator and ESetIterator also support prev()
.
Python-wrapped iterators also support direct comparison, so something like:
m=mset.begin() while m!=mset.end(): # do something m.next()
C++ iterators are often dereferenced to get information, eg
(*it)
. With Python these are all mapped to named methods, as
follows:
Iterator | Dereferencing method |
PositionIterator | get_termpos() |
PostingIterator | get_docid() |
TermIterator | get_term() |
ValueIterator | get_value() |
MSetIterator | get_docid() |
ESetIterator | get_term() |
Other methods, such as MSetIterator.get_document()
, are
available unchanged.
Many classes that support C++-style iterators also support Pythonic
iterators which do the same thing in a Python style. The following are
supported (where marked as default iterator, it means __iter__() does the right
thing so you can for instance use for term in document
to
iterate over terms in the Document):
Class | Method | Equivalent to | Iterator type |
MSet | default iterator | begin() | MSetIter |
ESet | default iterator | begin() | ESetIter |
Enquire | matching_terms() | get_matching_terms_begin() | TermIter |
Query | default iterator | get_terms_begin() | TermIter |
Database | allterms() | allterms_begin() (also as default iterator) | TermIter |
Database | postlist(tname) | postlist_begin(tname) | PostingIter |
Database | termlist(docid) | termlist_begin(docid) | TermIter |
Database | positionlist(docid, tname) | positionlist_begin(docid, tname) | PositionIter |
Document | values() | values_begin() | ValueIter |
Document | termlist() | termlist_begin() (also as default iterator) | TermIter |
QueryParser | stoplist() | stoplist_begin() | TermIter |
QueryParser | unstemlist(tname) | unstem_begin(tname) | TermIter |
The Pythonic iterators used to return lists representing the appropriate item when their next()
method is called, except PositionIter which just returns a single value. They now return Python objects, allowing lazy evaluation of properties where appropriate, and allowing a more pythonic access to attribute values. The sequence API is being maintained for now, so existing code should continue to work, but we recommend transitioning to the new API. The sequence API is planned to be removed at release 1.1.0 of Xapian.
Class | Returns |
MSetIter | [docid, weight, rank, percentage, document] |
ESetIter | [term, weight] |
TermIter | [term, wdf, termfreq, position iterator] |
PostingIter | [docid, doclength, wdf, position iterator] |
PositionIter | termpos |
ValueIter | [valueno, value] |
The lazy evaluation is mainly transparent, but does become visible in one situation: if you keep an object returned by an iterator, without evaluating its properties to force the lazy evaluation to happen, and then move the iterator forward, the object may no longer be able to efficiently perform the lazy evaluation. In this situation, an exception will be raised indicating that the information requested wasn't available. This will only happen for a few of the properties - most are either not evaluated lazily (because the underlying Xapian implementation doesn't evaluated them lazily, so there's no advantage in lazy evaluation), or can be accessed even after the iterator has moved. The simplest work around is simply to evaluate any properties you wish to use which are affected by this before moving the iterator. The complete set of iterator properties affected by this is:
MSet objects have some additional methods to simplify access (these work using the C++ array dereferencing):
Method name | Explanation |
get_hit(index) | returns MSetIterator at index |
get_document_percentage(index) | convert_to_percent(get_hit(index)) |
get_document(index) | get_hit(index).get_document() |
get_docid(index) | get_hit(index).get_docid() |
Additionally, the MSet has a property, mset.items
, which returns a
list of tuples representing the MSet; this may be more convenient than using
the MSetIterator. The members of the tuple are as follows.
Index | Contents |
xapian.MSET_DID | Document id |
xapian.MSET_WT | Weight |
xapian.MSET_RANK | Rank |
xapian.MSET_PERCENT | Percentage weight |
Two MSet objects are equal if they have the same number and maximum possible number of members, and if every document member of the first MSet exists at the same index in the second MSet, with the same weight.
The ESet has a property, eset.items
, which returns a list of
tuples representing the ESet; this may be more convenient than using the
ESetIterator. The members of the tuple are as follows.
Index | Contents |
xapian.ESET_TNAME | Term name |
xapian.ESET_WT | Weight |
Xapian::Auto::open_stub()
is wrapped as xapian.open_stub()
Xapian::Quartz::open()
is wrapped as xapian.quartz_open()
Xapian::InMemory::open()
is wrapped as xapian.inmemory_open()
Xapian::Remote::open()
is wrapped as xapian.remote_open()
(only
the TCP version is currently wrapped, the "program" version isn't).
In C++ there's a Xapian::Query constructor which takes a query operator and start/end iterators specifying a number of terms or queries, plus an optional parameter. In Python, this is wrapped to accept any Python sequence (for example a list or tuple) to give the terms/queries, and you can specify a mixture of terms and queries if you wish. For example:
subq = xapian.Query(xapian.Query.OP_AND, "hello", "world") q = xapian.Query(xapian.Query.OP_AND, [subq, "foo", xapian.Query("bar", 2)])
There is an additional method get_matching_terms()
which takes
an MSetIterator and returns a list of terms in the current query which
match the document given by that iterator. You may find this
more convenient than using the TermIterator directly.
Custom MatchDeciders can be created in Python; simply subclass xapian.MatchDecider, ensure you call the super-constructor, and define a __call__ method that will do the work. The simplest example (which does nothing useful) would be as follows:
class mymatchdecider(xapian.MatchDecider): def __init__(self): xapian.MatchDecider.__init__(self) def __call__(self, doc): return 1Last updated $Date: 2007-05-17 22:12:04 +0100 (Thu, 17 May 2007) $