\documentclass{howto}

% $Id: xml-howto.tex,v 1.8 2001/10/25 17:26:19 akuchling Exp $

% XXX not covered: c14n.py, xml.marshal, scripts, namespaces, TREX/schemas,
% XSLT, XPath
% XXX overview of parsers

\newcommand{\element}[1]{\code{#1}}
\newcommand{\attribute}[1]{\code{#1}}

\title{Python/XML HOWTO}

\release{0.6.1}

\author{A.M. Kuchling}
\authoraddress{\email{akuchlin@mems-exchange.org}}

\begin{document}
\maketitle

\begin{abstract}
\noindent
XML is the eXtensible Markup Language, a subset of SGML intended to
allow the creation and processing of application-specific markup
languages.  Python makes an excellent language for processing XML
data.  This document is a tutorial for the Python/XML package.  It
assumes you're already familiar with the structure and terminology of
XML.
\end{abstract}

\tableofcontents

\section{Introduction to XML}

XML, the eXtensible Markup Language, is a simplified dialect of SGML,
the Standardized General Markup Language.  XML is intended to be
reasonably simple to implement and use, and is already being used for
specifying markup languages for various new standards: MathML for
expressing mathematical equations, Synchronized Multimedia Integration
Language for multimedia presentations, and so forth.

SGML and XML represent a document by tagging the document's various
components with their function or meaning.  For example, a book
contains several parts: it has a title, one or more authors, the text
of the book, perhaps a preface or an index, and so forth.  A markup
languge for writing books would therefore have elements indicating
what the contents of the preface are, what the title is, and so forth.
This should not be confused with the physical details of how the
document is actually printed on paper.  The index might be printed
with narrow margins in a smaller font than the rest of the book, but
markup usually isn't (or shouldn't be, anyway) concerned with details
such as this.  Instead, other software will translate from the markup
language to a typesetting language such as \TeX, handling the
presentation details.

This section will provide a brief overview of XML and a few related
standards, but it's far from being complete because making it complete
would require a full-length book and not a short HOWTO. There's no
better way to get a completely accurate description than to read the
original W3C Recommendations; you can find links to them in
section~\ref{xml-links}, ``Related Links''.  If you already know what
XML is, you can skip the rest of this section.

Later sections of this HOWTO assume that you're familiar with XML
terminology.  Most sections will use XML terms such as \emph{element}
and \emph{attribute}.  Section~\ref{SAX} does not require that you
have experience with the Java SAX implentations.


\subsection{Elements, Attributes and Entities}

A markup language specified using XML looks a lot like HTML; a
document consists of a single \dfn{element}, which contains
sub-elements, which can have further sub-elements inside them.
Elements are indicated by \dfn{tags} in the text.  Tags are always
inside angle brackets \code{<}~\code{>}.  Elements can either contain content, or they can be empty:

\begin{itemize}

\item   An element can contain content between opening and closing
tags, as in \code{<name>Euryale</name>}, which is a \element{name}
element containing the data \samp{Euryale}. This content may be text
data, other XML elements, or a mixture of both. 

\item Elements can also be empty, containing nothing, and are
represented as a single tag ended with a slash.  For example,
\code{<stop/>} is an empty \element{stop} element.  Unlike HTML, XML
element names are case-sensitive; \element{stop} and \element{Stop}
are two different elements.

\end{itemize}

Opening and empty tags can also contain attributes, which specify
values associated with an element.  For example, in the XML text
\code{<name lang='greek'>Herakles</name>}, the \element{name} element
has a \attribute{lang} attribute which has a value of \samp{greek}.
Contrast this with \code{<name lang='latin'>Hercules</name>}, where
the attribute's value is \samp{latin}.

XML also includes \dfn{entities} as a shorthand for including a
particular character or a longer string.  Entity references always
begin with a \samp{\&} and end with a \samp{;}.  For example, a
particular Unicode character can be written as \code{\&\#4660;} using
its character code in decimal, or as \code{\&\#x1234;} using
hexadecimal.  It's also possible to define your own entities, making
\code{\&title;} expand to ``The Odyssey'', for example.  If you want to
include the \samp{\&} character in XML content, it must be written as
\code{\&amp;}.

\subsection{Well-Formed XML}

A legal XML document must, as a minimum, be \dfn{well-formed}: each
open tag must have a corresponding closing tag, and tags must nest
properly.  For example, \code{<b><i>text</b></i>} is not well-formed
because the \element{i} element should be enclosed inside the
\element{b} element, but instead the closing \code{</b>} tag is
encountered first.  This example can be made well-formed by swapping
the order of the closing tags, resulting in \code{<b><i>text</i></b>}.

If you've ever written HTML by hand, you may have acquired the habit
of being a bit sloppy about this.  Strictly speaking, HTML has exactly
the same rules about nesting tags as XML, but most Web browsers are
very forgiving of errors in HTML.  This is convenient for HTML
authors, but it makes it difficult to write programs to parse HTML
input, because the programs have to cope with all sorts of malformed
input.

The authors of the XML specification didn't want XML to fall into the
same trap, because it would make XML processing software much harder
to write.  Therefore, all XML parsers have to be strict and must
report an error if their input isn't well-formed.  The Expat parser
includes an executable program named \program{xmlwf} that parses the
contents of files and reports any well-formedness violations; it's
very handy for checking XML data that's been output from a program or
written by hand.


\subsection{DTDs}

Well-formedness just says that all tags nest properly and that every
opening tag is matched by a closing tag.  It says nothing about the
order of elements or about which elements can be contained inside other
elements.

The following XML, apparently representing a book, is well-formed but
it makes no logical sense:

\begin{verbatim}
<book>
  <index>  ... </index>
  <chapter> ... </chapter>
  <chapter> ... </chapter>
  <abstract>  ... </abstract>
  <chapter> ... </chapter>
  <preface> .... </preface>
</book>
\end{verbatim}

Prefaces don't come at the end of books, the index doesn't belong at
the front, and the abstract doesn't belong in the middle.
Well-formedness alone doesn't provide any way of enforcing that order.
You could write a Python program that took an XML file like this and
checked whether all the parts are in order, but then someone wanting
to understand what documents are legal would have to read your program.

Document Type Definitions, or \dfn{DTDs} for short, are a more concise
way of enforcing ordering and nesting rules. A DTD declares the
element names that are allowed, and how elements can be nested inside
each other.  To take an example from HTML, the \element{LI} element,
representing an entry in a list, can only occur inside certain
elements which represent lists, such as \element{OL} or \element{UL}.

The DTD also specifies the attributes that can be provided for each
element, the default value for each attribute, and whether the
attribute can be omitted.  A \dfn{validating parser} can take a
document and a DTD, and check whether the document is legal according
to the DTD's rules.

Note that it's quite possible to get useful work done without using a
validating parser and writing a DTD.  You might decide that just
writing well-formed XML and checking it with a Python program is all
you need.  

A DTD lists the supported elements, the order in which elements must
occur, and the possible attributes for each element.  Here's a
fragment from an imaginary DTD for writing books:

\begin{verbatim}
<!ELEMENT book (abstract?, preface, chapter*, appendix?)>
<!ELEMENT abstract ...>
<!ELEMENT chapter ...>
<!ATTLIST chapter id    ID    #REQUIRED 
                  title CDATA #IMPLIED>
\end{verbatim}

The first line declares the \element{book} element, and specifies the
elements that can occur inside it and the order in which the
subelements must be provided.  DTDs borrow from regular expression
notation in order to express how elements can be repeated; \samp{?}
means an element must occur 0 or 1 times, \samp{*} is 0 or more times,
and \samp{+} means the element must occur 1 or more times.  For
example, the \element{abstract} and \element{appendix} elements are
optional inside a \element{book} element.  Exactly one
\element{preface} element has to be present, and it can be followed by
any number of \element{chapter} elements; having no chapters at all
would be legal.

The \code{ATTLIST} declaration specifies attributes for the
\element{chapter} element.  Chapters can have two attributes,
\attribute{id} and \attribute{title}.  \attribute{title} contains
character data (CDATA) and is optional (that's what \samp{\#IMPLIED}
means, for obscure historical reasons).  \attribute{id} must contain
an ID value, and it's required and not optional.  

A validating parser could take this DTD and a sample document, and
report whether the document is \dfn{valid} according to the rules of
the DTD.  A document is valid if all the elements occur in the right
order, and in the right number of repetitions.

\subsection{Related Links}
\label{xml-links}

For the full details of XML's syntax, the definitive source is the XML
1.0 specification, available on the Web at
\url{http://www.w3.org/TR/xml-spec.html}.  However, like all
specifications it's quite formal and isn't intended to be a friendly
introduction or a tutorial.  The annotated version of the standard, at
\url{http://www.xml.com/xml/pub/a/axml/axmlintro.html}, is quite helpful
in clarifying the specification's intent.  There are also many more
informal tutorials and books available to introduce you to XML at
greater length.

The XML Cover Pages, at \url{http://xml.coverpages.org}, are an
extensive collection of links to XML and SGML resources, including a
news page that's updated every few days.  If you can only remember one
XML-related URL, remember this one.  Cafe con Leche, 
at \url{http://www.ibiblio.org/xml/}, is another good resource.

The xml-dev mailing list is a high-traffic list for implementation and
development; see \url{http://www.xml.org/xml/xmldev.shtml} for
archives and subscription information.  Be warned: Some people might
find the discussion too focused on inventing new standards and tools,
not on applying existing standards.  


\section{Related Standards}

XML 1.0 is the basic standard, but people have built many, \emph{many}
additional standards and tools on top of XML or to be used with XML.
This section will quickly introduce some of these related
technologies, paying particular attention to those that are supported
by the Python/XML package.

%\subsection{XPath and XPointer}

%XXX write section on XPath and XPointer

\subsection{XSLT}

XML documents are often transformed from one format to another.  These
transformations can be minor, such as changing all \element{OL}
elements into \element{UL} elements, or major, such as translating a
DocBook document into HTML so it can be displayed in a Web browser.
You can write a separate Python program to do each transformation as
you need it, and at times that will be the most appropriate option,
but an alternative approach is to use XSL, the Extensible Stylesheet
Language.

XSL is really two standards: XSLT, XSL Transformations; and XSL-FO,
XSL Formatting Objects.  XSLT is used much more often than XSL-FO,
because XSL-FO is intended primarily for rendering XML for printing
onto paper.  XSLT is a general tool for transforming one XML document
into another document, and therefore can be used for more diverse
tasks.

To use XSLT, you have to write a \dfn{stylesheet}, which is itself an
XML document written in the XSLT DTD.  The source document is turned
into a tree structure, and the stylesheet specifies the transformation
you want to perform by selecting some elements from the tree and
rearranging them.  

The Python/XML package includes 4XSLT, an XSLT processor written by
Fourthought, Inc., letting you write Python programs that apply an
XSLT stylesheet to a document.

\subsubsection{Related Links}

The W3C's XSL page is at \url{http://www.w3.org/Style/XSL/}, and links
to the XSLT specifications and to friendlier tutorials.


%\subsection{XML Schemas}

% XXX write a section on XSchema


%\subsection{RDF}

% XXX write a section on RDF; point users toward Redfoot.


\section{Installing the XML Toolkit}

Releases are available from
\url{http://sourceforge.net/projects/pyxml/}.
Windows users should download the appropriate precompiled version.
Linux users can either download an RPM, or install from source.  Users
on other platfoms have no choice but to install from source.

To compile from source on a \UNIX{} platform, simply perform the
following steps.

\begin{enumerate}

\item Download the latest version of the source distribution from
\url{http://sourceforge.net/projects/pyxml}.  Unpack it with the
following command.

\begin{verbatim}
gzip -dc xml-package.tgz | tar -xvf -
\end{verbatim}

\item Run \code{python setup.py install}.  In order to run this,
you'll need to have a C compiler installed, and it should be the same
one that was used to build your Python installation. On a Unix system,
this operation may require superuser permissions. \code{setup.py}
supports a number of different commands and options; invoke
\code{setup.py} without any arguments to see a help message.

\end{enumerate}

If you're using Python 1.5, you'll have to install the Distutils
first, available from
\url{http://www.python.org/sigs/distutils-sig}. The PyXML package
still works with Python 1.5.2, but the Unicode support is much less
powerful.  Versions of Python after 1.5.2 added very complete Unicode
support, so if you're going to be doing serious XML work you should
use the latest version of Python 2.x.  At this writing, the latest
released version is Python 2.1.

If you have difficulty installing this software, send a problem report
to the XML-SIG mailing list describing the problem, or submit a bug report
at \url{http://sourceforget.net/projects/pyxml}.

There are various demonstration programs in the \file{demo/} directory
of the Python/XML source distribution.  You may wish to look at them
to get an idea of what's possible with the XML tools, and as a source
of example code.


\subsection{Related Links}

The Python/XML Topic Guide, at
\url{http://pyxml.sourceforge.net/topics/} is the starting point for
Python-related XML topics; it links to software, mailing lists,
documentation, etc.

\section{SAX: The Simple API for XML}
\label{SAX}

The Simple API for XML isn't a standard in the formal sense that XML
or ANSI C are.  Rather, SAX is an informal specification originally
designed by David Megginson with input from many people on the xml-dev
mailing list.  SAX defines an event-driven interface for parsing XML.
To use SAX, you must create Python class instances which implement a
specified interface, and the parser will then call various methods on
those objects.

This HOWTO describes version 2 of SAX (also referred to as
SAX2). Earlier versions of this text explained SAX1, which is now only
of historical interest.

SAX is most suitable for purposes where you want to read through an
entire XML document from beginning to end, and perform some
computation such as building a data structure or summarizing the
contained information (computing an average value of a certain
element, for example).  SAX is not very convenient if you want to
modify the document structure by changing how elements are nested,
though it would be straightforward to write a SAX program that simply
changed element contents or attributes.  For example, you wouldn't
want to re-order chapters in a book using SAX, but you might want to
extract the contents of all \element{name} elements with the attribute
\attribute{lang} set to 'greek'.

One advantage of SAX is speed and simplicity.  Let's say
you've defined a complicated DTD for listing comic books, and you wish
to scan through your collection and list everything written by Neil
Gaiman.  For this specialized task, there's no need to expend effort
examining elements for artists and editors and colourists, because
they're irrelevant to the search.  You can therefore write a class
instance which ignores all elements that aren't \element{writer}.

Another advantage of SAX is that you don't have the whole document
resident in memory at any one time, which matters if you are
processing really huge documents.  

SAX defines 4 basic interfaces. A SAX-compliant XML parser can be
passed any objects that support these interfaces, and will call
various methods as data is processed.  Your task, therefore, is to
implement those interfaces that are relevant to your application.

The SAX interfaces are:

\begin{tableii}{c|p{4in}}{code}{Interface}{Purpose}

\lineii{ContentHandler}{Called for general document events.  This
interface is the heart of SAX; its methods are called for the start of
the document, the start and end of elements, and for the characters of
data contained inside elements.
}

\lineii{DTDHandler}{Called to handle DTD events required for basic
parsing.  This means notation declarations (XML spec section 4.7) and
unparsed entity declarations (XML spec section 4).
}

\lineii{EntityResolver}{Called to resolve references to external
entities.  If your documents will have no external entity references,
you don't need to implement this interface.}

\lineii{ErrorHandler}{Called for error handling.  The parser will call
methods from this interface to report all warnings and errors.}

\end{tableii}

Python doesn't support the concept of interfaces, so the interfaces
listed above are implemented as Python classes.  The default method
implementations are defined to do nothing---the method body is just a
Python \code{pass} statement---so usually you can simply ignore methods
that aren't relevant to your application. 

Pseudo-code for using SAX looks something like this:
\begin{verbatim}
# Define your specialized handler classes
from xml.sax import ContentHandler, ...
class docHandler(ContentHandler):
    ...

# Create an instance of the handler classes
dh = docHandler()

# Create an XML parser
parser = ...

# Tell the parser to use your handler instance
parser.setContentHandler(dh)

# Parse the file; your handler's methods will get called
parser.parse(sys.stdin)
\end{verbatim}

\subsection{Starting Out}

Let's follow the earlier example of a comic book collection, using a
simple DTD-less format. Here's a sample document for a collection
consisting of a single issue:

\begin{verbatim}
<collection>
  <comic title="Sandman" number='62'>
    <writer>Neil Gaiman</writer>
    <penciller pages='1-9,18-24'>Glyn Dillon</penciller>
    <penciller pages="10-17">Charles Vess</penciller>
  </comic>
</collection>
\end{verbatim}

An XML document must have a single root element; this is the
\samp{collection} element.  It has one child \element{comic} element
for each issue; the book's title and number are given as attributes of
the \element{comic} element.  The \element{comic} element can in turn
contain several other elements such as \element{writer} and
\element{penciller} listing the writer and artists responsible for the
issue.  There may be several artists or writers for a single issue.

Let's start off with something simple: a document handler named
\class{FindIssue} that reports whether a given issue is in the
collection.

\begin{verbatim}
from xml.sax import saxutils

class FindIssue(saxutils.DefaultHandler):
    def __init__(self, title, number):
        self.search_title, self.search_number = title, number
\end{verbatim}

The \class{DefaultHandler} class inherits from all four interfaces:
\class{ContentHandler}, \class{DTDHandler}, \class{EntityResolver},
and \class{ErrorHandler}.  This is what you should use if you want to
just write a single class that wraps up all the logic for your
parsing.  You could also subclass each interface individually and
implement separate classes for each purpose.  Neither of the two
approaches is always ``better'' than the other; mostly it's a matter
of taste.

Since this class is doing a search, an instance needs to know what it's searching for.  The desired title and issue number are passed to the
\class{FindIssue} constructor, and stored as part of the instance.

Now let's override some of the parsing methods.
This simple search only requires looking at the attributes of a given
element, so only the \method{startElement} method is relevant.

\begin{verbatim}
    def startElement(self, name, attrs):
        # If it's not a comic element, ignore it
        if name != 'comic': return

        # Look for the title and number attributes (see text)
        title = attrs.get('title', None)
        number = attrs.get('number', None)
        if (title == self.search_title and 
	    number == self.search_number):
            print title, '#' + str(number), 'found'
\end{verbatim}

The \method{startElement()} method is passed a string giving the name
of the element, and an instance containing the element's attributes.
Attributes are accessed using 
methods from the \class{AttributeList} interface, which
includes most of the semantics of Python dictionaries.  

To summarize, \method{startElement()} function looks for
\element{comic} elements, and compares the specified \attribute{title}
and \attribute{number} attributes to the search values.  If they
match, a message is printed out.

\method{startElement()} is called for every single element in the
document.  If you added \code{print 'Starting element:', name} to the
top of  \method{startElement()}, you would get the following output.

\begin{verbatim}
Starting element: collection
Starting element: comic
Starting element: writer
Starting element: penciller
Starting element: penciller
\end{verbatim}

To actually use the class, we need top-level code that creates
instances of a parser and of \class{FindIssue}, associates the parser
and the handler, and then calls a parser method to process the input.

\begin{verbatim}
from xml.sax import make_parser
from xml.sax.handler import feature_namespaces

if __name__ == '__main__':
    # Create a parser
    parser = make_parser()

    # Tell the parser we are not interested in XML namespaces
    parser.setFeature(feature_namespaces, 0)

    # Create the handler
    dh = FindIssue('Sandman', '62')

    # Tell the parser to use our handler
    parser.setContentHandler(dh)

    # Parse the input
    parser.parse(file)
\end{verbatim}

The \function{make_parser} class can automate the job of creating
parsers.  There are already several XML parsers available to Python,
and more might be added in future.  \file{xmllib.py} is included as
part of the Python standard library, so it's always available, but
it's also not particularly fast.  A faster version of \file{xmllib.py}
is included in \module{xml.parsers}.  The \module{xml.parsers.expat}
module is faster still, so it's obviously a preferred choice if it's
available.  \function{make_parser} determines which parsers are
available and chooses the fastest one, so you don't have to know what
the different parsers are, or how they differ. (You can also tell
\function{make_parser} to try a list of parsers, if you want to use a
specific one).

SAX2 supports XML namespaces.  If namespace processing is active,
parsers won't call \method{startElement()}, but instead will call a
method named \method{startElementNS()}. Since our \class{FindIssue}
content handler doesn't implement the namespace-aware methods, we
request that namespace processing is deactivated.  The default of this
setting varies from parser to parser, so you should always set it to a
safe value (unless your handler supports both namespace-aware and
-unaware processing).

Once you've created a parser instance, calling the
\method{setContentHandler()} method tells the parser what to use as
the content handler.  There are similar methods for setting the other
handlers: \method{setDTDHandler()}, \method{setEntityResolver()}, and
\method{setErrorHandler()}.

If you run the above code with the sample XML document, it'll print
\code{Sandman \#62 found.}  

\subsection{Error Handling}

Now, try running the above code with this file as input:
\begin{verbatim}
<collection>
  &foo;
  <comic title="Sandman" number='62'>
</collection>
\end{verbatim}

The \code{\&foo;} entity is unknown, and the \element{comic} element
isn't closed (if it was empty, there would be a \samp{/} before the
closing \samp{>}. As a result, you get a
\exception{SAXParseException}, e.g.

\begin{verbatim}
xml.sax._exceptions.SAXParseException: undefined entity at None:2:2
\end{verbatim}

The default code for the \class{ErrorHandler} interface automatically
raises an exception for any error; if that is what you want, you don't
need to implement an error handler class at all.  Otherwise, you can
provide your own version of the \class{ErrorHandler} interface, at
minimum overriding the \method{error()} and \method{fatalError()}
methods.  The minimal implementation for each method can be a single
line.  The methods in the \class{ErrorHandler}
interface---\method{warning()}, \method{error()}, and
\method{fatalError()}---are all passed a single argument, an exception
instance.  The exception will always be a subclass of
\exception{SAXException}, and calling \code{str()} on it will produce
a readable error message explaining the problem.

For example, if you just want to continue running if a recoverable
error occurs, simply define the \method{error()} method to print the
exception it's passed:

\begin{verbatim}
    def error(self, exception):
        import sys
        sys.stderr.write("\%s\n" \% exception)
\end{verbatim}

With this definition, non-fatal errors will result in an error message,
whereas fatal errors will continue to produce a traceback.

\subsection{Searching Element Content}

Let's tackle a slightly more complicated task: printing out all issues
written by a certain author.  This now requires looking at element
content, because the writer's name is inside a \element{writer}
element: \code{<writer>Peter Milligan</writer>}.

The search will be performed using the following algorithm:

\begin{enumerate}
\item 
The \method{startElement} method will be more complicated.  For
\element{comic} elements, the handler has to save the title and
number, in case this comic is later found to match the search
criterion.  For \element{writer} elements, it sets a
\code{inWriterContent} flag to true, and sets a \code{writerName}
attribute to the empty string.

\item Characters outside of XML tags must be processed.  When
\code{inWriterContent} is true, these characters must be added to the
\code{writerName} string.

\item When the \element{writer} element is finished, we've now
collected all of the element's content in the \code{writerName}
attribute, so we can check if the name matches the one we're searching 
for, and if so, print the information about this comic.  We must also
set \code{inWriterContent} back to false.
\end{enumerate}

Here's the first part of the code; this implements step 1.

\begin{verbatim}
from xml.sax import ContentHandler
import string

def normalize_whitespace(text):
    "Remove redundant whitespace from a string"
    return string.join(string.split(text), ' ')

class FindWriter(ContentHandler):
    def __init__(self, search_name):
        # Save the name we're looking for
        self.search_name = normalize_whitespace(search_name)

        # Initialize the flag to false
        self.inWriterContent = 0

    def startElement(self, name, attrs):
        # If it's a comic element, save the title and issue
        if name == 'comic':
            title = normalize_whitespace(attrs.get('title', ""))
            number = normalize_whitespace(attrs.get('number', ""))
            self.this_title = title
            self.this_number = number

        # If it's the start of a writer element, set flag
        elif name == 'writer':
            self.inWriterContent = 1
            self.writerName = ""
\end{verbatim}

The \method{startElement()} method has been discussed previously.  Now
we have to look at how the content of elements is processed.  

The \function{normalize_whitespace()} function is important, and
you'll probably use it in your own code.  XML treats whitespace very
flexibly; you can include extra spaces or newlines wherever you like.
This means that you must normalize the whitespace before comparing
attribute values or element content; otherwise the comparison might
produce an incorrect result due to the content of two elements having
different amounts of whitespace.

\begin{verbatim}
    def characters(self, ch):
        if self.inWriterContent:
            self.writerName = self.writerName + ch
\end{verbatim}

The \method{characters()} method is called for characters that aren't
inside XML tags.  \var{ch} is a string of characters. It is not
necessarily a byte string; parsers may also provide a buffer object
that is a slice of the full document, or they may pass Unicode
objects (as the expat parser does in Python 2.0).

You also shouldn't assume that all the characters are passed in a
single function call.  In the example above, there might be only one
call to \method{characters()} for the string \samp{Peter Milligan}, or
it might call \method{characters()} once for each character.  Another,
more realistic example: if the content contains an entity reference,
as in \samp{Wagner \&amp; Seagle}, the parser might call the method
three times; once for \samp{Wagner\ }, once for \samp{\&}, represented
by the entity reference, and again for \samp{\ Seagle}.

For step 2 of the algorithm, \method{characters()} only has to
check \code{inWriterContent}, and if it's true, add the characters to
the string being built up.

Finally, when the \element{writer} element ends, the entire name has
been collected, so we can compare it to the name we're searching for.

\begin{verbatim}
    def endElement(self, name):
        if name == 'writer':
            self.inWriterContent = 0
            self.writerName = normalize_whitespace(self.writerName)
            if self.search_name == self.writerName:
                print 'Found:', self.this_title, self.this_number
\end{verbatim}

To avoid being confused by differing whitespace, the
\function{normalize_whitespace()} function is called.  This can be
done because we know that leading and trailing whitespace are
insignificant for this application.  

End tags can't have attributes on them, so there's no \var{attrs}
parameter to the \method{endElement()} method.  Empty elements with
attributes, such as \samp{<arc name="Season of Mists"/>}, will result
in a call to \method{startElement()}, followed immediately by a call
to \method{endElement()}.

XXX how are external entities handled?  Anything special need to be
done for them?

\subsection{Related Links}

The SAX home page is \url{http://www.megginson.com/SAX/}.
This has the most recent copy of the
specification, and lists SAX implementations for various languages and
platforms.  At the moment it's somewhat Java-centric, though.


\section{DOM: The Document Object Model}
\label{DOM}

The Document Object Model specifies a tree-based representation for an
XML document, as opposed to the event-driven processing provided by
SAX.  Both approaches have their uses.

A top-level \class{Document} instance is the root of the tree, and has
a single child which is the top-level \class{Element} instance; this
\class{Element} has child nodes representing the content and any
sub-elements, which may in turn have further children and so forth.
There are different classes for everything that can be found in an XML
document, so in addition to the \class{Element} class, there are also
classes such as \class{Text}, \class{Comment}, \class{CDATASection},
\class{EntityReference}, and so on.  Tree nodes provide methods for
accessing the parent and child nodes, accessing element and attribute
values, insert and delete nodes, and converting the tree back into XML.

The DOM is often useful for modifying XML documents, because you can
create a DOM tree, modify it by adding new nodes and moving subtrees
around, and then produce a new XML document as output.  On the other
hand, while the DOM doesn't require that the entire tree be resident
in memory at one time, the Python DOM implementation currently keeps
the whole tree in RAM.  This means you may not have enough memory to
process very large documents as a DOM tree.  A SAX handler, on the
other hand, can potentially churn through amounts of data far larger
than the available RAM.

\subsection{Getting A DOM Tree}

The easiest way to get a DOM tree is to have it built for you. PyXML
offers two alternative implementations of the DOM,
\module{xml.dom.minidom} and \code{4DOM}. \module{xml.dom.minidom} is
included in Python 2. It is a minimal implementation, which means it
does not provide all interfaces and operations required by the DOM
standard.  \code{4DOM} (XXX reference) is a complete implementation of
DOM Level 2 (which is currently work in progress), so we will use that
in the examples.

One of the modules in the \module{xml.dom} package is
\module{xml.dom.ext.reader.Sax2}, which provides the functions
\function{FromXmlStream}, \function{FromXml}, \function{FromXmlFile},
and \function{FromXmlUrl} which will construct a DOM tree from their
input (a file-like object, a string, a file name, and a URL,
respectively). They all return a DOM \class{Document} object.
% XXX these functions seem to be deprecated!  why?

\begin{verbatim}
import sys
from xml.dom.ext.reader.Sax2 import FromXmlStream

# parse the document
doc = FromXmlStream(sys.stdin)
\end{verbatim}

\subsection{Manipulating The Tree}

This HOWTO can't be a complete introduction to the Document Object
Model, because there are lots of interfaces and lots of
methods. Luckily, the DOM Recommendation is quite readable as
specifications go, so I'd recommend that you read it to get a complete
picture of the available interfaces.  This section will only be a
partial overview.

The Document Object Model represents a XML document as a tree of
nodes, represented by an instance of some subclass of the \class{Node}
class.  Some subclasses of \class{Node} are \class{Element},
\class{Text}, and \class{Comment}.  

We'll use a single example document throughout this section.  Here's
the sample:

\begin{verbatim}
<?xml version="1.0" encoding="iso-8859-1"?>
<xbel>  
  <?processing instruction?>
  <desc>No description</desc>
  <folder>
    <title>XML bookmarks</title>
    <bookmark href="http://www.python.org/sigs/xml-sig/" >
      <title>SIG for XML Processing in Python</title>
    </bookmark>
  </folder>
</xbel>
\end{verbatim}

Converted to a DOM tree, this document could produce the following tree:

\begin{verbatim}
Element xbel None
   Text #text '  \012  '
   ProcessingInstruction processing 'instruction'
   Text #text '\012  '
   Element desc None
      Text #text 'No description'
   Text #text '\012  '
   Element folder None
      Text #text '\012    '
      Element title None
         Text #text 'XML bookmarks'
      Text #text '\012    '
      Element bookmark None
         Text #text '\012      '
         Element title None
            Text #text 'SIG for XML Processing in Python'
         Text #text '\012    '
      Text #text '\012  '
   Text #text '\012'
\end{verbatim}

This isn't the only possible tree, because different parsers may
differ in how they generate \class{Text} nodes; any of the
\class{Text} nodes in the above tree might be split into multiple
nodes.)

\subsubsection{The \class{Node} class}

e'll start by considering the basic \class{Node} class.  All the
other DOM nodes---\class{Document}, \class{Element}, \class{Text},
and so forth---are subclasses of \class{Node}.  It's possible to
perform many tasks using just the interface provided by \class{Node}.

XXX table of attributes and methods
            readonly attribute  DOMString            nodeName;
                     attribute  DOMString            nodeValue;
                                                           // raises(DOMException) on setting
                                                           // raises(DOMException) on retrieval
            readonly attribute  unsigned short       nodeType;
            readonly attribute  Node                 parentNode;
            readonly attribute  NodeList             childNodes;
            readonly attribute  Node                 firstChild;
            readonly attribute  Node                 lastChild;
            readonly attribute  Node                 previousSibling;
            readonly attribute  Node                 nextSibling;
            readonly attribute  NamedNodeMap         attributes;
            readonly attribute  Document             ownerDocument;

            Node                      insertBefore(in Node newChild,
                                                   in Node refChild)
                                                   raises(DOMException);
            Node                      replaceChild(in Node newChild,
                                                   in Node oldChild)
                                                   raises(DOMException);
            Node                      removeChild(in Node oldChild)
                                                  raises(DOMException);
            Node                      appendChild(in Node newChild)
                                                  raises(DOMException);
            boolean                   hasChildNodes();
            Node                      cloneNode(in boolean deep);


\subsubsection{\class{Document}, \class{Element}, and \class{Text} nodes}

The base of the entire tree is the \class{Document} node.  Its
\member{documentElement} attribute contains the \class{Element} node
for the root element.  The \class{Document} node may have additional
children, such as \class{ProcessingInstruction} nodes; the complete
list of children XXX.


\subsection{Walking Over The Entire Tree}

The \module{xml.dom} package also includes various helper classes for
common tasks such as walking over trees.

The \class{Walker} class

Introduction to the walker class

\subsection{Related Links}

The DOM Level 1 Recommendation is at
\url{http://www.w3.org/TR/1998/REC-DOM-Level-1-19981001/}.  Unlike
most standards, this one is actually pretty readable, particularly if
you're only interested in the Core XML interfaces.

Level 2 of the DOM has also been defined, adding more specialized
features such as support for XML namespaces, events, and ranges.  DOM
Level 3 is still being worked on, and will add yet more features.  The
World Wide Web Consortium's DOM page at \url{http://www.w3.org/DOM/}
has pointers to the specifications and current drafts for these higher
DOM levels.

\end{document}
