This is ../info/internals.info, produced by makeinfo version 4.8 from
internals/internals.texi.

INFO-DIR-SECTION XEmacs Editor
START-INFO-DIR-ENTRY
* Internals: (internals).       XEmacs Internals Manual.
END-INFO-DIR-ENTRY

   Edition History:

   Created November 1995 (?) by Ben Wing.  XEmacs Internals Manual
Version 1.0, March, 1996.  XEmacs Internals Manual Version 1.1, March,
1997.  XEmacs Internals Manual Version 1.4, March, 2001.  XEmacs
Internals Manual Version 21.5, October, 2004.

   Copyright (C) 1992 - 2004 Ben Wing.  Copyright (C) 1996, 1997 Sun
Microsystems.  Copyright (C) 1994 - 1998, 2002, 2003 Free Software
Foundation.  Copyright (C) 1994, 1995 Board of Trustees, University of
Illinois.

   Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided that the
entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that this permission notice may be stated in a
translation approved by the Foundation.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided also
that the section entitled "GNU General Public License" is included
exactly as in the original, and provided that the entire resulting
derived work is distributed under the terms of a permission notice
identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that the section entitled "GNU General Public License"
may be included in a translation approved by the Free Software
Foundation instead of in the original English.


File: internals.info,  Node: Windows I18N Introduction,  Next: Modules for Interfacing with MS Windows,  Prev: Windows Build Flags,  Up: Interface to MS Windows

37.3 Windows I18N Introduction
==============================

*Abstract:* This page provides an overview of the aspects of the Win32
internationalization API that are relevant to XEmacs, including the
basic distinction between multibyte and Unicode encodings.  Also
included are pointers to how XEmacs should make use of this API.

   The Win32 API is quite well-designed in its handling of strings
encoded for various character sets.  The API is geared around the idea
that two different methods of encoding strings should be supported.
These methods are called multibyte and Unicode, respectively.  The
multibyte encoding is compatible with ASCII strings and is a more
efficient representation when dealing with strings containing primarily
ASCII characters, but it has a great number of serious deficiencies and
limitations, including that it is very difficult and error-prone to work
with strings in this encoding, and any particular string in a multibyte
encoding can only contain characters from a very limited number of
character sets.  The Unicode encoding rectifies all of these
deficiencies, but it is not compatible with ASCII strings (in other
words, an existing program will not be able to handle the encoded
strings unless it is explicitly modified to do so), and it takes up
twice as much memory space as multibyte encodings when encoding a purely
ASCII string.

   Multibyte encodings use a variable number of bytes (either one or
two) to represent characters.  ASCII characters are also represented by
a single byte with its high bit not set, and non-ASCII characters are
represented by one or two bytes, the first of which always has its high
bit set.  (The second byte, when it exists, may or may not have its high
bit set.)  There is no single multibyte encoding.  Instead, there is
generally one encoding per non-ASCII character set.  Such an encoding is
capable of representing (besides ASCII characters, of course) only
characters from one (or possibly two) particular character sets.

   Multibyte encoding makes processing of strings very difficult.  For
example, given a pointer to the beginning of a character within a
string, finding the pointer to the beginning of the previous character
may require backing up all the way to the beginning of the string, and
then moving forward.  Also, an operation such as separating out the
components of a path by searching for backslashes will fail if it's
implemented in the simplest (but not multibyte-aware) fashion, because
it may find what appears to be a backslash, but which is actually the
second byte of a two-byte character.  Also, the limited number of
character sets that any particular multibyte encoding can represent
means that loss of data is likely if a string is converted from the
XEmacs internal format into a multibyte format.

   For these reasons, the C code in XEmacs should never do any sort of
work with multibyte encoded strings (or with strings in any external
encoding for that matter).  Strings should always be maintained in the
internal encoding, which is predictable, and converted to an external
encoding only at the point where the string moves from the XEmacs C
code and enters a system library function.  Similarly, when a string is
returned from a system library function, it should be immediately
converted into the internal coding before any operations are done on it.

   Unicode, unlike multibyte encodings, is a fixed-width encoding where
every character is represented using 16 bits.  It is also capable of
encoding all the characters from all the character sets in common use in
the world.  The predictability and completeness of the Unicode encoding
makes it a very good encoding for strings that may contain characters
from many character sets mixed up with each other.  At the same time, of
course, it is incompatible with routines that expect ASCII characters
and also incompatible with general string manipulation routines, which
will encounter a great number of what would appear to be embedded nulls
in the string.  It also takes twice as much room to encode strings
containing primarily ASCII characters.  This is why XEmacs does not use
Unicode or similar encoding internally for buffers.

   The Win32 API cleverly deals with the issue of 8 bit vs. 16 bit
characters by declaring a type called `"TCHAR"' which specifies a
generic character, either 8 bits or 16 bits.  Generally `TCHAR' is
defined to be the same as the simple C type `char', unless the
preprocessor constant `UNICODE' is defined, in which case `TCHAR' is
defined to be `WCHAR', which is a 16 bit type.  Nearly all functions in
the Win32 API that take strings are defined to take strings that are
actually arrays of `TCHAR's.  There is a type `LPTSTR' which is defined
to be a string of `TCHAR's and another type `LPCTSTR' which is a const
string of `TCHAR's.  The theory is that any program that uses `TCHAR's
exclusively to represent characters and does not make assumptions about
the size of a `TCHAR' or the way that the characters are encoded should
work transparently regardless of whether the `UNICODE' preprocessor
constant is defined, which is to say, regardless of whether 8 bit
multibyte or 16 bit Unicode characters are being used.  The way that
this is actually implemented is that every Win32 API function that takes
a string as an argument actually maps to one of two functions which are
suffixed with an `A' (which stands for ANSI, and means multibyte
strings) or `W' (which stands for wide, and means Unicode strings).
The mapping is, of course, controlled by the same `UNICODE'
preprocessor constant.  Generally all structures containing strings in
them actually map to one of two different kinds of structures, with
either an `A' or a `W' suffix after the structure name.

   Unfortunately, not all of the implementations of the Win32 API
implement all of the functionality described above.  In particular,
Windows 95 does not implement very much Unicode functionality.  It does
implement functions to convert multibyte-encoded strings to and from
Unicode strings, and provides Unicode versions of certain low-level
functions like `ExtTextOut()'.  In fact, all of the rest of the Unicode
versions of API functions are just stubs that return an error.
Conversely, all versions of Windows NT completely implement all the
Unicode functionality, but some versions (especially versions before
Windows NT 4.0) don't implement much of the multibyte functionality.
For this reason, as well as for general code cleanliness, XEmacs needs
to be written in such a way that it works with or without the `UNICODE'
preprocessor constant being defined.

   Getting XEmacs to run when all strings are Unicode primarily
involves removing any assumptions made about the size of characters.
Remember what I said earlier about how the point of conversion between
internally and externally encoded strings should occur at the point of
entry or exit into or out of a library function.  With this in mind, an
externally encoded string in XEmacs can be treated simply as an
arbitrary sequence of bytes of some length which has no particular
relationship to the length of the string in the internal encoding.

   #### The rest of this is *out-of-date* and needs to be written to
reference the actual coding systems or aliases that we currently use.

   [[ To facilitate this, the enum `external_data_format', which is
declared in `lisp.h', is expanded to contain three new formats, which
are `FORMAT_LOCALE', `FORMAT_UNICODE' and `FORMAT_TSTR'.
`FORMAT_LOCALE' always causes encoding into a multibyte string
consistent with the encoding of the current locale.  The functions to
handle locales are different under Unix and Windows and locales are a
process property under Unix and a thread property under Windows, but
the concepts are basically the same.  `FORMAT_UNICODE' of course causes
encoding into Unicode and `FORMAT_TSTR' logically maps to either
`FORMAT_LOCALE' or `FORMAT_UNICODE' depending on the `UNICODE'
preprocessor constant.

   Under Unix the behavior of `FORMAT_TSTR' is undefined and this
particular format should not be used.  Under Windows however
`FORMAT_TSTR' should be used for pretty much all of the Win32 API
calls.  The other two formats should only be used in particular APIs
that specifically call for a multibyte or Unicode encoded string
regardless of the `UNICODE' preprocessor constant.  String constants
that are to be passed directly to Win32 API functions, such as the
names of window classes, need to be bracketed in their definition with
a call to the macro `TEXT'.  This awfully named macro, which comes out
of the Win32 API, appropriately makes a string of either regular or
wide chars, which is to say this string may be prepended with an `L'
(causing it to be a wide string) depending on the `UNICODE'
preprocessor constant.

   By the way, if you're wondering what happened to `FORMAT_OS', I
think that this format should go away entirely because it is too vague
and should be replaced by more specific formats as they are defined.  ]]

   Use Qnative for Unix conversion, Qmswindows_tstr for Windows ...

   String constants that are to be passed directly to Win32 API
functions, such as the names of window classes, need to be bracketed in
their definition with a call to the macro XETEXT. This appropriately
makes a string of either regular or wide chars, which is to say this
string may be prepended with an L (causing it to be a wide string)
depending on XEUNICODE_P.


File: internals.info,  Node: Modules for Interfacing with MS Windows,  Next: CHANGES from 21.4-windows branch,  Prev: Windows I18N Introduction,  Up: Interface to MS Windows

37.4 Modules for Interfacing with MS Windows
============================================

There are two different general Windows-related include files in src.

   Uses are approximately:

`syswindows.h'
     Wrapper around `<windows.h>', including missing defines as
     necessary.  Includes stuff needed on both Cygwin and native
     Windows, regardless of window system chosen.  Includes definitions
     needed for Unicode conversion/encapsulation, and other
     Mule-related stuff, plus various other prototypes and
     Windows-specific, but not GUI-specific, stuff.

`console-msw.h'
     Used on both Cygwin and native Windows, but only when native window
     system (as opposed to X) chosen.  Includes `syswindows.h'.

   Summary of files:

`console-msw.h'
     include file for native windowing (otherwise, `console-x.h', etc.)

`console-msw.c, frame-msw.c, etc.'
     native windowing, as above

`process-nt.c'
     subprocess support for native OS (otherwise, `process-unix.c')

`nt.c'
     support routines used under native OS

`win32.c'
     support routines used under both OS environments

`syswindows.h'
     support header for both environments

`nt/xemacs.mak'
     Makefile for VC++ (otherwise, `src/Makefile.in.in')

`s/windowsnt.h'
     s header for basic native-OS defines, VC++ compiler

`s/mingw32.h'
     s header for basic native-OS defines, GCC/MinGW compiler

`s/cygwin.h'
     s header for basic Cygwin defines

`s/win32-native.h'
     s header for basic native-OS defines, all compilers

`s/win32-common.h'
     s header for defines for both OS environments

`intl-win32.c'
     internationalization functions for both OS environments

`intl-encap-win32.c'
     Unicode encapsulation functions for both OS environments

`intl-auto-encap-win32.c'
     Auto-generated Unicode encapsulation functions

`intl-auto-encap-win32.h'
     Auto-generated Unicode encapsulation headers


File: internals.info,  Node: CHANGES from 21.4-windows branch,  Prev: Modules for Interfacing with MS Windows,  Up: Interface to MS Windows

37.5 CHANGES from 21.4-windows branch (probably obsolete)
=========================================================

This node contains the `CHANGES-msw' log that Andy Piper kept while he
was maintaining the Windows branch of 21.4.  These changes have
(presumably) long since been merged to both 21.4 and 21.5, but let's not
throw the list away yet.

CHANGES-msw
===========

This file briefly describes all mswindows-specific changes to XEmacs in
the OXYMORON series of releases. The mswindows release branch contains
additional changes on top of the mainline XEmacs release. These changes
are deemed necessary for XEmacs to be fully functional under mswindows.
It is not intended that these changes cause problems on UNIX systems,
but they have not been tested on UNIX platforms. Caveat Emptor.

   See the file `CHANGES-release' for a full list of mainline changes.

to XEmacs 21.4.9 "Informed Management (Windows)"
================================================

   * Fix layout of widgets so that the search dialog works.

   * Fix focus capture of widgets under X.

to XEmacs 21.4.8 "Honest Recruiter (Windows)"
=============================================

   * All changes from 21.4.6 and 21.4.7.

   * Make sure revert temporaries are not visiting files. Suggested by
     Mike Alexander.

   * File renaming fix from Mathias Grimmberger.

   * Fix printer metrics on windows 95 from Jonathan Harris.

   * Fix layout of widgets so that the search dialog works.

   * Fix focus capture of widgets under X.

   * Buffers tab doc fixes from John Palmieri.

   * Sync with FSF custom `:set-after' behavior.

   * Virtual window manager freeze fix from Rick Rankin.

   * Fix various printing problems.

   * Enable windows printing on cygwin.

to XEmacs 21.4.7 "Economic Science (Windows)"
=============================================

   * All changes from 21.4.6.

   * Fix problems with auto-revert with noconfirm.

   * Undo autoconf 2.5x changes.

   * Undo 21.4.7 process change.

   to XEmacs 21.4.6 "Common Lisp (Windows)"

   * Made native registry entries match the installer.

   * Fixed mousewheel lockups.

   * Frame iconifcation fix from Adrian Aichner.

   * Fixed some printing problems.

   * Netinstaller updated to support kit revisions.

   * Fixed customize popup menus.

   * Fixed problems with too many dialog popups.

   * Netinstaller fixed to correctly upgrade shortcuts when upgrading
     core XEmacs.

   * Fix for virtual window managers from Adrian Aichner.

   * Installer registers all C++ file types.

   * Short-filename fix from Peter Arius.

   * Fix for GC assertions from Adrian Aichner.

   * Winclient DDE client from Alastair Houghton.

   * Fix event assert from Mike Alexander.

   * Warning removal noticed by Ben Wing.

   * Redisplay glyph height fix from Ben Wing.

   * Printer margin fix from Jonathan Harris.

   * Error dialog fix suggested by Thomas Vogler.

   * Fixed revert-buffer to not revert in the case that there is
     nothing to be done.

   * Glyph-baseline fix from Nix.

   * Fixed clipping of wide glyphs in non-zero-length extents.

   * Windows build fixes.

   * Fixed `:initial-focus' so that it works.

to XEmacs 21.4.5 "Civil Service (Windows)"
==========================================

   * Fixed a scrollbar problem when selecting the frame with focus.

   * Fixed `mswindows-shell-execute' under cygwin.

   * Added a new function `mswindows-cygwin-to-win32-path' for JDE.

   * Added support for dialog-based directory selection.

   * The installer version has been updated to the 21.5 netinstaller.
     The 21.5 installer now does proper dde file association and adds
     uninstall capability.

   * Handle leak fix from Mike Alexander.

   * New release build script.


File: internals.info,  Node: Interface to the X Window System,  Next: Dumping,  Prev: Interface to MS Windows,  Up: Top

38 Interface to the X Window System
***********************************

Mostly undocumented.

* Menu:

* Lucid Widget Library::        An interface to various widget sets.
* Modules for Interfacing with X Windows::


File: internals.info,  Node: Lucid Widget Library,  Next: Modules for Interfacing with X Windows,  Prev: Interface to the X Window System,  Up: Interface to the X Window System

38.1 Lucid Widget Library
=========================

Lwlib is extremely poorly documented and quite hairy.  The author(s)
blame that on X, Xt, and Motif, with some justice, but also sufficient
hypocrisy to avoid drawing the obvious conclusion about their own work.

   The Lucid Widget Library is composed of two more or less independent
pieces.  The first, as the name suggests, is a set of widgets.  These
widgets are intended to resemble and improve on widgets provided in the
Motif toolkit but not in the Athena widgets, including menubars and
scrollbars.  Recent additions by Andy Piper integrate some "modern"
widgets by Edward Falk, including checkboxes, radio buttons, progress
gauges, and index tab controls (aka notebooks).

   The second piece of the Lucid widget library is a generic interface
to several toolkits for X (including Xt, the Athena widget set, and
Motif, as well as the Lucid widgets themselves) so that core XEmacs
code need not know which widget set has been used to build the
graphical user interface.

* Menu:

* Generic Widget Interface::    The lwlib generic widget interface.
* Scrollbars::
* Menubars::
* Checkboxes and Radio Buttons::
* Progress Bars::
* Tab Controls::


File: internals.info,  Node: Generic Widget Interface,  Next: Scrollbars,  Prev: Lucid Widget Library,  Up: Lucid Widget Library

38.1.1 Generic Widget Interface
-------------------------------

In general in any toolkit a widget may be a composite object.  In Xt,
all widgets have an X window that they manage, but typically a complex
widget will have widget children, each of which manages a subwindow of
the parent widget's X window.  These children may themselves be
composite widgets.  Thus a widget is actually a tree or hierarchy of
widgets.

   For each toolkit widget, lwlib maintains a tree of `widget_values'
which mirror the hierarchical state of Xt widgets (including Motif,
Athena, 3D Athena, and Falk's widget sets).  Each `widget_value' has
`contents' member, which points to the head of a linked list of its
children.  The linked list of siblings is chained through the `next'
member of `widget_value'.

                +-----------+
                | composite |
                +-----------+
                      |
                      | contents
                      V
                  +-------+ next +-------+ next +-------+
                  | child |----->| child |----->| child |
                  +-------+      +-------+      +-------+
                                     |
                                     | contents
                                     V
                              +-------------+ next +-------------+
                              | grand child |----->| grand child |
                              +-------------+      +-------------+

     The `widget_value' hierarchy of a composite widget with two simple
     children and one composite child.

   The `widget_instance' structure maintains the inverse view of the
tree.  As for the `widget_value', siblings are chained through the
`next' member.  However, rather than naming children, the
`widget_instance' tree links to parents.

                +-----------+
                | composite |
                +-----------+
                      A
                      | parent
                      |
                  +-------+ next +-------+ next +-------+
                  | child |----->| child |----->| child |
                  +-------+      +-------+      +-------+
                                     A
                                     | parent
                                     |
                              +-------------+ next +-------------+
                              | grand child |----->| grand child |
                              +-------------+      +-------------+

     The `widget_value' hierarchy of a composite widget with two simple
     children and one composite child.

   This permits widgets derived from different toolkits to be updated
and manipulated generically by the lwlib library. For instance
`update_one_widget_instance' can cope with multiple types of widget and
multiple types of toolkit. Each element in the widget hierarchy is
updated from its corresponding `widget_value' by walking the
`widget_value' tree.  This has desirable properties.  For example,
`lw_modify_all_widgets' is called from `glyphs-x.c' and updates all the
properties of a widget without having to know what the widget is or
what toolkit it is from.  Unfortunately this also has its hairy
properties; the lwlib code quite complex. And of course lwlib has to
know at some level what the widget is and how to set its properties.

   The `widget_instance' structure also contains a pointer to the root
of its tree.  Widget instances are further confi


File: internals.info,  Node: Scrollbars,  Next: Menubars,  Prev: Generic Widget Interface,  Up: Lucid Widget Library

38.1.2 Scrollbars
-----------------


File: internals.info,  Node: Menubars,  Next: Checkboxes and Radio Buttons,  Prev: Scrollbars,  Up: Lucid Widget Library

38.1.3 Menubars
---------------


File: internals.info,  Node: Checkboxes and Radio Buttons,  Next: Progress Bars,  Prev: Menubars,  Up: Lucid Widget Library

38.1.4 Checkboxes and Radio Buttons
-----------------------------------


File: internals.info,  Node: Progress Bars,  Next: Tab Controls,  Prev: Checkboxes and Radio Buttons,  Up: Lucid Widget Library

38.1.5 Progress Bars
--------------------


File: internals.info,  Node: Tab Controls,  Prev: Progress Bars,  Up: Lucid Widget Library

38.1.6 Tab Controls
-------------------


File: internals.info,  Node: Modules for Interfacing with X Windows,  Prev: Lucid Widget Library,  Up: Interface to the X Window System

38.2 Modules for Interfacing with X Windows
===========================================

     Emacs.ad.h

   A file generated from `Emacs.ad', which contains XEmacs-supplied
fallback resources (so that XEmacs has pretty defaults).

     EmacsFrame.c
     EmacsFrame.h
     EmacsFrameP.h

   These modules implement an Xt widget class that encapsulates a frame.
This is for ease in integrating with Xt.  The EmacsFrame widget covers
the entire X window except for the menubar; the scrollbars are
positioned on top of the EmacsFrame widget.

   *Warning:* Abandon hope, all ye who enter here.  This code took an
ungodly amount of time to get right, and is likely to fall apart
mercilessly at the slightest change.  Such is life under Xt.

     EmacsManager.c
     EmacsManager.h
     EmacsManagerP.h

   These modules implement a simple Xt manager (i.e. composite) widget
class that simply lets its children set whatever geometry they want.
It's amazing that Xt doesn't provide this standardly, but on second
thought, it makes sense, considering how amazingly broken Xt is.

     EmacsShell-sub.c
     EmacsShell.c
     EmacsShell.h
     EmacsShellP.h

   These modules implement two Xt widget classes that are subclasses of
the TopLevelShell and TransientShell classes.  This is necessary to deal
with more brokenness that Xt has sadistically thrust onto the backs of
developers.

     xgccache.c
     xgccache.h

   These modules provide functions for maintenance and caching of GC's
(graphics contexts) under the X Window System.  This code is junky and
needs to be rewritten.

     select-msw.c
     select-x.c
     select.c
     select.h

   This module provides an interface to the X Window System's concept of
"selections", the standard way for X applications to communicate with
each other.

     xintrinsic.h
     xintrinsicp.h
     xmmanagerp.h
     xmprimitivep.h

   These header files are similar in spirit to the `sys*.h' files and
buffer against different implementations of Xt and Motif.

   * `xintrinsic.h' should be included in place of `<Intrinsic.h>'.

   * `xintrinsicp.h' should be included in place of `<IntrinsicP.h>'.

   * `xmmanagerp.h' should be included in place of `<XmManagerP.h>'.

   * `xmprimitivep.h' should be included in place of `<XmPrimitiveP.h>'.

     xmu.c
     xmu.h

   These files provide an emulation of the Xmu library for those systems
(i.e. HPUX) that don't provide it as a standard part of X.

     ExternalClient-Xlib.c
     ExternalClient.c
     ExternalClient.h
     ExternalClientP.h
     ExternalShell.c
     ExternalShell.h
     ExternalShellP.h
     extw-Xlib.c
     extw-Xlib.h
     extw-Xt.c
     extw-Xt.h

   These files provide the "external widget" interface, which allows an
XEmacs frame to appear as a widget in another application.  To do this,
you have to configure with `--external-widget'.

   `ExternalShell*' provides the server (XEmacs) side of the connection.

   `ExternalClient*' provides the client (other application) side of
the connection.  These files are not compiled into XEmacs but are
compiled into libraries that are then linked into your application.

   `extw-*' is common code that is used for both the client and server.

   Don't touch this code; something is liable to break if you do.


File: internals.info,  Node: Dumping,  Next: Future Work,  Prev: Interface to the X Window System,  Up: Top

39 Dumping
**********

* Menu:

* Dumping Justification::
* Overview::
* Data descriptions::
* Dumping phase::
* Reloading phase::
* Remaining issues::


File: internals.info,  Node: Dumping Justification,  Next: Overview,  Prev: Dumping,  Up: Dumping

39.1 Dumping Justification
==========================

The C code of XEmacs is just a Lisp engine with a lot of built-in
primitives useful for writing an editor.  The editor itself is written
mostly in Lisp, and represents around 100K lines of code.  Loading and
executing the initialization of all this code takes a bit a time (five
to ten times the usual startup time of current xemacs) and requires
having all the lisp source files around.  Having to reload them each
time the editor is started would not be acceptable.

   The traditional solution to this problem is called dumping: the build
process first creates the lisp engine under the name `temacs', then
runs it until it has finished loading and initializing all the lisp
code, and eventually creates a new executable called `xemacs' including
both the object code in `temacs' and all the contents of the memory
after the initialization.

   This solution, while working, has a huge problem: the creation of the
new executable from the actual contents of memory is an extremely
system-specific process, quite error-prone, and which interferes with a
lot of system libraries (like malloc).  It is even getting worse
nowadays with libraries using constructors which are automatically
called when the program is started (even before `main()') which tend to
crash when they are called multiple times, once before dumping and once
after (IRIX 6.x `libz.so' pulls in some C++ image libraries thru
dependencies which have this problem).  Writing the dumper is also one
of the most difficult parts of porting XEmacs to a new operating system.
Basically, `dumping' is an operation that is just not officially
supported on many operating systems.

   The aim of the portable dumper is to solve the same problem as the
system-specific dumper, that is to be able to reload quickly, using only
a small number of files, the fully initialized lisp part of the editor,
without any system-specific hacks.


File: internals.info,  Node: Overview,  Next: Data descriptions,  Prev: Dumping Justification,  Up: Dumping

39.2 Overview
=============

The portable dumping system has to:

  1. At dump time, write all initialized, non-quickly-rebuildable data
     to a file [Note: currently named `xemacs.dmp', but the name will
     change], along with all information needed for the reloading.

  2. When starting xemacs, reload the dump file, relocate it to its new
     starting address if needed, and reinitialize all pointers to this
     data.  Also, rebuild all the quickly rebuildable data.

   Note: As of 21.5.18, the dump file has been moved inside of the
executable, although there are still problems with this on some systems.


File: internals.info,  Node: Data descriptions,  Next: Dumping phase,  Prev: Overview,  Up: Dumping

39.3 Data descriptions
======================

The more complex task of the dumper is to be able to write memory blocks
on the heap (lisp objects, i.e. lrecords, and C-allocated memory, such
as structs and arrays) to disk and reload them at a different address,
updating all the pointers they include in the process.  This is done by
using external data descriptions that give information about the layout
of the blocks in memory.

   The specification of these descriptions is in lrecord.h.  A
description of an lrecord is an array of struct memory_description.
Each of these structs include a type, an offset in the block and some
optional parameters depending on the type.  For instance, here is the
string description:

     static const struct memory_description string_description[] = {
       { XD_BYTECOUNT,         offsetof (Lisp_String, size) },
       { XD_OPAQUE_DATA_PTR,   offsetof (Lisp_String, data), XD_INDIRECT(0, 1) },
       { XD_LISP_OBJECT,       offsetof (Lisp_String, plist) },
       { XD_END }
     };

   The first line indicates a member of type Bytecount, which is used by
the next, indirect directive.  The second means "there is a pointer to
some opaque data in the field `data'".  The length of said data is
given by the expression `XD_INDIRECT(0, 1)', which means "the value in
the 0th line of the description (welcome to C) plus one".  The third
line means "there is a Lisp_Object member `plist' in the Lisp_String
structure".  `XD_END' then ends the description.

   This gives us all the information we need to move around what is
pointed to by a memory block (C or lrecord) and, by transitivity,
everything that it points to.  The only missing information for dumping
is the size of the block.  For lrecords, this is part of the
lrecord_implementation, so we don't need to duplicate it.  For C blocks
we use a struct sized_memory_description, which includes a size field
and a pointer to an associated array of memory_description.


File: internals.info,  Node: Dumping phase,  Next: Reloading phase,  Prev: Data descriptions,  Up: Dumping

39.4 Dumping phase
==================

Dumping is done by calling the function `pdump()' (in `dumper.c') which
is invoked from Fdump_emacs (in `emacs.c').  This function performs a
number of tasks.

* Menu:

* Object inventory::
* Address allocation::
* The header::
* Data dumping::
* Pointers dumping::


File: internals.info,  Node: Object inventory,  Next: Address allocation,  Prev: Dumping phase,  Up: Dumping phase

39.4.1 Object inventory
-----------------------

The first task is to build the list of the objects to dump.  This
includes:

   * lisp objects

   * other memory blocks (C structures, arrays. etc)

   We end up with one `pdump_block_list_elt' per object group (arrays
of C structs are kept together) which includes a pointer to the first
object of the group, the per-object size and the count of objects in the
group, along with some other information which is initialized later.

   These entries are linked together in `pdump_block_list' structures
and can be enumerated thru either:

  1. the `pdump_object_table', an array of `pdump_block_list', one per
     lrecord type, indexed by type number.

  2. the `pdump_opaque_data_list', used for the opaque data which does
     not include pointers, and hence does not need descriptions.

  3. the `pdump_desc_table', which is a vector of
     `memory_description'/`pdump_block_list' pairs, used for non-opaque
     C memory blocks.

   This uses a marking strategy similar to the garbage collector.  Some
differences though:

  1. We do not use the mark bit (which does not exist for generic
     memory blocks anyway); we use a big hash table instead.

  2. We do not use the mark function of lrecords but instead rely on the
     external descriptions.  This happens essentially because we need to
     follow pointers to generic memory blocks and opaque data in
     addition to Lisp_Object members.

   This is done by `pdump_register_object()', which handles Lisp_Object
variables, and `pdump_register_block()' which handles generic memory
blocks (C structures, arrays, etc.), which both delegate the
description management to `pdump_register_sub()'.

   The hash table doubles as a map object to pdump_block_list_elmt (i.e.
allows us to look up a pdump_block_list_elmt with the object it points
to).  Entries are added with `pdump_add_block()' and looked up with
`pdump_get_block()'.  There is no need for entry removal.  The hash
value is computed quite simply from the object pointer by
`pdump_make_hash()'.

   The roots for the marking are:

  1. the `staticpro''ed variables (there is a special
     `staticpro_nodump()' call for protected variables we do not want to
     dump).

  2. the Lisp_Object variables registered via
     `dump_add_root_lisp_object' (`staticpro()' is equivalent to
     `staticpro_nodump()' + `dump_add_root_lisp_object()').

  3. the data-segment memory blocks registered via `dump_add_root_block'
     (for blocks with relocatable pointers), or `dump_add_opaque' (for
     "opaque" blocks with no relocatable pointers; this is just a
     shortcut for calling `dump_add_root_block' with a NULL
     description).

  4. the pointer variables registered via `dump_add_root_block_ptr',
     each of which points to a block of heap memory (generally a C
     structure or array).  Note that `dump_add_root_block_ptr' is not
     technically necessary, as a pointer variable can be seen as a
     special case of a data-segment memory block and registered using
     `dump_add_root_block'.  Doing it this way, however, would require
     another level of static structures declared.  Since pointer
     variables are quite common, `dump_add_root_block_ptr' is provided
     for convenience.  Note also that internally we have to treat it
     separately from `dump_add_root_block' rather than writing the
     former as a call to the latter, since we don't have support for
     creating and using memory descriptions on the fly - they must all
     be statically declared in the data-segment.

   This does not include the GCPRO'ed variables, the specbinds, the
catchtags, the backlist, the redisplay or the profiling info, since we
do not want to rebuild the actual chain of lisp calls which end up to
the dump-emacs call, only the global variables.

   Weak lists and weak hash tables are dumped as if they were their
non-weak equivalent (without changing their type, of course).  This has
not yet been a problem.


File: internals.info,  Node: Address allocation,  Next: The header,  Prev: Object inventory,  Up: Dumping phase

39.4.2 Address allocation
-------------------------

The next step is to allocate the offsets of each of the objects in the
final dump file.  This is done by `pdump_allocate_offset()' which is
called indirectly by `pdump_scan_by_alignment()'.

   The strategy to deal with alignment problems uses these facts:

  1. real world alignment requirements are powers of two.

  2. the C compiler is required to adjust the size of a struct so that
     you can have an array of them next to each other.  This means you
     can have an upper bound of the alignment requirements of a given
     structure by looking at which power of two its size is a multiple.

  3. the non-variant part of variable size lrecords has an alignment
     requirement of 4.

   Hence, for each lrecord type, C struct type or opaque data block the
alignment requirement is computed as a power of two, with a minimum of
2^2 for lrecords.  `pdump_scan_by_alignment()' then scans all the
`pdump_block_list_elmt''s, the ones with the highest requirements
first.  This ensures the best packing.

   The maximum alignment requirement we take into account is 2^8.

   `pdump_allocate_offset()' only has to do a linear allocation,
starting at offset 256 (this leaves room for the header and keeps the
alignments happy).


File: internals.info,  Node: The header,  Next: Data dumping,  Prev: Address allocation,  Up: Dumping phase

39.4.3 The header
-----------------

The next step creates the file and writes a header with a signature and
some random information in it.  The `reloc_address' field, which
indicates at which address the file should be loaded if we want to avoid
post-reload relocation, is set to 0.  It then seeks to offset 256 (base
offset for the objects).


File: internals.info,  Node: Data dumping,  Next: Pointers dumping,  Prev: The header,  Up: Dumping phase

39.4.4 Data dumping
-------------------

The data is dumped in the same order as the addresses were allocated by
`pdump_dump_data()', called from `pdump_scan_by_alignment()'.  This
function copies the data to a temporary buffer, relocates all pointers
in the object to the addresses allocated in step Address Allocation,
and writes it to the file.  Using the same order means that, if we are
careful with lrecords whose size is not a multiple of 4, we are ensured
that the object is always written at the offset in the file allocated
in step Address Allocation.


File: internals.info,  Node: Pointers dumping,  Prev: Data dumping,  Up: Dumping phase

39.4.5 Pointers dumping
-----------------------

A bunch of tables needed to reassign properly the global pointers are
then written.  They are:

  1. the pdump_root_block_ptrs dynarr

  2. the pdump_opaques dynarr

  3. a vector of all the offsets to the objects in the file that
     include a description (for faster relocation at reload time)

  4. the pdump_root_objects and pdump_weak_object_chains dynarrs.

   For each of the dynarrs we write both the pointer to the variables
and the relocated offset of the object they point to.  Since these
variables are global, the pointers are still valid when restarting the
program and are used to regenerate the global pointers.

   The `pdump_weak_object_chains' dynarr is a special case.  The
variables it points to are the head of weak linked lists of lisp objects
of the same type.  Not all objects of this list are dumped so the
relocated pointer we associate with them points to the first dumped
object of the list, or Qnil if none is available.  This is also the
reason why they are not used as roots for the purpose of object
enumeration.

   Some very important information like the `staticpros' and
`lrecord_implementations_table' are handled indirectly using
`dump_add_opaque' or `dump_add_root_block_ptr'.

   This is the end of the dumping part.


File: internals.info,  Node: Reloading phase,  Next: Remaining issues,  Prev: Dumping phase,  Up: Dumping

39.5 Reloading phase
====================

39.5.1 File loading
-------------------

The file is mmap'ed in memory (which ensures a PAGESIZE alignment, at
least 4096), or if mmap is unavailable or fails, a 256-bytes aligned
malloc is done and the file is loaded.

   Some variables are reinitialized from the values found in the header.

   The difference between the actual loading address and the
reloc_address is computed and will be used for all the relocations.

39.5.2 Putting back the pdump_opaques
-------------------------------------

The memory contents are restored in the obvious and trivial way.

39.5.3 Putting back the pdump_root_block_ptrs
---------------------------------------------

The variables pointed to by pdump_root_block_ptrs in the dump phase are
reset to the right relocated object addresses.

39.5.4 Object relocation
------------------------

All the objects are relocated using their description and their offset
by `pdump_reloc_one'.  This step is unnecessary if the reloc_address is
equal to the file loading address.

39.5.5 Putting back the pdump_root_objects and pdump_weak_object_chains
-----------------------------------------------------------------------

Same as Putting back the pdump_root_block_ptrs.

39.5.6 Reorganize the hash tables
---------------------------------

Since some of the hash values in the lisp hash tables are
address-dependent, their layout is now wrong.  So we go through each of
them and have them resorted by calling `pdump_reorganize_hash_table'.


File: internals.info,  Node: Remaining issues,  Prev: Reloading phase,  Up: Dumping

39.6 Remaining issues
=====================

The build process will have to start a post-dump xemacs, ask it the
loading address (which will, hopefully, be always the same between
different xemacs invocations) [[unfortunately, not true on Linux with
the ExecShield feature]] and relocate the file to the new address.
This way the object relocation phase will not have to be done, which
means no writes in the objects and that, because of the use of mmap, the
dumped data will be shared between all the xemacs running on the
computer.

   Some executable signature will be necessary to ensure that a given
dump file is really associated with a given executable, or random
crashes will occur.  Maybe a random number set at compile or configure
time thru a define.  This will also allow for having
differently-compiled xemacsen on the same system (mule and no-mule
comes to mind).

   The DOC file contents should probably end up in the dump file.


File: internals.info,  Node: Future Work,  Next: Future Work Discussion,  Prev: Dumping,  Up: Top

40 Future Work
**************

* Menu:

* Future Work -- General Suggestions::
* Future Work -- Elisp Compatibility Package::
* Future Work -- Drag-n-Drop::
* Future Work -- Standard Interface for Enabling Extensions::
* Future Work -- Better Initialization File Scheme::
* Future Work -- Keyword Parameters::
* Future Work -- Property Interface Changes::
* Future Work -- Toolbars::
* Future Work -- Menu API Changes::
* Future Work -- Removal of Misc-User Event Type::
* Future Work -- Mouse Pointer::
* Future Work -- Extents::
* Future Work -- Version Number and Development Tree Organization::
* Future Work -- Improvements to the `xemacs.org' Website::
* Future Work -- Keybindings::
* Future Work -- Byte Code Snippets::
* Future Work -- Lisp Stream API::
* Future Work -- Multiple Values::
* Future Work -- Macros::
* Future Work -- Specifiers::
* Future Work -- Display Tables::
* Future Work -- Making Elisp Function Calls Faster::
* Future Work -- Lisp Engine Replacement::
* Future Work -- Better Rendering Support::


File: internals.info,  Node: Future Work -- General Suggestions,  Next: Future Work -- Elisp Compatibility Package,  Prev: Future Work,  Up: Future Work

40.1 Future Work - General Suggestions
======================================

Jamie Zawinski's XEmacs Wishlist
--------------------------------

This document is based on Jamie Zawinski's xemacs wishlist
(http://www.jwz.org/doc/xemacs-wishlist.html).        Throughout this
page, "I" refers to Jamie.

   The list has been substantially reformatted and edited to fit the
needs       of this site. If you have any soul at all, you'll go check
out the       original. OK? You should also check out some other
wishlists
(http://www.xemacs.org/Releases/Public-21.2/execution.html#wishlists).

About the List
..............

I've ranked these (roughly) from easiest to hardest; though of all of
them, I think the debugger improvements would be the most useful. I
think the combination of emacs+gdb is the best Unix development
environment currently available, but it's still lamentably primitive
and extremely frustrating (much like Unix itself), especially if you
know what kinds of features more modern integrated debuggers have.

XEmacs Wishlist
...............

*Improve the keyboard macro system.*
     Keyboard macros are one of the most useful concepts that emacs has
     to offer, but there's room for improvement.

    *Make it possible to embed one macro inside of another.*
          Often, I'll define a keyboard macro, and then realize that
          I've left something out, or that there's more that I need to
          do; for example, I may define a macro that does something to
          the current line, and then realize that I want to apply it to
          a lot of lines. So, I'd like this to work:

               C-x (
               ; start macro #1
               ...
               ; (do stuff)
               C-x )
               ; done with macro #1
               ...
               ; (do stuff)
               C-x (
               ; start macro #2
               C-x e
               ; execute macro #1 (splice it into macro #2)
               C-s foo
               ; move forward to the next spot
               C-x )
               ; done with macro #2
               C-u 1000 C-x e
               ; apply the new macro

          That is, simply, one should be able to wrap new text around an
          existing macro. I can't tell you how many times I've defined
          a complex macro but left out the "`C-n C-a'" at the end...

          Yes, you can accomplish this with M-x name-last-kbd-macro, but
          that's a pain. And it's also more permanent than I'd often
          like.

    *Make it possible to correct errors when defining a macro.*
          Right now, the act of defining a macro stops if you get an
          error while defining it, and all of the characters you've
          already typed into the macro are gone. It needn't be that
          way. I think that, when that first error occurs, the user
          should be given the option of taking the last command off of
          the macro and trying again.

          The macro-reader knows where the bounds of multi-character
          command sequences are, and it could even keep track of the
          corresponding undo records; rubbing out the previous entry on
          the macro could also undo any changes that command had made.
          (This should also work if the macro spans multiple buffers,
          and should restore window configurations as well.)

          You'd want multi-level undo for this as well, so maybe the
          way to go would be to add some new key sequence which was
          used only as the back-up-inside-a-keyboard-macro-definition
          command.

          I'm not totally sure that this would end up being very usable;
          maybe it would be too hard to deal with. Which brings us to:

    *Make it possible to edit a keyboard macro after it has been defined.*
          I only just discovered `edit-kbd-macro' (`C-x C-k').  It is
          very, very cool.

          The trick it does of showing the command which will be
          executed is somewhat error-prone, as it can only look up
          things in the current map or the global map; if the macro
          changed buffers, it wouldn't be displaying the right
          commands. (One of the things I often use macros for is
          operating on many files at once, by bringing up a dired buffer
          of those files, editing them, and then moving on to the next.)

          However, if the act of recording a macro also kept track of
          the actual commands that had gotten executed, it could make
          use of that info as well.

          Another way of editing a macro, other than as text in a
          buffer, would be to have a command which single-steps a
          macro: you would lean on the space bar to watch the macro
          execute one character (command?)  at a time, and then when
          you reached the point you wanted to change, you could do some
          gesture to either: insert some keystrokes into the middle of
          the macro and then continue; or to replace the rest of the
          macro from here to the end; or something.

          Another similar hack might be to convert a macro to the
          equivalent lisp code, so that one could tweak it later in
          ways that would be too hard to do from the keyboard (wrapping
          parts of it in `while' loops or something.) (`M-x
          insert-kbd-macro' isn't really what I'm talking about here: I
          mean insert the list of commands, not the list of keystrokes.)

*Save my wrists!*
     In the spirit of the ``teach-extended-commands-p'' variable, it
     would be interesting if emacs would keep track of what are the
     commands I use most often, perhaps grouped by proximity or mode -
     it would then be more obvious which commands were most likely
     candidates for placement on a toolbar, or popup menu, or just a
     more convenient key binding.

     Bonus points if it figures out that I type "`bt\n'" and
     "`ret\ny\n'" into my `*gdb*' buffer about a hundred thousand times
     a day.

*XmCreateFileSelectionBox*
     The thing that "File/Open..." pops up has excellent _hack_ value,
     but as a user interface, it's an abomination. Isn't it time
     someone added a real file selection dialog already? (For the
     Motifly-challenged, the Athena-based file selector that GhostView
     uses seems adequate.)

*Improve the toolbar system.*
     It's great that XEmacs has a toolbar, but it's damn near impossible
     to customize it.

    *Make it easy to define new toolbar buttons.*
          Currently, to define a toolbar button that has a text
          equivalent, one must edit a pixmap, and put the text there!
          That's prohibitive.  One should be able to add some kind of
          generic toolbar button, with a plain icon or none at all, but
          which has a text label, without having to use a paint program.

    *Make it easy to have customized, mode-local toolbars.*
          In my `c-mode-hook', for example, I can add a couple of new
          keybindings, and delete a few others, and to do that, I don't
          have to duplicate the entire definition of the `c-mode-map'.
          Making mode-local additions and subtractions to the toolbars
          should be as easy.

    *Make it easy to have customized, mode-local popup menus.*
          The same situation holds for the right-mouse-button popup
          menu; one should be able to add new commands to those menus
          without difficulty.  One problem is that each mode which does
          have a popup menu implements it in a different way...

*Make the External Widget work.*
     About half of the work is done to make a replacement for the
     `XmText' widget which offloads editing responsibility to an
     external Emacs process. Someone should finish that. The benefit
     here would be that then, any Motif program could be linked such
     that all editing happened with a real Emacs behind it. (If you're
     Athena-minded, flavor with `Text' instead of `XmText' - it's
     probably easy to make it work with both.)

     The part of this that is done already is the ability to run an
     Emacs screen on a Window object that has been created by another
     process (this is what the `ExternalClient.c' and `ExternalShell.c'
     stuff is.) What is left to be done is, adding the
     text-widget-editor aspects of this.

     First, the emacs screen being displayed on that window would have
     to be one without a modeline, and one which behaved sensibly in
     the context of "I am a small multi-line text area embedded in a
     dialog box" as opposed to "I am a full-on text editor and lord of
     all that I survey."

     Second, the API that the (non-emacs-aware) user of the `XmText'
     widget expects would need to be implemented: give the caller the
     ability to pull the edited text string back out, and so on.  The
     idea here being, hooking up emacs as the widget editor should be as
     transparent as possible.

*Bring the debugger interface into the eighties.*
     Some of you may have seen my `gdb-highlight.el' package, that I
     posted to gnu.emacs.sources last month. I think it's really cool,
     but there should be a lot more work in that direction.  For those
     of you who haven't seen it, what it does is watch text that gets
     inserted into the `*gdb*' buffer and make very nearly everything
     be clickable and have a context-sensitive menu. Generally, the
     types that are noticed are:

        * function names;

        * variable and parameter names;

        * structure slots;

        * source file names;

        * type names;

        * breakpoint numbers;

        * stack frame numbers.

     Any time one of those objects is presented in the `*gdb*' buffer,
     it is mousable. Clicking middle button on it takes some default
     action (edits the function, selects the stack frame, disables the
     breakpoint, ...) Clicking the right button pops up a menu of
     commands, including commands specific to the object under the
     mouse, and/or other objects on the same line.

     So that's all well and good, and I get far more joy out of what
     this code does for me than I expected, but there are still a bunch
     of limitations. The debugger interface needs to do much, much more.

    *Make gdbsrc-mode not suck.*
          The idea behind `gdbsrc-mode' is on the side of the angels:
          one should be able to focus on the source code and not on the
          debugger buffer, absolutely. But the implementation is just
          awful.

          First and foremost, it should not change "modes" (in the more
          general sense). Any commands that it defines should be on
          keys which are exclusively used for that purpose, not keys
          which are normally self-inserting. I can't be the only person
          who usually has occasion to actually _edit_ the sources which
          the debugger has chosen to display! Switching into and out of
          `gdbsrc-mode' is prohibitive.

          I want to be looking at my sources at all times, yet I don't
          want to have to give up my source-editing gestures. I think
          the right way to accomplish this is to put the gdbsrc
          commands on the toolbar and on popup menus; or to let the
          user define their own keys (I could see devoting my
          <kp_enter> key to "step", or something common like that.)

          Also it's extremely frustrating that one can't turn off
          gdbsrc mode once it has been loaded, without exiting and
          restarting emacs; that alone means that I'd probably never
          take the time to learn how to use it, without first having
          taken the time to repair it...

    *Make it easier access to variable values.*
          I want to be able to double-click on a variable name to
          highlight it, and then drag it to the debugger window to have
          its value printed.

          I want gestures that let me write as well as read: for
          example, to store value A into slot B.

    *Make all breakpoints visible.*
          Any time there is a running gdb which has breakpoints, the
          buffers holding the lines on which those breakpoints are set
          should have icons in them. These icons should be
          context-sensitive: I should be able to pop up a menu to
          enable or disable them, to delete them, to change their
          commands or conditions.

          I should also be able to _move_ them. It's annoying when you
          have a breakpoint with a complex condition or command on it,
          and then you realize that you really want it to be at a
          different location. I want to be able to drag-and-drop the
          icon to its new home.

    *Make a debugger status display window.*
             *  I want a window off to the side that shows persistent
               information - it should have a pane which is a
               drag-editable, drag-reorderable representation of the
               elements on gdb's "display" list; they should be
               displayed here instead of being just dumped in with the
               rest of the output in the `*gdb*' buffer.

             *  I want a pane that displays the current call-stack and
               nothing else. I want a pane that displays the arguments
               and locals of the currently-selected frame and nothing
               else. I want these both to update as I move around on
               the stack.

             *  Since the unfortunate reality is that excavating this
               information from gdb can be slow, it would be a good
               idea for these panes to have a toggle button on them
               which meant "stop updating", so that when I want to move
               fast, I can, but I can easily get the display back when
               I need it again.

          The reason for all of this is that I spend entirely too much
          time scrolling around in the `*gdb*' buffer; with
          gdb-highlight, I can just click on a line in the backtrace
          output to go to that frame, but I find that I spend a lot of
          time _looking_ for that backtrace: since it's mixed in with
          all the other random output, I waste time looking around for
          things (and usually just give up and type "`bt'" again, then
          thrash around as the buffer scrolls, and I try to find the
          lower frames that I'm interested in, as they have invariably
          scrolled off the window already...

    *Save and restore breakpoints across emacs/debugger sessions.*
          This would be especially handy given that gdb leaks like a
          sieve, and with a big program, I only get a few dozen
          relink-and-rerun attempts before gdb has blown my swap space.

    *Keep breakpoints in sync with source lines.*
          When a program is recompiled and then reloaded into gdb, the
          breakpoints often end up in less-than-useful places. For
          example, when I edit text which occurs in a file anywhere
          before a breakpoint, emacs is aware that the line of the bp
          hasn't changed, but just that it is in a different place
          relative to the top of the file. Gdb doesn't know this, so
          your breakpoints end up getting set in the wrong places
          (usually the maximally inconvenient places, like _after_ a
          loop instead of _inside_ it). But emacs knows, so emacs should
          inform the debugger, and move the breakpoints back to the
          places they were intended to be.

     (Possibly the OOBR stuff does some of this, but can't tell, because
     I've never been able to get it to do anything but beep at me and
     mumble about environments. I find it pretty funny that the manual
     keeps explaining to me how intuitive it is, without actually
     giving me a clue how to launch it...)

*Add better dialog box features.*
     It'd be nice to be able to create more complex dialog boxes from
     emacs-lisp: ones with checkboxes, radio button groups, text
     fields, and popup menus.

*Add embeddable dialog boxes.*
     One of the things that the now-defunct Energize code (the C side of
     it, that is) could do was embed a dialog box between the toolbar
     and the main text area - buffers could have control panels
     associated with them, that had all kinds of complex behavior.

*Make the mark-stack be visible.*
     You know, I've encountered people who have been using emacs for
     years, and never use the mark stack for navigation. I can't live
     without it; "`C-u C-SPC'" is among my most common gestures.

       1.  It would be a lot easier to realize what's going to happen
          if the marks on the mark stack were visible. They could be
          displayed as small "caret" glyphs, for example; something
          large enough to be visible, but not easily mistaken for a
          character or for the cursor.

       2.  The marks and the selected region should be visible in the
          scrollbar as well - I don't remember where I first saw this
          idea, but it's very cool: there's a second,
          less-strongly-rendered "thumb" in the scrollbar which
          indicates the position and size of the selection; and there
          are tiny tick-marks which indicate the positions of the saved
          points.

       3.  Markers which are in registers (`point-to-register', `C-x
          /') should be displayed differently (more prominent.)

       4.  It'd be cool if you could pick up markers and move them
          around, to adjust the points you'll be coming back to later.

*Write a new garbage collector.*
     The emacs GC is very primitive; it is also, fortunately, a rather
     well isolated module, and it would not be a very big task to swap
     it with a new one (once that new one was written, that is.) Someone
     should go bone up on modern GC techniques, and then just dive right
     in...

*Add support for lexical scope to the emacs-lisp runtime.*
     Yadda yadda, this list goes to eleven.

Subject: *Re: XEmacs wishlist* Date: Wed, 14 May 1997 16:18:23 -0700
From: Jamie Zawinski <jwz@netscape.com> Newsgroups: comp.emacs.xemacs,
comp.emacs

   Andreas Schwab wrote:

     _Use `C-u C-x (': _

     _start-kbd-macro:
     Non-nil arg (prefix arg) means append to last macro defined; This
     begins by re-executing that macro as if you typed it again. _

   Cool, I didn't know it did that...

   But it only lets you append. I often want to prepend, or embed the
macro multiple times (motion 1, C-x e, motion 2, C-x e, motion 3.)

21.2 Showstoppers
-----------------

Author: Ben Wing (mailto:ben@xemacs.org)

   DISTRIBUTION ISSUES

   A. Unified Source Tarball.

   Packages go under root/lib/xemacs/xemacs-packages and no one ever has
to mess with -package-path and the result can be moved from one
directory to another pre- or post-install.

   Unified Binary Tarballs with Packages.

   Same principles as above.

   If people complain, we can also provide split binary tarballs
(architecture dependent and independent) and place these files in a
subdirectory so as not to confuse the majority just looking for one
tarball.

   Under Windows, we need to provide a WISE-style GUI setup program.
It's already there but needs some work so you can select "all" packages
easily (should be the default).

   Parallel Root and Package Trees.

   If the user downloads separately, the main source and the packages,
he will naturally untar them into the same directory. This results in
the parallel root and package structure. We should support this as a
"last resort," i.e., if we find no packages anywhere and are about to
resign ourselves to not having packages, then look for a parallel
package tree. The user who sets things up like this should be able to
either run in place or "make install" and get a proper installed
XEmacs. Never should the user have to touch -package-path.

   II. WINDOWS PRINTING

   Looks like the internals are done but not the GUI. This must be
working in 21.2.

   III. WINDOWS MULE

   Basic support should be there. There's already a patch to get things
started and I'll be doing more work to make this real.

   IV. GUTTER ETC.

   This stuff needs to be "stable" and generally free from bugs. Any
APIs we create need to be well-reviewed or marked clearly as
experimental.

   V. PORTABLE DUMPER

   Last bits need to be cleaned up. This should be made the "default"
for a while to flush-out problems. Under Microsoft Windows, Portable
Dumper must be the default in 21.2 because of the problems with the
existing dump process.

   COMMENT: I'd like to feature freeze this pretty soon and create a
21.3 tree where all of my major overhauls of Mule-related stuff will go
in. At the same time or around, we need to do the move-around in the
repository (or create a new one) and "upgrade" to the latest CVS server.


File: internals.info,  Node: Future Work -- Elisp Compatibility Package,  Next: Future Work -- Drag-n-Drop,  Prev: Future Work -- General Suggestions,  Up: Future Work

40.2 Future Work - Elisp Compatibility Package
==============================================

Author: Ben Wing (mailto:ben@xemacs.org)

   A while ago I created a package called Sysdep, which aimed to be a
forward compatibility package for Elisp.  The idea was that instead of
having to write your package using the oldest version of Emacs that you
wanted to support, you could use the newest XEmacs API, and then simply
load the Sysdep package, which would automatically define the new API in
terms of older APIs as necessary.  The idea of this package was good,
but its design wasn't perfect, and it wasn't widely adopted.  I propose
a new package called Compat that corrects the design flaws in Sysdep,
and hopefully will be adopted by most of the major packages.

   In addition, this package will provide macros that can be used to
bracket code as necessary to disable byte compiler warnings generated as
a result of supporting the APIs of different versions of Emacs; or
rather the Compat package strives to provide useful constructs to make
doing this support easier, and these constructs have the side effect of
not causing spurious byte compiler warnings.  The idea here is that it
should be possible to create well-written, clean, and understandable
Elisp that supports both older and newer APIs, and has no byte compiler
warnings.  Currently many warnings are unavoidable, and as a result,
they are simply ignored, which also causes a lot of legitimate warnings
to be ignored.

   The approach taken by the Sysdep package to make sure that the newest
API was always supported was fairly simple: when the Sysdep package was
loaded, it checked for the existence of new API functions, and if they
weren't defined, it defined them in terms of older API functions that
were defined.  This had the advantage that the checks for which API
functions were defined were done only once at load time rather than each
time the function was called.  However, the fact that the new APIs were
globally defined caused a lot of problems with unwanted interactions,
both with other versions of the Sysdep package provided as part of other
packages, and simply with compatibility code of other sorts in packages
that would determine whether an API existed by checking for the
existence of certain functions within that API.  In addition, the Sysdep
package did not scale well because it defined all of the functions that
it supported, regardless of whether or not they were used.

   The Compat package remedies the first problem by ensuring that the
new APIs are defined only within the lexical scope of the packages that
actually make use of the Compat package.  It remedies the second problem
by ensuring that only definitions of functions that are actually used
are loaded.  This all works roughly according to the following scheme:

  1.  Part of the Compat package is a module called the Compat
     generator.  This module is actually run as an additional step
     during byte compilation of a package that uses Compat.  This can
     happen either through the makefile or through the use of an
     `eval-when-compile' call within the package code itself.  What the
     generator does is scan all of the Lisp code in the package,
     determine which function calls are made that the Compat package
     knows about, and generates custom `compat' code that conditionally
     defines just these functions when the package is loaded.  The
     custom `compat' code can either be written to a separate Lisp file
     (for use with multi-file packages), or inserted into the beginning
     of the Lisp file of a single file package.  (In the latter case,
     the package indicates where this generated code should go through
     the use of magic comments that mark the beginning and end of the
     section.  Some will say that doing this trick is bad juju, but I
     have done this sort of thing before, and it works very well in
     practice).

  2.  The functions in the custom `compat' code have their names
     prefixed with both the name of the package and the word `compat',
     ensuring that there will be no name space conflicts with other
     functions in the same package, or with other packages that make
     use of the Compat package.

  3.  The actual definitions of the functions in the custom `compat'
     code are determined at run time.  When the equivalent API already
     exists, the wrapper functions are simply defined directly in terms
     of the actual functions, so that the only run time overhead from
     using the Compat package is one additional function call.
     (Alternatively, even this small overhead could be avoided by
     retrieving the definitions of the actual functions and supplying
     them as the definitions of the wrapper functions.  However, this
     appears to me to not be completely safe.  For example, it might
     have bad interactions with the advice package).

  4.  The code that wants to make use of the custom `compat' code is
     bracketed by a call to the construct `compat-execute'.  What this
     actually does is lexically bind all of the function names that are
     being redefined with macro functions by using the Common Lisp
     macro macrolet.  (The definition of this macro is in the CL
     package, but in order for things to work on all platforms, the
     definition of this macro will presumably have to be copied and
     inserted into the custom `compat' code).


   In addition, the Compat package should define the macro
`compat-if-fboundp'.  Similar macros such as `compile-when-fboundp' and
`compile-case-fboundp' could be defined using similar principles).  The
`compat-if-fboundp' macro behaves just like an `(if (fboundp ...) ...)'
clause when executed, but in addition, when it's compiled, it ensures
that the code inside the `if-true' sub-block will not cause any byte
compiler warnings about the function in question being unbound.  I
think that the way to implement this would be to make
`compat-if-fboundp' be a macro that does what it's supposed to do, but
which defines its own byte code handler, which ensures that the
particular warning in question will be suppressed.  (Actually ensuring
that just the warning in question is suppressed, and not any others,
might be rather tricky.  It certainly requires further thought).

   Note: An alternative way of avoiding both warnings about unbound
functions and warnings about obsolete functions is to just call the
function in question by using `funcall', instead of calling the
function directly.  This seems rather inelegant to me, though, and
doesn't make it obvious why the function is being called in such a
roundabout manner.  Perhaps the Compat package should also provide a
macro `compat-funcall', which works exactly like `funcall', but which
indicates to anyone reading the code why the code is expressed in such
a fashion.

   If you're wondering how to implement the part of the Compat generator
where it scans Lisp code to find function calls for functions that it
wants to do something about, I think the best way is to simply process
the code using the Lisp function `read' and recursively descend any
lists looking for function names as the first element of any list
encountered.  This might extract out a few more functions than are
actually called, but it is almost certainly safer than doing anything
trickier like byte compiling the code, and attempting to look for
function calls in the result.  (It could also be argued that the names
of the functions should be extracted, not only from the first element of
lists, but anywhere `symbol' occurs.  For example, to catch places
where a function is called using `funcall' or `apply'.  However, such
uses of functions would not be affected by the surrounding macrolet
call, and so there doesn't appear to be any point in extracting them).


File: internals.info,  Node: Future Work -- Drag-n-Drop,  Next: Future Work -- Standard Interface for Enabling Extensions,  Prev: Future Work -- Elisp Compatibility Package,  Up: Future Work

40.3 Future Work - Drag-n-Drop
==============================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* I propose completely redoing the drag-n-drop interface
to make it powerful and extensible enough to support such concepts as
drag over and drag under visuals and context menus invoked when a drag
is done with the right mouse button, to allow drop handlers to be
defined for all sorts of graphical elements including buffers, extents,
mode lines, toolbar items, menubar items, glyphs, etc., and to allow
different packages to add and remove drop handlers for the same drop
sites without interfering with each other.  The changes are extensive
enough that I think they can only be implemented in version 22, and the
drag-n-drop interface should remain experimental until then.

   The new drag-n-drop interface centers around the twin concepts of
"drop site" and "drop handler".  A "drop site" specifies a particular
graphical element where an object can be dropped onto, and a "drop
handler" encapsulates all of the behavior that happens when such an
object is dragged over and dropped onto a drop site.

   Each drop site has an object associated with it which is passed to
functions that are part of the drop handlers associated with that site.
The type of this object depends on the graphical element that comprises
the drop site.  The drop site object can be a buffer, an extent, a
glyph, a menu path, a toolbar item path, etc.  (These last two object
types are defined in Lisp Interface Changes (lisp-interface.html) in
the sections on menu and toolbar API changes.  If we wanted to allow
drops onto other kinds of drop sites, for example mode lines, we would
have to create corresponding path objects).  Each such object type
should be able to be accessed using the generalized property interface
defined above, and should have a property called `drop-handlers'
associated with it that specifies all of the drop handlers associated
with the drop site.  Normally, this property is not accessed directly,
but instead by using the drop handler API defined below, and Lisp
packages should not make any assumptions about the format of the data
contained in the `drop-handlers' property.

   Each drop handler has an object of type `drop-handler' associated
with it, whose primary purpose is to be a container for the various
properties associated with a particular drop handler.  These could
include, for example, a function invoked when the drop occurs, a context
menu invoked when a drop occurs as a result of a drag with the right
mouse button, functions invoked when a dragged object enters, leaves, or
moves within a drop site, the shape that the mouse pointer changes to
when an object is dragged over a drop site that allows this particular
object to be dropped onto it, the MIME types (actually a regular
expression matching the MIME types) of the allowable objects that can be
dropped onto the drop site, a "package tag" (a symbol specifying the
package that created the drop handler, used for identification
purposes), etc.  The drop handler object is passed to the functions that
are invoked as a result of a drag or a drop, most likely indirectly as
one of the properties of the drag or drop event passed to the function.
Properties of a drop handler object are accessed and modified in the
standard fashion using the generalized property interface.

   A drop handler is added to a drop site using the `add-drop-handler'
function.  The drop handler itself can either be created separately
using the `make-drop-handler' function and then passed in as one of the
parameters to `add-drop-handler', or it will be created automatically
by the `add-drop-handler' function, if the drop handler argument is
omitted, but keyword arguments corresponding to the valid keyword
properties for a drop handler are specified in the `add-drop-handler'
call.  Other functions, such as `find-drop-handler', `add-drop-handler'
(when specifying a drop handler before which the drop handler in
question is to be added), `remove-drop-handler' etc. should be defined
with obvious semantics.  All of these functions take or return a drop
site object which, as mentioned above, can be one of several object
types corresponding to graphical elements.  Defined drop handler
functions locate a particular drop handler using either the `MIME-type'
or `package-tag' property of the drop handler, as defined above.

   Logically, the drop handlers associated with a particular drop site
are an ordered list.  The first drop handler whose specified MIME type
matches the MIME type of the object being dragged or dropped controls
what happens to this object.  This is important particularly because the
specified MIME type of the drop handler can be a regular expression
that, for example, matches all audio objects with any sub-type.

   In the current drag-n-drop API, there is a distinction made between
objects with an associated MIME type and objects with an associated URL.
I think that this distinction is arbitrary, and should not exist.  All
objects should have a MIME type associated with them, and a new
XEmacs-specific MIME type should be defined for URLs, file names, etc.
as necessary.  I am not even sure that this is necessary, however, as
the MIME specification may specify a general concept of a pointer or
link to an object, which is exactly what we want.  Also in some cases
(for example, the name of a file that is locally available), the pointer
or link will have another MIME type associated with it, which is the
type of the object that is being pointed to.  I am not quite sure how we
should handle URL and file name objects being dragged, but I am positive
that it needs to be integrated with the mechanism used when an object
itself is being dragged or dropped.

   As is described in a separate page (misc-user-event.html), the
`misc-user-event' event type should be removed and split up into a
number of separate event types.  Two such event types would be
`drag-event' and `drop-event'.  A drop event is used when an object is
actually dropped, and a drag event is used if a function is invoked as
part of the dragging process.  (Such a function would typically be used
to control what are called "drag under visuals", which are changes to
the appearance of the drop site reflecting the fact that a compatible
object is being dragged over it).  The drag events and drop events
encapsulate all of the information that is pertinent to the drag or
drop action occurring, including such information as the actual MIME
type of the object in question, the drop handler that caused a function
to be invoked, the mouse event (or possibly even a keyboard event)
corresponding to the user's action that is causing the drag or drop,
etc.  This event is always passed to any function that is invoked as a
result of the drag or drop.  There should never be any need to refer to
the `current-mouse-event' variable, and in fact, this variable should
not be changed at all during a drag or a drop.


File: internals.info,  Node: Future Work -- Standard Interface for Enabling Extensions,  Next: Future Work -- Better Initialization File Scheme,  Prev: Future Work -- Drag-n-Drop,  Up: Future Work

40.4 Future Work - Standard Interface for Enabling Extensions
=============================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* Apparently, if you know the name of a package (for
example, `fusion'), you can load it using the `require' function, but
there's no standard way to turn it on or turn it off.  The only way to
figure out how to do that is to go read the source file, where
hopefully the comments at the start tell you the appropriate magic
incantations that you need to run in order to turn the extension on or
off.  There really needs to be standard functions, such as
`enable-extension' and `disable-extension', to do this sort of thing.
It seems like a glaring omission that this isn't currently present, and
it's really surprising to me that nobody has remarked on this.

   The easy part of this is defining the interface, and I think it
should be done as soon as possible.  When the package is loaded, it
simply calls some standard function in the package system, and passes
it the names of enable and disable functions, or perhaps just one
function that takes an argument specifying whether to enable or
disable.  In any case, this data is kept in a table which is used by the
`enable-extension' and `disable-extension' function.  There should also
be functions such as `extension-enabled-p' and
`enabled-extension-list', and so on with obvious semantics.  The hard
part is actually getting packages to obey this standard interface, but
this is mitigated by the fact that the changes needed to support this
interface are so simple.

   I have been conceiving of these enabling and disabling functions as
turning the feature on or off globally.  It's probably also useful to
have a standard interface returning a extension on or off in just the
particular buffer.  Perhaps then the appropriate interface would involve
registering a single function that takes an argument that specifies
various things, such as turn off globally, turn on globally, turn on or
off in the current buffer, etc.

   Part of this interface should specify the correct way to define
global key bindings.  The correct rule for this, of course, is that the
key bindings should not happen when the package is loaded, which is
often how things are currently done, but only when the extension is
actually enabled.  The key bindings should go away when the extension is
disabled.  I think that in order to support this properly, we should
expand the keymap interface slightly, so that in addition to other
properties associated with each key binding is a list of shadow
bindings.  Then there should be a function called
`define-key-shadowing', which is just like `define-key' but which also
remembers the previous key binding in a shadow list.  Then there can be
another function, something like `undefine-key', which restores the
binding to the most recently added item on the shadow list.  There are
already hash tables associated with each key binding, and it should be
easy to stuff additional values, such as a shadow list, into the hash
table.  Probably there should also be functions called
`global-set-key-shadowing' and `global-unset-key-shadowing' with
obvious semantics.

   Once this interface is defined, it should be easy to expand the
custom package so it knows about this interface.  Then it will be
possible to put all sorts of extensions on the options menu so that
they could be turned off and turned on very easily, and then when you
save the options out to a file, the design settings for whether these
extensions are enabled or not are saved out with it.  A whole lot of
custom junk that's been added to a lot of different packages could be
removed.  After doing this, we might want to think of a way to classify
extensions according to how likely we think the user will want to use
them.  This way we can avoid the problem of having a list of 100
extensions and the user not being able to figure out which ones might
be useful.  Perhaps the most useful extensions would appear immediately
on the extensions menu, and the less useful ones would appear in a
submenu of that, and another submenu might contain even less useful
extensions.  Of course the package authors might not be too happy with
this, but the users probably will be.  I think this at least deserves a
thought, although it's possible you might simply want to maintain a
list on the web site of extensions and a judgment on first of all, how
commonly a user might want this extension, and second of all, how well
written and bug-free the package is.  Both of these sorts of judgments
could be obtained by doing user surveys if need be.


File: internals.info,  Node: Future Work -- Better Initialization File Scheme,  Next: Future Work -- Keyword Parameters,  Prev: Future Work -- Standard Interface for Enabling Extensions,  Up: Future Work

40.5 Future Work - Better Initialization File Scheme
====================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* A proposal is outlined for converting XEmacs to use the
`.xemacs' subdirectory for its initialization files instead of putting
them in the user's home directory.  In the process, a general
pre-initialization scheme is created whereby all of the initialization
parameters, such as the location of the initialization files, whether
these files are loaded or not, where the initial frame is created, etc.
that are currently specified by command line arguments, by environment
variables, and other means, can be specified in a uniform way using
Lisp code.  Reasonable default behavior for everything will still be
provided, and the older, simpler means can be used if desired.
Compatibility with the current location and name of the initialization
file, and the current ill-chosen use for the `.xemacs' directory is
maintained, and the problem of how to gracefully migrate a user from the
old scheme into the new scheme while still allowing the user to use GNU
Emacs or older versions of XEmacs is solved.  A proposal for changing
the way that the initial frame is mapped is also outlined; this would
allow the user's initialization file to control the way that the initial
frame appears without resorting to hacks, while still making echo area
messages visible as they appear, and allowing the user to debug errors
in the initialization file.

Principles in the new scheme
----------------------------

  1.  XEmacs has a defined "pre-initialization process".  This process,
     whose purpose is to compute the values of the parameters that
     control how the initializiaton process proceeds, occurs as early
     as possible after the Lisp engine has been initialized, and in
     particular, it occurs before any devices have been opened, or
     before any initialization parameters are set that could reasonably
     be expected to be changed.  In fact, the pre-initialization
     process should take care of setting these parameters.  The code
     that implements the pre-initialization process should be written
     in Lisp and should be called from the Lisp function
     `normal-top-level', and the general way that the user customizes
     this process should also be done using Lisp code.

  2.  The pre-initialization process involves a number of properties,
     for example the directory containing the user initialization files
     (normally the `.xemacs' subdirectory), the name of the user init
     file, the name of the custom init file, where and what type the
     initial device is, whether and when the initial frame is mapped,
     etc.  A standard interface is provided for getting and setting the
     values of these properties using functions such as
     `set-pre-init-property', `pre-init-property', etc.  At various
     points during the pre-initialization process, the value of many of
     these properties can be undecided, which means that at the end of
     the process, the value of these properties will be derived from
     other properties in some fashion that is specific to each property.

  3.  The default values of these properties are set first from the
     registry under Windows, then from environment variables, then from
     command line switches, such as `-q' and `-nw'.

  4.  One of the command line switches is `-pre-init', whose value is a
     Lisp expression to be evaluated at pre-initialization time,
     similar to the `-eval' command line switch.  This allows any
     pre-initialization property to be set from the command line.

  5.  Let's define the term "to determine a pre-initialization
     property" to mean if the value of a property is undetermined, it
     is computed and set according to a rule that is specific to the
     property.  Then after the pre-init properties are initialized from
     the registry, from the environment variables, from command line
     arguments, two of the pre-init properties (specifically the init
     file directory and the location of the "pre-init file") are
     determined.  The purpose of the pre-init file is to contain Lisp
     code that is run at pre-initialization time, and to control how
     the initialization proceeds.  It is a bit similar to the standard
     init file, but the code in the pre-init file shouldn't do anything
     other than set pre-init properties.  Executing any code that does
     I/O might not produce expected results because the only device that
     will exist at the time is probably a stream device connected to the
     standard I/O of the XEmacs process.

  6.  After the pre-init file has been run, all of the rest of the
     pre-init properties are determined, and these values are then used
     to control the initialization process.  Some of the rules used in
     determining specific properties are:

       1.  If the `.xemacs' sub-directory exists, and it's not
          obviously a package root (which probably means that it
          contains a file like `init.el' or `pre-init.el', or if
          neither of those files is present, then it doesn't contain
          any sub-directories or files that look like what would be in
          a package root), then it becomes the value of the init file
          directory.  Otherwise the user's home directory is used.

       2.  If the init file directory is the user's home directory,
          then the init file is called `.emacs'.  Otherwise, it's
          called `init.el'.

       3.  If the init file directory is the user's home directory,
          then the pre-init file is called `.xemacs-pre-init.el'.
          Otherwise it's called `pre-init.el'. (One of the reasons for
          this rule has to do with the dialog box that might be
          displayed at startup.  This will be described below.)

       4.  If the init file directory is the user's home directory,
          then the custom init file is called `.xemacs-custom-init.el'.
          Otherwise, it's called `custom-init.el'.


  7.  After the first normal device is created, but before any frames
     are created on it, the XEmacs initialization code checks to see if
     the old init file scheme is being used, which is to say that the
     init file directory is the same as the user's home directory.  If
     that's the case, then normally a dialog box comes up (or a
     question is asked on the terminal if XEmacs is being run in a
     non-windowing mode) which asks if the user wants to migrate his
     initialization files to the new scheme.  The possible responses
     are *Yes*, *No*, and *No, and don't ask this again*.  If this last
     response is chosen, then the file `.xemacs-pre-init.el' in the
     user's home directory is created or appended to with a line of
     Lisp code that sets up a pre-init property indicating that this
     dialog box shouldn't come up again.  If the *Yes* option is
     chosen, then any package root files in `.xemacs' are moved into
     `.xemacs/packages', the file `.emacs' is moved into
     `.xemacs/init.el' and `.emacs' in the home directory becomes a
     symlink to this file.  This way some compatibility is still
     maintained with GNU Emacs and older versions of XEmacs.  The code
     that implements this has to be written very carefully to make sure
     that it doesn't accidentally delete or mess up any of the files
     that get moved around.


The custom init file
--------------------

The "custom init file" is where the custom package writes its options.
This obviously needs to be a separate file from the standard init file.
It should also be loaded before the init file rather than after, as is
usually done currently, so that the init file can override these
options if it wants to.

Frame mapping
-------------

In addition to the above scheme, the way that XEmacs handles mapping the
initial frame should be changed.  However, this change perhaps should be
delayed to a later version of XEmacs because of the user visible changes
that it entails and the possible breakage in people's init files that
might occur. (For example, if the rest of the scheme is implemented in
21.2, then this part of the scheme might want to be delayed until
version 22.)  The basic idea is that the initial frame is not created
before the initialization file is run, but instead a banner frame is
created containing the XEmacs logo, a button that allows the user to
cancel the execution of the init file and an area where messages that
are output in the process of running this file are displayed.  This area
should contain a number of lines, which makes it better than the current
scheme where only the last message is visible.  After the init file is
done, the initial frame is mapped.  This way the init file can make face
changes and other such modifications that affect initial frame and then
have the initial frame correctly come up with these changes and not see
any frame dancing or other problems that exist currently.

   There should be a function that allows the initialization file to
explicitly create and map the first frame if it wants to.  There should
also be a pre-init property that controls whether the banner frame
appears (of course it defaults to true) a property controlling when the
initial frame is created (before or after the init file, defaulting to
after), and a property controlling whether the initial frame is mapped
(normally true, but will be false if the `-unmapped' command line
argument is given).

   If an error occurs in the init file, then the initial frame should
always be created and mapped at that time so that the error is displayed
and the debugger has a place to be invoked.


File: internals.info,  Node: Future Work -- Keyword Parameters,  Next: Future Work -- Property Interface Changes,  Prev: Future Work -- Better Initialization File Scheme,  Up: Future Work

40.6 Future Work - Keyword Parameters
=====================================

Author: Ben Wing (mailto:ben@xemacs.org)

   NOTE: These changes are partly motivated by the various
user-interface changes elsewhere in this document, and partly for Mule
support.  In general the various APIs in this document would benefit
greatly from built-in keywords.

   I would like to make keyword parameters an integral part of Elisp.
The idea here is that you use the `&amp;key' identifier in the
parameter list of a function and all of the following parameters
specified are keyword parameters.  This means that when these arguments
are specified in a function call, they are immediately preceded in the
argument list by a "keyword", which is a symbol beginning with the `:'
character.  This allows any argument to be specified independently of
any other argument with no need to place the arguments in any
particular order.  This is particularly useful for functions that take
many optional parameters; using keyword parameters makes the code much
cleaner and easier to understand.

   The `cl' package already provides keyword parameters of a sort, but
I would like to make this more integrated and useable in a standard
fashion.  The interface that I am proposing is essentially compatible
with the keyword interface in Common Lisp, but it may be a subset of the
Common Lisp functionality, especially in the first implementation.
There is one departure from the Common Lisp specification that I would
like to make in order to make it much easier to add keyword parameters
to existing functions with optional parameters, and in general, to make
optional and keyword parameters coexist more easily.  The Common Lisp
specification indicates that if a function has both optional and keyword
parameters, the optional parameters are always processed before the
keyword parameters.  This means, for example, that if a function has
three required parameters, two optional parameters, and some number of
keyword parameters following, and the program attempts to call this
function by passing in the three required arguments, and then some
keyword arguments, the first keyword specified and the argument
following it get assigned to the first and second optional parameters as
specified in the function definition.  This is certainly not what is
intended, and means that if a function defines both optional and keyword
parameters, any calls of this function must specify `nil' for all of
the optional arguments before using any keywords.  If the function
definition is later changed to add more optional parameters, all
existing calls to this function that use any keyword arguments will
break.  This problem goes away if we simply process keyword parameters
before the optional parameters.

   The primary changes needed to support the keyword syntax are:

  1.  The subr object type needs to be modified to contain additional
     slots for the number and names of any keyword parameters.

  2.  The implementation of the `funcall' function needs to be modified
     so that it knows how to process keyword parameters.  This is the
     only place that will require very much intricate coding, and much
     of the logic that would need to be added can be lifted directly
     from the `cl' code.

  3.  A new macro, similar to the `DEFUN' macro, and probably called
     `DEFUN_WITH_KEYWORDS', needs to be defined so that built-in Lisp
     primitives containing keywords can be created.  Now, the
     `DEFUN_WITH_KEYWORDS' macro should take an additional parameter
     which is a string, which consists of the part of the lambda list
     declaration for this primitive that begins with the `&amp;key'
     specifier.  This string is parsed in the `DEFSUBR' macro during
     XEmacs initialization, and is converted into the appropriate
     structure that needs to be stored into the subr object.  In
     addition, the MAX_ARGS parameter of the `DEFUN' macro needs to be
     incremented by the number of keyword parameters and these
     parameters are passed to the C function simply as extra parameters
     at the end.  The `DEFSUBR' macro can sort out the actual number of
     required, optional and keyword parameters that the function takes,
     once it has parsed the keyword parameter string.  (An alternative
     that might make the declaration of a primitive a little bit easier
     to understand would involve adding another parameter to the
     `DEFUN_WITH_KEYWORDS' macro that specifies the number of keyword
     parameters.  However, this would require some additional
     complexity in the preprocessor definition of the
     `DEFUN_WITH_KEYWORDS' macro, and probably isn't worth
     implementing).

  4.  The byte compiler would have to be modified slightly so that it
     knows about keyword parameters when it parses the parameter
     declaration of a function.  For example, so that it issues the
     correct warnings concerning calls to that function with incorrect
     arguments.

  5.  The `make-docfile' program would have to be modified so that it
     generates the correct parameter lists for primitives defined using
     the `DEFUN_WITH_KEYWORDS' macro.

  6.  Possibly other aspects of the help system that deal with function
     descriptions might have to be modified.

  7.  A helper function might need to be defined to make it easier for
     primitives that use both the `&amp;rest' and `&amp;key' specifiers
     to parse their argument lists.


Internal API for C primitives with keywords - necessary for many of the new Mule APIs being defined.
----------------------------------------------------------------------------------------------------

       DEFUN_WITH_KEYWORDS (Ffoo, "foo", 2, 5, 6, ALLOW_OTHER_KEYWORDS,
           (ichi, ARG_NIL), (ni, ARG_NIL), (san, ARG_UNBOUND), 0,
           (arg1, arg2, arg3, arg4, arg5)
           )
       {
         ...
       }

       -> C fun of 12 args:

       (arg1, ... arg5, ichi, ..., roku, other keywords)

       Circled in blue is actual example declaration

       DEFUN_WITH_KEYWORDS (Ffoo, "foo", 1,2,0 (bar, baz) <- arg list
       [ MIN ARGS, MAX ARGS, something that could be REST, SPECIFY_DEFAULT or
       REST_SPEC]

       [#KEYWORDS [ ALLOW_OTHER, SPECIFY_DEFAULT, ALLOW_OTHER_SPECIFY_DEFAULT
       6, ALLOW_OTHER_SPECIFY_DEFAULT,

       (ichi, 0) (ni, 0), (san, DEFAULT_UNBOUND), (shi, "t"), (go, "5"),
       (roku, "(current-buffer)")
       <- specifies arguments, default values (string to be read into Lisp
          data during init; then forms evalled at fn ref time.

       ,0 <- [INTERACTIVE SPEC] )

       LO = Lisp_Object

       -> LO Ffoo (LO bar, LO baz, LO ichi, LO ni, LO san, LO shi, LO go,
                   LO roku, int numkeywords, LO *other_keywords)

       #define DEFUN_WITH_KEYWORDS (fun, funstr, minargs, maxargs, argspec, \
                #args, num_keywords, keywordspec, keywords, intspec) \
       LO fun (DWK_ARGS (maxargs, args) \
               DWK_KEYWORDS (num_keywords, keywordspec, keywords))

       #define DWK_KEYWORDS (num_keywords, keywordspec, keywords) \
               DWK_KEYWORDS ## keywordspec (keywords)
               DWK_OTHER_KEYWORDS ## keywordspec)

       #define DWK_KEYWORDS_ALLOW_OTHER (x,y)
               DWK_KEYWORDS (x,y)

       #define DWK_KEYWORDS_ALLOW_OTHER_SPECIFICATIONS (x,y)
               DWK_KEYWORDS_SPECIFY_DEFAULT (x,y)

       #define DWK_KEYWORDS_SPECIFY_DEFAULT (numkey, key)
               ARGLIST_CAR ## numkey key

       #define ARGLT_GRZ (x,y) LO CAR x, LO CAR y


File: internals.info,  Node: Future Work -- Property Interface Changes,  Next: Future Work -- Toolbars,  Prev: Future Work -- Keyword Parameters,  Up: Future Work

40.7 Future Work - Property Interface Changes
=============================================

Author: Ben Wing (mailto:ben@xemacs.org)

   In my past work on XEmacs, I already expanded the standard property
functions of `get', `put', and `remprop' to work on objects other than
symbols and defined an additional function `object-plist' for this
interface.  I'd like to expand this interface further and advertise it
as the standard way to make property changes in objects, especially the
new objects that are going to be defined in order to support the added
user interface features of version 22.  My proposed changes are as
follows:

  1.  A new concept associated with each property called a "default
     value" is introduced.  (This concept already exists, but not in a
     well-defined way.) The default value is the value that the
     property assumes for certain value retrieval functions such as
     `get' when it is "unbound", which is to say that its value has not
     been explicitly specified. Note: the way to make a property
     unbound is to call `remprop'.  Note also that for some built-in
     properties, setting the property to its default value is
     equivalent to making it unbound.

  2.  The behavior of the `get' function is modified.  If the `get'
     function is called on a property that is unbound and the third,
     optional DEFAULT argument is `nil', then the default value of the
     property is returned.  If the DEFAULT argument is not `nil', then
     whatever was specified as the value of this argument is returned.
     For the most part, this is upwardly compatible with the existing
     definition of `get' because all user-defined properties have an
     initial default value of `nil'.  Code that calls the `get'
     function and specifies `nil' for the DEFAULT argument, and expects
     to get `nil' returned if the property is unbound, is almost
     certainly wrong anyway.

  3.  A new function, `get1' is defined.  This function does not take a
     default argument like the `get' function.  Instead, if the property
     is unbound, an error is signaled.  Note: `get' can be implemented
     in terms of `get1'.

  4.  New functions `property-default-value' and `property-bound-p' are
     defined with the obvious semantics.

  5.  An additional function `property-built-in-p' is defined which
     takes two arguments, the first one being a symbol naming an object
     type, and the second one specifying a property, and indicates
     whether the property name has a built-in meaning for objects of
     that type.

  6.  It is not necessary, or even desirable, for all object types to
     allow user-defined properties.  It is always possible to simulate
     user-defined properties for an object by using a weak hash table.
     Therefore, whether an object allows a user to define properties or
     not should depend on the meaning of the object.  If an object does
     not allow user-defined properties, the `put' function should
     signal an error, such as `undefined-property', when given any
     property other than those that are predefined.

  7.  A function called `user-defined-properties-allowed-p' should be
     defined with the obvious semantics.  (See the previous item.)

  8.  Three more functions should be defined, called
     `built-in-property-name-list', `property-name-list', and
     `user-defined-property-name-list'.


   Another idea:

     (define-property-method
       predicate object-type
       predicate cons :(KEYWORD)  (all lists beginning with KEYWORD)

       :put putfun
       :get
       :remprop
       :object-props
       :clear-properties
       :map-properties

       e.g. (define-property-method 'hash-table
              :put #'(lambda (obj key value) (puthash key obj value)))


File: internals.info,  Node: Future Work -- Toolbars,  Next: Future Work -- Menu API Changes,  Prev: Future Work -- Property Interface Changes,  Up: Future Work

40.8 Future Work - Toolbars
===========================

* Menu:

* Future Work -- Easier Toolbar Customization::
* Future Work -- Toolbar Interface Changes::


File: internals.info,  Node: Future Work -- Easier Toolbar Customization,  Next: Future Work -- Toolbar Interface Changes,  Prev: Future Work -- Toolbars,  Up: Future Work -- Toolbars

40.8.1 Future Work - Easier Toolbar Customization
-------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* One of XEmacs' greatest strengths is its ability to be
customized endlessly.  Unfortunately, it is often too difficult to
figure out how to do this.  There has been some recent work like the
Custom package, which helps in this regard, but I think there's a lot
more work that needs to be done.  Here are some ideas (which certainly
could use some more thought).

   Although there is currently an `edit-toolbar' package, it is not
well integrated with XEmacs, and in general it is much too hard to
customize the way toolbars look.  I would like to see an interface that
works a bit like the way things work under Windows, where you can
right-click on a toolbar to get a menu of options that allows you to
change aspects of the toolbar.  The general idea is that if you
right-click on an item itself, you can do things to that item, whereas
if you right-click on a blank part of a toolbar, you can change the
properties of the toolbar.  Some of the items on the right-click menu
for a particular toolbar button should be specified by the button
itself.  Others should be standard.  For example, there should be an
*Execute* item which simply does what would happen if you left-click on
a toolbar button.  There should probably be a *Delete* item to get rid
of the toolbar button and a *Properties* item, which brings up a
property sheet that allows you to do things like change the icon and
the command string that's associated with the toolbar button.

   The options to change the appearance of the toolbar itself should
probably appear both on the context menu for specific buttons, and on
the menu that appears when you click on a blank part of the toolbar.
That way, if there isn't a blank part of the toolbar, you can still
change the toolbar appearance.  As for what appears in these items, in
Outlook Express, for example, there are three different menu items, one
of which is called *Buttons*, which brings up, or pops up a window
which allows you to edit the toolbar, which for us could pop up a new
frame, which is running `edit-toolbar.el'.  The second item is called
*Align*, which contains a submenu that says *Top*, *Bottom*, *Left*,
and *Right*, which will be just like setting the default toolbar
position.  The third one says *Text Labels*, which would just let you
select whether there are captions or not.  I think all three of these
are useful and are easy to implement in XEmacs.  These things also need
to be integrated with custom so that a user can control whether these
options apply to all sessions, and in such a case can save the settings
out to an options file.  `edit-toolbar.el' in particular needs to
integrate with custom.  Currently it has some sort of hokey stuff of
its own, which it saves out to a `.toolbar' file.  Another useful
option to have, once we draw the captions dynamically rather than using
pre-generated ones, would be the ability to change the font size of the
captions.  I'm sure that Kyle, for one, would appreciate this.

   (This is incomplete.....)


File: internals.info,  Node: Future Work -- Toolbar Interface Changes,  Prev: Future Work -- Easier Toolbar Customization,  Up: Future Work -- Toolbars

40.8.2 Future Work - Toolbar Interface Changes
----------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   I propose changing the way that toolbars are specified to make them
more flexible.

  1.  A new format for the vector that specifies a toolbar item is
     allowed.  In this format, the first three items of the vector are
     required and are, respectively, a caption, a glyph list, and a
     callback.  The glyph list and callback arguments are the same as
     in the current toolbar item specification, and the caption is a
     string specifying the caption text placed below the toolbar glyph.
     The caption text is required so that toolbar items can be
     identified for the purpose of retrieving and changing their
     property values.  Putting the caption first also makes it easy to
     distinguish between the new and the old toolbar item vector
     formats.  In the old format, the first item, the glyph list, is
     either a list or a symbol.  In the new format, the first item is a
     string.  In the new format, following the three required items,
     are optional keyword items specified using keywords in the same
     format as the menu item vector format.  The keywords that should
     be predefined are: `:help-echo', `:context-menu',
     `:drop-handlers', and `:enabled-p'.  The `:enabled-p' and
     `:help-echo' keyword arguments are the same as the third and
     fourth items in the old toolbar item vector format.  The
     `:context-menu' keyword is a list in standard menu format that
     specifies additional items that will appear when the context menu
     for the toolbar item is popped up.  (Typically, this happens when
     the right mouse button is clicked on the toolbar item).  The
     `:drop-handlers' keyword is for use by the new drag-n-drop
     interface (see Drag-n-Drop Interface Changes (drag-n-drop.html) ),
     and is not normally specified or modified directly.

  2.  Conceivably, there could also be keywords that are associated
     with a toolbar itself, rather than with a particular toolbar item.
     These keyword properties would be specified using keywords and
     arguments that occur before any toolbar item vectors, similarly to
     how things are done in menu specifications.  Possible properties
     could include `:captioned-p' (whether the captions are visible
     under the toolbar), `:glyphs-visible-p' (whether the toolbar
     glyphs are visible), and `:context-menu' (additional items that
     will appear on the context menus for all toolbar items and
     additionally will appear on the context menu that is popped up
     when the right mouse button is clicked over a portion of the
     toolbar that does not have any toolbar buttons in it).  The
     current standard practice with regards to such properties seems to
     be to have separate specifiers, such as `left-toolbar-width',
     `right-toolbar-width', `left-toolbar-visible-p',
     `right-toolbar-visible-p', etc.  It could easily be argued that
     there should be no such toolbar specifiers and that all such
     properties should be part of the toolbar instantiator itself.  In
     this scheme, the only separate specifiers that would exist for
     individual properties would be default values.  There are a lot of
     reasons why an interface change like this makes sense.  For
     example, currently when VM sets its toolbar, it also sets the
     toolbar width and similar properties.  If you change which edge of
     the frame the VM toolbar occurs in, VM will also have to go and
     modify all of the position-specific toolbar specifiers for all of
     the other properties associated with a toolbar.  It doesn't really
     seem to make sense to me for the user to be specifying the width
     and visibility and such of specific toolbars that are attached to
     specific edges because the user should be free to move the
     toolbars around and expect that all of the toolbar properties
     automatically move with the toolbar. (It is also easy to imagine,
     for example, that a toolbar might not be attached to the edge of
     the frame at all, but might be floating somewhere on the user's
     screen).  With an interface where these properties are separate
     specifiers, this has to be done manually.  Currently, having the
     various toolbar properties be inside of toolbar instantiators
     makes them difficult to modify, but this will be different with
     the API that I propose below.

  3.  I propose an API for modifying toolbar and toolbar item
     properties, as well as making other changes to toolbar
     instantiators, such as inserting or deleting toolbar items.  This
     API is based around the concept of a path.  There are two kinds of
     paths here - "toolbar paths" and "toolbar item paths".  Each kind
     of path is an object (of type `toolbar-path' and
     `toolbar-item-path', respectively) whose properties specify the
     location in a toolbar instantiator where changes to the
     instantiator can be made.  A toolbar path, for example, would be
     created using the `make-toolbar-path' function, which takes a
     toolbar specifier (or optionally, a symbol, such as `left',
     `right', `default', or `nil', which refers to a particular
     toolbar), and optionally, parameters such as the locale and the
     tag set, which specify which actual instantiator inside of the
     toolbar specifier is to be modified.  A toolbar item path is
     created similarly using a function called
     `make-toolbar-item-path', which takes a toolbar specifier and a
     string naming the caption of the toolbar item to be modified, as
     well as, of course, optionally the locale and tag set parameters
     and such.

     The usefulness of these path objects is as arguments to functions
     that will use them as pointers to the place in a toolbar
     instantiator where the modification should be made.  Recall, for
     example, the generalized property interface described above.  If a
     function such as `get' or `put' is called on a toolbar path or
     toolbar item path, it will use the information contained in the
     path object to retrieve or modify a property located at the end of
     the path.  The toolbar path objects can also be passed to new
     functions that I propose defining, such as `add-toolbar-item',
     `delete-toolbar-item', and `find-toolbar-item'.  These functions
     should be parallel to the functions for inserting, deleting,
     finding, etc. items in a menu.  The toolbar item path objects can
     also be passed to the drop-handler functions defined in
     Drag-n-Drop Interface Changes (drag-n-drop.html) to retrieve or
     modify the drop handlers that are associated with a toolbar item.
     (The idea here is that you can drag an object and drop it onto a
     toolbar item, just as you could onto a buffer, an extent, a menu
     item, or any other graphical element).

  4.  We should at least think about allowing for separate default and
     buffer-local toolbars.  The user should either be able to position
     these toolbars one above the other, or side by side, occupying a
     single toolbar line.  In the latter case, the boundary between the
     toolbars should be draggable, and if a toolbar takes up more room
     than is allocated for it, there should be arrows that appear on
     one or both sides of the toolbar so that the items in the toolbar
     can be scrolled left or right.  (For that matter, this sort of
     interface should exist even when there is only one toolbar that is
     on a particular toolbar line, because the toolbar may very well
     have more items than can be displayed at once, and it's silly in
     such a case if it's impossible to access the items that are not
     currently visible).

  5.  The default context menu for toolbars (which should be specified
     using a specifier called `default-toolbar-context-menu' according
     to the rules defined above) should contain entries allowing the
     user to modify the appearance of a toolbar.  Entries would
     include, for example, whether the toolbar is captioned, whether
     the glyphs for the toolbar are visible (if the toolbar is
     captioned but its glyphs are not visible, the toolbar appears as
     nothing but text; you can set things up this way, for example, in
     Netscape), an option that brings up a package for editing the
     contents of a toolbar, an option to allow the caption face to be
     dchanged (perhaps thorough jan `edit-faces' or `custom'
     interface), etc.



File: internals.info,  Node: Future Work -- Menu API Changes,  Next: Future Work -- Removal of Misc-User Event Type,  Prev: Future Work -- Toolbars,  Up: Future Work

40.9 Future Work - Menu API Changes
===================================

Author: Ben Wing (mailto:ben@xemacs.org)

  1.  I propose making a specifier for the menubar associated with the
     frame.  The specifier should be called `default-menubar' and
     should replace the existing `current-menubar' variable.  This
     would increase the power of the menubar interface and bring it in
     line with the toolbar interface.  (In order to provide proper
     backward compatibility, we might have to complete the symbol value
     handler mechanism (symbol-value-handlers.html))

  2.  I propose an API for modifying menu instantiators similar to the
     API composed above for toolbar instantiators.  A new object called
     a "menu path" (of type `menu-path') can be created using the
     `make-menu-path' function, and specifies a location in a particular
     menu instantiator where changes can be made.  The first argument to
     `make-menu-path' specifies which menu to modify and can be a
     specifier, a value such as `nil' (which means to modify the default
     menubar associated with the selected frame), or perhaps some other
     kind of specification referring to some other menu, such as the
     context menus invoked by the right mouse button.  The second
     argument to `make-menu-path', also required, is a list of zero or
     more strings that specifies the particular menu or menu item in
     the instantiator that is being referred to.  The remaining
     arguments are optional and would be a locale, a tag set, etc.  The
     menu path object can be passed to `get', `put' or other standard
     property functions to access or modify particular properties of a
     menu or a menu item.  It can also be passed to expanded versions
     of the existing functions such as `find-menu-item',
     `delete-menu-item', `add-menu-button', etc.  (It is really a shame
     that `add-menu-item' is an obsolete function because it is a much
     better name than `add-menu-button').  Finally, the menu path
     object can be passed to the drop-handler functions described in
     Drag-n-Drop Interface Changes (drag-n-drop.html) to access or
     modify the drop handlers that are associated with a particular
     menu item.

  3.  New keyword properties should be added to the menu item vector.
     These include `:help-echo', `:context-menu' and `:drop-handlers',
     with similar semantics to the corresponding keywords for toolbar
     items.  (It may seem a bit strange at first to have a context menu
     associated with a particular menu item, but it is a user interface
     concept that exists both in Open Look and in Windows, and really
     makes a lot of sense if you give it a bit of thought).  These
     properties may not actually be implemented at first, but at least
     the keywords for them should be defined.



File: internals.info,  Node: Future Work -- Removal of Misc-User Event Type,  Next: Future Work -- Mouse Pointer,  Prev: Future Work -- Menu API Changes,  Up: Future Work

40.10 Future Work - Removal of Misc-User Event Type
===================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* This page describes why the misc-user event type should
be split up into a number of different event types, and how to do this.

   The misc-user event should not exist as a single event type.  It
should be split up into a number of different event types: one for
scrollbar events, one for menu events, and one or two for drag-n-drop
events.  Possibly there will be other event types created in the
future.  The reason for this is that the misc-user event was a bad
design choice when I made it, and it has only gotten worse with
Oliver's attempts to add features to it to make it be used for
drag-n-drop.  I know that there was originally a separate drag-n-drop
event type, and it was folded into the misc-user event type on my
recommendation, but I have now realized the error of my ways.  I had
originally created a single event type in an attempt to prevent some
Lisp programs from breaking because they might have a case statement
over various event types, and would not be able to handle new event
types appearing.  I think now that these programs simply need to be
written in a way to handle new event types appearing.  It's not very
hard to do this.  You just use predicates instead of doing a case
statement over the event type.  If we preserve the existing predicate
called `misc-user-event-p', and just make sure that it evaluates to
true when given any user event type other than the standard simple
ones, then most existing code will not break either when we split the
event types up like this, or if we add any new event types in the
future.

   More specifically, the only clean way to design the misc-user event
type would be to add a sub-type field to it, and then have the nature
of all the other fields in the event type be dependent on this
sub-type.  But then in essence, we'd just be reimplementing the whole
event-type scheme inside of misc-user events, which would be rather
pointless.


File: internals.info,  Node: Future Work -- Mouse Pointer,  Next: Future Work -- Extents,  Prev: Future Work -- Removal of Misc-User Event Type,  Up: Future Work

40.11 Future Work - Mouse Pointer
=================================

* Menu:

* Future Work -- Abstracted Mouse Pointer Interface::
* Future Work -- Busy Pointer::


File: internals.info,  Node: Future Work -- Abstracted Mouse Pointer Interface,  Next: Future Work -- Busy Pointer,  Prev: Future Work -- Mouse Pointer,  Up: Future Work -- Mouse Pointer

40.11.1 Future Work - Abstracted Mouse Pointer Interface
--------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* We need to create a new image format that allows
standard pointer shapes to be specified in a way that works on all
Windows systems.  I suggest that this be called `pointer', which has
one tag associated with it, named `:data', and whose value is a string.
The possible strings that can be specified here are predefined by
XEmacs, and are guaranteed to work across all Windows systems.  This
means that we may need to provide our own definition for pointer shapes
that are not standard on some systems.  In particular, there are a lot
more standard pointer shapes under X than under Windows, and most of
these pointer shapes are fairly useful.  There are also a few pointer
shapes (I think the hand, for example) on Windows, but not on X.
Converting the X pointer shapes to Windows should be easy because the
definitions of the pointer shapes are simply XBM files, which we can
read under Windows.  Going the other way might be a little bit more
difficult, but it should still not be that hard.

   While we're at it, we should change the image format currently called
`cursor-font' to `x-cursor-font', because it only works under X
Windows.  We also need to change the format called `resource' to be
`mswindows-resource'.  At least in the case of `cursor-font', the old
value should be maintained for compatibility as an obsolete alias.  The
`resource' format was added so recently that it's possible that we can
just change it.


File: internals.info,  Node: Future Work -- Busy Pointer,  Prev: Future Work -- Abstracted Mouse Pointer Interface,  Up: Future Work -- Mouse Pointer

40.11.2 Future Work - Busy Pointer
----------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   Automatically make the mouse pointer switch to a busy shape (watch
signal) when XEmacs has been "busy" for more than, e.g. 2 seconds.
Define the "busy time" as the time since the last time that XEmacs was
ready to receive input from the user.  An implementation might be:

  1. Set up an asynchronous timeout, to signal after the busy time;
     these are triggered through a call to QUIT so they will be
     triggered even when the code is busy doing something.

  2. We already have an "emacs_is_blocking" flag when we are waiting for
     input.  In the same place, when we are about to block and wait for
     input (regardless of whether input is already present), maybe call
     a hook, which in this case would remove the timer and put back the
     normal mouse shape.  Then when we exit the blocking stage (we got
     some input), call another hook, which in this case will start the
     timer.  Note that we don't want these "blocking" hooks to be
     triggered just because of an accept-process-output or some similar
     thing that retrieves events, only to put them back onto a queue
     for later processing.  Maybe we want some sort of flag that's
     bound by those routines saying that we aren't really waiting for
     input.  Making that flag Lisp-accessible allows it to be set by
     similar sorts of Lisp routines (if there are any?) that loop
     retrieving events but defer them, or only drain the queue, or
     whatnot.  #### Think about whether it would make some sense to try
     and be more clever in our determinations of what counts as "real
     waiting for user input", e.g.  whether the event gets dispatched
     (unfortunately this occurs way too late, we want to know to remove
     the busy cursor *before* getting an event), maybe whether there
     are any events waiting to be processed or we'll truly block, etc.
     (e.g. one possibility if there is input on the queue already when
     we "block" for input, don't remove the busy- wait pointer, but
     trigger the removal of it when we dispatch a user event).


File: internals.info,  Node: Future Work -- Extents,  Next: Future Work -- Version Number and Development Tree Organization,  Prev: Future Work -- Mouse Pointer,  Up: Future Work

40.12 Future Work - Extents
===========================

* Menu:

* Future Work -- Everything should obey duplicable extents::


File: internals.info,  Node: Future Work -- Everything should obey duplicable extents,  Prev: Future Work -- Extents,  Up: Future Work -- Extents

40.12.1 Future Work - Everything should obey duplicable extents
---------------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   A lot of functions don't properly track duplicable extents.  For
example, the `concat' function does, but the `format' function does
not, and extents in keymap prompts are not displayed either.  All of
the functions that generate strings or string-like entities should
track the extents that are associated with the strings.  Currently this
is difficult because there is no general mechanism implemented for doing
this.  I propose such a general mechanism, which would not be hard to
implement, and would be easy to use in other functions that build up
strings.

   The basic idea is that we create a C structure that is analogous to a
Lisp string in that it contains string data and lists of extents for
that data.  Unlike standard Lisp strings, however, this structure (let's
call it `lisp_string_struct') can be incrementally updated and its
allocation is handled explicitly so that no garbage is generated.  (This
is important for example, in the event-handling code which would want to
use this structure, but needs to not generate any garbage for efficiency
reasons).  Both the string data and the list of extents in this string
are handled using dynarrs so that it is easy to incrementally update
this structure.  Functions should exist to create and destroy instances
of `lisp_string_struct' to generate a Lisp string from a
`lisp_string_struct' and vice-versa to append a sub-string of a Lisp
string to a `lisp_string_struct', to just append characters to a
`lisp_string_struct', etc.  The only thing possibly tricky about
implementing these functions is implementing the copying of extents from
a Lisp string into a `lisp_string_struct'.  However, there is already a
function `copy_string_extents()' that does basically this exact thing,
and it should be easy to create a modified version of this function.


File: internals.info,  Node: Future Work -- Version Number and Development Tree Organization,  Next: Future Work -- Improvements to the `xemacs.org' Website,  Prev: Future Work -- Extents,  Up: Future Work

40.13 Future Work - Version Number and Development Tree Organization
====================================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* The purpose of this proposal is to present a coherent
plan for how development branches in XEmacs are managed.  This will
cover such issues as stable versus experimental branches, creating new
branches, synchronizing patches between branches, and how version
numbers are assigned to branches.

   A development branch is defined to be a linear series of releases of
the XEmacs code base, each of which is derived from the previous one.
When the XEmacs development tree is forked and two branches are created
where there used to be one, the branch that is intended to be more
stable and have fewer changes made to it is considered the one that
inherits the parent branch, and the other branch is considered to have
begun at the branching point.  The less stable of the two branches will
eventually be forked again, while this will not happen usually to the
more stable of the two branches, and its development will eventually
come to an end.  This means that every branch has a definite ending
point.  For example, the 20.x branch began at the point when the
released 19.13 code tree was split into a 19.x and a 20.x branch, and a
20.x branch will end when the last 20.x release (probably numbered 20.5
or 20.6) is released.

   I think that there should always be three active development
branches at any time.  These branches can be designated the stable, the
semi-stable, and the experimental branches.  This situation has existed
in the current code tree as soon as the 21.0 development branch was
split.  In this situation, the stable branch is the 20.x series.  The
semi-stable branch is the 21.0 release and the stability releases that
follow.  The experimental branch is the branch that was created as the
result of the 21.0 development branch split.  Typically, the stable
branch has been released for a long period of time.  The semi-stable
branch has been released for a short period of time, or is about to be
released, and the experimental branch has not yet been released, and
will probably not be released for awhile.  The conditions that should
hold in all circumstances are:

  1.  There should be three active branches.

  2.  The experimental branch should never be in feature freeze.


   The reason for the second condition is to ensure that active
development can always proceed and is never throttled, as is happening
currently at the end of the 21.0 release cycle.  What this means is
that as soon as the experimental branch is deemed to be stable enough
to go into feature freeze:

  1.  The current stable branch is made inactive and all further
     development on it ceases.

  2.  The semi-stable branch, which by now should have been released
     for a fair amount of time, and should be fairly stable, gets
     renamed to the stable branch.

  3.  The experimental branch is forked into two branches, one of which
     becomes the semi-stable branch, and the other, the experimental
     branch.


   The stable branch is always in high resistance, which is to say that
the only changes that can be made to the code are important bug fixes
involving a small amount of code where it should be clear just by
reading the code that no destabilizing code has been introduced.  The
semi-stable branch is in low resistance, which means that no major
features can be added, but except right before a release fairly major
code changes are allowed.  Features can be added if they are
sufficiently small, if they are deemed sufficiently critical due to
severe problems that would exist if the features were not added (for
example, replacement of the unexec mechanism with a portable solution
would be a feature that could be added to the semi-stable branch
provided that it did not involve an overly radical code re-architecture,
because otherwise it might be impossible to build XEmacs on some
architectures or with some compilers), or if the primary purpose of the
new feature is to remedy an incompleteness in a recent architectural
change that was not finished in a prior release due to lack of time (for
example, abstracting the mouse pointer and list-of-colors interfaces,
which were left out of 21.0).  There is no feature resistance in place
in the experimental branch, which allows full development to proceed at
all times.

   In general, both the stable and semi-stable branches will contain
previous net releases.  In addition, there will be beta releases in all
three branches, and possibly development snapshots between the beta
releases.  It's obviously necessary to have a good version numbering
scheme in order to keep everything straight.

   First of all, it needs to be immediately clear from the version
number whether the release is a beta release or a net release.  Steve
has proposed getting rid of the beta version numbering system, which I
think would be a big mistake.  Furthermore, the net release version
number and beta release version number should be kept separate, just as
they are now, to make it completely clear where any particular release
stands.  There may be alternate ways of phrasing a beta release other
than something like 21.0 beta 34, but in all such systems, the beta
number needs to be zero for any release version.  Three possible
alternative systems, none of which I like very much, are:

  1.  The beta number is simply an extra number in the regular version
     number.  Then, for example, 21.0 beta 34 becomes 21.0.34.  The
     problem is that the release version, which would simply be called
     21.0, appears to be earlier than 21.0 beta 34.

  2.  The beta releases appear as later revisions of earlier releases.
     Then, for example, 21.1 beta 34 becomes 21.0.34, and 21.0 beta 34
     would have to become 21.-1.34.  This has both the obvious ugliness
     of negative version numbers and the problem that it makes beta
     releases appear to be associated with their previous releases,
     when in fact they are more closely associated with the following
     release.

  3.  Simply make the beta version number be negative.  In this scheme,
     you'd start with something like -1000 as the first beta, and then
     21.0 beta 34 would get renumbered to 21.0.-968.  Obviously, this
     is a crazy and convoluted scheme as well, and we would be best to
     avoid it.


   Currently, the between-beta snapshots are not numbered, but I think
that they probably should be.  If appropriate scripts are handled to
automate beta release, it should be very easy to have a version number
automatically updated whenever a snapshot is made.  The number could be
added either as a separate snapshot number, and you'd have 21.0 beta 34
pre 1, which becomes before 21.0 beta 34; or we could make the beta
number be floating point, and then the same snapshot would have to be
called 21.0 beta 33.1.  The latter solution seems quite kludgey to me.

   There also needs to be a clear way to distinguish, when a net
release is made, which branch the release is a part of.  Again, three
solutions come to mind:

  1.  The major version number reflects which development branch the
     release is in and the minor version number indicates how many
     releases have been made along this branch.  In this scheme, 21.0
     is always the first release of the 21 series development branch,
     and when this branch is split, the child branch that becomes the
     experimental branch gets version numbers starting with 22.  This
     scheme is the simplest, and it's the one I like best.

  2.  We move to a three-part version number.  In this scheme, the
     first two numbers indicate the branch, and the third number
     indicates the release along the branch.  In this scheme, we have
     numbers like 21.0.1, which would be the second release in the 21.0
     series branch, and 21.1.2, which would be the third release in the
     21.1 series branch.  The major version number then gets increased
     only very occasionally, and only when a sufficiently major
     architectural change has been made, particularly one that causes
     compatibility problems with code written for previous branches.  I
     think schemes like this are unnecessary in most circumstances,
     because usually either the major version number ends up changing
     so often that the second number is always either zero or one, or
     the major version number never changes, and as such becomes
     useless.  By the time the major version number would change, the
     product itself has changed so much that it often gets renamed.
     Furthermore, it is clear that the two version number scheme has
     been used throughout most of the history of Emacs, and recently we
     have been following the two number scheme also.  If we introduced a
     third revision number, at this point it would both confuse
     existing code that assumed there were two numbers, and would look
     rather silly given that the major version number is so high and
     would probably remain at the same place for quite a long time.

  3.  A third scheme that would attempt to cross the two schemes would
     keep the same concept of major version number as for the three
     number scheme, and would compress the second and third numbers of
     the three number scheme into one number by using increments of
     ten.  For example, the current 21.x branch would have releases No.
     21.0, 21.1, etc.  The next branch would be No. 21.10, 21.11, etc.
     I don't like this scheme very much because it seems rather
     kludgey, and also because it is not used in any other product as
     far as I know.

  4.  Another scheme that would combine the second and third numbers in
     the three number scheme would be to have the releases in the
     current 21.x series be numbered 21.0, then 21.01, then 22.02, etc.
     The next series is 21.1, then 21.11, then 21.12, etc.  This is
     similar to the way that version numbers are done for DOS in
     Windows.  I also think that this scheme is fairly silly because,
     like the previous scheme, its only purpose is to avoid increasing
     the major version number very much.  But given that we have
     already have a fairly large major version number, there doesn't
     seem to be any particular problem with increasing this number by
     one every year or two.  Some people will object that by doing
     this, it becomes impossible to tell when a change is so major that
     it causes a lot of code breakage, but past releases have not been
     accurate indicators of this.  For example, 19.12 caused a lot of
     code breakage, but 20.0 caused less, and 21.0 caused less still.
     In the GNU Emacs world, there were byte code changes made between
     19.28 and 19.29, but as far as I know, not between 19.29 and 20.0.


   With three active development branches, synchronizing code changes
between the branches is obviously somewhat of a problem.  To make things
easier, I propose a few general guidelines:

  1.  Merging between different branches need not happen that often.  It
     should not happen more often than necessary to avoid undue burden
     on the maintainer, but needs to be done at all defined
     checkpoints.  These checkpoints need to be noted in all of the
     places that track changes along the branch, for example, in all of
     the change logs and in all of the CVS tags.

  2.  Every code change that can be considered a self-contained unit, no
     matter how large or small, needs to have a change log entry,
     preferably a single change log entry associated with it.  This is
     an absolute requirement.  There should be no code changes without
     an associated change log entry.  Otherwise, it is highly likely
     that patches will not be correctly synchronized across all
     versions, and will get lost.  There is no need for change log
     entries to contain unnecessary detail though, and it is important
     that there be no more change log entries than necessary, which
     means that two or more change log entries associated with a single
     patch need to be grouped together if possible.  This might imply
     that there should be one global change log instead of change logs
     in each directory, or at the very least, the number of separate
     change logs should be kept to a minimum.

  3.  The patch that is associated with each change log entry needs to
     be kept around somewhere.  The reason for this is that when
     synchronizing code from some branch to some earlier branch, it is
     necessary to go through each change log entry and decide whether a
     change is worthy to make it into a more stable branch.  If so, the
     patch associated with this change needs to be individually applied
     to the earlier branch.

  4.  All changes made in more stable branches get merged into less
     stable branches unless the change really is completely unnecessary
     in the less stable branch because it is superseded by some other
     change.  This will probably mean more developers making changes to
     the semi-stable branch than to the experimental branch.  This
     means that developers should strive to do their development in the
     most stable branch that they expect their code to go into.  An
     alternative to this which is perhaps more workable is simply to
     insist that all developers make all patches based off of the
     experimental branch, and then later merge these patches down to
     the more stable branches as necessary.  This means, however, that
     submitted patches should never be combinations of two or more
     unrelated changes.  Whenever such patches are submitted, they
     should either be rejected (which should apply to anybody who
     should know better, which probably means everybody on the beta
     list and anybody else who is a regular contributor), or the
     maintainer or some other designated party needs to filter the
     combined patch into separate patches, one per logical change.

  5.  The maintainer should keep all the patches around in some data
     base, and the patches should be given an identifier consisting of
     the author of the patch, the date the patch was submitted, and
     some other identifying characteristic, such as a number, in case
     there is more than one patch on the same date by the same author.
     The database should hopefully be correctly marked at all times
     with something indicating which branches the patch has been
     applied to, and this database should hopefully be publicly visible
     so that patch authors can determine whether their patches have
     been applied, and whether their patches have been received, so
     that patches do not get needlessly resubmitted.

  6.  Global automatable changes such as textual renaming, reordering,
     and additions or deletions of parameters in function calls should
     still be allowed, even with multiple development branches.
     (Sometimes these are necessary for code cleanliness, and in the
     long run, they save a lot of time, even through they may cause
     some headaches in the short-term.)  In general, when such changes
     are made, they should occur in a separate beta version that
     contains only such changes and no other patches, and the changes
     should be made in both the semi-stable and experimental branches
     at the same time.  The description of the beta version should make
     it very clear that the beta is comprised of such changes.  The
     reason for doing these things is to make it easier for people to
     diff between beta versions in order to figure out the changes that
     were made without the diff getting cluttered up by these code
     cleanliness changes that don't change any actual behavior.



File: internals.info,  Node: Future Work -- Improvements to the `xemacs.org' Website,  Next: Future Work -- Keybindings,  Prev: Future Work -- Version Number and Development Tree Organization,  Up: Future Work

40.14 Future Work - Improvements to the `xemacs.org' Website
============================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   The `xemacs.org' web site is the face that XEmacs presents to the
outside world.  In my opinion, its most important function is to present
information about XEmacs in such a way that solicits new XEmacs users
and co-contributors.  Existing members of the XEmacs community can
probably find out most of the information they want to know about XEmacs
regardless of what shape the web site is in, or for that matter, perhaps
even if the web site doesn't exist at all.  However, potential new users
and co-contributors who go to the XEmacs web site and find it out of
date and/or lacking the information that they need are likely to be
turned away and may never return.  For this reason, I think it's
extremely important that the web site be up-to-date, well-organized, and
full of information that an inquisitive visitor is likely to want to
know.

   The current XEmacs web site needs a lot of work if it is to meet
these standards.  I don't think it's reasonable to expect one person to
do all of this work and make continual updates as needed, especially
given the dismal record that the XEmacs web site has had.  The proper
thing to do is to place the web site itself under CVS and allow many of
the core members to remotely check files in and out.  This way, for
example, Steve could update the part of the site that contains the
current release status of XEmacs. (Much of this could be done by a
script that Steve executes when he sends out a beta release
announcement which automatically HTML-izes the mail message and puts it
in the appropriate place on the web site.  There are programs that are
specifically designed to convert email messages into HTML, for example
`mhonarc'.)  Meanwhile, the `xemacs.org' mailing list administrator
(currently Jason Mastaler, I think) could maintain the part of the site
that describes the various mailing lists and other addresses at
`xemacs.org'.  Someone like me (perhaps through a proxy typist) could
maintain the part of the site that specifies the future directions that
XEmacs is going in, etc., etc.

   Here are some things that I think it's very important to add to the
web site.

  1.  A page describing in detail how to get involved in the XEmacs
     development process, how to submit and where to submit various
     patches to the XEmacs core or associated packages, how to contact
     the maintainers and core developers of XEmacs and the maintainers
     of various packages, etc.

  2.  A page describing exactly how to download, compile, and install
     XEmacs, and how to download and install the various binary
     distributions.  This page should particularly cover in detail how
     exactly the package system works from an installation standpoint
     and how to correctly compile and install under Microsoft Windows
     and Cygwin.  This latter section should cover what compilers are
     needed under Microsoft Windows and Cygwin, and how to get and
     install the Cygwin components that are needed.

  3.  A page describing where to get the various ancillary libraries
     that can be linked with XEmacs, such as the JPEG, TIFF, PNG,
     X-Face, DBM, and other libraries.  This page should also cover how
     to correctly compile it and install these libraries, including
     under Microsoft Windows (or at least it should contain pointers to
     where this information can be found).  Also, it should describe
     anything that needs to be specified as an option to `configure' in
     order for XEmacs to link with and make use of these libraries or
     of Motif or CDE.  Finally, this page should list which versions of
     the various libraries are required for use with the various
     different beta versions of XEmacs.  (Remember, this can change
     from beta to beta, and someone needs to keep a watchful eye on
     this).

  4.  Pointers to any other sites containing information on XEmacs.
     This would include, for example, Hrvoje's XEmacs on Windows FAQ
     and my Architecting XEmacs web site.  (Presumably, most of the
     information in this section will be temporary.  Eventually, these
     pages should be integrated into the main XEmacs web site).

  5.  A page listing the various sub-projects in the XEmacs development
     process and who is responsible for each of these sub-projects, for
     example development of the package system, administration of the
     mailing lists, maintenance of stable XEmacs versions, maintenance
     of the CVS web interface, etc.  This page should also list all of
     the packages that are archived at `xemacs.org' and who is the
     maintainer or maintainers for each of these packages.


Other Places with an XEmacs Presence
------------------------------------

We should try to keep an XEmacs presence in all of the major places on
the web that are devoted to free software or to the "open source"
community.  This includes, for example, the open source web site at
http://opensource.oreilly.com (../../opensource.oreilly.com/default.htm)
(I'm already in the process of contacting this site), the Freshmeat site
at http://www.freshmeat.net (../../www.freshmeat.net/default.htm), the
various announcement news groups (for example, comp.os.linux.announce
(news:comp.os.linux.announce), and the Windows announcement news group)
etc.


File: internals.info,  Node: Future Work -- Keybindings,  Next: Future Work -- Byte Code Snippets,  Prev: Future Work -- Improvements to the `xemacs.org' Website,  Up: Future Work

40.15 Future Work - Keybindings
===============================

* Menu:

* Future Work -- Keybinding Schemes::
* Future Work -- Better Support for Windows Style Key Bindings::
* Future Work -- Misc Key Binding Ideas::


File: internals.info,  Node: Future Work -- Keybinding Schemes,  Next: Future Work -- Better Support for Windows Style Key Bindings,  Prev: Future Work -- Keybindings,  Up: Future Work -- Keybindings

40.15.1 Future Work - Keybinding Schemes
----------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* We need a standard mechanism that allows a different
global key binding schemes to be defined.  Ideally, this would be the
keyboard action interface (keyboard-actions.html) that I have proposed,
however this would require a lot of work on the part of mode
maintainers and other external Elisp packages and will not be rady in
the short term.  So I propose a very kludgy interface, along the lines
of what is done in Viper currently.  Perhaps we can rip that key munging
code out of Viper and make a separate extension that implements a global
key binding scheme munging feature.  This way a key binding scheme could
rearrange all the default keys and have all sorts of other code, which
depends on the standard keys being in their default location, still
work.


File: internals.info,  Node: Future Work -- Better Support for Windows Style Key Bindings,  Next: Future Work -- Misc Key Binding Ideas,  Prev: Future Work -- Keybinding Schemes,  Up: Future Work -- Keybindings

40.15.2 Future Work - Better Support for Windows Style Key Bindings
-------------------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* This page describes how we could create an XEmacs
extension that modifies the global key bindings so that a Windows user
would feel at home when using the keyboard in XEmacs.  Some of these
bindings don't conflict with standard XEmacs keybindings and should be
added by default, or at the very least under Windows, and probably under
X Windows as well. Other key bindings would need to be implemented in a
Windows compatibility extension which can be enabled and disabled on the
fly, following the conventions outlined in Standard interface for
enabling extensions (enabling-extensions.html) Ideally, this should be
implemented using the keyboard action interface (keyboard-actions.html)
but these wil not be available in the short term, so we will have to
resort to some awful kludges, following the model of Michael Kifer's
Viper mode.

   We really need to make XEmacs provide standard Windows key bindings
as much as possible.  Currently, for example, there are at least two
packages that allow the user to make a selection using the shifted arrow
keys, and neither package works all that well, or is maintained.  There
should be one well-written piece of code that does this, and it should
be a standard part of XEmacs.  In fact, it should be turned on by
default under Windows, and probably under X as well. (As an aside here,
one point of contention in how to implement this involves what happens
if you select a region using the shifted arrow keys and then hit the
regular arrow keys.  Does the region remain selected or not?  I think
there should be a variable that controls which of these two behaviors
you want.  We can argue over what the default value of this variable
should be.  The standard Windows behavior here is to keep the region
selected, but move the insertion point elsewhere, which is unfortunately
impossible to implement in XEmacs.)

   Some thought should be given to what to do about the standard Windows
control and alt key bindings.  Under NTEmacs, there is a variable that
controls whether the alt key behaves like the Emacs meta key, or whether
it is passed on to the menu as in standard Windows programs.  We should
surely implement this and put this option on the *Options* menu.
Making `Alt-f' for example, invoke the *File* menu, is not all that
disruptive in XEmacs, because the user can always type `ESC f' to get
the meta key functionality.  Making `Control-x', for example, do *Cut*,
is much, much more problematic, of course, but we should consider how
to implement this anyway.  One possibility would be to move all of the
current Emacs control key bindings onto control-shift plus a key, and
to make the simple control keys follow the Windows standard as much as
possible.  This would mean, for example, that we would have the
following keybindings:
`Control-x' ==> *Cut*
`Control-c' ==> *Copy*
`Control-v' ==> *Paste*
`Control-z' ==> *Undo*
`Control-f' ==> *Find*
`Control-a' ==> *Select All*
`Control-s' ==> *Save*
`Control-p' ==> *Print*
`Control-y' ==> *Redo*
(this functionality _is_ available in XEmacs with Kyle Jones' `redo.el'
package, but it should be better integrated)
`Control-n' ==> *New*
`Control-o' ==> *Open*
`Control-w' ==> *Close Window*
The changes described in the previous paragraph should be put into an
extension named `windows-keys.el' (see Standard interface for enabling
extensions (enabling-extensions.html)) so that it can be enabled and
disabled on the fly using a menu item and can be selected as the
default for a particular user in their custom options file. Once this
is implemented, the Windows installer should also be modified so that
it brings up a dialog box that allows the user to make a selection of
which key binding scheme they would prefer as the default, either the
XEmacs standard bindings, Vi bindings (which would be Viper mode),
Windows-style bindings, Brief, CodeWright, Visual C++, or whatever we
manage to implement.


File: internals.info,  Node: Future Work -- Misc Key Binding Ideas,  Prev: Future Work -- Better Support for Windows Style Key Bindings,  Up: Future Work -- Keybindings

40.15.3 Future Work - Misc Key Binding Ideas
--------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   * M-123 ... do digit arg

   * However, M-( group commands together until M-)

   * Nested M-() are allowed.

   * Number repeating plus () repeats N times each group of commands as
     a unit.

   * M-() by itself forms an anonymous macro, and there should be a
     command to repeat, like VI (execute macro), but when no () before,
     it repeats the last command of same amount of complication - or
     more like, somewhere there is a repeats all command back to make
     to act that stopping like VI's dot command.

   * C-numbers switches to a particular window.  maybe 1-3 or 1-4 does
     this.

   * C-4 or 5 to 9 (or ()? maybe reserved) switches to a particular
     frame.

   * Possibly C-Sh-numbers select more windows or frames.

   * M-C-1 M-C-2 M-C-3 M-C-4 M-C-5 M-C-6 M-C-7 M-C-8 M-C-9 M-C-0

     maybe should be execute anonymous macros (other possibility is
     insert register but you can easily simulate with a keyboard macro)

   * What about C-S M-C-S M-S??

   * I think there should be default fun key binding for *ILLEGIBLE*
     similar to what I have - load, save, cut, copy, paste, kill line,
     start/end macro, do macro


File: internals.info,  Node: Future Work -- Byte Code Snippets,  Next: Future Work -- Lisp Stream API,  Prev: Future Work -- Keybindings,  Up: Future Work

40.16 Future Work - Byte Code Snippets
======================================

Author: Ben Wing (mailto:ben@xemacs.org)

   * For use in time critical (e.g. redisplay) places such as display
     tables - a simple piece of code is evalled, e.g.
          (int-to-char (1+ c))
     where c is the arg, specbound.

   * can be compiled like
          (byte-compile-snippet (int-to-char (1+ c)) (c))
                                                     ^^^
                                          environment of local vars

   * need eval with bindings (not hard to implement) (extendable when
     lexical scoping present)

   * What's the return value of byte-compile-snippet?  (Look to see how
     this might be implemented)

* Menu:

* Future Work -- Autodetection::
* Future Work -- Conversion Error Detection::
* Future Work -- Unicode::
* Future Work -- BIDI Support::
* Future Work -- Localized Text/Messages::


File: internals.info,  Node: Future Work -- Autodetection,  Next: Future Work -- Conversion Error Detection,  Prev: Future Work -- Byte Code Snippets,  Up: Future Work -- Byte Code Snippets

40.16.1 Future Work - Autodetection
-----------------------------------

There are various proposals contained here.

New Implementation of Autodetection Mechanism
---------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   The current auto detection mechanism in XEmacs Mule has many
problems. For one thing, it is wrong too much of the time. Another
problem, although easily fixed, is that priority lists are fixed rather
than varying, depending on the particular locale; and finally, it
doesn't warn the user when it's not sure of the encoding or when there's
a mistake made during decoding. In both of these situations the user
should be presented with a list of likely encodings and given the
choice, rather than simply proceeding anyway and giving a result that is
likely to be wrong and may result in data corruption when the file is
saved out again.

   All coding systems are categorized according to their type. Currently
this includes ISO2022, Big 5, Shift-JIS, UTF8 and a few others. In the
future there will be many more types defined and this mechanism will be
generalized so that it is easily extendable by the Lisp programmer.

   In general, each coding system type defines a series of subtypes
which are handled differently for the purpose of detection. For
example, ISO 2022 defines many different subtypes such as 7 bit, 8 bit,
locking shift, designating and so on. UCS2 may define subtypes such as
normal and byte reversed.

   The detection engine works conceptually by calling the detection
methods of all of the defined coding system types in parallel on
successive chunks of data (which may, for example, be 4K in size, but
where the size makes no difference except for optimization purposes)
and watching the results until either a definite answer is determined
or the end of data is reached. The way the definite answer is
determined will be defined below. The detection method of the coding
system type is passed some data and a chunk of memory, which the method
uses to store its current state (and which is maintained separately for
each coding system type by the detection engine between successive
calls to the coding system type's detection method). Its return value
should be an alist consisting of a list of all of the defined subtypes
for that coding system type along with a level of likelihood and a list
of additional properties indicating certain features detected in the
data. The extra properties returned are defined entirely by the
particular coding system type and are used only in the algorithm
described below under "user control." However, the levels of likelihood
have a standard meaning as follows:

   Level 4 means "near certainty" and typically indicates that a
signature has been detected, usually at the beginning of the data,
indicating that the data is encoded in this particular coding system
type. An example of this would be the byte order mark at the beginning
of UCS2 encoded data or the GZIP mark at the beginning of GZIP data.

   Level 3 means "highly likely" and indicates that tell-tale signs have
been discovered in the data that are characteristic of this particular
coding system type. Examples of this might be ISO 2022 escape sequences
or the current Unicode end of line markers at regular intervals.

   Level 2 means "strongly statistically likely" indicating that
statistical analysis concludes that there's a high chance that this
data is encoded according to this particular type. For example, this
might mean that for UCS2 data, there is a high proportion of null bytes
or other repeated bytes in the odd-numbered bytes of the data and a
high variance in the even-numbered bytes of the data. For Shift-JIS,
this might indicate that there were no illegal Shift-JIS sequences and
a fairly high occurrence of common Shift-JIS characters.

   Level 1 means "weak statistical likelihood" meaning that there is
some indication that the data is encoded in this coding system type. In
fact, there is a reasonable chance that it may be some other type as
well. This means, for example, that no illegal sequences were
encountered and at least some data was encountered that is purposely
not in other coding system types. For Shift-JIS data, this might mean
that some bytes in the range 128 to 159 were encountered in the data.

   Level 0 means "neutral" which is to say that there's either not
enough data to make any decision or that the data could well be
interpreted as this type (meaning no illegal sequences), but there is
little or no indication of anything particular to this particular type.

   Level -1 means "weakly unlikely" meaning that some data was
encountered that could conceivably be part of the coding system type
but is probably not. For example, successively long line-lengths or
very rarely-encountered sequences.

   Level -2 means "strongly unlikely" meaning that typically a number
of illegal sequences were encountered.

   The algorithm to determine when to stop and indicate that the data
has been detected as a particular coding system uses a priority list,
which is typically specified as part of the language environment
determined from the current locale or the user's choice. This priority
list consists of a list of coding system subtypes, along with a minimum
level required for positive detection and optionally additional
properties that need to be present. Using the return values from all of
the detection methods called, the detection engine looks through this
priority list until it finds a positive match. In this priority list,
along with each subtype is a particular coding system to return when
the subtype is encountered. (For example, in a Japanese-language
environment particular subtypes of ISO 2022 will be associated with the
Japanese coding system version of those subtypes). It is perfectly
legal and quite common in fact, to list the same subtype more than once
in the priority list with successively lower requirements. Other facts
that can be listed in the priority list for a subtype are "reject",
meaning that the data should never be detected as this subtype, or
"ask", meaning that if the data is detected to be this subtype, the
user will be asked whether they actually mean this. This latter
property could be used, for example, towards the bottom of the priority
list.

   In addition there is a global variable which specifies the minimum
number of characters required before any positive match is reported.
There may actually be more than one such variable for different sources
of data, for example, detection of files versus detection of subprocess
data.

   Whenever a file is opened and detected to be a particular coding
system, the subtype, the coding system and the associated level of
likelihood will be prominently displayed either in the echo area or in
a status box somewhere.

   If no positive match is found according to the priority list, or if
the matches that are found have the "ask" property on them, then the
user will be presented with a list of choices of possible encodings and
asked to choose one. This list is typically sorted first by level of
likelihood, and then within this, by the order in which the subtypes
appear in the priority list. This list is displayed in a special kind
of dialog box or other buffer allowing the user, in addition to just
choosing a particular encoding, to view what the file would look like
if it were decoded according to the type.

   Furthermore, whenever a file is decoded according to a particular
type, the decoding engine keeps track of status values that are output
by the coding system type's decoding method. Generally, this status
will be in the form of errors or warnings of various levels, some of
which may be severe enough to stop the decoding entirely, and some of
which may either indicate definitely malformed data but from which it's
possible to recover, or simply data that appears rather questionable.
If any of these status values are reported during decoding, the user
will be informed of this and asked "are you sure?"  As part of the "are
you sure" dialog box or question, the user can display the results of
the decoding to make sure it's correct. If the user says "no, they're
not sure," then the same list of choices as previously mentioned will
be presented.

RFC: Autodetection
------------------

Also appeared under heading "Implementation of Coding System Priority
Lists in Various Locales" ?

   Author: Stephen Turnbull (mailto:stephen@xemacs.org)

   Date: 11/1/1999 2:48 AM

     >>>>> "Hrvoje" == Hrvoje Niksic <hniksic@srce.hr> writes:

         [Ben sez:]

         >> You are perfectly free to set up your XEmacs like this, but
         >> XEmacs/Mule *will* autodetect by default if there is no
         >> Content-Type: info and no reason to believe we are dealing with
         >> binary files.

         Hrvoje> In that case, it will be a serious mistake to make
         Hrvoje> --with-mule the default, ever.  I think more care should
         Hrvoje> be shown in meeting the need of European users.

   Hrvoje, I don't understand what you are worrying about.  I suspect
you are worrying about Handa's hyperactive and obstinate Mule, not what
Ben has in mind.  Yes, Ben has said "better guessing," but that's
simply not reasonable without substantial language environment
information.  I think trying to detect Latin-1 vs Latin-2 in the POSIX
locale would be a big mistake, I think trying to guess Big 5 v. Shift
JIS in a European locale would be a big mistake.

   If Ben doesn't mean "more appropriate use of language environment
information" when he writes "better guessing," I, as much as you, want
to see how he plans to do that.  Ben?  ("Yes/no/oops I need to think
about it" is good enough if you have specifics you intend to put in the
RFC you're planning to present.)

   Let me give a formal proposal of what I would like to see in the
autodetection specification.

  1. Definitions

       1. "Autodetection" means detecting and making available to Mule
          the external file's encoding.  See (5), below.  It doesn't
          imply any specific actions based on that information.

       2. The "default" case is POSIX locale, and no environment
          information in ~/.emacs.

          N.B.  This *will* cause breakage for all 1-byte users because
          the default case can no longer assume Latin-1.  You *may* be
          able to use the TTY font or the Xt -font option to fake this,
          and default to iso8859-1; I would hope that we would not use
          such a kludge in the beta versions, although it might be
          satisfactory for general use.  In particular, encodings like
          VISCII (Vietnamese) and I believe KOI-8 (Cyrillic) are not
          ISO-2022-clean, but using C1 control characters as a heuristic
          for detecting binary files is useful.

          If we do allow it, I think that XEmacs should bitch and warn
          that the practices of implicitly specifying language
          environment by -font and defaulting on TTYs is deprecated and
          likely to be obsoleted.

       3. The "European" case is any Latin-* locale, either implied by
          setlocale() and friends or set in ~/.emacs.  Latin-1 is
          specifically not given precedence over other Latin-*, or
          non-Latin or non-ISO-8859 for that matter.  I suspect but am
          not sure that this case extends to all ISO-8859 encodings, and
          possibly to non-ISO-8859 single-byte encodings like KOI-8r (in
          particular when combined in a class with ISO-8859 encodings).

       4. The "CJK" case is any CJK locale.  Japanese is specifically
          not given precedence over other Asian locales.

       5. For completeness, define the "Unicode" case (Unicode
          unfortunately has lots of junk such as precomposed characters,
          language tags, and directionality indicators in it; we
          probably don't care yet, but we should also not claim
          compliance) and the "general" case (which has a lot of
          features similar to Unicode, but lacks the advantage of a
          unified encoding).  This proposal has no idea how to handle
          the special features of these, or even if that matters.  The
          general case includes stuff that nobody here really knows how
          it works, like Tibetan and Ethiopic.

     Each of the following cases is given in the order of priority of
     detection.  I'm not sure I'm serious about the top priority given
     the (optional) Unicode detection.  This may be appropriate if Ben
     is right that ISO-2022 is going to disappear, but possibly not
     until then (two two-byte sequences out of 65536 is probably 1.99
     too many).  It probably isn't too risky if (6)(c) is taken pretty
     seriously; a Unicode file should contain _no_ private use
     characters unless the encoding is explicitly specified, and that's
     a block of 1/10 of the code space, which should help a lot in
     detecting binary files.

  2. Default locale

       1. Some Unicode (fixed width; maybe UTF-8, too?) may optionally
          be detected by the byte-order-mark magic (if the first two
          bytes are 0xFE 0xFF, the file is Unicode text, if 0xFF 0xFE,
          it is wrong-endian Unicode; if legal in UTF-8, it would be
          0xFE 0xBB 0xBF, either-endian).  This is probably an
          optimization that should not be on by default yet.

       2. ISO-2022 encodings will be detected as long as they use
          explicit designation of all non-ASCII character sets.  This
          means that many 7-bit ISO-2022 encodings would be detected
          (eg, ISO-2022-JP), but EUC-JP and X Compound Text would not,
          because they implicitly designate character sets.

          N.B. Latin-1 will be detected as binary, as for any Latin-*.

          N.B. An explicit ISO-2022 designation is semantically
          equivalent to a Content-Type: header.  It is more dangerous
          because shorter, but I think we should recognize them by
          default despite the slight risk; XEmacs is a text editor.

          N.B. This is unlikely to be as dangerous as it looks at first
          glance.  Any file that includes an 8-bit-set byte before the
          first valid designation should be detected as binary.

       3. Binary files will be detected (eg, presence of NULs, other
          non-whitespace control characters, absurdly long lines, and
          presence of bytes >127).

       4. Everything else is ASCII.

       5. Newlines will be detected in text files.

  3. European locales

       1. Unicode may optionally be detected by the byte-order-mark
          magic.

       2. ISO-2022 encodings will be detected as long as they use
          explicit designation of all non-ASCII character sets.

       3. A locale-specific class of 1-byte character sets (eg,
          '(Latin-1)) will be detected.

          N.B.  The reason for permitting a class is for cases like
          Cyrillic where there are both ISO-8859 encodings and
          incompatible encodings (KOI-8r) in common use.  If you want to
          write a Latin-1 v. Latin-2 detector, be my guest, but I don't
          think it would be easy or accurate.

       4. Binary files will be detected per (2)(c), except that only
          8-bit bytes out of the encoding's range imply binary.

       5. Everything else is ASCII.

       6. Newlines will be detected in text files.

  4. CJK locales

       1. Unicode may optionally be detected by the byte-order-mark
          magic.

       2. ISO-2022 encodings will be detected as long as they use
          explicit designation of all non-ASCII character sets.

       3. A locale-specific class of multi-byte and wide-character
          encodings will be detected.  N.B. No 1-byte character sets
          (eg, Latin-1) will be detected.  The reason for a class is to
          allow the Japanese to let Mule do the work of choosing EUC v.
          SJIS.

       4. Binary files will be detected per (3)(d).

       5. Everything else is ASCII.

       6. Newlines will be detected in text files.

  5. Unicode and general locales; multilingual use

       1. Hopefully a system general enough to handle (2)-(4) will
          handle these, too, but we should watch out for gotchas like
          Unicode "plane 14" tags which (I think _both_ Ben and Olivier
          will agree) have no place in the internal representation, and
          thus must be treated as out-of-band control sequences.  I
          don't know if all such gotchas will be as easy to dispose of.

       2. An explicit coding system priority list will be provided to
          allow multilingual users to autodetect both Shift JIS and Big
          5, say, but this ability is not promised by Mule, since it
          would involve (eg) heuristics like picking a set of code
          points that are frequent in Shift JIS and uncommon in Big 5
          and betting that a file containing many characters from that
          set is Shift JIS.

  6. Relationship to decoding semantics

       1. Autodetection should be run on every input stream unless the
          user explicitly disables it.

       2. The (conceptual) default procedure is

       3. Read the file into the buffer

          Announce the result of autodetection to the user.

          User may request decoding, with autodetected encoding(s)
          given priority in a list of available encodings.

          zations (see (e) below) should avoid introducing data tion
          that this default procedure would avoid.

          sly, it can't be perfect if any autodecoding is done; like
          Hrvoje should have an easily available option to  to this
          default (or an optimized approximation which t actually read
          the whole file into a buffer) or simply y everything as
          binary (with the "font" for binary files a user option).

       4. This implies that we should be detecting conditions in the
          tail of the file which violate the implicit assumptions of the
          coding system autodetected (eg, in UTF-8 illegal UTF-8
          sequences, including those corresponding to surrogates) should
          raise a warning; the buffer should probably be made read-only
          and the user prompted.

          This could be taken to extremes, like checking by table
          whether all characters in a Japanese file are actually
          legitimate JIS codes; that's insane (and would cause corporate
          encodings to be recognized as binary).  But we should think
          about the idea that autodetection shouldn't mean XEmacs can't
          change its mind.

       5. A flexible means for the user to delegate the decision
          (conditional on the result of autodetection) to decode or not
          to XEmacs or a Lisp program should be provided (eg, the
          coding priority list and/or a file-coding-alist).

       6. Optimized operations (eg, the current lstreams) should be
          provided, with the recognition that if they depend on sampling
          the file they are risky.

       7. Mule should provide a reasonable set of default delegations
          (as in (d) above) for as many locales as possible.

  7. Implementation

       1. I think all the decision logic suggested above can be
          accomplished through a coding-priority-list and appropriate
          initializations for different language environments, and a
          file-coding-alist.

       2. Many of the tests on the file's tail shouldn't be very
          expensive; in particular, all of the ones I've suggested are
          O(n) although they might involve moderate-sized auxiliary
          tables for efficiency (eg, 64kB for a single Unicode-oriented
          test).

   Other comments:

   It might be reasonable given Hrvoje's objections to require that any
autodetection that could cause data loss (any coding system that
involves escape sequences, and only those AFAIK: by design translation
to Unicode is invertible) by default prompt the user (presumable with a
novice-like ability to retain the prompt, always default to binary, or
always default to the autodetected encoding) in the future, at least in
locales that don't need it (POSIX, Latin-any).

   Ben thinks that we can remember the input data; I think it's going to
be hard to comprehensively test that a highly optimized version works.
Good design will help, but ISO-2022 is enormously complex, and there
are many encodings that violate even its lax assumptions.  On the other
hand, memory is the only way to get non-rewindable streams right.

   Hrvoje himself said he would like to have an XEmacs that
distinguishes between Latin-1 and Latin-2 text.  Where it is possible
to do that, this is exactly what autodetection of ISO-2022 and Unicode
gives you.  Many people would want that, even at some risk of binary
corruption.

   >> Once again I remind you that XEmacs is a *text* editor.  There
>> are lots of files that potentially may have Japanese etc. in     >>
them without this marked, e.g. C or Elisp files in the XEmacs     >>
source.  Surely you're not arguing that we interpret even these     >>
files as binary by default?

   Hrvoje> I am.  If I want to see Japanese, I'll setup my     Hrvoje>
environment that way.  But I don't, and neither do 99% of     Hrvoje>
Croatian users.  I can't speak for French, Italian, and     Hrvoje>
others, but I'd assume similar.

   Hrvoje> If there is Japanese in the source files, I will see it as
 Hrvoje> escape sequences, which is perfectly fine, because I don't
Hrvoje> read Japanese.

   And some (European) people will have their terminals scrambled,
because Shift-JIS contains sequences that can change the state of XTerm
(as do fixed-width Unicode and Big5).  This may also be a problem with
some Windows-12xx encodings; I'm not sure they all are ISO-2022-clean.
(This isn't a problem for XEmacs native X11 frames or native MS-Windows
frames, and the XEmacs sources themselves are all in 7-bit ISO-2022 now
IIRC.  But it is a potential source of great frustration for many
users.)

   I think that should be considered too, although it is presumably
lower priority than the data corruption of binary files.

Response to RFC: Autodetection
------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   Date: 11/1/1999 7:24 AM

   Stephen, thank you very much for writing this up.  I think it is a
good start, and definitely moving in the direction I would like to see
things going: more proposals, less arguing. (aka "more light, less
heat") However, I have some suggestions for cleaning this up:

   You should try to make it more layered.  For example, you might have
one section devoted to the workings of autodetection, which starts out
like this (the section numbers below are totally arbitrary):

Section 5
.........

`Autodetect()' is a function whose arguments are (1) a readable stream,
(2) some hints indicating how the autodetection is to proceed, and (3)
a value indicating the maximum number of characters to examine at the
beginning of the stream.  (Possibly, the value in (3) may be some
special symbol indicating that we only go as far as the next line, or a
certain number of lines ahead; this would be used as part of
"continuous autodetection", e.g. we are decoding the results of an
interactive terminal session, where the user may periodically switch
encodings, line terminations, etc. as different programs get run and/or
telnet or similar sessions are entered into and exited.) We assume the
stream is rewindable; if not, insert a "rewinding" stream in front of
the non-rewinding stream; this kind of stream automatically buffers the
data as necessary.  [You can use pseudo-code terminology here.  No need
for straight C or ELisp.]  [Then proceed to describe what the hints
look like - e.g. you could portray it as a property list or whatever.
The idea is that, for each locale, there is a corresponding hints value
that is used at least by default.  The hints structure also has to be
set up to allow for two or more competing hints specifications to be
merged together.  For example, the extension of a file might provide an
additional hint or hints about how to interpret the data of that file,
and the caller of `autodetect()', when calling `autodetect()' on such a
file, would need to have a way of gracefully merging the default hints
corresponding to the locale with the more specific hints provided by the
extension.  Furthermore, users like Hrvoje might well want to provide
their own hints to supplement and override parts of the generic hints -
e.g. "I don't ever want to see non-European encodings decoded; treat
them as binary instead".]  [Then describe algorithmically how the
autodetection works.  First, you could describe it more generally, i.e.
presenting an algorithmic overview, then you could discuss in detail
exactly how autodetection of a particular type of external encoding
works - e.g. "for iso2022, we first look for an escape character,
followed by a byte in this range [. ... .] etc."]

Section 6
.........

This section describes the concept of a locale in XEmacs, and how it is
derived from the user's environment.  A locale in XEmacs is a pair, a
country and a language, together determining the handling of
locale-specific areas of XEmacs.  All locale-specific areas in XEmacs
make use of this XEmacs locale, and do not attempt to derive the locale
from any other sources.  The user is free to change the current locale
at any time; accessor and mutator functions are provided to do this so
that various locale-specific areas can optionally be changed together
with it.

   [Then you describe how the XEmacs locale is extracted from .emacs,
from `setlocale()', from the LANG environment variables, from -font, or
wherever else.  All other sections assume this dirty work is done and
never even mention it]

Section 7
.........

[Here you describe the default `autodetect()' hints value corresponding
to each possible locale.  You should probably use a schematic
description here, e.g.  an actual Lisp property list, liberally
commented.]

Section 8 etc.
..............

[Other sections cover anything I've missed.  By being very careful to
separate out the layers, you simultaneously introduce more rigor
(easier to catch bugs) and make it easier for someone else to
understand it completely.]

Better Algorithm, More Flexibility, Different Levels of Certainty
-----------------------------------------------------------------

Much More Flexible Coding System Priority List, per-Language Environment
------------------------------------------------------------------------

User Ability to Select Encoding when System Unsure or Encounters Errors
-----------------------------------------------------------------------

Another Autodetection Proposal
------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   however, in general the detection code has major problems and needs
lots of work:

   * instead of merely "yes" or "no" for particular categories, we need
     a more flexible system, with various levels of likelihood.
     Currently I've created a system with six levels, as follows:

     [see `file-coding.h']

     Let's consider what this might mean for an ASCII text detector.
     (In order to have accurate detection, especially given the
     iteration I proposed below, we need active detectors for *all*
     types of data we might reasonably encounter, such as ASCII text
     files, binary files, and possibly other sorts of ASCII files, and
     not assume that simply "falling back to no detection" will work at
     all well.)

     An ASCII text detector DOES NOT report ASCII text as level 0, since
     that's what the detector is looking for.  Such a detector ideally
     wants all bytes in the range 0x20 - 0x7E (no high bytes!), except
     for whitespace control chars and perhaps a few others; LF, CR, or
     CRLF sequences at regular intervals (where "regular" might mean an
     average < 100 chars and 99% < 300 for code and other stuff of the
     "text file w/line breaks" variety, but for the "text file w/o line
     breaks" variety, excluding blank lines, averages could easily be
     600 or more with 2000-3000 char "lines" not so uncommon); similar
     statistical variance between odds and evens (not Unicode);
     frequent occurrences of the space character; letters more common
     than non-letters; etc.  Also checking for too little variability
     between frequencies of characters and for exclusion of particular
     characters based on character ranges can catch ASCII encodings
     like base-64, UUEncode, UTF-7, etc.  Granted, this doesn't even
     apply to everything called "ASCII", and we could potentially
     distinguish off ASCII for code, ASCII for text, etc. as separate
     categories.  However, it does give us a lot to work off of, in
     deciding what likelihood to choose - and it shows there's in fact
     a lot of detectable patterns to look for even in something
     seemingly so generic as ASCII.  The detector would report most text
     files in level 1 or level 2.  EUC encodings, Shift-JIS, etc.
     probably go to level -1 because they also pass the EOL test and
     all other tests for the ASCII part of the text, but have lots of
     high bytes, which in essence turn them into binary.  Aberrant text
     files like something in BASE64 encoding might get placed in level
     0, because they pass most tests but fail dramatically the
     frequency test; but they should not be reported as any lower,
     because that would cause explicit prompting, and the user should
     be able any valid text file without prompting.  The escape
     sequences and the base-64-type checks might send 7-bit iso2022 to
     0, but probably not -1, for similar reasons.

   * The assumed algorithm for the above detection levels is to in
     essence sort categories first by detection level and then by
     priority.  Perhaps, however, we would want smarter algorithms, or
     at least something user-controllable - in particular, when (other
     than no category at level 0 or greater) do we prompt the user to
     pick a category?

   * Improvements in how the detection algorithm works: we want to
     handle lots of different ways something could be encoded,
     including multiple stacked encodings.  trying to specify a series
     of detection levels (check for base64 first, then check for gzip,
     then check for an i18n decoding, then for crlf) won't generally
     work.  for example, what about the same encoding appearing more
     than once? for example, take euc-jp, base64'd, then gzip'd, then
     base64'd again: this could well happen, and you could specify the
     encodings specifically as base64|gzip|base64|euc-jp, but we'd like
     to autodetect it without worrying about exactly what order these
     things appear in.  we should allow for iterating over
     detection/decoding cycles until we reach some maximum (we got
     stuck in a loop, due to incorrect category tables or detection
     algorithms), have no reported detection levels over -1, or we end
     up with no change after a decoding pass (i.e. the coding system
     associated with a chosen category was `no-conversion' or something
     equivalent).  it might make sense to divide things into two phases
     (internal and external), where the internal phase has a separate
     category list and would probably mostly end up handling EOL
     detection; but the i think about it, the more i disagree.  with
     properly written detectors, and properly organized tables (in
     general, those decodings that are more "distinctive" and thus
     detectable with greater certainty go lower on the list), we
     shouldn't need two phases.  for example, let's say the example
     above was also in CRLF format.  The EOL detector (which really
     detects *plain text* with a particular EOL type) would return at
     most level 0 for all results until the text file is reached,
     whereas the base64, gzip or euc-jp decoders will return higher.
     Once the text file is reached, the EOL detector will return 0 or
     higher for the CRLF encoding, and all other detectors will return
     0 or lower; thus, we will successfully proceed through CRLF
     decoding, or at worst prompt the user. (The only
     external-vs-internal distinction that might make sense here is to
     favor coding systems of the correct source type over those that
     require conversion between external and internal; if done right,
     this could allow the CRLF detector to return level 1 for all
     CRLF-encoded text files, even those that look like Base-64 or
     similar encoding, so that CRLF encoding will always get decoded
     without prompting, but not interfere with other decoders.  On the
     other hand, this external-vs-internal distinction may not matter
     at all - with automatic internal-external conversion, CRLF
     decoding can occur before or after decoding of euc-jp, base64,
     iso2022, or similar, without any difference in the final results.)

     #### What are we trying to say?  In base64, the CRLF decoding
     before base64 decoding is irrelevant, they will be thrown out as
     whitespace is not significant in base64.

     [sjt considers all of this to be rather bogus.  Ideas like "greater
     certainty" and "distinctive" can and should be quantified.  The
     issue of proper table organization should be a question of
     optimization.]

     [sjt wonders if it might not be a good idea to use Unicode's
     newline character as the internal representation so that (for
     non-Unicode coding systems) we can catch EOL bugs on Unix too.]

   * There need to be two priority lists and two
     category->coding-system lists.  Once is general, the other
     category->langenv-specific.  The user sets the former, the langenv
     category->the latter.  The langenv-specific entries take precedence
     category->over the others.  This works similarly to the
     category->category->Unicode charset priority list.

   * The simple list of coding categories per detectors is not enough.
     Instead of coding categories, we need parameters.  For example,
     Unicode might have separate detectors for UTF-8, UTF-7, UTF-16,
     and perhaps UCS-4; or UTF-16/UCS-4 would be one detection type.
     UTF-16 would have parameters such as "little-endian" and "needs
     BOM", and possibly another one like "collapse/expand/leave alone
     composite sequences" once we add this support.  Usually these
     parameters correspond directly to a coding system parameter.
     Different likelihood values can be specified for each parameter as
     well as for the detection type as a whole.  The user can specify
     particular coding systems for a particular combination of
     detection type and parameters, or can give "default parameters"
     associated with a detection type.  In the latter case, we create a
     new coding system as necessary that corresponds to the detected
     type and parameters.

   * a better means of presentation.  rather than just coming up with
     the new file decoded according to the detected coding system,
     allow the user to browse through the file and conveniently reject
     it if it looks wrong; then detection starts again, but with that
     possibility removed.  in cases where certainty is low and thus
     more than one possibility is presented, the user can browse each
     one and select one or reject them all.

   * fail-safe: even after the user has made a choice, if they later on
     realize they have the wrong coding system, they can go back, and
     we've squirreled away the original data so they can start the
     process over.  this may be tricky.

   * using a larger buffer for detection.  we use just a small piece,
     which can give quite random results.  we may need to buffer up all
     the data we look through because we can't necessarily rewind.  the
     idea is we proceed until we get a result that's at least at a
     certain level of certainty (e.g. "probable") or we reached a
     maximum limit of how much we want to buffer.

   * dealing with interactive systems.  we might need to go ahead and
     present the data before we've finished detection, and then
     re-decode it, perhaps multiple times, as we get better detection
     results.

   * Clearly some of these are more important than others.  at the very
     least, the "better means of presentation" should be implemented as
     soon as possible, along with a very simple means of fail-safe
     whenever the data is readibly available, e.g. it's coming from a
     file, which is the most common scenario.

   ben [at least that's what sjt thinks]

   *****

   Author: Stephen Turnbull (mailto:stephen@xemacs.org)

   While this is clearly something of an improvement over earlier
designs, it doesn't deal with the most important issue: to do better
than categories (which in the medium term is mostly going to mean
"which flavor of Unicode is this?"), we need to look at statistical
behavior rather than ruling out categories via presence of specific
sequences.  This means the stream processor should

  1. keep octet distributions (octet, 2-, 3-, 4- octet sequences)

  2. in some kind of compressed form

  3. look for "skip features" (eg, characteristic behavior of leading
     bytes for UTF-7, UTF-8, UTF-16, Mule code)

  4. pick up certain "simple" regexps

  5. provide "triggers" to determine when statistical detectors should
     be invoked, such as octet count

  6. and "magic" like Unicode signatures or file(1) magic.

   -sjt


File: internals.info,  Node: Future Work -- Conversion Error Detection,  Next: Future Work -- Unicode,  Prev: Future Work -- Autodetection,  Up: Future Work -- Byte Code Snippets

40.16.2 Future Work - Conversion Error Detection
------------------------------------------------

"No Corruption" Scheme for Preserving External Encoding when Non-Invertible Transformation Applied
--------------------------------------------------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   A preliminary and simple implementation is:

     But you could implement it much more simply and usefully by just
     determining, for any text being decoded into mule-internal, can we
     go back and read the source again?  If not, remember the entire
     file (GNUS message, etc) in text properties.  Then, implement the
     UI interface (like Netscape's) on top of that.  This way, you have
     something that at least works, but it might be inefficient.  All we
     would need to do is work on making the underlying implementation
     more efficient.

   A more detailed proposal for avoiding binary file corruption is

     Basic idea: A coding system is a filter converting an entire input
     stream into an output stream. The resulting stream can be said to
     be "correspondent to" the input stream. Similarly, smaller units
     can correspond. These could potentially include zero width
     intervals on either side, but we avoid this.  Specifically, the
     coding system works like:

          loop (input) {

           Read bytes till we have enough to generate a translated character or a chars.

           This establishes a "correspondence" between the whole input and
           output more or less in minimal chunks.

          }

     We then do the following processing:

       1. Eliminate correspondences where one or the other of the I/O
          streams has a zero interval by combining with an adjacent
          interval;

       2. Group together all adjacent "identity" correspondences into as
          large groups as possible;

       3. Use text properties to store the non-identity correspondences
          on the characters. For identity correspondences, use a simple
          text property on all that contains no data but just indicates
          that the whole string of text is identity corresponded. (How
          do we define "identity"? Latin 1 or could it be something
          else? For example, Latin 2)?

       4. Figure out the procedures when text is inserted/deleted and
          copied or pasted.

       5. Figure out to save the file out making use of the
          correspondences. Allow ways of saving without
          correspondences, and doing a "save to buffer with and without
          correspondences."  Need to be clever when dealing with modal
          coding systems to parse the correspondences to get the
          internal state right.

Another Error-Catching Idea
---------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   Nov 4, 1999

   Finally, I don't think "save the input" is as hard as you make it
out to be.  Conceptually, in fact, it's simple: for each minimal group
of bytes where you cannot absolutely guarantee that an
external->internal transformation is reversible, you put a text
property on the corresponding internal character indicating the bytes
that generated this character.  We also put a text property on every
character, indicating the coding system that caused the transformation.
This latter text property is extremely efficient (e.g. in a buffer
with no data pasted from elsewhere, it will map to a single extent over
all the buffer), and the former cases should not be prevalent enough to
cause a lot of inefficiency, esp. if we define what "reversible" means
for each coding system in such a way that it correctly handles the most
common cases.  The hardest part, in fact, is making all the string/text
handling in XEmacs be robust w.r.t. text properties.

Strategies for Error Annotation and Coding Orthogonalization
------------------------------------------------------------

Author: Stephen Turnbull (mailto:stephen@xemacs.org)

   We really want to separate out a number of things.  Conceptually,
there is a nested syntax.

   At the top level is the ISO 2022 extension syntax, including charset
designation and invocation, and certain auxiliary controls such as the
ISO 6429 direction specification.  These are octet-oriented, with the
single exception (AFAIK) of the "exit Unicode" sequence which uses the
UTF's natural width (1 byte for UTF-7 and UTF-8, 2 bytes for UCS-2 and
UTF-16, and 4 bytes for UCS-4 and UTF-32).  This will be treated as a
(deprecated) special case in Unicode processing.

   The middle layer is ISO 2022 character interpretation.  This will
depend on the current state of the ISO 2022 registers, and assembles
octets into the character's internal representation.

   The lowest level is translating system control conventions.  At
present this is restricted to newline translation, but one could
imagine doing tab conversion or line wrapping here.  "Escape from
Unicode" processing would be done at this level.

   At each level the parser will verify the syntax.  In the case of a
syntax error or warning (such as a redundant escape sequence that
affects no characters), the parser will take some action, typically
inserting the erroneous octets directly into the output and creating an
annotation which can be used by higher level I/O to mark the affected
region.

   This should make it possible to do something sensible about
separating newline convention processing from character construction,
and about preventing ISO 2022 escape sequences from being recognized
inappropriately.

   The basic strategy will be to have octet classification tables, and
switch processing according to the table entry.

   It's possible that, by doing the processing with tables of functions
or the like, the parser can be used for both detection and translation.

Handling Writing a File Safely, Without Data Loss
-------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

     When writing a file, we need error detection; otherwise somebody
     will create a Unicode file without realizing the coding system of
     the buffer is Raw, and then lose all the non-ASCII/Latin-1 text
     when it's written out.  We need two levels

       1. first, a "safe-charset" level that checks before any actual
          encoding to see if all characters in the document can safely
          be represented using the given coding system.  FSF has a
          "safe-charset" property of coding systems, but it's stupid
          because this information can be automatically derived from
          the coding system, at least the vast majority of the time.
          What we need is some sort of
          alternative-coding-system-precedence-list, langenv-specific,
          where everything on it can be checked for safe charsets and
          then the user given a list of possibilities.  When the user
          does "save with specified encoding", they should see the same
          precedence list.  Again like with other precedence lists,
          there's also a global one, and presumably all coding systems
          not on other list get appended to the end (and perhaps not
          checked at all when doing safe-checking?).  safe-checking
          should work something like this: compile a list of all
          charsets used in the buffer, along with a count of chars
          used.  that way, "slightly unsafe" coding systems can perhaps
          be presented at the end, which will lose only a few characters
          and are perhaps what the users were looking for.

          [sjt sez this whole step is a crock.  If a universal coding
          system is unacceptable, the user had better know what he/she
          is doing, and explicitly specify a lossy encoding.  In
          principle, we can simply check for characters being writable
          as we go along.  Eg, via an "unrepresentable character
          handler."  We still have the buffer contents.  If we can't
          successfully save, then ask the user what to do.  (Do we ever
          simply destroy previous file version before completing a
          write?)]

       2. when actually writing out, we need error checking in case an
          individual char in a charset can't be written even though the
          charsets are safe.  again, the user gets the choice of other
          reasonable coding systems.

          [sjt - something is very confused, here; safe charsets should
          be defined as those charsets all of whose characters can be
          encoded.]

       3. same thing (error checking, list of alternatives, etc.) needs
          to happen when reading!  all of this will be a lot of work!

   Author: Stephen Turnbull (mailto:stephen@xemacs.org)

   I don't much like Ben's scheme.  First, this isn't an issue of I/O,
it's a coding issue.  It can happen in many places, not just on stream
I/O.  Error checking should take place on all translations.  Second,
the two-pass algorithm should be avoided if possible.  In some cases
(eg, output to a tty) we won't be able to go back and change the
previously output data.  Third, the whole idea of having a buffer full
of arbitrary characters which we're going to somehow shoehorn into a
file based on some twit user's less than informed idea of a coding
system is kind of laughable from the start.  If we're going to say that
a buffer has a coding system, shouldn't we enforce restrictions on what
you can put into it?  Fourth, what's the point of having safe charsets
if some of the characters in them are unsafe?  Fifth, what makes you
think we're going to have a list of charsets?  It seems to me that
there might be reasons to have user-defined charsets (eg, "German" vs
"French" subsets of ISO 8859/15).  Sixth, the idea of having language
environment determine precedence doesn't seem very useful to me.  Users
who are working with a language that corresponds to the language
environment are not going to run into safe charsets problems.  It's
users who are outside of their usual language environment who run into
trouble.  Also, the reason for specifying anything other than a
universal coding system is normally restrictions imposed by other users
or applications.  Seventh, the statistical feedback isn't terribly
useful.  Users rarely "want" a coding system, they want their file
saved in a useful way.  We could add a FORCE argument to conversions
for those who really want a specific coding system.  But mostly, a user
might want to edit out a few unsafe characters.  So (up to some
maximum) we should keep a list of unsafe text positions, and provide a
convenient function for traversing them.

   -sjt


File: internals.info,  Node: Future Work -- Unicode,  Next: Future Work -- BIDI Support,  Prev: Future Work -- Conversion Error Detection,  Up: Future Work -- Byte Code Snippets

40.16.3 Future Work - Unicode
-----------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   Following is an old proposal.  Unicode has been implemented already,
in a different fashion; but there are some ideas here for more general
support, e.g. properties of Unicode characters other than their mappings
to particular charsets.

   We recognize 128, [256], 128x128, [256x256] for source charsets;

   for Unicode, 256x256 or 16x256x256.

   In all cases, use tables of tables and substitute a default subtable
if entire row is empty.

   If destination is Unicode, either 16 or 32 bits.

   If destination is charset, either 8 or 16 bits.

   For the moment, since we only do 94, 96, 94x94 or 96x96, only do 128
or 128x128 for source charsets and use the range 33-126 or 32-127.
(Except ASCII - we special case that and have no table because we can
algorithmically translate)

   Also have a 16x256x256 table -> 32 bits of Unicode char properties.

   A particular charset contains two associated mapping tables, for both
directions.

   API is set-unicode-mapping:

     (set-unicode-mapping
          unicode char
          unicode charset-code charset-offset
          unicode vector of char
          unicode list of char
          unicode string of char
          unicode vector or list of codes charset-offset

   Establishes a mapping between a unicode codepoint (an integer) and
one or more chars in a charset.  The mapping is automatically
established in both directions.  Chars in a charset can be specified
either with an actual character or a codepoint (i.e. an integer)   and
the charset it's within.  If a sequence of chars or charset   points is
given, multiple mappings are established for consecutive   unicode
codepoints starting with the given one.  Charset codepoints   are
specified as most-significant x 256 + least significant, with   both
bytes in the range 33-126 (for 94 or 94x94) or 32-127 (for 96   or
96x96), unless an offset is given, which will be subtracted from   each
byte.  (Most common values are 128, for codepoints given with   the
high bit set, or -32, for codepoints given as 1-94 or 0-95.)

   Other APIs:

     (write-unicode-mapping file charset)

   Write the mapping table for a particular charset to the specified
file.  The tables are written in an internal format that allows for
efficient loading, for portability across platforms and XEmacs
invocations, for conserving space, for appending multiple tables one
directly after another with no need for a directory anywhere in the
file, and for reorganizing a file as in this format (with a magic
sequence at the beginning).  The data will be appended at the end of
a file, so that multiple tables can be written to a file; remove the
file first to avoid this.

     (write-unicode-properties file unicode-codepoint length)

   Write the Unicode properties (not including charset mappings) for
the specified range of contiguous Unicode codepoints to the end of
the file (i.e. append mode) in a binary format similar to what was
mentioned in the write-unicode-mapping description and with the same
features.

   Extension to set-unicode-mapping:

     (set-unicode-mapping
       list-or-vector-of-unicode-codepoints char
       ""                                   charset-code charset-offset
       ""                                   sequence of char
       ""                                   list-or-vector-of-codes
     charset-offset

   The first two forms are conceptually the inverse of the forms above
to specify characters for a contiguous range of Unicode codepoints.
These new forms let you specify the Unicode codepoints for a
contiguous range of chars in a charset.  "Contiguous" here means   that
if we run off the end of a row, we go to the first entry of the   next
row, rather than to an invalid code point.  For example, in a   94x94
charset, valid rows and columns are in the range 0x21-0x7e;   after
0x457c 0x457d 4x457e goes 0x4621, not something like 0x457f,   which is
invalid.

   The final two forms are the most general, letting you specify an
arbitrary set of both Unicode points and charset chars, and the two
are matched up just like a series of individual calls.  However, if
the lists or vectors do not have the same length, an error is
signaled.

     (load-unicode-mapping file &optional charset)

   If charset is omitted, loads all charset mapping tables found and
returns a list of the charsets found.  If charset is specified,
searches through the file for the appropriate mapping tables.  (This
is extremely fast because each entry in the file gives an offset to
the next one).  Returns t if found.

     (load-unicode-properties file unicode-codepoint)

     (list-unicode-entries file)

     (autoload-unicode-mapping charset)

   ...

   (unfinished)


File: internals.info,  Node: Future Work -- BIDI Support,  Next: Future Work -- Localized Text/Messages,  Prev: Future Work -- Unicode,  Up: Future Work -- Byte Code Snippets

40.16.4 Future Work - BIDI Support
----------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

  1. Use text properties to handle nesting levels, overrides
     BIDI-specific text properties (as per Unicode BIDI algorithm)
     computed at text insertion time.

  2. Lisp API for reordering a display line at redisplay time, possibly
     substitution of different glyphs (esp. mirroring of glyphs).

  3. Lisp API called after a display line is laid out, but only when
     reordering may be necessary (display engine checks for non-uniform
     BIDI text properties; can handle internally a line that's
     completely in one direction)

  4. Default direction is a buffer-local variable

  5. We concentrate on implementing Unicode BIDI algorithm.

  6. Display support for mirroring of entire window

  7. Display code keeps track of mirroring junctures so it can display
     double cursor.

  8. Entire layout of screen (on a per window basis) is exported as a
     Lisp API, for visual editing (also very useful for other purposes
     e.g. proper handling of word wrapping with proportional fonts,
     complex Lisp layout engines e.g. W3)

  9. Logical, visual, etc. cursor movement handled entirely in Lisp,
     using aforementioned API, plus a specifier for controlling how
     cursor is shown (e.g. split or not).


File: internals.info,  Node: Future Work -- Localized Text/Messages,  Prev: Future Work -- BIDI Support,  Up: Future Work -- Byte Code Snippets

40.16.5 Future Work - Localized Text/Messages
---------------------------------------------

NOTE: There is existing message translation in X Windows of menu names.
This is handled through X resources.  The files are in
`PACKAGES/mule-packages/locale/app-defaults/LOCALE/Emacs', where LOCALE
is `ja', `fr', etc.

   See lib-src/make-msgfile.lex.

   Long comment from jwz, some additions from ben marked "ben":

   (much of this comment is outdated, and a lot of it is actually
implemented)

40.16.6 Proposal for How This All Ought to Work
-----------------------------------------------

Author: Jamie Zawinski (mailto:jwz@jwz.org)

   this isn't implemented yet, but this is the plan-in-progress

   In general, it's accepted that the best way to internationalize is
for all messages to be referred to by a symbolic name (or number) and
come out of a table or tables, which are easy to change.

   However, with Emacs, we've got the task of internationalizing a huge
body of existing code, which already contains messages internally.

   For the C code we've got two options:

   * Use a Sun-like `gettext()' form, which takes an "english" string
     which appears literally in the source, and uses that as a hash key
     to find a translated string;

   * Rip all of the strings out and put them in a table.

   In this case, it's desirable to make as few changes as possible to
the C code, to make it easier to merge the code with the FSF version of
emacs which won't ever have these changes made to it.  So we should go
with the former option.

   The way it has been done (between 19.8 and 19.9) was to use
`gettext()', but *also* to make massive changes to the source code.
The goal now is to use `gettext()' at run-time and yet not require a
textual change to every line in the C code which contains a string
constant.  A possible way to do this is described below.

   (`gettext()' can be implemented in terms of `catgets()' for non-Sun
systems, so that in itself isn't a problem.)

   For the Lisp code, we've got basically the same options: put
everything in a table, or translate things implicitly.

   Another kink that lisp code introduces is that there are thousands
of third- party packages, so changing the source for all of those is
simply not an option.

   Is it a goal that if some third party package displays a message
which is one we know how to translate, then we translate it?  I think
this is a worthy goal.  It remains to be seen how well it will work in
practice.

   So, we should endeavor to minimize the impact on the lisp code.
Certain primitive lisp routines (the stuff in lisp/prim/, and
especially in `cmdloop.el' and `minibuf.el') may need to be changed to
know about translation, but that's an ideologically clean thing to do
because those are considered a part of the emacs substrate.

   However, if we find ourselves wanting to make changes to, say,
RMAIL, then something has gone wrong.  (Except to do things like remove
assumptions about the order of words within a sentence, or how
pluralization works.)

   There are two parts to the task of displaying translated strings to
the user: the first is to extract the strings which need to be
translated from the sources; and the second is to make some call which
will translate those strings before they are presented to the user.

   The old way was to use the same form to do both, that is,
`GETTEXT()' was both the tag that we searched for to build a catalog,
and was the form which did the translation.  The new plan is to
separate these two things more: the tags that we search for to build
the catalog will be stuff that was in there already, and the
translation will get done in some more centralized, lower level place.

   This program (`make-msgfile.c') addresses the first part, extracting
the strings.

   For the emacs C code, we need to recognize the following patterns:

       message ("string" ... )
       error ("string")
       report_file_error ("string" ... )
       signal_simple_error ("string" ... )
       signal_simple_error_2 ("string" ... )

       build_translated_string ("string")
       #### add this and use it instead of `build_string()' in some places.

       yes_or_no_p ("string" ... )
       #### add this instead of funcalling Qyes_or_no_p directly.

       barf_or_query_if_file_exists	#### restructure this
       check all callers of Fsignal	#### restructure these
       signal_error (Qerror ... )		#### change all of these to `error()'

       And we also parse out the `interactive' prompts from `DEFUN()' forms.

       #### When we've got a string which is a candidate for translation, we
       should ignore it if it contains only format directives, that is, if
       there are no alphabetic characters in it that are not a part of a `%'
       directive.  (Careful not to translate either "%s%s" or "%s: ".)

   For the emacs Lisp code, we need to recognize the following patterns:

       (message "string" ... )
       (error "string" ... )
       (format "string" ... )
       (read-from-minibuffer "string" ... )
       (read-shell-command "string" ... )
       (y-or-n-p "string" ... )
       (yes-or-no-p "string" ... )
       (read-file-name "string" ... )
       (temp-minibuffer-message "string")
       (query-replace-read-args "string" ... )

   I expect there will be a lot like the above; basically, any function
which is a commonly used wrapper around an eventual call to `message' or
`read-from-minibuffer' needs to be recognized by this program.

       (dgettext "domain-name" "string")		#### do we still need this?

       things that should probably be restructured:
         `princ' in `cmdloop.el'
         `insert' in `debug.el'
         face-interactive
         `help.el', `syntax.el' all messed up

   Author: Ben Wing (mailto:ben@xemacs.org)

   ben: (format) is a tricky case.  If I use format to create a string
that I then send to a file, I probably don't want the string translated.
On the other hand, If the string gets used as an argument to (y-or-n-p)
or some such function, I do want it translated, and it needs to be
translated before the %s and such are replaced.  The proper solution
here is for (format) and other functions that call gettext but don't
immediately output the string to the user to add the translated (and
formatted) string as a string property of the object, and have
functions that output potentially translated strings look for a
"translated string" property.  Of course, this will fail if someone
does something like

        (y-or-n-p (concat (if you-p "Do you " "Does he ")
          		(format "want to delete %s? " filename))))

   But you shouldn't be doing things like this anyway.

   ben: Also, to avoid excessive translating, strings should be marked
as translated once they get translated, and further calls to gettext
don't do any more translating.  Otherwise, a call like

        (y-or-n-p (format "Delete %s? " filename))

   would cause translation on both the pre-formatted and post-formatted
strings, which could lead to weird results in some cases (y-or-n-p has
to translate its argument because someone could pass a string to it
directly).  Note that the "translating too much" solution outlined
below could be implemented by just marking all strings that don't come
from a .el or .elc file as already translated.

   Menu descriptors: one way to extract the strings in menu labels
would be to teach this program about "^(defvar .*menu\n" forms; that's
probably kind of hard, though, so perhaps a better approach would be to
make this program recognize lines of the form

       "string" ... ;###translate

   where the magic token ";###translate" on a line means that the string
constant on this line should go into the message catalog.  This is
analogous to the magic ";###autoload" comments, and to the magic
comments used in the EPSF structuring conventions.

   ---- So this program manages to build up a catalog of strings to be
translated.  To address the second part of the problem, of actually
looking up the translations, there are hooks in a small number of low
level places in emacs.

   Assume the existence of a C function gettext(str) which returns the
translation of STR if there is one, otherwise returns STR.

   * `message()' takes a char* as its argument, and always filters it
     through `gettext()' before displaying it.

   * errors are printed by running the lisp function `display-error'
     which doesn't call `message' directly (it princ's to streams), so
     it must be carefully coded to translate its arguments.  This is
     only a few lines of code.

   * `Fread_minibuffer_internal()' is the lowest level interface to all
     minibuf interactions, so it is responsible for translating the
     value that will go into Vminibuf_prompt.

   * Fpopup_menu filters the menu titles through `gettext()'.

     The above take care of 99% of all messages the user ever sees.

   * The lisp function temp-minibuffer-message translates its arg.

   * query-replace-read-args is funny; it does   (setq from
     (read-from-minibuffer (format "%s: " string) ... ))   (setq to
     (read-from-minibuffer (format "%s %s with: " string from) ... ))

   What should we do about this?  We could hack query-replace-read-args
to translate its args, but might this be a more general problem?  I
don't think we ought to translate all calls to format.  We could just
change the calling sequence, since this is odd in that the first %s
wants to be translated but the second doesn't.

   Solving the "translating too much" problem:

   The concern has been raised that in this situation:

   * "Help" is a string for which we know a translation;

   * someone visits a file called Help, and someone does something
     contrived like (error buffer-file-name)

   then we would display the translation of Help, which would not be
correct.  We can solve this by adding a bit to Lisp_String objects
which identifies them as having been read as literal constants from a
.el or .elc file (as opposed to having been constructed at run time as
it would in the above case.)  To solve this:

   * `Fmessage()' takes a lisp string as its first argument.  If that
     string is a constant, that is, was read from a source file as a
     literal, then it calls `message()' with it, which translates.
     Otherwise, it calls `message_no_translate()', which does not
     translate.

   * `Ferror()' (actually, `Fsignal()' when condition is Qerror) works
     similarly.

   More specifically, we do:

      Scan specified C and Lisp files, extracting the following
     messages:

             C files:
                GETTEXT (...)
                DEFER_GETTEXT (...)
                DEFUN interactive prompts
             Lisp files:
                (gettext ...)
                (dgettext "domain-name" ...)
                (defer-gettext ...)
                (interactive ...)

     The arguments given to this program are all the C and Lisp source
     files of GNU Emacs.  .el and .c files are allowed.  There is no
     support for .elc files at this time, but they may be specified;
     the corresponding .el file will be used.  Similarly, .o files can
     also be specified, and the corresponding .c file will be used.
     This helps the makefile pass the correct list of files.

     The results, which go to standard output or to a file specified
     with -a or -o (-a to append, -o to start from nothing), are quoted
     strings wrapped in gettext(...).  The results can be passed to
     xgettext to produce a .po message file.

     However, we also need to do the following:

       1. Definition of Arg below won't handle a generalized argument
          as might appear in a function call.  This is fine for DEFUN
          and friends, because only simple arguments appear there; but
          it might run into problems if Arg is used for other sorts of
          functions.

       2. `snarf()' should be modified so that it doesn't output null
          strings and non-textual strings (see the comment at the top
          of `make-msgfile.c').

       3. parsing of (insert) should snarf all of the arguments.

       4. need to add set-keymap-prompt and deal with gettext of that.

       5. parsing of arguments should snarf all strings anywhere within
          the arguments, rather than just looking for a string as the
          argument.  This allows if statements as arguments to get
          parsed.

       6. `begin_paren_counting()' et al. should handle recursive entry.

       7. handle set-window-buffer and other such functions that take a
          buffer as the other-than-first argument.

       8. there is a fair amount of work to be done on the C code.
          Look through the code for #### comments associated with
          '#ifdef I18N3' or with an I18N3 nearby.

       9. Deal with `get-buffer-process' et al.

      10. Many of the changes in the Lisp code marked 'rewritten for
          I18N3 snarfing' should be undone once (5) is implemented.

      11. Go through the Lisp code in prim and make sure that all
          strings are gettexted as necessary.  This may reveal more
          things to implement.

      12. Do the equivalent of (8) for the Lisp code.

      13. Deal with parsing of menu specifications.


File: internals.info,  Node: Future Work -- Lisp Stream API,  Next: Future Work -- Multiple Values,  Prev: Future Work -- Byte Code Snippets,  Up: Future Work

40.17 Future Work - Lisp Stream API
===================================

Author: Ben Wing (mailto:ben@xemacs.org)

   Expose XEmacs internal lstreams to Lisp as stream objects.  (In
addition to the functions given below, each stream object has
properties that can be associated with it using the standard put, get
etc. API.  For GNU Emacs, where put and get have not been extended to
be general property functions, but work only on strings, we would have
to create functions set-stream-property, stream-property,
remove-stream-property, and stream-properties.  These provide the same
functionality as the generic get, put, remprop, and object-plist
functions under XEmacs)

   (Implement properties using a hash table, and *generalize* this so
that it is extremely easy to add a property interface onto any kind of
object)

     (write-stream STREAM STRING)

   Write the STRING to the STREAM.  This will signal an error if all the
bytes cannot be written.

     (read-stream STREAM &optional N SEQUENCE)

   Reads data from STREAM.  N specifies the number of bytes or
characters, depending on the stream.  SEQUENCE specifies where to write
the data into.  If N is not specified, data is read until end of file.
If SEQUENCE is not specified, the data is returned as a stream.  If
SEQUENCE is specified, the SEQUENCE must be large enough to hold the
data.

     (push-stream-marker STREAM)

   returns ID, probably a stream marker object

     (pop-stream-marker STREAM)

   backs up stream to last marker

     (unread-stream STREAM STRING)

   The only valid STREAM is an input stream in which case the data in
STRING is pushed back and will be read ahead of all other data.  In
general, there is no limit to the amount of data that can be unread or
the number of times that unread-stream can be called before another
read.

     (stream-available-chars STREAM)

   This returns the number of characters (or bytes) that can definitely
be read from the screen without an error.  This can be useful, for
example, when dealing with non-blocking streams when an attempt to read
too much data will result in a blocking error.

     (stream-seekable-p STREAM)

   Returns true if the stream is seekable.  If false, operations such as
seek-stream and stream-position will signal an error.  However, the
functions set-stream-marker and seek-stream-marker will still succeed
for an input stream.

     (stream-position STREAM)

   If STREAM is a seekable stream, returns a position which can be
passed to seek-stream.

     (seek-stream STREAM N)

   If STREAM is a seekable stream, move to the position indicated by N,
otherwise signal an error.

     (set-stream-marker STREAM)

   If STREAM is an input stream, create a marker at the current
position, which can later be moved back to.  The stream does not need
to be a seekable stream.  In this case, all successive data will be
buffered to simulate the effect of a seekable stream.  Therefore use
this function with care.

     (seek-stream-marker STREAM marker)

   Move the stream back to the position that was stored in the marker
object. (this is generally an opaque object of type stream-marker).

     (delete-stream-marker MARKER)

   Destroy the stream marker and if the stream is a non-seekable stream
and there are no other stream markers pointing to an earlier position,
frees up some buffering information.

     (delete-stream STREAM N)

     (delete-stream-marker STREAM ID)

     (close-stream stream)

   Writes any remaining data to the stream and closes it and the object
to which it's attached.  This also happens automatically when the
stream is garbage collected.

     (getchar-stream STREAM)

   Return a single character from the stream. (This may be a single byte
depending on the nature of the stream).  This is actually a macro with
an extremely efficient implementation (as efficient as you can get in
Emacs Lisp), so that this can be used without fear in a loop.  The
implementation works by reading a large amount of data into a vector
and then simply using the function AREF to read characters one by one
from the vector.  Because AREF is one of the primitives handled
specially by the byte interpreter, this will be very efficient.  The
actual implementation may in fact use the function
call-with-condition-handler to avoid the necessity of checking for
overflow.  Its typical implementation is to fetch the vector containing
the characters as a stream property, as well as the index into that
vector.  Then it retrieves the character and increments the value and
stores it back in the stream.  As a first implementation, we check to
see when we are reading the character whether the character would be
out of range.  If so, we read another 4096 characters, storing them
into the same vector, setting the index back to the beginning, and then
proceeding with the rest of the getchar algorithm.

     (putchar-stream STREAM CHAR)

   This is similar to getchar-stream but it writes data instead of
reading data.

     Function make-stream

   There are actually two stream-creation functions, which are:

     (make-input-stream TYPE PROPERTIES)
     (make-output-stream TYPE PROPERTIES)

   These can be used to create a stream that reads data, or writes data,
respectively.  PROPERTIES is a property list and the allowable
properties in it are defined by the type.  Possible types are:

  1. `file' (this reads data from a file or writes to a file)

     Allowable properties are:

    `:file-name'
          (the name of the file)

    `:create'
          (for output streams only, creates the file if it doesn't
          already exist)

    `:exclusive'
          (for output streams only, fails if the file already exists)

    `:append'
          (for output streams only; starts appending to the end of the
          file rather than overwriting the file)

    `:offset'
          (positions in bytes in the file where reading or writing
          should begin.  If unspecified, defaults to the beginning of
          the file or to the end of the file when :appended specified)

    `:count'
          (for input streams only, the number of bytes to read from the
          file before signaling "end of file".  If nil or omitted, the
          number of bytes is unlimited)

    `:non-blocking'
          (if true, reads or writes will fail if the operation would
          block.  This only makes sense for non-regular files).

  2. `process' (For output streams only, send data to a process.)

     Allowable properties are:

    `:process'
          (the process object)

  3. `buffer'  (Read from or write to a buffer.)

     Allowable properties are:

    `:buffer'
          (the name of the buffer or the buffer object.)

    `:start'
          (the position to start reading from or writing to.  If nil,
          use the buffer point.  If true, use the buffer's point and
          move point beyond the end of the data read or written.)

    `:end'
          (only for input streams, the position to stop reading at.  If
          nil, continue to the end of the buffer.)

    `:ignore-accessible'
          (if true, the default for :start and :end ignore any
          narrowing of the buffer.)

  4. `stream' (read from or write to a lisp stream)

     Allowable properties are:

    `:stream'
          (the stream object)

    `:offset'
          (the position to begin to be reading from or writing to)

    `:length'
          (For input streams only, the amount of data to read,
          defaulting to the rest of the data in the string.  Revise
          string for output streams only if true, the stream is resized
          as necessary to accommodate data written off the end,
          otherwise the writes will fail.

  5. `memory' (For output only, writes data to an internal memory
     buffer.  This is more lightweight than using a Lisp buffer.  The
     function memory-stream-string can be used to convert the memory
     into a string.)

  6. `debugging' (For output streams only, write data to the debugging
     output.)

  7. `stream-device' (During non-interactive invocations only, Read
     from or write to the initial stream terminal device.)

  8. `function' (For output streams only, send data by calling a
     function, exactly as with the STREAM argument to the print
     primitive.)

     Allowable Properties are:

    `:function'
          (the function to call.  The function is called with one
          argument, the stream.)

  9. `marker' (Write data to the location pointed to by a marker and
     move the marker past the data.)

     Allowable properties are:

    `:marker'
          (the marker object.)

 10. `decoding' (As an input stream, reads data from another stream and
     decodes it according to a coding system.  As an output stream
     decodes the data written to it according to a coding system and
     then writes results in another stream.)

     Properties are:

    `:coding-system'
          (the symbol of coding system object, which defines the
          decoding.)

    `:stream'
          (the stream on the other end.)

 11. `encoding' (As an input stream, reads data from another stream and
     encodes it according to a coding system.  As an output stream
     encodes the data written to it according to a coding system and
     then writes results in another stream.)

     Properties are:

    `:coding-system'
          (the symbol of coding system object, which defines the
          encoding.)

    `:stream'
          (the stream on the other end.)

   Consider

     (define-stream-type 'type
       :read-function
       :write-function
       :rewind-
       :seek-
       :tell-
       (?:buffer)

   Old Notes:

   Expose lstreams as hash (put get etc. properties) table.

       (write-stream stream string)
       (read-stream stream &optional n sequence)
       (make-stream ...)
       (push-stream-marker stream)
          returns ID prob a stream marker object
       (pop-stream-marker stream)
          backs up stream to last marker
       (unread-stream stream string)
       (stream-available-chars stream)
       (seek-stream stream n)
       (delete-stream stream n)
       (delete-stream-marker stream ic) can always be poe only nested if you
         have set stream marker

       (get-char-stream *generalizes* stream)

       a macro that tries to be efficient perhaps by reading the next
       e.g. 512 characters into a vector and arefing them.  Might check aref
       optimization for vectors in the byte interpreter.

       (make-stream 'process :process ... :type write)

       Consider

       (define-stream-type 'type
         :read-function
         :write-function
         :rewind-
         :seek-
         :tell-
         (?:buffer)


File: internals.info,  Node: Future Work -- Multiple Values,  Next: Future Work -- Macros,  Prev: Future Work -- Lisp Stream API,  Up: Future Work

40.18 Future Work - Multiple Values
===================================

Author: Ben Wing (mailto:ben@xemacs.org)

   On low level, all funs that can return multiple values are defined
with DEFUN_MULTIPLE_VALUES and have an extra parameter, a struct
mv_context *.

   It has to be this way to ensure that only the fun itself, and no
called funs, think they're called in an mv context.

   apply, funcall, eval might propagate their mv context to their
children?

   Might need eval-mv to implement calling a fun in an mv context.
Maybe also funcall_mv? apply_mv?

   Generally, just set up context appropriately.  Call fun (noticing
whether it's an mv-aware fun) and binding values on the way back or
passing them out.  (e.g. to multiple-value-bind)

Common Lisp multiple values, required for specifier improvements.
-----------------------------------------------------------------

The multiple return values from get-specifier should allow the
specifier value to be modified in the correct fashion (i.e.  should
interact correctly with all manner of changes from other callers) using
set-specifier.  We should check this and see if we need other return
values.  (how-to-add? inst-list?)

   In C, call multiple-values-context to get number of expected values,
and multiple-value-set (#, value) to get values other than the first.

   (Returns Qno_value, or something, if there are no values.

   #### Or should throw?  Probably not.  #### What happens if a fn
returns no values but the caller expects a #### value?

   Something like `funcall_with_multiple_values()' for setting up the
context.

   For efficiency, byte code could notice Ffuncall to m.v. functions and
sub in special opcodes during load in processing, if it mattered.


File: internals.info,  Node: Future Work -- Macros,  Next: Future Work -- Specifiers,  Prev: Future Work -- Multiple Values,  Up: Future Work

40.19 Future Work - Macros
==========================

Author: Ben Wing (mailto:ben@xemacs.org)

  1. Option to control whether beep really kills a macro execution.

  2. Recently defined macros are remembered on a stack, so accidentally
     defining another one doesn't fuck you up.  You can "rotate"
     anonymous macros or just pick one (numbered) to put on tags, so it
     works with execute macro - menu shows the anonymous macro, and
     lists some keystrokes.  Normally numbered but you can easily assign
     to named fun or to keyboard sequence or give it a number (or give
     it a letter accelerator?)


File: internals.info,  Node: Future Work -- Specifiers,  Next: Future Work -- Display Tables,  Prev: Future Work -- Macros,  Up: Future Work

40.20 Future Work - Specifiers
==============================

Author: Ben Wing (mailto:ben@xemacs.org)

Ideas To Work On When Their Time Has Come
-----------------------------------------

   * specifier-instance returns additional params (multiple-value) -
     the instantiator used, the associated tag set, the locale found
     in, a code that can be passed in as an additional param RESTART to
     restart an instantiation process, e.g. to allow an instantiator to
     "inherit" from another one higher up.  Also, domain can be 'global
     (look only in global specs) or "complex" - a list of the actual
     locales to look in (e.g. a buffer - frame - a device - 'global)

   * pragmatic-specifier-domain (locale) Converts a locale into a
     domain in a way that's "pragmatic" - does what most users expect
     will happen, but is not clean.  In particular, handling of
     "buffer" requires trickiness, as mentioned before.

   * ensure-instantiator-exists (specifier locale) Ensures an actual
     instantiator exists in a locale, so that it can later be futzed
     with.  If none exists, one is constructed by first calling
     pragmatic-specifier domain and then specifier-instance and
     fetching out the instantiator for this call.

   * map-modifying-instantiators (specifier fun &optional locale
     tag-set) Same args as map-specifier, but use the return value from
     the fun to replace the instantiator.  Called with three args
     (instantiator locale tag-set)

   * map-modifying-instantiators-force (specifier fun &optional locale
     tag-set) Same as previous, but calls ensure-instantiator-exists on
     each locale before processing.

   NOTE:  Can do preliminary implementation without Multiple Values -
instead create fun specifier-instance - that returns a list (and will
be deleted at some point)

specifier &c changes for glyphs
-------------------------------

  1.        * resizable vectors with funs to insert, delete elements
          (elements shift accordingly)

        * gap array vectors as an implementation of resizing vectors.

  2. You can `put' `get', etc. on vectors to modify properties within
     them.

  3. copy-over routines routines that carefully copy one complex item
     OVER another one, destroying the second in the process.  I wrote
     one for lists.  Need a general copy-over-tree.

  4. improvement to specifier mapping routines e.g.

     map-modifying-instantiator and its force versions below, so that we
     could implement in turns.

  5. put-specifier-property (specifier which finds the key, value
     instantiator in the locale, &opt locale possibly creating one
     tag-set) if necessary and goes into the vector, changes it, and
     puts it back into the specifier.

  6. Smarter add-spec-to-specifier

     If it notices that it's just replacing one instantiator with
     another, instead of just copy-tree the first one and throw away the
     other, use copy-over-tree to save lots of garbage when repeatedly
     called.

     ILLEGIBLE: GOTO LOO BUI BUGS LAST PNOTE

  7. When at image instantiate:
        * Some properties in the instantiators could be implemented
          through dynamically modifying an existing image instance
          (e.g. when the value of a slider or progress bar or text in a
          text field changes).  So when we hash, we only hash the part
          of the instantiator that cannot be dynamically modified (We
          might need to do something tricky here - allowing a :key
          property in hash tables or *ILLEGIBLE*).  Anyway, so we need
          to generate an image instance, and we mask off the dynamic
          properties and look up in our hash table, and we get
          something back!  But is it ours to modify?  (We already
          checked to see it wasn't exactly the same dynamic properties
          that it had)  Thus --

  8. Reference counting.  Somehow or other, each image instance in the
     cache needs to keep track of the instantiators that generated it.

   It might do this through some sort of special instantiator-reference
object.  This points to the instantiator, where in the hierarchy the
instantiator is etc.  When an instantiator gets removed, this
gu*ILLEGIBLE* values report not attached.  Somehow that gets
communicated back to the image instance in the cache.  So somehow or
other, the image instance in the cache knows who's using them and so
when you go and keep updating the slider value, by simply modifying an
instantiator, which efficiently changes the internal structure of this
specifier - eventually image instantiate notices that the image
instance it points has no other user and just modifiers it,  but in
complex situations, some optimizations get lost, but everything is
still correct.

   vs.

   Andy's set-image-instance-property, which achieves the same
optimizations much more easily, but

  1. falls apart in any more complicated system

  2. only works because of the way the caching system in XEmacs works.
     Any change (e.g. *ILLEGIBLE* more of making the caches GQ instead
     of GQ) is likely to make things stop working right in all but the
     simplest situation.

Specifier improvements for support of specifier inheritance (necessary for the new font mapping API)
----------------------------------------------------------------------------------------------------

'Fallback should be a locale/domain.

     (get-specifier specifier &optional locale)

     #### If locale is omitted, should it be (current-buffer) or 'global?
     #### Should argument not be optional?

   If a buffer is specified: find a window showing buffer by looking

   * at selected window

   * at other windows on selected frame

   * at selected windows on other frames in selected device

   * at other windows on ""

   * at selected windows on selected frames on other devices in selected
     console.

   * other windows sel from other devices sel con

   * ""       oth       ""           sel

   * sel win sel from sel dev oth con

   * oth win sel from sel dev oth con

   * sel win oth from sel dev oth con

   * oth win oth from sel dev oth con

   * sel win sel from oth dev oth con

   * oth win sel from oth dev oth con

   * oth win oth from oth dev oth con

   If none, use  buffer -> sel from -> etc.

     Returns multiple values
       second is instantiator
       third  is locale containing inst.
       fourth is tag set

     (restart-specifier-instance ...)

   like specifier-instance, but allows restarting the lookup, for
implementing inheritance, etc.  Obsoletes
specifier-matching-find-charset, or whatever it is.  The restart
argument is opaque, and is returned as a multiple value of
restart-specifier-instance.  (It's actually an integer with the low
bits holding the locale and the other bits count int to the list)
attached to the locale.)


File: internals.info,  Node: Future Work -- Display Tables,  Next: Future Work -- Making Elisp Function Calls Faster,  Prev: Future Work -- Specifiers,  Up: Future Work

40.21 Future Work - Display Tables
==================================

Author: Ben Wing (mailto:ben@xemacs.org)

   #### It would also be really nice if you could specify that the
characters come out in hex instead of in octal.  Mule does that by
adding a `ctl-hexa' variable similar to `ctl-arrow', but that's bogus -
we need a more general solution.  I think you need to extend the
concept of display tables into a more general conversion mechanism.
Ideally you could specify a Lisp function that converts characters, but
this violates the Second Golden Rule and besides would make things way
way way way slow.

   So instead, we extend the display-table concept, which was
historically limited to 256-byte vectors, to one of the following:

  1. A 256-entry vector, for backward compatibility;

  2. char-table, mapping characters to values;

  3. range-table, mapping ranges of characters to values;

  4. a list of the above.

   The fourth option allows you to specify multiple display tables
instead of just one.  Each display table can specify conversions for
some characters and leave others unchanged.  The way the character gets
displayed is determined by the first display table with a binding for
that character.  This way, you could call a function
`enable-hex-display' that adds a hex display-table to the list of
display tables for the current buffer.

   #### ...not yet implemented...  Also, we extend the concept of
"mapping" to include a printf-like spec.  Thus you can make all extended
characters show up as hex with a display table like this:

         #s(range-table data ((256 524288) (format "%x")))

   Since more than one display table is possible, you have great
flexibility in mapping ranges of characters.


File: internals.info,  Node: Future Work -- Making Elisp Function Calls Faster,  Next: Future Work -- Lisp Engine Replacement,  Prev: Future Work -- Display Tables,  Up: Future Work

40.22 Future Work - Making Elisp Function Calls Faster
======================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract: *This page describes many optimizations that can be made
to the existing Elisp function call mechanism without too much effort.
The most important optimizations can probably be implemented with only
a day or two of work.  I think it's important to do this work
regardless of whether we eventually decide to replace the Lisp engine.

   Many complaints have been made about the speed of Elisp, and in
particular about the slowness in executing function calls, and rightly
so.  If you look at the implementation of the `funcall' function,
you'll notice that it does an incredible amount of work.  Now logically,
it doesn't need to be so.  Let's look first from the theoretical
standpoint at what absolutely needs to be done to call a Lisp function.

   First, let's look at the situation that would exist if we were smart
enough to have made lexical scoping be the default language policy.  We
know at compile time exactly which code can reference the variables that
are the formal parameters for the function being called (specifically,
only the code that is part of that function's definition) and where
these references are.  As a result, we can simply push all the values of
the variables onto a stack, and convert all the variable references in
the function definition into stack references.  Therefore, binding
lexically-scoped parameters in preparation for a function call involves
nothing more than pushing the values of the parameters onto a stack and
then setting a new value for the frame pointer, at the same time
remembering the old one.  Because the byte-code interpreter has a
stack-based architecture, however, the parameter values have already
been pushed onto the stack at the time of the function call invocation.
Therefore, binding the variables involves doing nothing at all, other
than dealing with the frame pointer.

   With dynamic scoping, the situation is somewhat more complicated.
Because the parameters can be referenced anywhere, and these references
cannot be located at compile time, their values have to be stored into a
global table that maps the name of the parameter to its current value.
In Elisp, this table is called the "obarray".  Variable binding in
Elisp is done using the C function `specbind()'. (This stands for
"special variable binding" where "special" is the standard Lisp
terminology for a dynamically-scoped variable.)  What `specbind()'
does, essentially, is retrieve the old value of the variable out of the
obarray, remember the value by pushing it, along with the name of the
variable, onto what's called the "specpdl" stack, and then store the
new value into the obarray.  The term "specpdl" means "Special Variable
Pushdown List", where "Pushdown List" is an archaic computer science
term for a stack that used to be popular at MIT.  These binding
operations, however, should still not take very much time because of the
use of symbols, i.e. because the location in the obarray where the
variable's value is stored has already been determined (specifically, it
was determined at the time that the byte code was loaded and the symbol
created), so no expensive hash table lookups need to be performed.

   An actual function invocation in Elisp does a great deal more work,
however, than was just outlined above.  Let's just take a look at what
happens when one byte-compiled function invokes another byte-compiled
function, checking for places where unnecessary work is being done and
determining how to optimize these places.

  1.  The byte-compiled function's parameter list is stored in exactly
     the format that the programmer entered it in, which is to say as a
     Lisp list, complete with `&amp;optional' and `&amp;rest' keywords.
     This list has to be parsed for _every_ function invocation, which
     means that for every element in a list, the element is checked to
     see whether it's the `&amp;optional' or `&amp;rest' keywords, its
     surrounding cons cell is checked to make sure that it is indeed a
     cons cell, the `QUIT' macro is called, etc.  What should be
     happening here is that the argument list is parsed exactly once,
     at the time that the byte code is loaded, and converted into a C
     array.  The C array should be stored as part of the byte-code
     object.  The C array should also contain, in addition to the
     symbols themselves, the number of required and optional arguments.
     At function call time, the C array can be very quickly retrieved
     and processed.

  2.  For every variable that is to be bound, the `specbind()' function
     is called.  This actually does quite a lot of things, including:

       1.  Checking the symbol argument to the function to make sure
          it's actually a symbol.

       2.  Checking for specpdl stack overflow, and increasing its size
          as necessary.

       3.  Calling `symbol_value_buffer_local_info()' to retrieve
          buffer local information for the symbol, and then processing
          the return value from this function in a series of if
          statements.

       4.  Actually storing the old value onto the specpdl stack.

       5.  Calling `Fset()' to change the variable's value.



   The entire series of calls to `specbind()' should be inline and
merged into the argument processing code as a single tight loop, with no
function calls in the vast majority of cases.  The `specbind()' logic
should be streamlined as follows:

  1.  The symbol argument type checking is unnecessary.

  2.  The check for the specpdl stack overflow needs to be done only
     once, not once per argument.

  3.  All of the remaining logic should be boiled down as follows:

       1.  Retrieve the old value from the symbol's value cell.

       2.  If this value is a symbol-value-magic object, then call the
          real `specbind()' to do the work.

       3.  Otherwise, we know that nothing complicated needs to be
          done, so we simply push the symbol and its value onto the
          specpdl stack, and then replace the value in the symbol's
          value cell.

       4.  The only logic that we are omitting is the code in `Fset()'
          that checks to make sure a constant isn't being set.  These
          checks should be made at the time that the byte code for the
          function is loaded and the C array of parameters to the
          function is created.  (Whether a symbol is constant or not is
          generally known at XEmacs compile time.  The only issue here
          is with symbols whose names begin with a colon.  These
          symbols should simply be disallowed completely as parameter
          names.)



   Other optimizations that could be done are:

   *  At the beginning of the function that implements the byte-code
     interpreter (this is the Lisp primitive `byte-code'), the string
     containing the actual byte code is converted into an array of
     integers.  I added this code specifically for MULE so that the
     byte-code engine didn't have to deal with the complexities of the
     internal string format for text.  This conversion, however, is
     generally useful because on modern processors accessing 32-bit
     values out of an array is significantly faster than accessing
     unaligned 8-bit values.  This conversion takes time, though, and
     should be done once at load time rather than each time the byte
     code is executed.  This array should be stored in the byte-code
     object.  Currently, this is a bit tricky to do, because
     `byte-code' is not actually passed the byte-code object, but
     rather three of its elements.  We can't just change `byte-code' so
     that it is directly passed the byte-code object because this
     function, with its existing argument calling pattern, is called
     directly from compiled Elisp files.  What we can and should do,
     however, is create a subfunction that does take a byte-code object
     and actually implements the byte-code interpreter engine.
     Whenever the C code wants to execute byte code, it calls this
     subfunction.  `byte-code' itself also calls this subfunction after
     conjuring up an appropriate byte-code object and storing its
     arguments into this object.  With a small amount of work, it's
     possible to do this conjuring in such a way that it doesn't
     generate any garbage.

   *  At the end of a function call, the parameter bindings that have
     been done need to be undone.  This is standardly done by calling
     `unbind_to()'.  Just as for a `specbind()', this function does a
     lot of work that is unnecessary in the vast majority of cases, and
     it could also be inlined and streamlined.

   *  As part of each Elisp function call, a whole bunch of checks are
     done for a series of unlikely but possible conditions that may
     occur.  These include, for example,

        *  Calling the `QUIT' macro, which essentially involves
          checking a global volatile variable to see whether additional
          processing needs to be done.

        *  Checking whether a garbage collection needs to be done.

        *  Checking the variable `debug_on_next_call'.

        *  Checking for whether Elisp profiling is active.  (An
          additional optimization that's perhaps not worth the effort
          is to do some post-processing on the array of integers after
          it has been converted.  For example, whenever a 16-bit value
          occurs in the byte code, it has to be encoded as two separate
          8-bit values.  These values could be combined.  The tricky
          part here is that all of the places where a goto occurs
          across the place where this modification is made would have to
          have their offsets changed.  Other such optimizations can
          easily be imagined as well.)


   *  With a little bit smarter code, it should be possible to make a
     single trip variable that indicates whether any of these
     conditions is true.  This variable would be updated by any code
     that changes the actual variables whose values are checked in the
     various checks just mentioned.  (By the way, all of this is
     occurring in the C function `funcall_recording_as()'.)  There is a
     little bit of code between each of the checks.  This code would
     simply have to be duplicated between the two cases where this
     general trip variable is true and is false.  (Note: the
     optimization detailed in this item is probably not worth doing on
     the first pass.)



File: internals.info,  Node: Future Work -- Lisp Engine Replacement,  Next: Future Work -- Better Rendering Support,  Prev: Future Work -- Making Elisp Function Calls Faster,  Up: Future Work

40.23 Future Work - Lisp Engine Replacement
===========================================

* Menu:

* Future Work -- Lisp Engine Discussion::
* Future Work -- Lisp Engine Replacement -- Implementation::
* Future Work -- Startup File Modification by Packages::


File: internals.info,  Node: Future Work -- Lisp Engine Discussion,  Next: Future Work -- Lisp Engine Replacement -- Implementation,  Prev: Future Work -- Lisp Engine Replacement,  Up: Future Work -- Lisp Engine Replacement

40.23.1 Future Work - Lisp Engine Discussion
--------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract: *Recently there has been a great deal of talk on the
XEmacs mailing lists about potential changes to the XEmacs Lisp engine.
Usually the discussion has centered around the question which is better,
Common Lisp or Scheme?  This is certainly an interesting debate topic,
but it didn't seem to have much practical relevance to me, so I vowed to
stay out of the discussion.  Recently, however, it seems that people are
losing sight of the broader picture.  For example, nobody seems to be
asking the question, ""Would an extension language other than Lisp or
Scheme (perhaps not a Lisp variant at all) be more appropriate?"" Nor
does anybody seem to be addressing what I consider to be the most
fundamental question, is changing the extension language a good thing to
do?

   I think it would be a mistake at this point in XEmacs development to
begin any project involving fundamental changes to the Lisp engine or to
the XEmacs Lisp language itself.  It would take a huge amount of effort
to complete even part of this project, and would be a major drain on the
already-insufficient resources of the XEmacs development community.
Most of the gains that are purported to stem from a project such as this
could be obtained with far less effort by making more incremental
changes to the XEmacs core.  I think it would be an even bigger mistake
to change the actual XEmacs extension language (as opposed to just
changing the Lisp engine, making few, if any, externally visible
changes).  The only language change that I could possibly imagine
justifying would involve switching to some ubiquitous web language, such
as Java and JavaScript, or Perl.  (Even among those, I think Java would
be the only possibility that really makes sense).

   In the rest of this document I'll present the broader issues that
would be involved in changing the Lisp engine or extension language.
This should make clear why I've come to believe as I do.

Is everyone clear on the difference between interface and implementation?
-------------------------------------------------------------------------

There seems to be a great deal of confusion concerning the difference
between interface and implementation.  In the context of XEmacs,
changing the interface means switching to a different extension language
such as Common Lisp, Scheme, Java, etc.  Changing the implementation
means using a different Lisp engine.  There is obviously some relation
between these two issues, but there is no particular requirement that
one be changed if the other is changed.  It is quite possible, for
example, to imagine taking the underlying engine for any of the various
Lisp dialects in existence, and adapting it so that it implements the
same Elisp extension language that currently exists.  The vast majority
of the purported benefits that we would get from changing the extension
language could just as easily be obtained while making minimal changes
to the external Elisp interface.  This way nearly all existing Elisp
programs would continue to work, there would be no need to translate
Elisp programs into some other language or to simultaneously support two
incompatible Lisp variants, and there would be no need for users or
package authors to learn a new extension language that would be just as
unfamiliar to the vast majority of them as Elisp is.

Why should we change the Lisp engine?
-------------------------------------

Let's go over the possible reasons for changing the Lisp engine.

Speed.
......

Changing the Lisp engine might make XEmacs faster.  However, consider
the following.

  1.  XEmacs will get faster over time without any development effort
     at all because computers will get faster.

  2.  Perhaps the biggest causes of the slowness of XEmacs are not
     related to the Lisp engine at all.  It has been asserted, for
     example, that the slowness of XEmacs is primarily due to the
     redisplay mechanism, to the handling of insertion and deletion of
     text in a buffer, to the event loop, etc.  Nobody has done any
     real studies to determine what the actual cause of slowness is.

  3.  Emacs 18 seems plenty fast enough to most people.  However, Emacs
     18 also had a worse Lisp engine and a worse byte compiler than
     XEmacs.

  4.  Significant speed increases in the execution of Lisp code could be
     achieved without too much effort by working on the existing byte
     code interpreter and function call mechanism a bit.


Memory usage.
.............

A new Lisp engine with a better garbage collection mechanism might make
more efficient use of memory; for example, through the use of a
relocating garbage collector.  However, consider this:

  1.  A new Lisp engine would probably have a larger memory footprint,
     perhaps a significantly larger one.

  2.  The worst memory problems might not be due to Lisp object
     inefficiency at all.  The problems could simply be due mainly to
     the inefficient buffer representation.  Nobody has come up with
     any concrete numbers on where the real problem lies.


Robustness.
...........

A new Lisp engine might well be more robust.  (On the other hand, it
might not be.  It is not always easy to tell).  However, I think that
the biggest problems with robustness are in the part of the C code that
is not concerned with implementing the Lisp engine.  The redisplay
mechanism and the unexec mechanism are probably the biggest sources of
robustness problems.  I think the biggest robustness problems that are
related to the Lisp engine concern the use of GCPRO declarations.  The
entire GCPRO mechanism is ill-conceived and unsafe.  The only real way
to make this safe would be to do conservative garbage collection over
the C stack and to eliminate the GCPRO declarations entirely.  But how
many of the Lisp engines that are being considered have such a mechanism
built into them?

Maintainability.
................

A new Lisp engine might well improve the maintainability of XEmacs by
offloading the maintenance of the Lisp engine.  However, we need to make
very sure that this is, in fact, the case before embarking on a project
like this.  We would almost certainly have to make significant
modifications to any Lisp engine that we choose to integrate, and
without the active and committed support and cooperation of the
developers of that Lisp engine, the maintainability problem would
actually get worse.

Features.
.........

A new Lisp engine might have built in support for various features that
we would like to add to the XEmacs extension language, such as lexical
scoping and an object system.

Why would we want to change the extension language?
---------------------------------------------------

Possible reasons for changing the extension language include:

More standard.
..............

Switching to a language that is more standard and more commonly in use
would be beneficial for various reasons.  First of all, the language
that is more commonly used and more familiar would make it easier for
users to write their own extensions and in general, increase the
acceptance of XEmacs.  Also, an accepted standard probably has had a lot
more thought put into it than any language interface created by the
XEmacs developers themselves.  Furthermore, if our extension language is
being actively developed and supported, much of the work that we would
otherwise have to do ourselves is transferred elsewhere.

   However, both Scheme and Common Lisp flunk the familiarity test.
Neither language is being actively used for program development outside
of small research communities, and few prospective authors of XEmacs
extensions will be familiar with any Lisp variant for real world uses.
(I consider the argument that Scheme is often used in introductory
programming courses to be irrelevant.  Many existing programmers were
taught Pascal in their introductory programming courses.  How many of
them would actually be comfortable writing a program in Pascal?)
Furthermore, someone who wants to learn Lisp can't exactly go to their
neighborhood bookstore and pick up a book on this topic.

Ease of use.
............

There are endless arguments about which language is easiest to use.  In
practice, this largely boils down to which languages are most familiar.

Object oriented.
................

The object-oriented paradigm is the dominant one in use today for new
languages.  User interface concepts in particular are expressed very
naturally in an object-oriented system.  However, neither Scheme nor
Common Lisp has been designed with object orientation in mind.  There is
a standard object system for Common Lisp, but it is extremely complex
and difficult to understand.


File: internals.info,  Node: Future Work -- Lisp Engine Replacement -- Implementation,  Next: Future Work -- Startup File Modification by Packages,  Prev: Future Work -- Lisp Engine Discussion,  Up: Future Work -- Lisp Engine Replacement

40.23.2 Future Work - Lisp Engine Replacement - Implementation
--------------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   Let's take a look at the sort of work that would be required if we
were to replace the existing Elisp engine in XEmacs with some other
engine, for example, the Clisp engine.  I'm assuming here, of course,
that we are not going to be changing the interface here at the same
time, which is to say that we will be keeping the same Elisp language
that we currently have as the extension language for XEmacs, except
perhaps for incremental changes that we will make, such as lexical
scoping and proper structure support in an attempt to gradually move
the language towards an upwardly-compatible goal, such as Common Lisp.
I am writing this page primarily as food for thought.  I feel fairly
strongly that actually doing this work would be a big waste of effort
that would inevitably become a huge time sink on the part of nearly
everyone involved in XEmacs development, and not only for the ones who
were supposed to be actually doing the engine change.  I feel that most
of the desired changes that we want for the language and/or the engine
can be achieved with much less effort and time through incremental
changes to the existing code base.

   First of all, in order to make a successful Lisp engine change in
XEmacs, it is vitally important that the work be done through a series
of incremental stages where at the end of each stage XEmacs can be
compiled and run, and it works.  It is tempting to try to make the
change all at once, but this would be disastrous.  If the resulting
product worked at all, it would inevitably contain a huge number of
subtle and extremely difficult to track down bugs, and it would be next
to impossible to determine which of the myriad changes made introduced
the bug.

   Now let's look at what the possible stages of implementation could
be.

An Extra C Preprocessing Stage
..............................

The first step would be to introduce another preprocessing stage for the
XEmacs C code, which is done before the C compiler itself is invoked on
the code, and before the standard C preprocessor runs.  The C
preprocessor is simply not powerful enough to do many of the things we
would like to do in the C code.  The existing results of this have been
a combination of a lot of hacked up and tricky-to-maintain stuff (such
as the `DEFUN' macro, and the associated `DEFSUBR'), as well as code
constructs that are difficult to write.  (Consider for example,
attempting to do structured exception handling, such as catch/throw and
unwind-protect constructs), as well as code that is potentially or
actually unsafe (such as the uses of `alloca'), which could easily
cause stack overflow with large amounts of memory allocated in this
fashion.)  The problem is that the C preprocessor does not allow macros
to have the power of an actual language, such as C or Lisp.  What our
own preprocessor should do is allow us to define macros, whose
definitions are simply functions written in some language which are
executed at compile time, and whose arguments are the actual argument
for the macro call, as well as an environment which should have a data
structure representation of the C code in the file and allow this
environment to be queried and modified.  It can be debated what the
language should be that these extensions are written in.  Whatever the
language chosen, it needs to be a very standard language and a language
whose compiler or interpreter is available on all of the platforms that
we could ever possibly consider putting XEmacs to, which is basically to
say all the platforms in existence.  One obvious choice is C, because
there will obviously be a C compiler available, because it is needed to
compile XEmacs itself.  Another possibility is Perl, which is already
installed on most systems, and is universally available on all others.
This language has powerful text processing facilities which would
probably make it possible to implement the macro definitions more
quickly and easily; however, this might also encourage bad coding
practices in the macros (often simple text processing is not
appropriate, and more sophisticated parsing or recursive data structure
processing needs to be done instead), and we'd have to make sure that
the nested data structure that comprises the environment could be
represented well in Perl.  Elisp would not be a good choice because it
would create a bootstrapping problem.  Other possible languages, such as
Python, are not appropriate, because most programmers are unfamiliar
with this language (creating a maintainability problem) and the Python
interpreter would have to be included and compiled as part of the XEmacs
compilation process (another maintainability problem).  Java is still
too much in flux to be considered at this point.

   The macro facility that we will provide needs to add two features to
the language: the ability to define a macro, and the ability to call a
macro.  One good way of doing this would be to make use of special
characters that have no meaning in the C language (or in C++ for that
matter), and thus can never appear in a C file outside of comments and
strings.  Two obvious characters are the @ sign and the $ sign.  We
could, for example, use `@' defined to define new macros, and the `$'
sign followed by the macro name to call a macro.  (Proponents of Perl
will note that both of these characters have a meaning in Perl.  This
should not be a problem, however, because the way that macros are
defined and called inside of another macro should not be through the use
of any special characters which would in effect be extending the macro
language, but through function calls made in the normal way for the
language.)

   The program that actually implements this extra preprocessing stage
needs to know a certain amount about how to parse C code.  In
particular, it needs to know how to recognize comments, strings,
character constants, and perhaps certain other kinds of C tokens, and
needs to be able to parse C code down to the statement level.  (This is
to say it needs to be able to parse function definitions and to separate
out the statements, `if' blocks, `while' blocks, etc. within these
definitions.  It probably doesn't, however need to parse the contents
of a C expression.)  The preprocessing program should work first by
parsing the entire file into a data structure (which may just contain
expressions in the form of literal strings rather than a data structure
representing the parsed expression).  This data structure should become
the environment parameter that is passed as an argument to macros as
mentioned above.  The implementation of the parsing could and probably
should be done using `lex' and `yacc'.  One good idea is simply to
steal some of the `lex' and `yacc' code that is part of GCC.

   Here are some possibilities that could be implemented as part of the
preprocessing:

  1.  A proper way of doing the `DEFUN' macros.  These could, for
     example, take an argument list in the form of a Lisp argument list
     (complete with keyword parameters and other complex features) and
     automatically generate the appropriate `subr' structure, the
     appropriate C function definition header, and the appropriate call
     to the `DEFSUBR' initialization function.

  2.  A truly safe and easy to use implementation of the `alloca'
     function.  This could allocate the memory in any fashion it chooses
     (calling `malloc' using a large global array, or a series of such
     arrays, etc.) an `insert' in the appropriate places to
     automatically free up this memory.  (Appropriate places here would
     be at the end of the function and before any return statements.
     Non-local exits can be handled in the function that actually
     implements the non-local exit.)

  3.  If we allow for the possibility of having an arbitrary Lisp
     engine, we can't necessarily assume that we can call Lisp
     primitives implemented in C from other C functions by simply
     making a function all.  Perhaps something special needs to happen
     when this is done.  This could be handled fairly easily by having
     our new and improved `DEFUN' macro define a new macro for use when
     calling a primitive.

Make the Existing Lisp Engine be Self-contained.
................................................

The goal of this stage is to gradually build up a self-contained Lisp
engine out of the existing XEmacs core, which has no dependencies on any
of the code elsewhere in the XEmacs core, and has a well-defined and
black box-style interface.  (This is to say that the rest of the C code
should not be able to access the implementation of the Lisp engine, and
should make as few assumptions as possible about how this implementation
works).  The Lisp engine could, and probably should, be built up as a
separate library which can be compiled on its own without any of the
rest of the XEmacs C code, and can be tested in this configuration as
well.

   The creation of this engine library should be done as a series of
subsets, each of which moves more code out of the XEmacs core and into
the engine library, and XEmacs should be compilable and runnable between
each sub-step.  One possible series of sub-steps would be to first
create an engine that does only object allocation and garbage
collection, then as a second sub-step, move in the code that handles
symbols, symbol values, and simple binding, and then finally move in the
code that handles control structures, function calling, `byte-code'
execution, exception handling, etc.  (It might well be possible to
further separate this last sub-step).

Removal of Assumptions About the Lisp Engine Implementation
...........................................................

Currently, the XEmacs C code makes all sorts of assumptions about the
implementation of the Lisp engine, particularly in the areas of object
allocation, object representation, and garbage collection.  A different
Lisp engine may well have different ways of doing these implementations,
and thus the XEmacs C code must be rid of any assumptions about these
implementations.  This is a tough and tedious job, but it needs to be
done.  Here are some examples:

  1.  `GCPRO' must go.  The `GCPRO' mechanism is tedious, error-prone,
     unmaintainable, and fundamentally unsafe.  As anyone who has
     worked on the C Core of XEmacs knows, figuring out where to insert
     the `GCPRO' calls is an exercise in black magic, and debugging
     crashes as a result of incorrect `GCPROing' is an absolute
     nightmare.  Furthermore, the entire mechanism is fundamentally
     unsafe.  Even if we were to use the extra preprocessing stage
     detailed above to automatically generate `GCPRO' and `UNGCPRO'
     calls for all Lisp object variables occurring anywhere in the C
     code, there are still places where we could be bitten.  Consider,
     for example, code which calls `cons' and where the two arguments
     to this functions are both calls to the `append' function.  Now
     the `append' function generates new Lisp objects, and it also
     calls `QUIT', which could potentially execute arbitrary Lisp code
     and cause a garbage collection before returning control to the
     `append' function.  Now in order to generate the arguments to the
     `cons' function, the `append' function is called twice in a row.
     When the first `append' call returns, new Lisp data has been
     created, but has no `GCPRO' pointers to it.  If the second
     `append' call causes a garbage collection, the Lisp data from the
     first `append' call will be collected and recycled, which is
     likely to lead to obscure and impossible-to-debug crashes.  The
     only way around this would be to rewrite all function calls whose
     parameters are Lisp objects in terms of temporary variables, so
     that no such function calls ever contain other function calls as
     arguments.  This would not only be annoying to implement, even in
     a smart preprocessor, but would make the C code become incredibly
     slow because of all the constant updating of the `GCPRO' lists.

  2.  The only proper solution here is to completely do away with the
     `GCPRO' mechanism and simply do conservative garbage collection
     over the C stack.  There are already portable implementations of
     conservative pointer marking over the C stack, and these could
     easily be adapted for use in the Elisp garbage collector.  If, as
     outlined above, we use an extra preprocessing stage to create a
     new version of `alloca' that allocates its memory elsewhere than
     actually on the C stack, and we ensure that we don't declare any
     large arrays as local variables, but instead use `alloca', then we
     can be guaranteed that the C stack is small and thus that the
     conservative pointer marking stage will be fast and not very
     likely to find false matches.

  3.  Removing the `GCPRO' declarations as just outlined would also
     remove the assumption currently made that garbage collection can
     occur only in certain places in the C code, rather than in any
     arbitrary spot.  (For example, any time an allocation of Lisp data
     happens).  In order to make things really safe, however, we also
     have to remove another assumption as detailed in the following
     item.

  4.  Lisp objects might be relocatable.  Currently, the C code assumes
     that Lisp objects other than string data are not relocatable and
     therefore it's safe to pass around and hold onto the actual
     pointers for the C structures that implement the Lisp objects.
     Current code, for example, assumes that a `Lisp_Object' of type
     buffer and a C pointer to a `struct buffer' mean basically the
     same thing, and indiscriminately passes the two kinds of buffer
     pointers around.  With relocatable Lisp objects, the pointers to
     the C structures might change at any time.  (Remember, we are now
     assuming that a garbage collection can happen at basically any
     point).  All of the C code needs to be changed so that Lisp
     objects are always passed around using a Lisp object type, and the
     underlying pointers are only retrieved at the time when a
     particular data element out of the structure is needed.  (As an
     aside, here's another reason why Lisp objects, instead of
     pointers, should always be passed around.  If pointers are passed
     around, it's conceivable that at the time a garbage collection
     occurs, the only reference to a Lisp object (for example, a
     deleted buffer) would be in the form of a C pointer rather than a
     Lisp object.  In such a case, the conservative pointer marking
     mechanism might not notice the reference, especially if, in an
     attempt to eliminate false matches and make the code generally
     more efficient, it will be written so that it will look for actual
     Lisp object references.)

  5.  I would go a step farther and completely eliminate the macros that
     convert a Lisp object reference into a C pointer.  This way the
     only way to access an element out of a Lisp object would be to use
     the macro for that element, which in one atomic operation
     de-references the Lisp object reference and retrieves the value
     contained in the element.  We probably do need the ability to
     retrieve actual C pointers, though.  For example, in the case
     where an array is stored in a Lisp object, or simply for
     efficiency purposes where we might want some code to retrieve the
     C pointer for a Lisp object, and work on that directly to avoid a
     whole bunch of extra indirections.  I think the way to do this
     would be through the use of a special locking construct
     implemented as part of the extra preprocessor stage mentioned
     above.  This would essentially be what you might call a "lock
     block", just like a `while' block.  You'd write the word `lock'
     followed by a parenthesized expression that retrieves the C
     pointer and stores it into a variable that is scoped only within
     the lock block and followed in turn by some code in braces, which
     is the actual code associated with the lock block, and which can
     make use of this pointer.  While the code inside the lock block is
     executing, that particular pointer and the object pointed to by it
     is guaranteed not to be relocated.

  6.  If all the XEmacs C code were converted according to these rules,
     there would be no restrictions on the sorts of implementations
     that can be used for the garbage collector.  It would be possible,
     for example, to have an incremental asynchronous relocating
     garbage collector that operated continuously in another thread
     while XEmacs was running.

  7.  The C implementation of Lisp objects might not, and probably
     should not, be visible to the rest of the XEmacs C code.  It
     should theoretically be possible, for example, to implement Lisp
     objects entirely in terms of association lists, rather than using
     C structures in the standard way.  (This may be an extreme
     example, but it's good to keep in mind an example such as this
     when cleaning up the XEmacs C code).  The changes mentioned in the
     previous item would go a long way towards removing this
     assumption.  The only places where this assumption might still be
     made would be inside of the lock blocks where an actual pointer is
     retrieved.  (Also, of course, we'd have to change the way that
     Lisp objects are defined in C so that this is done with some
     function calls and new and improved macros rather than by having
     the XEmacs C code actually define the structures.  This sort of
     thing would probably have to be done in any case once the
     allocation mechanism is moved into a separate library.)  With some
     thought it should be possible to define the lock block interface
     in such a way as to remove any assumptions about the
     implementation of Lisp objects.

  8.  C code may not be able to call Lisp primitives that are defined
     in C simply by making standard C function calls.  There might need
     to be some wrapper around all such calls.  This could be achieved
     cleanly through the extra preprocessing step mentioned above, in
     line with the example described there.


Actually Replacing the Engine.
..............................

Once we've done all of the work mentioned in the previous steps (and
admittedly, this is quite a lot of work), we should have an XEmacs that
still uses what is essentially the old and previously existing Lisp
engine, but which is ready to have its Lisp engine replaced.  The
replacement might proceed as follows:

  1.  Identify any further changes that need to be made to the engine
     interface that we have defined as a result of the previous steps
     so that features and idiosyncrasies of various Lisp engines that
     we examine could be properly supported.

  2.  Pick a Lisp engine and write an interface layer that sits on top
     of this Lisp engine and makes it adhere to what I'll now call the
     XEmacs Lisp engine interface.

  3.  Strongly consider creating, if we haven't already done so, a test
     suite that can test the XEmacs Lisp engine interface when used
     with a stand-alone Lisp engine.

  4.  Test the hell out of the Lisp engine that we've chosen when
     combined with its XEmacs Lisp engine interface layer as a
     stand-alone program.

  5.  Now finally attach this stand-alone program to XEmacs itself.
     Debug and fix any further problems that ensue (and there
     inevitably will be such problems), updating the test suite as we
     go along so that if it were run again on the old and buggy
     interfaced Lisp engine, it would note the bug.



File: internals.info,  Node: Future Work -- Startup File Modification by Packages,  Prev: Future Work -- Lisp Engine Replacement -- Implementation,  Up: Future Work -- Lisp Engine Replacement

40.23.3 Future Work - Startup File Modification by Packages
-----------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   OK, we need to create a design document for all of this, including:

   PRINCIPLE #1: Whenever you have auto-generated stuff, *CLEARLY*
indicate this in comments around the stuff.  These comments get
searched for, and used to locate the existing generated stuff to
replace.  Custom currently doesn't do this.

   PRINCIPLE #2: Currently, lots of functions want to add code to the
.emacs. (e.g. I get prompted for my mail address from
add-change-log-entry, and then prompted if I want to make this
permanent).  There needs to be a Lisp API for working with arbitrary
code to be added to a user's startup.  This API hides all the details
of which file to put the fragment in, where in it, how to mark it with
magical comments of the right kind so that previous fragments can be
replaced, etc.

   PRINCIPLE #3: *ALL* generated stuff should be loaded before any
user-written init stuff.  This way the user can override the generated
settings.  Although in the case of customize, it may work when the
custom stuff is at the end of the init file, it surely won't work for
arbitrary code fragments (which typically do `setq' or the like).

   PRINCIPLE #4: As much as possible, generated stuff should be place in
separate files from non-generated stuff.  Otherwise it's inevitable
that some corruption is going to result.

   PRINCIPLE #5: Packages are encouraged, as much as possible, to work
within the customize model and store all their customizations there.
However, if they really need to have their own init files, these files
should be placed in .xemacs/, given normal names (e.g.
`saved-abbrevs.el' not .abbrevs), and there should be some magic
comment at the top of the file that causes it to get automatically
loaded while loading a user's init file. (Alternatively, the
above-named API could specify a function that lets a package specify
that they want such-and-such file loaded from the init file, and have
the specifics of this get handled correctly.)

   OVERARCHING GOAL: The overarching goal is to provide a unified
mechanism for packages to store state and setting information about the
user and what they were doing when XEmacs exited, so that the same or a
similar environment can be automatically set up the next time.  In
general, we are working more and more towards being a truly GUI app
where users' settings are easy to change and get remembered correctly
and consistently from one session to the next, rather than requiring
nasty hacking in elisp.

   Hrvoje, do you have any interest in this?  How about you, Martin?
This seems like it might be up your alley.  This stuff has been
ad-hocked since kingdom come, and it's high time that we make this work
properly so that it could be relied upon, and a lot of things could
"just work".


File: internals.info,  Node: Future Work -- Better Rendering Support,  Prev: Future Work -- Lisp Engine Replacement,  Up: Future Work

40.24 Future Work - Better Rendering Support
============================================

This section was written by Stephen Turnbull <stephen@xemacs.org>, so
don't blame Ben (or Eric and Matthias, for that matter).  Feel free to
add, edit, and share the blame, guys!

   As of late November 2004, this principally means adding support for
the `Xft' library, which provides a more robust _font configuration_
mechanism via Keith Packard's `fontconfig' library improved glyph
rendering, including antialiasing, via the `freetype' library, and
client-side rendering (saving bandwidth and server memory) via the
`XRender extension'.  In fact, patches which provide Xft support have
been available for several years, but the authors have been unwilling
to deal with several important issues which block integration.  These
are _Mule_, and more generally, _face_ support; _widget_ support
(including the toolbar and menubar); and _redisplay refactoring_.

   However, in late 2003 Eric Knauel
<knauel@informatik.uni-tuebingen.de> and Matthias Neubauer
<neubauer@informatik.uni-freiburg.de> put forward a relatively complete
patch which was robust to daily use in ISO 8859-1 locales, and Stephen
Turnbull began work on the integration issues.  At this point a
(private) CVS branch is available for Stephen's patch (branch point tag
`sjt-xft-bp', branch tag `sjt-xft'), and one may be made available for
the Knauel-Matthias patch soon.

* Menu:

* Better Rendering Support -- Review Criteria::
* Better Rendering Support -- Implementation::
* Better Rendering Support -- Current Status::
* Better Rendering Support -- Configuration with the Interim Patches::
* Better Rendering Support -- Modern Font Support::


File: internals.info,  Node: Better Rendering Support -- Review Criteria,  Next: Better Rendering Support -- Implementation,  Prev: Future Work -- Better Rendering Support,  Up: Future Work -- Better Rendering Support

40.24.1 Better Rendering Support - Review Criteria
--------------------------------------------------

Of course it's "unfair" to demand that the implementers of a nice
feature like anti-aliasing support deal with accumulated cruft of the
last few years, but somebody must, sometime soon.  Even core developers
are complaining about how slow XEmacs is in some applications, and there
is reason to believe that some of the problem is in redisplay.  Adding
more _ad hoc_ features to redisplay will make the whole module more
complex and unintelligible.  Even if it doesn't inherently further
detract from efficiency, it will surely make reform and refactoring
harder.

   Similar considerations apply to Mule support.  If Xft support is not
carefully designed, or implemented with Mule support soon, it will
undoubtedly make later Mule implementation far more difficult than it
needs to be, and require redundant work be done (_e.g._, on `Options'
menu support).

   Besides the design issue--and many users are requesting more
flexibility, primarily face support, from the widgets--with widget
support there is also an aesthetic issue.  It is horribly unimpressive
to have clunky bitmapped fonts on the decorations when pleasant
antialiased fonts are available in the buffer.

   Finally, these issues interact.  Widgets and faces are inherently
heavyweight objects, requiring orders of magnitude more computation than
simply displaying a string in a fixed font.  This will have an
efficiency impact, of course.  And they interact with each other; Mule
was designed for use in buffers and display in Emacs windows--but a
widget's content is usually not a buffer, and widgets need not be
displayed in a window, but may appear in other contexts, especially in
the gutters.  So specifiers will probably have to be reworked, in order
to properly support display of different faces in non-buffer, non-window
contexts.

