This is ../info/internals.info, produced by makeinfo version 4.8 from
internals/internals.texi.

INFO-DIR-SECTION XEmacs Editor
START-INFO-DIR-ENTRY
* Internals: (internals).       XEmacs Internals Manual.
END-INFO-DIR-ENTRY

   Edition History:

   Created November 1995 (?) by Ben Wing.  XEmacs Internals Manual
Version 1.0, March, 1996.  XEmacs Internals Manual Version 1.1, March,
1997.  XEmacs Internals Manual Version 1.4, March, 2001.  XEmacs
Internals Manual Version 21.5, October, 2004.

   Copyright (C) 1992 - 2004 Ben Wing.  Copyright (C) 1996, 1997 Sun
Microsystems.  Copyright (C) 1994 - 1998, 2002, 2003 Free Software
Foundation.  Copyright (C) 1994, 1995 Board of Trustees, University of
Illinois.

   Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided that the
entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that this permission notice may be stated in a
translation approved by the Foundation.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided also
that the section entitled "GNU General Public License" is included
exactly as in the original, and provided that the entire resulting
derived work is distributed under the terms of a permission notice
identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that the section entitled "GNU General Public License"
may be included in a translation approved by the Free Software
Foundation instead of in the original English.


File: internals.info,  Node: List of changed files in new Mule workspace,  Next: Changes to the MULE subsystems,  Prev: The Great Mule Merge of March 2002,  Up: The Great Mule Merge of March 2002

25.14.1 List of changed files in new Mule workspace
---------------------------------------------------

This node lists the files that were touched in the Great Mule Merge.

Deleted files
=============

     src/iso-wide.h
     src/mule-charset.h
     src/mule.c
     src/ntheap.h
     src/syscommctrl.h
     lisp/files-nomule.el
     lisp/help-nomule.el
     lisp/mule/mule-help.el
     lisp/mule/mule-init.el
     lisp/mule/mule-misc.el
     nt/config.h

Other deleted files
===================

These files were all zero-width and accidentally present.

     src/events-mod.h
     tests/Dnd/README.OffiX
     tests/Dnd/dragtest.el
     netinstall/README.xemacs
     lib-src/srcdir-symlink.stamp

New files
=========

     CHANGES-ben-mule
     README.ben-mule-21-5
     README.ben-separate-stderr
     TODO.ben-mule-21-5
     etc/TUTORIAL.{cs,es,nl,sk,sl}
     etc/unicode/*
     lib-src/make-mswin-unicode.pl
     lisp/code-init.el
     lisp/resize-minibuffer.el
     lisp/unicode.el
     lisp/mule/china-util.el
     lisp/mule/cyril-util.el
     lisp/mule/devan-util.el
     lisp/mule/devanagari.el
     lisp/mule/ethio-util.el
     lisp/mule/indian.el
     lisp/mule/japan-util.el
     lisp/mule/korea-util.el
     lisp/mule/lao-util.el
     lisp/mule/lao.el
     lisp/mule/mule-locale.txt
     lisp/mule/mule-msw-init.el
     lisp/mule/thai-util.el
     lisp/mule/thai.el
     lisp/mule/tibet-util.el
     lisp/mule/tibetan.el
     lisp/mule/viet-util.el
     src/charset.h
     src/intl-auto-encap-win32.c
     src/intl-auto-encap-win32.h
     src/intl-encap-win32.c
     src/intl-win32.c
     src/intl-x.c
     src/mule-coding.c
     src/text.c
     src/text.h
     src/unicode.c
     src/s/win32-common.h
     src/s/win32-native.h

Changed files
=============

"Too numerous to mention."  (Ben didn't write that, I did, but it's a
good guess that's the intent....)


File: internals.info,  Node: Changes to the MULE subsystems,  Next: Pervasive changes throughout XEmacs sources,  Prev: List of changed files in new Mule workspace,  Up: The Great Mule Merge of March 2002

25.14.2 Changes to the MULE subsystems
--------------------------------------

configure changes
=================

   * file-coding always compiled in.  eol detection is off by default on
     unix, non-mule, but can be enabled with configure option
     `--with-default-eol-detection' or command-line flag `-eol'.

   * code that selects which files are compiled is mostly moved to
     `Makefile.in.in'.  see comment in `Makefile.in.in'.

   * vestigial i18n3 code deleted.

   * new cygwin mswin libs imm32 (input methods), mpr (user name
     enumeration).

   * check for `link', `symlink'.

   * `vfork'-related code deleted.

   * fix `configure.usage'.  (delete `--with-file-coding',
     `--no-doc-file', add `--with-default-eol-detection',
     `--quick-build').

   * `nt/config.h' has been eliminated and everything in it merged into
     `config.h.in' and `s/windowsnt.h'.  see `config.h.in' for more
     info.

   * massive rewrite of `s/windowsnt.h', `m/windowsnt.h',
     `s/cygwin32.h', `s/mingw32.h'.  common code moved into
     `s/win32-common.h', `s/win32-native.h'.

   * in `nt/xemacs.mak', `nt/config.inc.samp', variable is called
     `MULE', not `HAVE_MULE', for consistency with sources.

   * define `TABDLY', `TAB3' in `freebsd.h' (#### from where?)


File: internals.info,  Node: Pervasive changes throughout XEmacs sources,  Next: Changes to specific subsystems,  Prev: Changes to the MULE subsystems,  Up: The Great Mule Merge of March 2002

25.14.3 Pervasive changes throughout XEmacs sources
---------------------------------------------------

   * all `#ifdef FILE_CODING' statements removed from code.

Changes to string processing
============================

   * new `qxe()' string functions that accept `Intbyte *' as arguments.
     These work exactly like the standard `strcmp()', `strcpy()',
     `sprintf()', etc. except for the argument declaration differences.
     We use these whenever we have `Intbyte *' strings, which is quite
     often.

   * new fun `build_intstring()' takes an `Intbyte *'.  also new funs
     `build_msg_intstring' (like `build_intstring()') and
     `build_msg_string' (like `build_string()') to do a `GETTEXT()'
     before building the string.  (elimination of old
     `build_translated_string()', replaced by `build_msg_string()').

   * function `intern_int()' for `Intbyte *' arguments, like `intern()'.

   * numerous places throughout code where `char *' replaced with
     something else, e.g. `Char_ASCII *', `Intbyte *', `Char_Binary *',
     etc.  same with unsigned `char *', going to `UChar_Binary *', etc.


File: internals.info,  Node: Changes to specific subsystems,  Next: Mule changes by theme,  Prev: Pervasive changes throughout XEmacs sources,  Up: The Great Mule Merge of March 2002

25.14.4 Changes to specific subsystems
--------------------------------------

Changes to the init code
========================

   * lots of init code rewritten to be mule-correct.

Changes to processes
====================

   * always call `egetenv()', never `getenv()', for mule correctness.

command line (`startup.el', `emacs.c')
======================================

   * new option `-eol' to enable auto EOL detection under non-mule unix.

   * new option `-nuni' (`--no-unicode-lib-calls') to force use of
     non-Unicode API's under Windows NT, mostly for debugging purposes.


File: internals.info,  Node: Mule changes by theme,  Next: File-coding rewrite,  Prev: Changes to specific subsystems,  Up: The Great Mule Merge of March 2002

25.14.5 Mule changes by theme
-----------------------------

   * the code that handles the details of processing multilingual text
     has been consolidated to make it easier to extend it.  it has been
     yanked out of various files (`buffer.h', `mule-charset.h',
     `lisp.h', `insdel.c', `fns.c', `file-coding.c', etc.) and put into
     `text.c' and `text.h'.  `mule-charset.h' has also been renamed
     `charset.h'.  all long comments concerning the representations and
     their processing have been consolidated into `text.c'.

   * major rewriting of file-coding.  it's mostly abstracted into coding
     systems that are defined by methods (similar to devices and
     specifiers), with the ultimate aim being to allow non-i18n coding
     systems such as gzip.  there is a "chain" coding system that
     allows multiple coding systems to be chained together.  (it
     doesn't yet have the concept that either end of a coding system
     can be bytes or chars; this needs to be added.)

   * large amounts of code throughout the code base have been
     Mule-ized, not just Windows code.

   * total rewriting of OS locale code.  it notices your locale at
     startup and sets the language environment accordingly, and calls
     `setlocale()' and sets `LANG' when you change the language
     environment.  new language environment properties `locale',
     `mswindows-locale', `cygwin-locale', `native-coding-system', to
     determine langenv from locale and vice-versa; fix all language
     environments (lots of language files).  langenv startup code
     rewritten.  many new functions to convert between locales,
     language environments, etc.

   * major overhaul of the way default values for the various coding
     system variables are handled.  all default values are collected
     into one location, a new file `code-init.el', which provides a
     unified mechanism for setting and querying what i call "basic
     coding system variables" (which may be aliases, parts of conses,
     etc.) and a mechanism of different configurations (Windows w/Mule,
     Windows w/o Mule, Unix w/Mule, Unix w/o Mule, unix w/o Mule but
     w/auto EOL), each of which specifies a set of default values.  we
     determine the configuration at startup and set all the values in
     one place.  (`code-init.el', `code-files.el', `coding.el', ...)

   * i copied the remaining language-specific files from fsf.  i made
     some minor changes in certain cases but for the most part the
     stuff was just copied and may not work.

   * ms windows mule support, with full unicode support.  required font,
     redisplay, event, other changes.  ime support from ikeyama.

Lisp-Visible Changes:
=====================

   * ensure that `escape-quoted' works correctly even without Mule
     support and use it for all auto-saves.  (`auto-save.el',
     `fileio.c', `coding.el', `files.el')

   * new var `buffer-file-coding-system-when-loaded' specifies the
     actual coding system used when the file was loaded
     (`buffer-file-coding-system' is usually the same, but may be
     changed because it controls how the file is written out).  use it
     in revert-buffer (`files.el', `code-files.el') and in new submenu
     File->Revert Buffer with Specified Encoding (`menubar-items.el').

   * improve docs on how the coding system is determined when a file is
     read in; improved docs are in both `find-file' and
     `insert-file-contents' and a reference to where to find them is in
     `buffer-file-coding-system-for-read'.  (`files.el',
     `code-files.el')

   * new (brain-damaged) FSF way of calling post-read-conversion (only
     one arg, not two) is supported, along with our two-argument way,
     as best we can.  (`code-files.el')

   * add inexplicably missing var `default-process-coding-system'.  use
     it.  get rid of former hacked-up way of setting these defaults
     using `comint-exec-hook'.  also fun
     `set-buffer-process-coding-system'.  (`code-process.el',
     `code-cmds.el', `process.c')

   * remove function `set-default-coding-systems'; replace with
     `set-default-output-coding-systems', which affects only the output
     defaults (`buffer-file-coding-system', output half of
     `default-process-coding-system').  the input defaults should not be
     set by this because they should always remain `undecided' in normal
     circumstances.  fix `prefer-coding-system' to use the new function
     and correct its docs.

   * fix bug in `coding-system-change-eol-conversion' (`code-cmds.el')

   * recognize all eol types in `prefer-coding-system' (`code-cmds.el')

   * rewrite `coding-system-category' to be correct (`coding.el')

Internal Changes
================

   * major improvements to eistring code, fleshing out of missing funs.

   * Separate encoding and decoding lstreams have been combined into a
     single coding lstream.  Functions` make_encoding_*_stream' and
     `make_decoding_*_stream' have been combined into
     `make_coding_*_stream', which takes an argument specifying whether
     encode or decode is wanted.

   * remove last vestiges of I18N3, I18N4 code.

   * ascii optimization for strings: we keep track of the number of
     ascii chars at the beginning and use this to optimize byte<->char
     conversion on strings.

   * `mule-misc.el', `mule-init.el' deleted; code in there either
     deleted, rewritten, or moved to another file.

   * `mule.c' deleted.

   * move non-Mule-specific code out of `mule-cmds.el' into
     `code-cmds.el'.  (`coding-system-change-text-conversion'; remove
     duplicate `coding-system-change-eol-conversion')

   * remove duplicate `set-buffer-process-coding-system'
     (`code-cmds.el')

   * add some commented-out code from FSF `mule-cmds.el'
     (`find-coding-systems-region-subset-p',
     `find-coding-systems-region', `find-coding-systems-string',
     `find-coding-systems-for-charsets', `find-multibyte-characters',
     `last-coding-system-specified', `select-safe-coding-system',
     `select-message-coding-system') (`code-cmds.el')

   * remove obsolete alias `pathname-coding-system', function
     `set-pathname-coding-system' (`coding.el')

   * remove coding-system property `doc-string'; split into
     `description' (short, for menu items) and `documentation' (long);
     correct coding system defns (`coding.el', `file-coding.c', lots of
     language files)

   * move coding-system-base into C and make use of internal info
     (`coding.el', `file-coding.c')

   * move `undecided' defn into C (`coding.el', `file-coding.c')

   * use `define-coding-system-alias', not `copy-coding-system'
     (`coding.el')

   * new coding system `iso-8859-6' for arabic

   * delete windows-1251 support from `cyrillic.el'; we do it
     automatically

   * remove `setup-*-environment' as per FSF 21

   * rewrite `european.el' with lang envs for each language, so we can
     specify the locale

   * fix corruption in `greek.el'

   * sync `japanese.el' with FSF 20.6

   * fix warnings in `mule-ccl.el'

   * move FSF compat Mule fns from `obsolete.el' to `mule-charset.el'

   * eliminate unused `truncate-string{-to-width}'

   * `make-coding-system' accepts (but ignores) the additional
     properties present in the fsf version, for compatibility.

   * i fixed the iso2022 handling so it will correctly read in files
     containing unknown charsets, creating a "temporary" charset which
     can later be overwritten by the real charset when it's defined.
     this allows iso2022 elisp files with literals in strange languages
     to compile correctly under mule.  i also added a hack that will
     correctly read in and write out the emacs-specific "composition"
     escape sequences, i.e. `ESC 0' through `ESC 4'.  this means that
     my workspace correctly compiles the new file `devanagari.el' that
     i added.

   * elimination of `string-to-char-list' (use `string-to-list')

   * elimination of junky `define-charset'

Selection
=========

   * fix msw selection code for Mule.  proper encoding for
     `RegisterClipboardFormat'.  store selection as `CF_UNICODETEXT',
     which will get converted to the other formats.  don't respond to
     destroy messages from `EmptyClipboard()'.

Menubar
=======

   * new items `Open With Specified Encoding', `Revert Buffer with
     Specified Encoding'

   * split Mule menu into `Encoding' (non-Mule-specific; includes new
     item to control EOL auto-detection) and `International' submenus on
     `Options', `International' on `Help'


Unicode support:
================

   * translation tables added in `etc/unicode'

   * new files `unicode.c', `unicode.el' containing unicode coding
     systems and support; old code ripped out of `file-coding.c'

   * translation tables read in at startup (NEEDS WORK TO MAKE IT MORE
     EFFICIENT)

   * support `CF_TEXT', `CF_UNICODETEXT' in `select.el'

   * encapsulation code added so that we can support both Windows 9x
     and NT in a single executable, determining at runtime whether to
     call the Unicode or non-Unicode API.  encapsulated routines in
     `intl-encap-win32.c' (non-auto-generated) and
     `intl-auto-encap-win32.[ch]' (auto-generated).  code generator in
     `lib-src/make-mswin-unicode.pl'.  changes throughout the code to
     use the wide structures (W suffix) and call the encapsulated Win32
     API routines (`qxe' prefix).  calling code needs to do proper
     conversion of text using new coding systems `Qmswindows_tstr',
     `Qmswindows_unicode', or `Qmswindows_multibyte'.  (the first
     points to one of the other two.)


File: internals.info,  Node: File-coding rewrite,  Next: General User-Visible Changes,  Prev: Mule changes by theme,  Up: The Great Mule Merge of March 2002

25.14.6 File-coding rewrite
---------------------------

The coding system code has been majorly rewritten.  It's abstracted into
coding systems that are defined by methods (similar to devices and
specifiers).  The types of conversions have also been generalized.
Formerly, decoding always converted bytes to characters and encoding the
reverse (these are now called "text file converters"), but conversion
can now happen either to or from bytes or characters.  This allows
coding systems such as `gzip' and `base64' to be written.  When
specifying such a coding system to an operation that expects a text
file converter (such as reading in or writing out a file), the
appropriate coding systems to convert between bytes and characters are
automatically inserted into the conversion chain as necessary.  To
facilitate creating such chains, a special coding system called "chain"
has been created, which chains together two or more coding systems.

   Encoding detection has also been abstracted.  Detectors are logically
separate from coding systems, and each detector defines one or more
categories.  (For example, the detector for Unicode defines categories
such as UTF-8, UTF-16, UCS-4, and UTF-7.) When a particular detector is
given a piece of text to detect, it determines likeliness values (seven
of them, from 3 [most likely] to -3 [least likely]; specific criteria
are defined for each possible value).  All detectors are run in parallel
on a particular piece of text, and the results tabulated together to
determine the actual encoding of the text.

   Encoding and decoding are now completely parallel operations, and the
former "encoding" and "decoding" lstreams have been combined into a
single "coding" lstream.  Coding system methods that were formerly
split in such a fashion have also been combined.


File: internals.info,  Node: General User-Visible Changes,  Next: General Lisp-Visible Changes,  Prev: File-coding rewrite,  Up: The Great Mule Merge of March 2002

25.14.7 General User-Visible Changes
------------------------------------

Search
======

   * make regex routines reentrant, since they're sometimes called
     reentrantly.  (see `regex.c' for a description of how.)  all global
     variables used by the regex routines get pushed onto a stack by the
     callers before being set, and are restored when finished.  redo the
     preprocessor flags controlling `REL_ALLOC' in conjunction with
     this.

Menubar
=======

   * move menu-splitting code (`menu-split-long-menu', etc.) from
     `font-menu.el' to `menubar-items.el' and redo its algorithm; use
     in various items with long generated menus; rename to remove
     `font-' from beginning of functions but keep old names as aliases

   * new fn `menu-sort-menu'

   * redo items `Grep All Files in Current Directory {and Below}' using
     stuff from sample `init.el'

   * `Debug on Error' and friends now affect current session only; not
     saved

   * `maybe-add-init-button' -> `init-menubar-at-startup' and call
     explicitly from `startup.el'

   * don't use `charset-registry' in `msw-font-menu.el'; it's only for X

Changes to key bindings
=======================

These changes are primarily found in `keymap.c', `keydefs.el', and
`help.el', but are found in many other files.

   * `M-home', `M-end' now move forward and backward in buffers; with
     <Shift>, stay within current group (e.g. all C files; same grouping
     as the gutter tabs).  (bindings
     `switch-to-{next/previous}-buffer[-in-group]' in `files.el')

     needed to move code from `gutter-items.el' to `buff-menu.el'
     that's used by these bindings, since `gutter-items.el' is loaded
     only when the gutter is active and these bindings (and hence the
     code) is not (any more) gutter specific.

   * new global vars global-tty-map and global-window-system-map
     specify key bindings for use only on TTY's or window systems,
     respectively.  this is used to make `ESC ESC' be keyboard-quit on
     window systems, but `ESC ESC ESC' on TTY's, where <Meta + arrow>
     keys may appear as `ESC ESC O A' or whatever.  `C-z' on window
     systems is now `zap-up-to-char', and `iconify-frame' is moved to
     `C-Z'.  `ESC ESC' is `isearch-quit'.  (`isearch-mode.el')

   * document `global-{tty,window-system}-map' in various places;
     display them when you do `C-h b'.

   * fix up function documentation in general for keyboard primitives.
     e.g. key-bindings now contains a detailed section on the steps
     prior to looking up in keymaps, i.e. `function-key-map',
     `keyboard-translate-table'. etc.  `define-key' and other obvious
     starting points indicate where to look for more info.

   * eliminate use and mention of grody `advertised-undo' and
     `deprecated-help'.  (`simple.el', `startup.el', `picture.el',
     `menubar-items.el')


File: internals.info,  Node: General Lisp-Visible Changes,  Next: User documentation,  Prev: General User-Visible Changes,  Up: The Great Mule Merge of March 2002

25.14.8 General Lisp-Visible Changes
------------------------------------

gzip support
============

The gzip protocol is now partially supported as a coding system.

   * new coding system `gzip' (bytes -> bytes); unfortunately, not quite
     working yet because it handles only the raw zlib format and not the
     higher-level gzip format (the zlib library is brain-damaged in
     that it provides low-level, stream-oriented API's only for raw
     zlib, and for gzip you have only high-level API's, which aren't
     useful for xemacs).

   * configure support (`--with-zlib').


File: internals.info,  Node: User documentation,  Next: General internal changes,  Prev: General Lisp-Visible Changes,  Up: The Great Mule Merge of March 2002

25.14.9 User documentation
--------------------------

Tutorial
========

   * massive rewrite; sync to FSF 21.0.106, switch focus to window
     systems, new sections on terminology and multiple frames, lots of
     fixes for current xemacs idioms.

   * german version from Adrian mostly matching my changes.

   * copy new tutorials from FSF (Spanish, Dutch, Slovak, Slovenian,
     Czech); not updated yet though.

   * eliminate `help-nomule.el' and `mule-help.el'; merge into one
     single tutorial function, fix lots of problems, put back in
     `help.el' where it belongs.  (there was some random junk in
     `help-nomule.el', `string-width' and `make-char'.  `string-width'
     is now in `subr.el' with a single definition, and `make-char' in
     `text.c'.)

Sample init file
================

   * remove forward/backward buffer code, since it's now standard.

   * when disabling `C-x C-c', make it display a message saying how to
     exit, not just beep and complain "undefined".


File: internals.info,  Node: General internal changes,  Next: Ben's TODO list,  Prev: User documentation,  Up: The Great Mule Merge of March 2002

25.14.10 General internal changes
---------------------------------

Changes to gnuclient and gnuserv
================================

   * clean up headers a bit.

   * use proper ms win idiom for checking for temp directory (`TEMP' or
     `TMP', not `TMPDIR').

Process changes
===============

   * Move `setenv' from packages; synch `setenv'/`getenv' with 21.0.105

Changes to I/O internals
========================

   * use `PATH_MAX' consistently instead of `MAXPATHLEN', `MAX_PATH',
     etc.

   * all code that does preprocessor games with C lib I/O functions
     (open, read) has been removed.  The code has been changed to call
     the correct function directly.  Functions that accept `Intbyte *'
     arguments for filenames and such and do automatic conversion to or
     from external format will be prefixed `qxe...()'.  Functions that
     are retrying in case of `EINTR' are prefixed `retry_...()'.
     `DONT_ENCAPSULATE' is long-gone.

   * never call `getcwd()' any more.  use our shadowed value always.

Changes to string processing
============================

   * the `doprnt.c' external entry points have been completely rewritten
     to be more useful and have more sensible names.  We now have, for
     example, versions that work exactly like `sprintf()' but return a
     `malloc()'ed string.

   * code in `print.c' that handles `stdout', `stderr' rewritten.

   * places that print to `stderr' directly replaced with
     `stderr_out()'.

   * new convenience functions `write_fmt_string()',
     `write_fmt_string_lisp()', `stderr_out_lisp()', `write_string()'.

Changes to Allocation, Objects, and the Lisp Interpreter
========================================================

   * automatically use "managed lcrecord" code when allocating.  any
     lcrecord can be put on a free list with `free_lcrecord()'.

   * `record_unwind_protect()' returns the old spec depth.

   * `unbind_to()' now takes only one arg.  use `unbind_to_1()' if you
     want the 2-arg version, with GC protection of second arg.

   * new funs to easily inhibit GC.  (`{begin,end}_gc_forbidden()') use
     them in places where gc is currently being inhibited in a more ugly
     fashion.  also, we disable GC in certain strategic places where
     string data is often passed in, e.g. `dfc' functions, `print'
     functions.

   * `make_buffer()' -> `wrap_buffer()' for consistency with other
     objects; same for `make_frame()' ->` wrap_frame()' and
     `make_console()' -> `wrap_console()'.

   * better documentation in condition-case.

   * new convenience funs `record_unwind_protect_freeing()' and
     `record_unwind_protect_freeing_dynarr()' for conveniently setting
     up an unwind-protect to `xfree()' or `Dynarr_free()' a pointer.

s/m files:
==========

   * removal of unused `DATA_END', `TEXT_END', `SYSTEM_PURESIZE_EXTRA',
     `HAVE_ALLOCA' (automatically determined)

   * removal of `vfork' references (we no longer use `vfork')

`make-docfile':
===============

   * clean up headers a bit.

   * allow `.obj' to mean equivalent `.c', just like for `.o'.

   * allow specification of a "response file" (a command-line argument
     beginning with @, specifying a file containing further command-line
     arguments) - a standard mswin idiom to avoid potential command-line
     limits and to simplify makefiles.  use this in `xemacs.mak'.

debug support
=============

   * (`cmdloop.el') new var breakpoint-on-error, which breaks into the C
     debugger when an unhandled error occurs noninteractively.  useful
     when debugging errors coming out of complicated make scripts, e.g.
     package compilation, since you can set this through an env var.

   * (`startup.el') new env var `XEMACSDEBUG', specifying a Lisp form
     executed early in the startup process; meant to be used for turning
     on debug flags such as `breakpoint-on-error' or
     `stack-trace-on-error', to track down noninteractive errors.

   * (`cmdloop.el') removed non-working code in `command-error' to
     display a backtrace on `debug-on-error'.  use
     `stack-trace-on-error' instead to get this.

   * (`process.c') new var `debug-process-io' displays data sent to and
     received from a process.

   * (`alloc.c') staticpros have name stored with them for easier
     debugging.

   * (`emacs.c') code that handles fatal errors consolidated and
     rewritten.  much more robust and correctly handles all fatal exits
     on mswin (e.g. aborts, not previously handled right).

`startup.el'
============

   * move init routines from `before-init-hook' or `after-init-hook';
     just call them directly (`init-menubar-at-startup',
     `init-mule-at-startup').

   * help message fixed up (divided into sections), existing problem
     causing incomplete output fixed, undocumented options documented.

`frame.el'
==========

   * delete old commented-out code.


File: internals.info,  Node: Ben's TODO list,  Next: Ben's README,  Prev: General internal changes,  Up: The Great Mule Merge of March 2002

25.14.11 Ben's TODO list (probably obsolete)
--------------------------------------------

These notes substantially overlap those in *Note Ben's README::.  They
should probably be combined.

April 11, 2002
==============

Priority:

  1. Finish checking in current mule ws.

  2. Start working on bugs reported by others and noticed by me:

        *   problems cutting and pasting binary data, e.g. from
          byte-compiler   instructions

        *   test suite failures

        *   process i/o problems w.r.t. eol: |uniq (e.g.) leaves ^M's
          at end of   line; running "bash" as shell-file-name doesn't
          work because it doesn't   like the extra ^M's.

March 20, 2002
==============

bugs:

   * TTY-mode problem.  When you start up in TTY mode, XEmacs goes
     through the loadup process and appears to be working - you see the
     startup screen pulsing through the different screens, and it
     appears to be listening (hitting a key stops the screen motion),
     but it's frozen - the screen won't get off the startup, key
     commands don't cause anything to happen. STATUS: In progress.

   * Memory ballooning in some cases.  Not yet understood.

   * other test suite failures?

   * need to review the handling of sounds.  seems that not everything
     is documented, not everything is consistently used where it's
     supposed to, some sounds are ugly, etc.  add sounds to `completer'
     as well.

   * redo with-trapping-errors so that the backtrace is stored away and
     only outputted when an error actually occurs (i.e. in the
     condition-case handler).  test. (use ding of various sorts as a
     helpful way of checking out what's going on.)

   * problems with process input: |uniq (for example) leaves ^M's at
     end of line.

   * carefully review looking up of fonts by charset, esp. wrt the last
     element of a font spec.

   * add package support to ignore certain files - *-util.el for
     languages.

   * review use of escape-quoted in auto_save_1() vs. the buffer's own
     coding system.

   * figure out how to get the total amount of data memory (i.e.
     everything but the code, or even including the code if can't
     distinguish) used by the process on each different OS, and use it
     in a new algorithm for triggering GC: trigger only when a certain
     % of the data size has been consed up; in addition, have a minimum.

   * fixed bugs???

        *   Occasional crash when freeing display structures.  The
          problem seems to   be this: A window has a "display line
          dynarr"; each display line has a   "display block dynarr".
          Sometimes this display block dynarr is getting   freed twice.
          It appears from looking at the code that sometimes a
          display line from somewhere in the dynarr gets added to the
          end - hence   two pointers to the same display block dynarr.
          need to review this   code.

August 29, 2001
===============

This is the most current list of priorities in `ben-mule-21-5'.
Updated often.

   high-priority:

*[input]*
        * support for WM_IME_CHAR.  IME input can work under -nuni if
          we use WM_IME_CHAR.  probably we should always be using this,
          instead of snarfing input using WM_COMPOSITION.  i'll check
          this out.

        * Russian C-x problem.  see above.

*[clean-up]*
        * make sure it compiles and runs under non-mule.  remember that
          some code needs the unicode support, or at least a simple
          version of it.

        * make sure it compiles and runs under pdump.  see below.

        * make sure it compiles and runs under cygwin.  see below.

        * clean up mswindows-multibyte, TSTR_TO_C_STRING.  expand dfc
          optimizations to work across chain.

        * eliminate last vestiges of codepage<->charset conversion and
          similar stuff.

*[other]*
        * test the "file-coding is binary only on Unix, no-Mule" stuff.

        * test that things work correctly in -nuni if the system
          environment is set to e.g. japanese - i should get japanese
          menus, japanese file names, etc.  same for russian, hebrew ...

        * cut and paste.  see below.

        * misc issues with handling lang environments.  see also August
          25, "finally: working on the `C-x' in ...".

             *   when switching lang env, needs to set keyboard layout.

             *   user var to control whether, when moving into text of a
                particular language, we set the appropriate keyboard
               layout.  we   would need to have a lisp api for
               retrieving and setting the   keyboard layout, set text
               properties to indicate the layout of   text, and have a
               way of dealing with text with no property on   it. (e.g.
               saved text has no text properties on it.) basically,
               we need to get a keyboard layout from a charset; getting
               a   language would do.  Perhaps we need a table that
               maps charsets   to language environments.

             *   test that the lang env is properly set at startup.
               test that   switching the lang env properly sets the C
               locale (call   `setlocale()', set `LANG', etc.) - a
               spawned subprogram   should have the new locale in its
               environment.

        * look through everything below and see if anything is missed
          in this priority list, and if so add it.  create a separate
          file for the priority list, so it can be updated as
          appropriate.

   mid-priority:

   * clean up the chain coding system.  its list should specify decode
     order, not encode; i now think this way is more logical.  it should
     check the endpoints to make sure they make sense.  it should also
     allow for the specification of "reverse-direction coding systems":
     use the specified coding system, but invert the sense of decode and
     encode.

   * along with that, places that take an arbitrary coding system and
     expect the ends to be anything specific need to check this, and add
     the appropriate conversions from byte->char or char->byte.

   * get some support for arabic, thai, vietnamese, japanese jisx 0212:
     at least get the unicode information in place and make sure we have
     things tied together so that we can display them.  worry about r2l
     some other time.

   * check the handling of `C-c'.  can XEmacs itself be interrupted with
     `C-c'?  is that impossible now that we are a window, not a console,
     app?  at least we should work something out with `i' so that if it
     receives a `C-c' or `C-break', it interrupts XEmacs, too.  check
     out how process groups work and if they apply only to console apps.
     also redo the way that XEmacs sends `C-c' to other apps.  the
     business of injecting code should be last resort.  we should try
     `C-c' first, and if that doesn't work, then the next time we try to
     interrupt the same process, use the injection method.


File: internals.info,  Node: Ben's README,  Prev: Ben's TODO list,  Up: The Great Mule Merge of March 2002

25.14.12 Ben's README (probably obsolete)
-----------------------------------------

These notes substantially overlap those in *Note Ben's TODO list::.
They should probably be combined.

   This may be of some historical interest as a record of Ben at work.
There may also be some useful suggestions as yet unimplemented.

oct 27, 2001
============

------- proposal for better buffer-switching commands:

   implement what VC++ currently has.  you have a single "switch"
command like `CTRL-TAB', which as long as you hold the <CTRL> button
down, brings successive buffers that are "next in line" into the current
position, bumping the rest forward.  once you release the <CTRL> key,
the chain is broken, and further `CTRL-TAB's will start from the
beginning again.  this way, frequently used buffers naturally move
toward the front of the chain, and you can switch back and forth between
two buffers using `CTRL-TAB'.  the only thing about `CTRL-TAB' is it's
a bit awkward.  the way to implement is to have modifier-up strokes
fire off a hook, like modifier-up-hook.  this is driven by event
dispatch, so there are no synchronization issues.  when `C-tab' is
pressed, the binding function does something like set a one-shot handler
on the modifier-up-hook (perhaps separate hooks for separate
modifiers?).

   to do this, we'd also want to change the buffer tabs so that they
maintain their own order.  in particular, they start out synched to the
regular order, but as you make changes, you don't want the tabs to
change order. (in fact, they may already do this.) selecting a
particular buffer from the buffer tabs DOES make the buffer go to the
head of the line.  the invariant is that if the tabs are displaying X
items, those X items are the first X items in the standard buffer list,
but may be in a different order. (it looks like the tabs may already
implement all of this.)

oct 26, 2001
============

necessary testing/changes:

   * test all eol detection stuff under windows w/ and w/o mule, unix
     w/ and w/o mule. (test configure flag, command-line flag, menu
     option) may need a way of pretending to be unix under cygwin.

   * test under windows w/ and w/o mule, cygwin w/ and w/o mule, cygwin
     x windows w/ and w/o mule.

   * test undecided-dos/unix/mac.

   * check `ESC ESC' works as `isearch-quit' under TTY's.

   * test `coding-system-base' and all its uses (grep for them).

   * menu item to revert to most recent auto save.

   * consider renaming `build_string' -> `build_intstring' and
     `build_c_string' to `build_string'. (consistent with
     `build_msg_string' et al; many more `build_c_string' than
     `build_string')

oct 20, 2001
============

fixed problem causing crash due to invalid internal-format data, fixed
an existing bug in `valid_char_p', and added checks to more quickly
catch when invalid chars are generated.  still need to investigate why
`mswindows-multibyte' is being detected.

   i now see why - we only process 65536 bytes due to a constant
`MAX_BYTES_PROCESSED_FOR_DETECTION'.  instead, we should have no limit
as long as we have a seekable stream.  we also need to write
`stderr_out_lisp()', used in the debug info routines i wrote.

   check once more about `DEBUG_XEMACS'.  i think debugging info should
be ON by default.  make sure it is.  check that nothing untoward will
result in a production system, e.g. presumably `assert()'s should not
really `abort()'.  (!! Actually, this should be runtime settable!  Use
a variable for this, and it can be set using the same `XEMACSDEBUG'
method.  In fact, now that I think of it, I'm sure that debugging info
should be on always, with runtime ways of turning on or off any funny
behavior.)

oct 19, 2001
============

fixed various bugs preventing packages from being able to be built.
still another bug, with `psgml/etc/cdtd/docbook', which contains some
strange characters starting around char pos 110,000.  It gets detected
as `mswindows-multibyte' (wrong! why?) and then invalid internal-format
data is generated.  need to fix `mswindows-multibyte' (and possibly add
something that signals an error as well; need to work on this
error-signalling mechanism) and figure out why it's getting detected as
such.  what i should do is add a debug var that outputs blow-by-blow
info of the detection process.

oct 9, 2001
===========

the stuff with `global-window-system-map' doesn't appear to work.  in
any case it needs better documentation. [DONE]

   `M-home', `M-end' do work, but cause cl-macs to get loaded.  why?

oct 8, 2001
===========

finished the coding system changes and they finally work!

   need to implement undecided-unix/dos/mac.  they should be easy to
do; it should be enough to specify an eol-type but not do-eol, but
check this.

   consider making the standard naming be foo-lf/crlf/cr, with
unix/dos/mac as aliases.

   print methods for coding systems should include some of the generic
properties. (also then fix print_..._within_print_method). [DONE]

   in a little while, go back and delete the
`text-file-wrapper-coding-system' code. (it'll be in CVS if necessary
to get at it.) [DONE]

   need to verify at some point that non-text-file coding systems work
properly when specified.  when gzip is working, this would be a good
test case. (and consider creating base64 as well!)

   remove extra crap from `coding-system-category' that checks for
chain coding systems. [DONE]

   perhaps make a primitive that gets at `coding-system-canonical'.
[DONE]

   need to test cygwin, compiling the mule packages, get unix-eol stuff
working.  frank from germany says he doesn't see a lisp backtrace when
he gets an error during temacs?  verify that this actually gets
outputted.

   consider putting the current language on the modeline, mousable so
it can be switched.  also consider making the coding system be mousable
and the line number (pick a line) and the percentage (pick a
percentage).

oct 6, 2001
===========

added code so that `debug_print()' will output a newline to the
mswindows debugging output, not just the console.  need to test. [DONE]

   working on problem where all files are being detected as binary.  the
problem may be that the undecided coding system is getting wrapped with
an auto-eol coding system, which it shouldn't be - but even in this
situation, we should get the right results!  check the
canonicalize-after-coding methods.  also,
`determine_real_coding_system' appears to be getting called even when
we're not detecting encoding.  also, undecided needs a print method to
show its params, and chain needs to be updated to show
`canonicalize_after_coding'.  check others as well. [DONE]

oct 5, 2001
===========

finished up coding system changes, testing.

   errors byte-compiling files in `iso-2022-7-bit'.  perhaps it's not
correctly detecting the encoding?

   noticed a problem in the dfc macros: we call
`get_coding_system_for_text_file' with `eol_wrap == 1', to allow for
auto-detection of the eol type; but this defeats the check and
short-circuit for unicode.

   still need to implement calling `determine_real_coding_system()' for
non-seekable streams.  to implement correctly, we need to do our own
buffering. [DONE, BUT WITHOUT BUFFERING]

oct 4, 2001
===========

implemented most stuff below.

   need to finish up changes to `make_coding_system_1'. (i changed the
way internal coding systems were handled; i need to create subsidiaries
for all types of coding systems, not just text ones.) there's a nasty
`xfree()' crash i was hitting; perhaps it'll go away once all stuff has
been rewritten.

   check under cygwin to make sure that when an error occurs during
loadup, a backtrace is output.

   as soon as andy releases his new setup, we should put it onto various
standard windows software repositories.

oct 3, 2001
===========

added `global-tty-map' and `global-window-system-map'.  add some stuff
to the maps, e.g. `C-x ESC' for repeat vs. `C-x ESC ESC' on TTY's, and
of course `ESC ESC' on window systems vs. `ESC ESC ESC' on TTY's. [TEST]

   was working on integrating the two `help-for-tutorial' versions
(mule, non-mule). [DONE, but test under non-Mule]

   was working on the file-coding changes.  need to think more about
`text-file-wrapper'.  conclusion i think is that
`get_coding_system_for_text_file' should wrap using a special coding
system type called a `text-file-wrapper', which inherits from chain,
and implements `canonicalize-after-decoding' to just return the
unwrapped coding system.  We need to implement inheritance of coding
systems, which will certainly come in extremely useful when coding
systems get implemented in Lisp, which should happen at some point.
(see existing docs about this.)  essentially, we have a way of
declaring that we inherit from some system, and the appropriate data
structures get created, perhaps just an extra inheritance pointer.  but
when we create the coding system, the extra data needs to be a stretchy
array of offsets, pointing to the type-specific data for the coding
system type and all its parents.  that means that in the methods
structure for a coding system (which perhaps should be expanded beyond
method, it's just a "class structure") is the index in these arrays of
offsets.  `CODING_SYSTEM_DATA()' can take any of the coding system
classes (rename type to class!) that make up this class.  similarly, a
coding system class inherits its methods from the class above unless
specifying its own method, and can call the superclass method at any
point by either just invoking its name, or conceivably by some macro
like

   `CALL_SUPER (method, (args))'

   similar mods would have to be made to coding stream structures.

   perhaps for the immediate we can just sort of fake things like we
currently do with undecided calling some stuff from chain.

oct 2, 2001
===========

need to implement support for iso-8859-15, i.e. iso-8859-1 + euro
symbol.  figure out how to fall back to iso-8859-1 as necessary.

   leave the current bindings the way they are for the moment, but bump
off `M-home' and `M-end' (hardly used), and substitute my buffer
movement stuff there. [DONE, but test]

   there's something to be said for combining block of 6 and paragraph,
esp. if we make the definition of "paragraph" be so that it skips by 6
when within code.  hmm.

   eliminate `advertised-undo' crap, and similar hacks. [DONE]

   think about obsolete stuff to be eliminated.  think about
eliminating or dimming obsolete items from `hyper-apropos' and
something similar in completion buffers.

sep 30, 2001
============

synched up the tutorials with FSF 21.0.105.  was rewriting them to favor
the cursor keys over the older `C-p', etc. keys.

   Got thinking about key bindings again.

  1. I think that `M-up/down' and `M-C-up/down' should be reversed.  I
     use scroll-up/down much more often than motion by paragraph.

  2. Should we eliminate move by block (of 6) and subsitute it for
     paragraph?  This would have the advantage that I could make
     bindings for buffer change (forward/back buffer, perhaps
     `M-C-up/down'.  with shift, `M-C-S-up/down' only goes within the
     same type (C files, etc.).  alternatively, just bump off
     `beginning-of-defun' from `C-M-home', since it's on `C-M-a'
     already.

   need someone to go over the other tutorials (five new ones, from FSF
21.0.105) and fix them up to correspond to the english one.

   shouldn't shift-motion work with `C-a' and such as well as arrows?

sep 29, 2001
============

`charcount_to_bytecount' can also be made to scream - as can
`scan_buffer', `buffer_mule_signal_inserted_region', others?  we should
start profiling though before going too far down this line.

   Debug code that causes no slowdown should in general remain in the
executable even in the release version because it may be useful (e.g.
for people to see the event output).  so `DEBUG_XEMACS' should be
rethought.  things like use of `msvcrtd.dll' should be controlled by
error_checking on.  maybe `DEBUG_XEMACS' controls general debug code
(e.g. use of `msvcrtd.dll', asserts abort, error checking), and the
actual debugging code should remain always, or be conditonalized on
something else (e.g. `DEBUGGING_FUNS_PRESENT').

   doc strings in dumped files are displayed with an extra blank line
between each line.  presumably this is recent?  i assume either the
change to detect-coding-region or the double-wrapping mentioned below.

   error with `coding-system-property' on `iso-2022-jp-dos'.  problem
is that that coding system is wrapped, so its type shows up as `chain',
not `iso-2022'.  this is a general problem, and i think the way to fix
it is to in essence do late canonicalization - similar in spirit to
what was done long ago, `canonicalize_when_code', except that the new
coding system (the wrapper) is created only once, either when the
original cs is created or when first needed.  this way, operations on
the coding system work like expected, and you get the same results as
currently when decoding/encoding.  the only thing tricky is handling
`canonicalize-after-coding' and the ever-tricky double-wrapping problem
mentioned below.  i think the proper solution is to move the
autodetection of eol into the main autodetect type.  it can be asked to
autodetect eol, coding, or both.  for just coding, it does like it
currently does.  for just eol, it does similar to what it currently does
but runs the detection code that `convert-eol' currently does, and
selects the appropriate `convert-eol' system.  when it does both eol
and coding, it does something on the order of creating two more
autodetect coding systems, one for eol only and one for coding only, and
chains them together.  when each has detected the appropriate value, the
results are combined.  this automatically eliminates the double-wrapping
problem, removes the need for complicated `canonicalize-after-coding'
stuff in chain, and fixes the problem of autodetect not having a
seekable stream because hidden inside of a chain. (we presume that in
the both-eol-and-coding case, the various autodetect coding streams can
communicate with each other appropriately.)

   also, we should solve the problem of internal coding systems floating
around and clogging up the list simply by having an "internal" property
on cs's and an internal param to `coding-system-list' (optional; if not
given, you don't get the internal ones). [DONE]

   we should try to reduce the size of the from-unicode tables (the
dominant memory hog in the tables).  one obvious thing is to not store
a whole emchar as the mapped-to value, but a short that encodes the
octets. [DONE]

sep 28, 2001
============

need to merge up to latest in trunk.

   add unicode charsets for all non-translatable unicode chars; probably
want to extend the concept of charsets to allow for dimension 3 and
dimension 4 charsets.  for the moment we should stick with just
dimension 3 charsets; otherwise we run past the current maximum of 4
bytes per emchar. (most code would work automatically since it uses`
MAX_EMCHAR_LEN'; the trickiness is in certain code that has intimate
knowledge of the representation.  e.g. `bufpos_to_bytind()' has to
multiply or divide by 1, 2, 3, or 4, and has special ways of handling
each number.  with 5 or 6 bytes per char, we'd have to change that code
in various ways.) 96x96x96 = 884,000 or so, so with two 96x96x96
charsets, we could tackle all Unicode values representable by UTF-16
and then some - and only these codepoints will ever have assigned
chars, as far as we know.

   need an easy way of showing the current language environment.  some
menus need to have the current one checked or whatever. [DONE]

   implement unicode surrogates.

   implement `buffer-file-coding-system-when-loaded' - make sure
`find-file', `revert-file', etc. set the coding system [DONE]

   verify all the menu stuff [DONE]

   implemented the entirely-ascii check in buffers.  not sure how much
gain it'll get us as we already have a known range inside of which is
constant time, and with pure-ascii files the known range spans the whole
buffer.  improved the comment about how `bufpos-to-bytind' and
vice-versa work. [DONE]

   fix double-wrapping of `convert-eol': when undecided converts itself
to something with a non-autodetect eol, it needs to tell the adjacent
`convert-eol' to reduce itself to nothing.

   need menu item for find file with specified encoding. [DONE]

   renamed coding systems mswindows-### to windows-### to follow the
standard in rfc1345. [DONE]

   implemented `coding-system-subsidiary-parent' [DONE] `HAVE_MULE' ->
`MULE' in files in `nt/' so that depend checking works [DONE]

   need to take the smarter `search-all-files-in-dir' stuff from my
sample init file and put it on the grep menu [DONE]

   added item for revert w/specified encoding; mostly works, but needs
fixes.  in particular, you get the correct results, but
`buffer-file-coding-system' does not reflect things right.  also, there
are too many entries.  need to split into submenus.  there is already
split code out there; see if it's generalized and if not make it so.
it should only split when there's more than a specified number, and
when splitting, split into groups of a specified size, not into a
specified number of groups. [DONE]

   too many entries in the langenv menus; need to split. [DONE]

sep 27, 2001
============

NOTE: `M-x grep' for make-string causes crash now.  something
definitely to do with string changes.  check very carefully the diffs
and put in those sledgehammer checks. [DONE]

   fix font-lock bug i introduced. [DONE]

   added optimization to strings (keeps track of # of bytes of ascii at
the beginning of a string).  perhaps should also keep an all-ascii flag
to deal with really large (> 2 MB) strings.  rewrite code to count
ascii-begin to use the 4-or-8-at-a-time stuff in
`bytecount_to_charcount'.

   Error: `M-q' is causing Invalid Regexp error on the above paragraph.
It's not in working.  I assume it's a side effect of the string stuff.
VERIFY!  Write sledgehammer checks for strings. [DONE]

   revamped the locale/init stuff so that it tries much harder to get
things right.  should test a bit more.  in particular, test out
Describe Language on the various created environments and make sure
everything looks right.

   should change the menus: move the submenus on `Edit->Mule' directly
under `Edit'.  add a menu entry on `File' to say "Reload with specified
encoding ->".  [DONE]

   Also `Find File' with specified encoding -> Also entry to change the
EOL settings for Unix, and implement it.

   `decode-coding-region' isn't working because it needs to insert a
binary (char->byte) converter. [DONE]

   chain should be rearranged to be in decoding order; similar for
source/sink-type, other things?

   the detector should check for a magic cookie even without a seekable
input.  (currently its input is not seekable, because it's hidden
within a chain.  #### See what we can do about this.)

   provide a way to display various settings, e.g. the current category
mappings and priority (see mule-diag; get this working so it's in the
path); also a way to print out the likeliness results from a detection,
perhaps a debug flag.

   problem with `env', which causes path issues due to `env' in
packages.  move env code to process, sync with fsf 21.0.105, check that
the autoloads in `env' don't cause problems. [DONE]

   8-bit iso2022 detection appears broken; or at least, mule-canna.c is
not so detected.

sep 25, 2001
============

something else to do is review the font selection and fix it so that
(e.g.)  JISX-0212 can be displayed.

   also, text in widgets needs to be drawn by us so that the correct
fonts will be displayed even in multi-lingual text.

sep 24, 2001
============

the detection system is now properly abstracted.  the detectors have
been rewritten to include multiple levels of abstraction.  now we just
need detectors for ascii, binary, and latin-x, as well as more
sophisticated detectors in general and further review of the general
algorithm for doing detection. (#### Is this written up anywhere?)
after that, consider adding error-checking to decoding (VERY IMPORTANT)
and verifying the binary correctness of things under unix no-mule.

sep 23, 2001
============

began to fix the detection system - adding multiple levels of likelihood
and properly abstracting the detectors.  the system is in place except
for the abstraction of the detector-specific data out of the struct
detection_state.  we should get things working first before tackling
that (which should not be too hard).  i'm rewriting algorithms here
rather than just converting code, so it's harder.  mostly done with
everything, but i need to review all detectors except iso2022 and make
them properly follow the new way.  also write a no-conversion detector.
also need to look into the `recode' package and see how (if?) they
handle detection, and maybe copy some of the algorithms.  also look at
recent FSF 21.0 and see if their algorithms have improved.

sep 22, 2001
============

   * fixed gc bugs from yesterday.

   * fixed truename bug.

   * close/finalize stuff works.

   * eliminated notyet stuff in syswindows.h.

   * eliminated special code in tstr_to_c_string.

   * fixed pdump problems. (many of them, mostly latent bugs, ugh)

   * fixed cygwin `sscanf' problems in
     `parse-unicode-translation-table'. (NOT a `sscanf' bug, but subtly
     different behavior w.r.t. whitespace in the format string,
     combined with a debugger that sucks ROCKS!! and consistently
     outputs garbage for variable values.)

   main stuff to test is the handling of EOF recognition vs. binary
(i.e. check what the default settings are under Unix).  then we may have
something that WORKS on all platforms!!!  (Also need to test Windows
non-Mule)

sep 21, 2001
============

finished redoing the close/finalize stuff in the lstream code.  but i
encountered again the nasty bug mentioned on sep 15 that disappeared on
its own then.  the problem seems to be that the finalize method of some
of the lstreams is calling `Lstream_delete()', which calls
`free_managed_lcrecord()', which is a no-no when we're inside of
garbage-collection and the object passed to `free_managed_lcrecord()'
is unmarked, and about to be released by the gc mechanism - the free
lists will end up with `xfree()'d objects on them, which is very bad.
we need to modify `free_managed_lcrecord()' to check if we're in gc and
the object is unmarked, and ignore it rather than move it to the free
list. [DONE]

   (#### What we really need to do is do what Java and C# do w.r.t.
their finalize methods: For objects with finalizers, when they're about
to be freed, leave them marked, run the finalizer, and set another bit
on them indicating that the finalizer has run.  Next GC cycle, the
objects will again come up for freeing, and this time the sweeper
notices that the finalize method has already been called, and frees
them for good (provided that a finalize method didn't do something to
make the object alive again).)

sep 20, 2001
============

redid the lstream code so there is only one coding stream.  combined the
various doubled coding stream methods into one; i'm a little bit unsure
of this last part, though, as the results of combining the two together
seem unclean.  got it to compile, but it crashes in loadup.  need to go
through and rehash the close vs. finalize stuff, as the problem was
stuff getting freed too quickly, before the canonicalize-after-decoding
was run.  should eliminate entirely `CODING_STATE_END' and use a
different method (close coding stream).  rewrite to use these two.  make
sure they're called in the right places.  `Lstream_close' on a stream
should *NOT* do finalizing.  finalize only on delete. [DONE]

   in general i'd like to see the flags eliminated and converted to
bit-fields.  also, rewriting the methods to take advantage of rejecting
should make it possible to eliminate much of the state in the various
methods, esp. including the flags.  need to test this is working,
though - reduce the buffer size down very low and try files with only
CRLF's in them, with one offset by a byte from the other, and see if we
correctly handle rejection.

   still have the problem with incorrectly truenaming files.

sep 19, 2001
============

bug reported: crash while closing lstreams.

   the lstream/coding system close code needs revamping.  we need to
document that order of closing lstreams is very important, and make
sure we're consistent.  furthermore, chain and undecided lstreams need
to close their underneath lstreams when they receive the EOF signal
(there may be data in the underneath streams waiting to come out), not
when they themselves are closed. [DONE]

   (if only we had proper inheritance.  i think in any case we should
simulate it for the chain coding stream - write things in such a way
that undecided can use the chain coding stream and not have to duplicate
anything itself.)

   in general we need to carefully think through the closing process to
make sure everything always works correctly and in the right order.
also check very carefully to make sure there are no dangling pointers
to deleted objects floating around.

   move the docs for the lstream functions to the functions themselves,
not the header files.  document more carefully what exactly
`Lstream_delete()' means and how it's used, what the connections are
between `Lstream_close('), `Lstream_delete()', `Lstream_flush()',
`lstream_finalize', etc. [DONE]

   additional error-checking: consider deadbeefing the memory in objects
stored in lcrecord free lists; furthermore, consider whether lifo or
fifo is correct; under error-checking, we should perhaps be doing fifo,
and setting a minimum number of objects on the lists that's quite large
so that it's highly likely that any erroneous accesses to freed objects
will go into such deadbeefed memory and cause crashes.  also, at the
earliest available opportunity, go through all freed memory and check
for any consistency failures (overwrites of the deadbeef), crashing if
so.  perhaps we could have some sort of id for each block, to easier
trace where the offending block came from.  (all of these ideas are
present in the debug system malloc from VC++, plus more stuff.)  there's
similar code i wrote sitting somewhere (in `free-hook.c'? doesn't
appear so.  we need to delete the blocking stuff out of there!).  also
look into using the debug system malloc from VC++, which has lots of
cool stuff in it.  we even have the sources.  that means compiling under
pdump, which would be a good idea anyway.  set it as the default. (but
then, we need to remove the requirement that Xpm be a DLL, which is
extremely annoying.  look into this.)

   test the windows code page coding systems recently created.

   problems reading my mail files - 1personal appears to hang, others
come up with lots of ^M's.  investigate.

   test the enum functions i just wrote, and finish them.

   still pdump problems.

sep 18, 2001
============

critical-quit broken sometime after aug 25.

   * fixed critical quit.

   * fixed process problems.

   * print routines work. (no routine for ccl, though)

   * can read and write unicode files, and they can still be read by
     some other program

   * defaults should come up correctly - mswindows-multibyte is general.

   still need to test matej's stuff.  seems ok with multibyte stuff but
needs more testing.

sep 17, 2001
============

!!!!! something broken with processes !!!!! cannot send mail anymore.
must investigate.

sep 17, 2001
============

on mon/wed nights, stop *BEFORE* 11pm.  Otherwise i just start getting
woozy and can't concentrate.

   just finished getting assorted fixups to the main branch committed,
so it will compile under C++ (Andy committed some code that broke C++
builds).  cup'd the code into the fixtypes workspace, updated the tags
appropriately.  i've created the appropriate log message, sitting in
fixtypes.txt in /src/xemacs; perhaps it should go into a README.  now i
just have to build on everything (it's currently building), verify it's
ok, run patcher-mail, commit, send.

   my mule ws is also very close.  need to:

   * test the new print routines.

   * test it can read and write unicode files, and they can still be
     read by some other program.

   * try to see if unicode can be auto-detected properly.

   * test it can read and write multibyte files in a few different
     formats.  currently can't recognize them, but if you set the cs
     right, it should work.

   * examine the test files sent by matej and see if we can handle them.

sep 15, 2001
============

more eol fixing.  this stuff is utter crap.

   currently we wrap coding systems with `convert-eol-autodetect' when
we create them in `make_coding_system_1'.  i had a feeling that this
would be a problem, and indeed it is - when autodetecting with
`undecided', for example, we end up with multiple layers of eol
conversion.  to avoid this, we need to do the eol wrapping *ONLY* when
we actually retrieve a coding system in places such as
`insert-file-contents'.  these places are `insert-file-contents', load,
process input, `call-process-internal',
`encode/decode/detect-coding-region', database input, ...

   (later) it's fixed, and things basically work.  NOTE: for some
reason, adding code to wrap coding systems with `convert-eol-lf' when
`eol-type == lf' results in crashing during garbage collection in some
pretty obscure place - an lstream is free when it shouldn't be.  this
is a bad sign.  i guess something might be getting initialized too
early?

   we still need to fix the canonicalization-after-decoding code to
avoid problems with coding systems like `internal-7' showing up.
basically, when `eol==lf' is detected, nil should be returned, and the
callers should handle it appropriately, eliding when necessary.  chain
needs to recognize when it's got only one (or even 0) items in the
chain, and elide out the chain.

sep 11, 2001: the day that will live in infamy
==============================================

rewrite of sep 9 entry about formats:

   when calling `make-coding-system', the name can be a cons of
`(format1 .  format2)', specifying that it decodes `format1->format2'
and encodes the other way.  if only one name is given, that is assumed
to be `format1', and the other is either `external' or `internal'
depending on the end type.  normally the user when decoding gives the
decoding order in formats, but can leave off the last one, `internal',
which is assumed.  a multichain might look like gzip|multibyte|unicode,
using the coding systems named `gzip', `(unicode . multibyte)' and
`unicode'.  the way this actually works is by searching for
gzip->multibyte; if not found, look for gzip->external or
gzip->internal. (In general we automatically do conversion between
internal and external as necessary: thus gzip|crlf does the expected,
and maps to gzip->external, external->internal, crlf->internal, which
when fully specified would be
gzip|external:external|internal:crlf|internal - see below.)  To
forcibly fit together two converters that have explicitly specified and
incompatible names (say you have unicode->multibyte and
iso8859-1->ebcdic and you know that the multibyte and iso8859-1 in this
case are compatible), you can force-cast using :, like this:
ebcdic|iso8859-1:multibyte|unicode. (again, if you force-cast between
internal and external formats, the conversion happens automatically.)

sep 10, 2001
============

moved the autodetection stuff (both codesys and eol) into particular
coding systems - `undecided' and `convert-eol' (type == `autodetect').
needs lots of work.  still need to search through the rest of the code
and find any remaining auto-detect code and move it into the undecided
coding system.  need to modify make-coding-system so that it spits out
auto-detecting versions of all text-file coding systems unless we say
not to.  need eliminate entirely the EOF flag from both the stream info
and the coding system; have only the original-eof flag.  in
coding_system_from_mask, need to check that the returned value is not of
type `undecided', falling back to no-conversion if so.  also need to
make sure we wrap everything appropriate for text-files - i removed the
wrapping on set-coding-category-list or whatever (need to check all
those files to make sure all wrapping is removed).  need to review
carefully the new code in `undecided' to make sure it works are
preserves the same logic as previously.  need to review the closing and
rewinding behavior of chain and undecided (same - should really
consolidate into helper routines, so that any coding system can embed a
chain in it) - make sure the dynarr's are getting their data flushed
out as necessary, rewound/closed in the right order, no missing steps,
etc.

   also split out mule stuff into `mule-coding.c'.  work done on
`configure'/`xemacs.mak'/`Makefile's not done yet.  work on
`emacs.c'/`symsinit.h' to interface with the new init functions not
done yet.

   also put in a few declarations of the way i think the abstracted
detection stuff ought to go.  DON'T WORK ON THIS MORE UNTIL THE REST IS
DEALT WITH AND WE HAVE A WORKING XEMACS AGAIN WITH ALL EOL ISSUES
NAILED.

   really need a version of `cvs-mods' that reports only the current
directory.  WRITE THIS!  use it to implement a better `cvs-checkin'.

sep 9, 2001
===========

implemented a gzip coding system.  unfortunately, doesn't quite work
right because it doesn't handle the gzip headers - it just reads and
writes raw zlib data.  there's no function in the library to skip past
the header, but we do have some code out of the library that we can
snarf that implements header parsing.  we need to snarf that, store it,
and output it again at the beginning when encoding.  in the process, we
should create a "get next byte" macro that bails out when there are no
more.  using this, we set up a nice way of doing most stuff statelessly
- if we have to bail, we reject everything back to the sync point.
also need to fix up the autodetection of zlib in configure.in.

   BIG problems with eol.  finished up everything i thought i would
need to get eol stuff working, but no - when you have
mswindows-unicode, with its eol set to autodetect, the detection
routines themselves do the autodetect (first), and fail (they report CR
on CRLF because of the NULL byte between the CR and the LF) since
they're not looking at ascii data.  with a chain it's similarly bad.
for mswindows-multibyte, for example, which is a chain
unicode->unicode-to-multibyte, autodetection happens inside of the
chain, both when unicode and unicode-to-multibyte are active.  we could
twiddle around with the eol flags to try to deal with this, but it's
gonna be a big mess, which is exactly what we're trying to avoid.  what
we basically want is to entirely rip out all EOL settings from either
the coding system or the stream (yes, there are two!  one might saw
autodetect, and then the stream contains the actual detected value).
instead, we simply create an eol-autodetect coding system - or rather,
it's part of the convert-eol coding system.  convert-eol, type =
autodetect, does autodetection the first time it gets data sent to it to
decode, and thereafter sets a stream parameter indicating the actual eol
type for this stream.  this means that all autodetect coding systems, as
created by `make-coding-system', really are chains with a convert-eol
at the beginning.  only subsidiary xxx-unix has no wrapping at all.
this should allow eof detection of gzip, unicode, etc.  for that
matter, general autodetection should be entirely encapsulated inside of
the `autodetect' coding system, with no eol-autodetection - the chain
becomes convert-eol (autodetect) -> autodetect or perhaps backwards.
the generic autodetect similarly has a coding-system in its stream
methods, and needs somehow or other to insert the detected
coding-system into the chain.  either it contains a chain inside of it
(perhaps it *IS* a chain), or there's some magic involving
canonicalization-type switcherooing in the middle of a decode.  either
way, once everything is good and done and we want to save the coding
system so it can be used later, we need to do another sort of
canonicalization - converting auto-detect-type coding systems into the
detected systems.  again, a coding-system method, with some magic
currently so that subsidiaries get properly used rather than something
that's new but equivalent to subsidiaries. (#### perhaps we could use a
hash table to avoid recreating coding systems when not necessary.  but
that would require that coding systems be immutable from external, and
i'm not sure that's the case.)

   i really think, after all, that i should reverse the naming of
everything in chain and source-sink-type - they should be
decoding-centric.  later on, if/when we come up with the proper way to
make it totally symmetrical, we'll be fine whether before then we were
encoding or decoding centric.

sep 9, 2001
===========

investigated eol parameter.

   implemented handling in `make-coding-system' of `eol-cr' and
`eol-crlf'.  fixed calls everywhere to `Fget_coding_system' /
`Ffind_coding_system' to reject non-char->byte coding systems.

   still need to handle "query eol type using coding-system-property"
so it magically returns the right type by parsing the chain.

   no work done on formats, as mentioned below.  we should consider
using : instead of || to indicate casting.

early sep 9, 2001
=================

renamed some codesys properties: `list' in chain -> chain; `subtype' in
unicode -> type.  everything compiles again and sort of works; some CRLF
problems that may resolve themselves when i finish the convert-eol
stuff.  the stuff to create subsidiaries has been rewritten to use
chains; but i still need to investigate how the EOL type parameter is
used.  also, still need to implement this: when a coding system is
created, and its eol type is not autodetect or lf, a chain needs to be
created and returned.  i think that what needs to happen is that the
eol type can only be set to autodetect or lf; later on this should be
changed to simply be either autodetect or not (but that would require
ripping out the eol converting stuff in the various coding systems),
and eventually we will do the work on the detection mechanism so it can
do chain detection; then we won't need an eol autodetect setting at
all.  i think there's a way to query the eol type of a coding system;
this should check to see if the coding system is a chain and there's a
convert-eol at the front; if so, the eol type comes from the type of
the convert-eol.

   also check out everywhere that `Fget_coding_system' or
`Ffind_coding_system' is called, and see whether anything but a
char->byte system can be tolerated.  create a new function for all the
places that only want char->byte, something like
`get_coding_system_char_to_byte_only'.

   think about specifying formats in make-coding-system.  perhaps the
name can be a cons of (format1, format2), specifying that it encodes
format1->format2 and decodes the other way.  if only one name is given,
that is assumed to be format2, and the other is either `byte' or `char'
depending on the end type.  normally the user when decoding gives the
decoding order in formats, but can leave off the last one, `char',
which is assumed.  perhaps we should say `internal' instead of `char'
and `external' instead of byte.  a multichain might look like
gzip|multibyte|unicode, using the coding systems named `gzip',
`(unicode . multibyte)' and `unicode'.  we would have to allow
something where one format is given only as generic byte/char or
internal/external to fit with any of the same byte/char type.  when
forcibly fitting together two converters that have explicitly specified
and incompatible names (say you have unicode->multibyte and
iso8859-1->ebcdic and you know that the multibyte and iso8859-1 in this
case are compatible), you can force-cast using ||, like this:
ebcdic|iso8859-1||multibyte|unicode.  this will also force
external->internal translation as necessary:
unicode|multibyte||crlf|internal does unicode->multibyte,
external->internal, crlf->internal.  perhaps you'd need to put in the
internal translation, like this:
unicode|multibyte|internal||crlf|internal, which means
unicode->multibyte, external->internal (multibyte is compatible with
external); force-cast to crlf format and convert crlf->internal.

even later: Sep 8, 2001
=======================

chain doesn't need to set character mode, that happens automatically
when the coding systems are created.  fixed chain to return correct
source/sink type for itself and to check the compatibility of
source/sink types in its chain.  fixed decode/encode-coding-region to
check the source and sink types of the coding system performing the
conversion and insert appropriate byte->char/char->byte converters (aka
"binary" coding system).  fixed set-coding-category-system to only
accept the traditional encode-char-to-byte types of coding systems.

   still need to extend chain to specify the parameters mentioned below,
esp. "reverse".  also need to extend the print mechanism for chain so it
prints out the chain.  probably this should be general: have a new
method to return all properties, and output those properties.  you
could also implement a read syntax for coding systems this way.

   still need to implement `convert-eol' and finish up the rest of the
eol stuff mentioned below.

later September 7, 2001 (more like Sep 8)
=========================================

moved many `Lisp_Coding_System *' params to `Lisp_Object'.  In general
this is the way to go, and if we ever implement a copying GC, we will
never want to be passing direct pointers around.  With no
error-checking, we lose no cycles using `Lisp_Object's in place of
pointers - the `Lisp_Object' itself is nothing but a pointer, and so
all the casts and "dereferences" boil down to nothing.

   Clarified and cleaned up the "character mode" on streams, and
documented who (caller or object itself) has the right to be setting
character mode on a stream, depending on whether it's a read or write
stream.  changed `conversion_end_type' method and `enum
source_sink_type' to return encoding-centric values, rather than
decoding-centric.  for the moment, we're going to be entirely
encoding-centric in everything; we can rethink later.  fixed coding
systems so that the decode and encode methods are guaranteed to receive
only full characters, if that's the source type of the data, as per
conversion_end_type.

   still need to fix the chain method so that it correctly sets the
character mode on all the lstreams in it and checks the source/sink
types to be compatible.  also fix `decode-coding-string' and friends to
put the appropriate byte->character (i.e. `no-conversion') coding
systems on the ends as necessary so that the final ends are both
character.  also add to chain a parameter giving the ability to switch
the direction of conversion of any particular item in the chain (i.e.
swap encoding and decoding).  i think what we really want to do is
allow for arbitrary parameters to be put onto a particular coding
system in the chain, of which the only one so far is
swap-encode-decode.  don't need too much codage here for that, but make
the design extendable.

September 7, 2001
=================

just added a return value from the decode and encode methods of a coding
system, so that some of the data can get rejected.  fixed the calling
routines to handle this.  need to investigate when and whether the
coding lstream is set to character mode, so that the decode/encode
methods only get whole characters.  if not, we should do so, according
to the source type of these methods.  also need to implement the
convert_eol coding system, and fix the subsidiary coding systems (and
in general, any coding system where the eol type is specified and is
not LF) to be chains involving convert_eol.

   after everything is working, need to remove eol handling from
encode/decode methods and eventually consider rewriting (simplifying)
them given the reject ability.

September 5, 2001
=================

   * need to organize this.  get everything below into the TODO list.
     CVS the TODO list frequently so i can delete old stuff.  prioritize
     it!!!!!!!!!

   * move `README.ben-mule...' to `STATUS.ben-mule...'; use `README'
     for intro, overview of what's new, what's broken, how to use the
     features, etc.

   * need a global and local `coding-category-precedence' list, which
     get merged.

   * finished the BOM support.  also finished something not listed
     below, expansion to the auto-generator of Unicode-encapsulation to
     support bracketing code with `#if ... #endif', for Cygwin and MINGW
     problems, e.g.  This is tested; appears to work.

   * need to add more multibyte coding systems now that we have various
     properties to specify them.  need to add DEFUN's for mac-code-page
     and ebcdic-code-page for completeness.  need to rethink the whole
     way that the priority list works.  it will continue to be total
     junk until multiple levels of likeliness get implemented.

   * need to finish up the stuff about the various defaults. [need to
     investigate more generally where all the different default values
     are that control encoding. (there are six places or so.) need to
     list them in `make-coding-system' docs and put pointers elsewhere.
     [[[[#### what interface to specify that this default should be
     unicode?  a "Unicode" language environment seems too drastic, as
     the language environment controls much more.]]]] even skipping the
     Unicode stuff here, we need to survey and list the variables that
     control coding page behavior and determine how they need to be set
     for various possible scenarios:

        *   total binary: no detection at all.

        *   raw-text only: wants only autodetection of line endings,
          nothing else.

        *   "standard Windows environment": tries for Unicode, falls
          back on   code page encoding.

        *   some sort of East European environment, and Russian.

        *   some sort of standard Japanese Windows environment.

        *   standard Chinese Windows environments (traditional and
          simplified)

        *   various Unix environments (European, Japanese, Russian,
          etc.)

        *   Unicode support in all of these when it's reasonable

These really require multiple likelihood levels to be fully
implementable.  We should see what can be done ("gracefully fall back")
with single likelihood level.  need lots of testing.

   * need to fix the truename problem.

   * lots of testing: need to test all of the stuff above and below
     that's recently been implemented.

September 4, 2001
=================

mostly everything compiles.  currently there is a crash in
`parse-unicode-translation-table', and Cygwin/Mule won't run.  it may
well be a bug in the `sscanf()' in Cygwin.

   working on today:

   * adding BOM support for Unicode coding systems.  mostly there, but
     need to finish adding BOM support to the detection routines.  then
     test.

   * adding properties to `unicode-to-multibyte' to specify the coding
     system in various flexible ways, e.g. directly specified code page
     or ansi or oem code page of specified locale, current locale,
     user-default or system-default locale.  need to test.

   * creating a `multibyte' coding system, with the same parameters as
     unicode-to-multibyte and which resolves at coding-system-creation
     time to the appropriate chain.  creating the underlying mechanism
     to allow such under-the-scenes switcheroo.  need to test.

   * set default-value of `buffer-file-coding-system' to
     mswindows-multibyte, as Matej said it should be.  need to test.
     need to investigate more generally where all the different default
     values are that control encoding. (there are six places or so.)
     need to list them in make-coding-system docs and put pointers
     elsewhere. #### what interface to specify that this default should
     be unicode?  a "Unicode" language environment seems too drastic, as
     the language environment controls much more.

   * thinking about adding multiple levels of certainty to the detection
     schemes, instead of just a mask.  eventually, we need to totally
     abstract things, but that can easier be done in many steps. (we
     need multiple levels of likelihood to more reasonably support a
     Windows environment with code-page type files.  currently, in order
     to get them detected, we have to put them first, because they can
     look like lots of other things; but then, other encodings don't get
     detected.  with multiple levels of likelihood, we still put the
     code-page categories first, but they will return low levels of
     likelihood.  Lower-down encodings may be able to return higher
     levels of likelihood, and will get taken preferentially.)

   * making it so you cannot disable file-coding, but you get an
     equivalent default on Unix non-Mule systems where all defaults are
     `binary'.  need to test!!!!!!!!!

   Matej (mostly, + some others) notes the following problems, and here
are possible solutions:

   * he wants the defaults to work right. [figure out what those
     defaults are.  i presume they are auto-detection of data in current
     code page and in unicode, and new files have current code page set
     as their output encoding.]

   * too easy to lose data with incorrect encodings. [need to set up an
     error system for encoding/decoding.  extremely important but a
     little tricky to implement so let's deal with other issues now.]

   * EOL isn't always detected correctly. [#### ?? need examples]

   * truename isn't working: `c:\t.txt' and `c:\tmp.txt' have the same
     truename.  [should be easy to fix]

   * unicode files lose the BOM mark. [working on this]

   * command-line utilities use OEM. [actually it seems more
     complicated.  it seems they use the codepage of the console.  we
     may be able to set that, e.g. to UTF8, before we invoke a command.
     need to investigate.]

   * no way to handle unicode characters not recognized as charsets. [we
     need to create something like 8 private 2-dimensional charsets to
     handle all BMP Unicode chars.  Obviously this is a stopgap
     solution.  Switching to Unicode internal will ultimately make life
     far easier and remove the BMP limitation.  but for now it will
     work.  we translate all characters where we have charsets into
     chars in those charsets, and the remainder in a unicode charset.
     that way we can save them out again and guarantee no data loss with
     unicode.  this creates font problems, though ...]

   * problems with xemacs font handling. [xemacs font handling is not
     sophisticated enough.  it goes on a charset granularity basis and
     only looks for a font whose name contains the corresponding windows
     charset in it.  with unicode this fails in various ways.  for one
     the granularity needs to be single character, so that those unicode
     charsets mentioned above work; and it needs to query the font to
     see what unicode ranges it supports, rather than just looking at
     the charset ending.]

August 28, 2001
===============

working on getting everything to compile again: Cygwin, non-MULE,
pdump.  not there yet.

   `mswindows-multibyte' is now defined using chain, and works.
removed most vestiges of the `mswindows-multibyte' coding system type.

   file-coding is on by default; should default to binary only on Unix.
Need to test. (Needs to compile first :-)

August 26, 2001
===============

I've fixed the issue of inputting non-ASCII text under -nuni, and done
some of the work on the Russian <C-x> problem - we now compute the
other possibilities.  We still need to fix the key-lookup code, though,
and that code is unfortunately a bit ugly.  the best way, it seems, is
to expand the command-builder structure so you can specify different
interpretations for keys. (if we do find an alternative binding, though,
we need to mess with both the command builder and this-command-keys, as
does the function-key stuff.  probably need to abstract that munging
code.)

   high-priority:

*[currently doing]*
        * support for `WM_IME_CHAR'.  IME input can work under `-nuni'
          if we use `WM_IME_CHAR'.  probably we should always be using
          this, instead of snarfing input using `WM_COMPOSITION'.  i'll
          check this out.

        * Russian <C-x> problem.  see above.

*[clean-up]*
        * make sure it compiles and runs under non-mule.  remember that
          some code needs the unicode support, or at least a simple
          version of it.

        * make sure it compiles and runs under pdump.  see below.

        * clean up `mswindows-multibyte', `TSTR_TO_C_STRING'.  see
          below. [DONE]

        * eliminate last vestiges of codepage<->charset conversion and
          similar stuff.

*[other]*
        * cut and paste.  see below.

        * misc issues with handling lang environments.  see also August
          25, "finally: working on the C-x in ...".
             *   when switching lang env, needs to set keyboard layout.

             *   user var to control whether, when moving into text of a
                particular language, we set the appropriate keyboard
               layout.  we   would need to have a lisp api for
               retrieving and setting the   keyboard layout, set text
               properties to indicate the layout of   text, and have a
               way of dealing with text with no property on   it. (e.g.
               saved text has no text properties on it.) basically,
               we need to get a keyboard layout from a charset; getting
               a   language would do.  Perhaps we need a table that
               maps charsets   to language environments.

             *   test that the lang env is properly set at startup.
               test that   switching the lang env properly sets the C
               locale (call   setlocale(), set LANG, etc.) - a spawned
               subprogram should have   the new locale in its
               environment.

        * look through everything below and see if anything is missed
          in this priority list, and if so add it.  create a separate
          file for the priority list, so it can be updated as
          appropriate.

   mid-priority:

   * clean up the chain coding system.  its list should specify decode
     order, not encode; i now think this way is more logical.  it should
     check the endpoints to make sure they make sense.  it should also
     allow for the specification of "reverse-direction coding systems":
     use the specified coding system, but invert the sense of decode and
     encode.

   * along with that, places that take an arbitrary coding system and
     expect the ends to be anything specific need to check this, and add
     the appropriate conversions from byte->char or char->byte.

   * get some support for arabic, thai, vietnamese, japanese jisx 0212:
     at least get the unicode information in place and make sure we have
     things tied together so that we can display them.  worry about r2l
     some other time.

August 25, 2001
===============

There is actually more non-Unicode-ized stuff, but it's basically
inconsequential. (See previous note.) You can check using the file
nmkun.txt (#### RENAME), which is just a list of all the routines that
have been split. (It was generated from the output of `nmake
unicode-encapsulate', after removing everything from the output but the
function names.) Use something like

     grep -F -f ../nmkun.txt -w [a-hj-z]*.[ch]  |m

   in the source directory, which does a word match and skips
`intl-unicode-win32.[ch]' and `intl-win32.[ch]', which have a whole lot
of references to these, unavoidably.  It effectively detects what needs
to be changed because changed versions either begin `qxe...' or end
with A or W, and in each case there's no whole-word match.

   The nasty bug has been fixed below.  The `-nuni' option now works -
all specially-written code to handle the encapsulation has been tested
by some operation (fonts by loadup and checking the output of
`(list-fonts "")'; devmode by printing; dragdrop tests other stuff).

   NOTE: for `-nuni' (Win 95), areas need work:

   * cut and paste.  we should be able to receive Unicode text if it's
     there, and we should be able to receive it even in Win 95 or
     `-nuni'.  we should just check in all circumstances.  also, under
     95, when we put some text in the clipboard, it may or may not also
     be automatically enumerated as unicode.  we need to test this out
     and/or just go ahead and manually do the unicode enumeration.

   * receiving keyboard input.  we get only a single byte, but we should
     be able to correlate the language of the keyboard layout to a
     particular code page, so we can then decode it correctly.

   * `mswindows-multibyte'.  still implemented as its own thing.  should
     be done as a chain of (encoding) unicode | unicode-to-multibyte.
     need to turn this on, get it working, and look into optimizations
     in the dfc stuff. (#### perhaps there's a general way to do these
     optimizations???  something like having a method on a coding
     system that can specify whether a pure-ASCII string gets rendered
     as pure-ASCII bytes and vice-versa.)

   ALSO:

   * we have special macros `TSTR_TO_C_STRING' and such because formerly
     the `DFC' macros didn't know about external stuff that was Unicode
     encoded and would call `strlen()' on them.  this is fixed, so now
     we should undo the special macros, make em normal, removal the
     comments about this, and make sure it works. [DONE]

   * finally: working on the `C-x' in Russian key layout problem.  in
     the process will probably end up doing work on cleaning up the
     handling of keyboard layouts, integrating or deleting the FSF
     stuff, adding code to change the keyboard layout as we move in and
     out of text in different languages (implemented as a
     post-command-hook; we need something like
     internal-post-command-hook if not already there, for internal
     stuff that doesn't want to get mixed up with the regular
     post-command-hook; similar for pre-command-hook).  also, when
     langenv changes, ways to set the keyboard layout appropriately.

   * i think the stuff above is higher priority than the other stuff
     mentioned below.  what i'm aiming for is to be able to input and
     work with multiple languages without weird glitches, both under 95
     and NT.  the problems above are all basic impediments to such work.
     we assume for the moment that the user can make use of the existing
     file i/o conversion stuff, and put that lower in priority, after
     the basic input is working.

   * i should get my modem connected and write up what's going on and
     send it to the lists; also cvs commit my workspaces and get more
     testers.

   August 24, 2001:

   All code has been Unicode-ized except for some stuff in console-msw.c
that deals with console output.  Much of the Unicode-encapsulation
stuff, particularly the hand-written stuff, really needs testing.  I
added a new command-line option, `-nuni', to force use of all ANSI
calls - `XE_UNICODEP' evaluates to false in this case.

   There is a nasty bug that appeared recently, probably when the event
code got Unicode-ized - bad interactions with OS sticky modifiers.
Hold the shift key down and release it, then instead of affecting the
next char only, it gets permanently stuck on (until you do a regular
shift+char stroke).  This needs to be debugged.

   Other things on agenda:

   * go through and prioritize what's listed below.

   * make sure the pdump code can compile and work.  for the moment we
     just don't try to dump any Unicode tables and load them up each
     time.  this is certainly fast but ...

   * there's the problem that XEmacs can't be run in a directory with
     non-ASCII/Latin-1 chars in it, since it will be doing Unicode
     processing before we've had a chance to load the tables.  In fact,
     even finding the tables in such a situation is problematic using
     the normal commands.  my idea is to eventually load the stuff
     extremely extremely early, at the same time as the pdump data gets
     loaded.  in fact, the unicode table data (stored in an efficient
     binary format) can even be stuck into the pdump file (which would
     mean as a resource to the executable, for windows).  we'd need to
     extend pdump a bit: to allow for attaching extra data to the pdump
     file. (something like `pdump_attach_extra_data (addr, length)'
     returns a number of some sort, an index into the file, which you
     can then retrieve with `pdump_load_extra_data()', which returns an
     addr (`mmap()'ed or loaded), and later you
     `pdump_unload_extra_data()' when finished.  we'd probably also need
     `pdump_attach_extra_data_append()', which appends data to the data
     just written out with `pdump_attach_extra_data()'.  this way,
     multiple tables in memory can be written out into one contiguous
     table. (we'd use the tar-like trick of allowing new blocks to be
     written without going back to change the old blocks - we just rely
     on the end of file/end of memory.) this same mechanism could be
     extracted out of pdump and used to handle the non-pdump situation
     (or alternatively, we could just dump either the memory image of
     the tables themselves or the compressed binary version).  in the
     case of extra unicode tables not known about at compile time that
     get loaded before dumping, we either just dump them into the image
     (pdump and all) or extract them into the compressed binary format,
     free the original tables, and treat them like all other tables.

   * `C-x b' when using a Russian keyboard layout.  XEmacs currently
     tries to interpret `C+cyrillic char', which causes an error.  We
     want `C-x b' to still work even when the keyboard normally
     generates Cyrillic.  What we should do is expand the keyboard
     event structure so that it contains not only the actual char, but
     what the char would have been in various other keyboard layouts,
     and in contexts where only certain keystrokes make sense (creating
     control chars, and looking up in keymaps), we proceed in order,
     processing each of them until we get something.  order should be
     something like: current keyboard layout; layout of the current
     language environment; layout of the user's default language;
     layout of the system default language; layout of US English.

   * reading and writing Unicode files.  multiple problems:

        *   EOL's aren't handled right.  for the moment, just fix the
          Unicode coding systems; later on, create EOL-only coding
          systems:

            1.     they would be character->character and operate next
               to the     internal data; this means that coding systems
               need to be able     to handle ends of lines that are
               either CR, LF, or CRLF.      usually this isn't a
               problem, as they are just characters     like any other
               and get encoded appropriately.  however,     coding
               systems that are line-oriented need to recognize any
               of the three as line endings.

            2.     we'd also have to complete the stuff that handles
               coding     systems where either end can be byte or char
               (four     possibilities total; use a single enum such as
                  `ENCODES_CHAR_TO_BYTE', `ENCODES_BYTE_TO_BYTE',
               etc.).

            3.     we'd need ways of specifying the chaining of coding
               systems.      e.g. when reading a coding system, a user
               can specify more     than one with a | symbol between
               them.  when a context calls     for a coding system and
               a chain is needed, the `chain' coding     system is
               useful; but we should really expand the contexts
               where a list of coding systems can be given, and whenever
                  possible try to inline the chain instead of using a
                 surrounding `chain' coding system.

            4.     the `chain' needs some work so that it passes all
               sorts of     lstream commands down to the chain inside
               it - it should be     entirely transparent and the fact
               that there's actually a     surrounding coding system
               should be invisible.  more general     coding system
               methods might need to be created.

            5.     important: we need a way of specifying how detecting
               works     when we have more than one coding system.  we
               might need more     than a single priority list.  need
               to think about this.

        *   Unicode files beginning with the BOM are not recognized as
          such.    we need to fix this; but to make things sensible, we
          really need   to add the idea of different levels of
          confidence regarding   what's detected.  otherwise, Unicode
          says "yes this is me" but   others higher up do too.  in the
          process we should probably   finish abstracting the detection
          system and fix up some   stupidities in it.

        *   When writing a file, we need error detection; otherwise
          somebody   will create a Unicode file without realizing the
          coding system   of the buffer is Raw, and then lose all the
          non-ASCII/Latin-1   text when it's written out.  We need two
          levels

            1.     first, a "safe-charset" level that checks before any
               actual     encoding to see if all characters in the
               document can safely     be represented using the given
               coding system.  FSF has a     "safe-charset" property of
               coding systems, but it's stupid     because this
               information can be automatically derived from     the
               coding system, at least the vast majority of the time.
                 What we need is some sort of
               alternative-coding-system-precedence-list,
               langenv-specific,     where everything on it can be
               checked for safe charsets and     then the user given a
               list of possibilities.  When the user     does "save
               with specified encoding", they should see the same
               precedence list.  Again like with other precedence lists,
                  there's also a global one, and presumably all coding
               systems     not on other list get appended to the end
               (and perhaps not     checked at all when doing
               safe-checking?).  safe-checking     should work
               something like this: compile a list of all     charsets
               used in the buffer, along with a count of chars
               used.  that way, "slightly unsafe" charsets can perhaps
               be     presented at the end, which will lose only a few
               characters     and are perhaps what the users were
               looking for.

            2.     when actually writing out, we need error checking in
               case an     individual char in a charset can't be
               written even though the     charsets are safe.  again,
               the user gets the choice of other     reasonable coding
               systems.

            3.     same thing (error checking, list of alternatives,
               etc.) needs     to happen when reading!  all of this
               will be a lot of work!


Announcement, August 20, 2001:
==============================

I'm looking for testers.  There is a complete and fast implementation
in C of Unicode conversion, translations for almost all of the
standardly-defined charsets that load up automatically and
instantaneously at runtime, coding systems supporting the common
external representations of Unicode [utf-16, ucs-4, utf-8,
little-endian versions of utf-16 and ucs-4; utf-7 is sitting there with
abort[]s where the coding routines should go, just waiting for somebody
to implement], and a nice set of primitives for translating
characters<->codepoints and setting the priority lists used to control
codepoint->char lookup.

   It's so far hooked into one place: the Windows IME.  Currently I can
select the Japanese IME from the thing on my tray pad in the lower
right corner of the screen, and type Japanese into XEmacs, and you get
Japanese in XEmacs - regardless of whether you set either your current
or global system locale to Japanese,and regardless of whether you set
your XEmacs lang env as Japanese.  This should work for many other
languages, too - Cyrillic, Chinese either Traditional or Simplified,
and many others, but YMMV.  There may be some lurking bugs (hardly
surprising for something so raw).

   To get at this, checkout using `ben-mule-21-5', NOT the simpler
*`mule-21-5'.  For example

   cvs -d :pserver:xemacs@cvs.xemacs.org:/usr/CVSroot checkout -r
ben-mule-21-5  xemacs

   or you get the idea.  the `-r ben-mule-21-5' is important.

   I keep track of my progress in a file called README.ben-mule-21-5 in
the root directory of the source tree.

   WARNING: Pdump might not work. Will be fixed rsn.

August 20, 2001
===============

   * still need to sort out demand loading, binary format, etc.  figure
     out what the goals are and how we're going to achieve them.  for
     the moment let's just say that running XEmacs in a directory with
     Japanese or other weird characters in the name is likely to cause
     problems under MS Windows, but once XEmacs is initialized (and
     before processing init files), all Unicode support is there.

   * wrote the size computation routines, although not yet tested.

   * lots more abstraction of coding systems; almost done.

   * UNICODE WORKS!!!!!

August 19, 2001
===============

Still needed on the Unicode support:

   * demand loading: load the Unicode table data the first time a
     conversion needs to be done.

   * maybe: table size computation: figure out how big the in-memory
     tables actually are.

   * maybe: create a space-efficient binary format for the data, and a
     way to dump out an existing charset's data into this binary format.
     it should allow for many such groups of data to be appended
     together in one file, such that you can just append the new data
     onto the end and not have to go back and modify anything
     previously. (like how tar archives work, and how the UFS? for
     CD-R's and CD-RW's works.)

   * maybe: figure out how to be able to access the Unicode tables at
     `init_intl()' time, before we know how to get at data-directory;
     that way we can handle the need for unicode conversions that come
     up very early, for example if XEmacs is run from a directory
     containing Japanese in it.  Presumably we'd want to generalize the
     stuff in `pdump.c' that deals with the dumper file, so that it can
     handle other files - putting the file either in the directory of
     the executable or in a resource, maybe actually attached to the
     pdump file itself - or maybe we just dump the data into the actual
     executable.  With pdump we could extend pdump to allow for data
     that's in the pdump file but not actually mapped at startup,
     separate from the data that does get mapped - and then at runtime
     the pointer gets restored not with a real pointer but an offset
     into the file; another pdump call and we get some way to access
     the data. (tricky because it might be in a resource, not a file.
     we might have to just tell pdump to mmap or whatever the data in,
     and then tell pdump to release it.)

   * fix multibyte to use unicode.  at first, just reverse
     `mswindows-multibyte-to-unicode' to be `unicode-to-multibyte';
     later implement something in chain to allow for reversal, for
     declaring the ends of the coding systems, etc.

   * actually make sure that the IME stuff is working!!!

   Other things before announcing:

   * change so that the Unicode tables are not pdumped.  This means we
     need to free any table data out there.  Make sure that pdump
     compiles and try to finish the pretty-much-already-done stuff
     already with `XD_STRUCT_ARRAY' and dynamic size computation; just
     need to see what's going on with `LO_LINK'.

August 14, 2001
===============

To do a diff between this workspace and the mainline, use the most
recent sync tags, currently:

     cvs diff -r main-branch-ben-mule-21-5-aug-11-2001-sync -r ben-mule-21-5-post-aug-11-2001-sync

   Unicode support:

   Unicode support is important for supporting many languages under
Windows, such as Cyrillic, without resorting to translation tables for
particular Windows-specific code pages.  Internally, all characters in
Windows can be represented in two encodings: code pages and Unicode.
With Unicode support, we can seamlessly support all Windows characters.
Currently, the test in the drive to support Unicode is if IME input
works properly, since it is being converted from Unicode.

   Unicode support also requires that the various Windows API's be
"Unicode-encapsulated", so that they automatically call the ANSI or
Unicode version of the API call appropriately and handle the size
differences in structures.  What this means is:

   * first, note that Windows already provides a sort of encapsulation
     of all API's that deal with text.  All such API's are underlyingly
     provided in two versions, with an A or W suffix (ANSI or "wide"
     i.e. Unicode), and the compile-time constant UNICODE controls which
     is selected by the unsuffixed API.  Same thing happens with
     structures.  Unfortunately, this is compile-time only, not
     run-time, so not sufficient. (Creating the necessary run-time
     encoding is not conceptually difficult, but very time-consuming to
     write.  It adds no significant overhead, and the only reason it's
     not standard in Windows is conscious marketing attempts by
     Microsoft to cripple Windows 95.  FUCK MICROSOFT!  They even
     describe in a KnowledgeBase article exactly how to create such an
     API [although we don't exactly follow their procedure], and point
     out its usefulness; the procedure is also described more generally
     in Nadine Kano's book on Win32 internationalization - written SIX
     YEARS AGO!  Obviously Microsoft has such an API available
     internally.)

   * what we do is provide an encapsulation of each standard Windows API
     call that is split into A and W versions.  current theory is to
     avoid all preprocessor games; so we name the function with a prefix
     - "qxe" currently - and require callers to use the prefixed name.
     Callers need to explicitly use the W version of all structures, and
     convert text themselves using `Qmswindows_tstr'.  the qxe
     encapsulated version will automatically call the appropriate A or W
     version depending on whether we're running on 9x or NT, and copy
     data between W and A versions of the structures as necessary.

   * We require the caller to handle the actual translation of text to
     avoid possible overflow when dealing with fixed-size Windows
     structures.  There are no such problems when copying data between
     the A and W versions because ANSI text is never larger than its
     equivalent Unicode representation.

   * We allow for incremental creation of the encapsulated routines by
     using the coding system `Qmswindows_tstr_notyet'.  This is an
     alias for `Qmswindows_multibyte', i.e. it always converts to ANSI;
     but it indicates that it will be changed to `Qmswindows_tstr' when
     we have a qxe version of the API call that the data is being
     passed to and change the code to use the new function.

   Besides creating the encapsulation, the following needs to be done
for Unicode support:

   * No actual translation tables are fed into XEmacs.  We need to
     provide glue code to read the tables in `etc/unicode'.  See
     `etc/unicode/README' for the interface to implement.

   * Fix pdump.  The translation tables for Unicode characters function
     as unions of structures with different numbers of indirection
     levels, in order to be efficient.  pdump doesn't yet support such
     unions.  `charset.h' has a general description of how the
     translation tables work, and the pdump code has constants added
     for the new required data types, and descriptions of how these
     should work.

   * ultimately, there's no end to additional work (composition, bidi
     reordering, glyph shaping/ordering, etc.), but the above is enough
     to get basic translation working.

   Merging this workspace into the trunk requires some work.  ChangeLogs
have not yet been created.  Also, there is a lot of additional code in
this workspace other than just Windows and Unicode stuff.  Some of the
changes have been somewhat disruptive to the code base, in particular:

   * the code that handles the details of processing multilingual text
     has been consolidated to make it easier to extend it.  it has been
     yanked out of various files (`buffer.h', `mule-charset.h',
     `lisp.h', `insdel.c', `fns.c', `file-coding.c', etc.) and put into
     `text.c' and `text.h'.  `mule-charset.h' has also been renamed
     `charset.h'.  all long comments concerning the representations and
     their processing have been consolidated into `text.c'.

   * `nt/config.h' has been eliminated and everything in it merged into
     `config.h.in' and `s/windowsnt.h'.  see `config.h.in' for more
     info.

   * `s/windowsnt.h' has been completely rewritten, and `s/cygwin32.h'
     and `s/mingw32.h' have been largely rewritten.  tons of dead
     weight has been removed, and stuff common to more than one file
     has been isolated into `s/win32-common.h' and `s/win32-native.h',
     similar to what's already done for usg variants.

   * large amounts of code throughout the code base have been Mule-ized,
     not just Windows code.

   * `file-coding.c/.h' have been largely rewritten (although still
     mostly syncable); see below.

June 26, 2001
=============

ben-mule-21-5

   this contains all the mule work i've been doing.  this includes
mostly work done to get mule working under ms windows, but in the
process i've [of course] fixed a whole lot of other things as well,
mostly mule issues.  the specifics:

   * it compiles and runs under windows and should basically work.  the
     stuff remaining to do is (a) improved unicode support (see below)
     and (b) smarter handling of keyboard layouts.  in particular, it
     should (1) set the right keyboard layout when you change your
     language environment; (2) optionally (a user var) set the
     appropriate keyboard layout as you move the cursor into text in a
     particular language.

   * i added a bunch of code to better support OS locales.  it tries to
     notice your locale at startup and set the language environment
     accordingly (this more or less works), and call setlocale() and set
     LANG when you change the language environment (may or may not
     work).

   * major rewriting of file-coding.  it's mostly abstracted into coding
     systems that are defined by methods (similar to devices and
     specifiers), with the ultimate aim being to allow non-i18n coding
     systems such as gzip.  there is a "chain" coding system that allows
     multiple coding systems to be chained together. (it doesn't yet
     have the concept that either end of a coding system can be bytes or
     chars; this needs to be added.)

   * unicode support.  very raw.  a few days ago i wrote a complete and
     efficient implementation of unicode translation.  it should be very
     fast, and fairly memory-efficient in its tables.  it allows for
     charset priority lists, which should be language-environment
     specific (but i haven't yet written the glue code).  it works in
     preliminary testing, but obviously needs more testing and work.
     as of yet there is no translation data added for the standard
     charsets.  the tables are in etc/unicode, and all we need is a bit
     of glue code to process them.  see etc/unicode/README for the
     interface to implement.

   * support for unicode in windows is partly there.  this will work
     even on windows 95.  the basic model is implemented but it needs
     finishing up.

   * there is a preliminary implementation of windows ime support
     courtesy of ikeyama.

   * if you want to get cyrillic working under windows (it appears to
     "work" but the wrong chars currently appear), the best way is to
     add unicode support for iso-8859-5 and use it in redisplay-msw.c.
     we are already passing unicode codepoints to the text-draw routine
     (ExtTextOutW).  (ExtTextOutW and GetTextExtentPoint32W are
     implemented on both 95 and NT.)

   * i fixed the iso2022 handling so it will correctly read in files
     containing unknown charsets, creating a "temporary" charset which
     can later be overwritten by the real charset when it's defined.
     this allows iso2022 elisp files with literals in strange languages
     to compile correctly under mule.  i also added a hack that will
     correctly read in and write out the emacs-specific "composition"
     escape sequences, i.e. `ESC 0' through `ESC 4'.  this means that
     my workspace correctly compiles the new file `devanagari.el' that
     i added (see below).

   * i copied the remaining language-specific files from fsf.  i made
     some minor changes in certain cases but for the most part the stuff
     was just copied and may not work.

   * i fixed `post-read-conversion' in coding systems to follow fsf
     conventions. (i also support our convention, for the moment.  a
     kludge, of course.)

   * `make-coding-system' accepts (but ignores) the additional
     properties present in the fsf version, for compatibility.


File: internals.info,  Node: Consoles; Devices; Frames; Windows,  Next: The Redisplay Mechanism,  Prev: Multilingual Support,  Up: Top

26 Consoles; Devices; Frames; Windows
*************************************

* Menu:

* Introduction to Consoles; Devices; Frames; Windows::
* Point::
* Window Hierarchy::
* The Window Object::
* Creating a New Console/Device/Frame Type::
* Modules for the Basic Displayable Lisp Objects::


File: internals.info,  Node: Introduction to Consoles; Devices; Frames; Windows,  Next: Point,  Prev: Consoles; Devices; Frames; Windows,  Up: Consoles; Devices; Frames; Windows

26.1 Introduction to Consoles; Devices; Frames; Windows
=======================================================

A window-system window that you see on the screen is called a "frame"
in Emacs terminology.  Each frame is subdivided into one or more
non-overlapping panes, called (confusingly) "windows".  Each window
displays the text of a buffer in it. (See above on Buffers.) Note that
buffers and windows are independent entities: Two or more windows can
be displaying the same buffer (potentially in different locations), and
a buffer can be displayed in no windows.

   A single display screen that contains one or more frames is called a
"display".  Under most circumstances, there is only one display.
However, more than one display can exist, for example if you have a
"multi-headed" console, i.e. one with a single keyboard but multiple
displays. (Typically in such a situation, the various displays act like
one large display, in that the mouse is only in one of them at a time,
and moving the mouse off of one moves it into another.) In some cases,
the different displays will have different characteristics, e.g. one
color and one mono.

   XEmacs can display frames on multiple displays.  It can even deal
simultaneously with frames on multiple keyboards (called "consoles" in
XEmacs terminology).  Here is one case where this might be useful: You
are using XEmacs on your workstation at work, and leave it running.
Then you go home and dial in on a TTY line, and you can use the
already-running XEmacs process to display another frame on your local
TTY.

   Thus, there is a hierarchy console -> display -> frame -> window.
There is a separate Lisp object type for each of these four concepts.
Furthermore, there is logically a "selected console", "selected
display", "selected frame", and "selected window".  Each of these
objects is distinguished in various ways, such as being the default
object for various functions that act on objects of that type.  Note
that every containing object remembers the "selected" object among the
objects that it contains: e.g. not only is there a selected window, but
every frame remembers the last window in it that was selected, and
changing the selected frame causes the remembered window within it to
become the selected window.  Similar relationships apply for consoles
to devices and devices to frames.


File: internals.info,  Node: Point,  Next: Window Hierarchy,  Prev: Introduction to Consoles; Devices; Frames; Windows,  Up: Consoles; Devices; Frames; Windows

26.2 Point
==========

Recall that every buffer has a current insertion position, called
"point".  Now, two or more windows may be displaying the same buffer,
and the text cursor in the two windows (i.e. `point') can be in two
different places.  You may ask, how can that be, since each buffer has
only one value of `point'?  The answer is that each window also has a
value of `point' that is squirreled away in it.  There is only one
selected window, and the value of "point" in that buffer corresponds to
that window.  When the selected window is changed from one window to
another displaying the same buffer, the old value of `point' is stored
into the old window's "point" and the value of `point' from the new
window is retrieved and made the value of `point' in the buffer.  This
means that `window-point' for the selected window is potentially
inaccurate, and if you want to retrieve the correct value of `point'
for a window, you must special-case on the selected window and retrieve
the buffer's point instead.  This is related to why
`save-window-excursion' does not save the selected window's value of
`point'.


File: internals.info,  Node: Window Hierarchy,  Next: The Window Object,  Prev: Point,  Up: Consoles; Devices; Frames; Windows

26.3 Window Hierarchy
=====================

If a frame contains multiple windows (panes), they are always created
by splitting an existing window along the horizontal or vertical axis.
Terminology is a bit confusing here: to "split a window horizontally"
means to create two side-by-side windows, i.e. to make a _vertical_ cut
in a window.  Likewise, to "split a window vertically" means to create
two windows, one above the other, by making a _horizontal_ cut.

   If you split a window and then split again along the same axis, you
will end up with a number of panes all arranged along the same axis.
The precise way in which the splits were made should not be important,
and this is reflected internally.  Internally, all windows are arranged
in a tree, consisting of two types of windows, "combination" windows
(which have children, and are covered completely by those children) and
"leaf" windows, which have no children and are visible.  Every
combination window has two or more children, all arranged along the same
axis.  There are (logically) two subtypes of windows, depending on
whether their children are horizontally or vertically arrayed.  There is
always one root window, which is either a leaf window (if the frame
contains only one window) or a combination window (if the frame contains
more than one window).  In the latter case, the root window will have
two or more children, either horizontally or vertically arrayed, and
each of those children will be either a leaf window or another
combination window.

   Here are some rules:

  1. Horizontal combination windows can never have children that are
     horizontal combination windows; same for vertical.

  2. Only leaf windows can be split (obviously) and this splitting does
     one of two things: (a) turns the leaf window into a combination
     window and creates two new leaf children, or (b) turns the leaf
     window into one of the two new leaves and creates the other leaf.
     Rule (1) dictates which of these two outcomes happens.

  3. Every combination window must have at least two children.

  4. Leaf windows can never become combination windows.  They can be
     deleted, however.  If this results in a violation of (3), the
     parent combination window also gets deleted.

  5. All functions that accept windows must be prepared to accept
     combination windows, and do something sane (e.g. signal an error
     if so).  Combination windows _do_ escape to the Lisp level.

  6. All windows have three fields governing their contents: these are
     "hchild" (a list of horizontally-arrayed children), "vchild" (a
     list of vertically-arrayed children), and "buffer" (the buffer
     contained in a leaf window).  Exactly one of these will be
     non-`nil'.  Remember that "horizontally-arrayed" means
     "side-by-side" and "vertically-arrayed" means "one above the
     other".

  7. Leaf windows also have markers in their `start' (the first buffer
     position displayed in the window) and `pointm' (the window's
     stashed value of `point'--see above) fields, while combination
     windows have `nil' in these fields.

  8. The list of children for a window is threaded through the `next'
     and `prev' fields of each child window.

  9. *Deleted windows can be undeleted*.  This happens as a result of
     restoring a window configuration, and is unlike frames, displays,
     and consoles, which, once deleted, can never be restored.
     Deleting a window does nothing except set a special `dead' bit to
     1 and clear out the `next', `prev', `hchild', and `vchild' fields,
     for GC purposes.

 10. Most frames actually have two top-level windows--one for the
     minibuffer and one (the "root") for everything else.  The modeline
     (if present) separates these two.  The `next' field of the root
     points to the minibuffer, and the `prev' field of the minibuffer
     points to the root.  The other `next' and `prev' fields are `nil',
     and the frame points to both of these windows.  Minibuffer-less
     frames have no minibuffer window, and the `next' and `prev' of the
     root window are `nil'.  Minibuffer-only frames have no root
     window, and the `next' of the minibuffer window is `nil' but the
     `prev' points to itself. (#### This is an artifact that should be
     fixed.)


File: internals.info,  Node: The Window Object,  Next: Creating a New Console/Device/Frame Type,  Prev: Window Hierarchy,  Up: Consoles; Devices; Frames; Windows

26.4 The Window Object
======================

Windows have the following accessible fields:

`frame'
     The frame that this window is on.

`mini_p'
     Non-`nil' if this window is a minibuffer window.

`buffer'
     The buffer that the window is displaying.  This may change often
     during the life of the window.

`dedicated'
     Non-`nil' if this window is dedicated to its buffer.

`pointm'
     This is the value of point in the current buffer when this window
     is selected; when it is not selected, it retains its previous
     value.

`start'
     The position in the buffer that is the first character to be
     displayed in the window.

`force_start'
     If this flag is non-`nil', it says that the window has been
     scrolled explicitly by the Lisp program.  This affects what the
     next redisplay does if point is off the screen: instead of
     scrolling the window to show the text around point, it moves point
     to a location that is on the screen.

`last_modified'
     The `modified' field of the window's buffer, as of the last time a
     redisplay completed in this window.

`last_point'
     The buffer's value of point, as of the last time a redisplay
     completed in this window.

`left'
     This is the left-hand edge of the window, measured in columns.
     (The leftmost column on the screen is column 0.)

`top'
     This is the top edge of the window, measured in lines.  (The top
     line on the screen is line 0.)

`height'
     The height of the window, measured in lines.

`width'
     The width of the window, measured in columns.

`next'
     This is the window that is the next in the chain of siblings.  It
     is `nil' in a window that is the rightmost or bottommost of a
     group of siblings.

`prev'
     This is the window that is the previous in the chain of siblings.
     It is `nil' in a window that is the leftmost or topmost of a group
     of siblings.

`parent'
     Internally, XEmacs arranges windows in a tree; each group of
     siblings has a parent window whose area includes all the siblings.
     This field points to a window's parent.

     Parent windows do not display buffers, and play little role in
     display except to shape their child windows.  Emacs Lisp programs
     usually have no access to the parent windows; they operate on the
     windows at the leaves of the tree, which actually display buffers.

`hscroll'
     This is the number of columns that the display in the window is
     scrolled horizontally to the left.  Normally, this is 0.

`use_time'
     This is the last time that the window was selected.  The function
     `get-lru-window' uses this field.

`display_table'
     The window's display table, or `nil' if none is specified for it.

`update_mode_line'
     Non-`nil' means this window's mode line needs to be updated.

`base_line_number'
     The line number of a certain position in the buffer, or `nil'.
     This is used for displaying the line number of point in the mode
     line.

`base_line_pos'
     The position in the buffer for which the line number is known, or
     `nil' meaning none is known.

`region_showing'
     If the region (or part of it) is highlighted in this window, this
     field holds the mark position that made one end of that region.
     Otherwise, this field is `nil'.


File: internals.info,  Node: Creating a New Console/Device/Frame Type,  Next: Modules for the Basic Displayable Lisp Objects,  Prev: The Window Object,  Up: Consoles; Devices; Frames; Windows

26.5 Creating a New Console, Device, or Frame Type
==================================================

Unfortunately, at the present time, only the console abstraction is at
all well-maintained.  Device and frame internals are referred to from
many places in the redisplay and console code.  The best that can be
done therefore is to create a whole new console type, even though much
code will be shared.  (Ben Wing has complained about the code
duplication in the GTK+ v1 console, and probably would not be happy with
the unpublished Qt console or Andrew Choi's Carbon console, but it's
hard to see how those consoles could have been done better without
fixing the abstractions in the X (actually Xt), MS Windows, and tty
consoles as well as doing a complete refactoring of the console, device,
and frame code.)

   What is desireable is sharing console, device, and frame methods
across platforms in a more general way, reducing the amount of
duplicated code by pulling it back into the redisplay engine proper or
the Lisp modules as appropriate.  For example, we should be able to use
`make-frame-on-device' to share a single X connection among GTK, X, and
Xft frames.  Xft is partially implemented, but GTK would be much harder
(impossible?) because it has its own event loop.  (Xft shares the Xt
event loop with the X console.)

   The way all methods get added to the console type is uncool.  A
console should be a composite, which indicates a collection of I/O
resources "used together."  It should indicate where operations that
change "focus" search for targets, _i.e._, by default new frames are
created on the selected device of the console where the input that
invoked the command was received, `pop-to-buffer-other-window' only
considers existing windows on the same device of the same console,
_etc._  But it should be possible to assemble consoles out of component
input channels, imaging devices, and multimedia (audio) channels.

   The following notes may provide some guidance to those who wish to
create new console types (_i.e._, port the redisplay code to a new
platform).  They are based on an unsuccessful attempt to refactor the
Xft code into a new console type while sharing most routines with the X
console.  (For clarification or suggestions, feel free to write Stephen
Turnbull <stephen@xemacs.org>.)

   The first thing to realize is that the naming of many modules
obscures their relationship to the console abstraction.  For example,
the `objects-CONSOLE-TYPE', `redisplay-CONSOLE-TYPE', and
`glyphs-CONSOLE-TYPE' series of modules have nothing to do with Lisp
objects and little to do with redisplay, respectively.  Rather they
implement abstractions used for rendering on each console type, such as
fonts and colors (`objects') and string and graphics drawing primitives
(`redisplay').  These modules are conceptually part of the console
implementations, not part of redisplay or Lisp.

   Public methods of a console are implemented as C functions declared
`static', following a rigid naming convention:
`CONSOLE-TYPE_METHOD-NAME'.  Methods are bound to the console type in
the `console_type_create_FILE' functions for each console component
(console, device, and frame) using the `CONSOLE_HAS_METHOD' family of
macros.  Methods for displaying images are bound using the
`IIFORMAT_HAS_DEVMETHOD' family of macros.  Methods are invoked using
the `CONMETH', `DEVMETH', and `FRAMEMETH' families of macros, which
look up the relevant methods in the object's table of methods.

   *N.B.* All of the object tables are actually references to console
method tables.  To create a variant of an existing console, there is a
`CONSOLE_INHERITS_METHOD' constructor, but this actually constructs the
name of the parent's method pointer and stores in the derived console
type's method table.  Of course this is time-efficient, and since there
are few console types it is a neglible waste of space.  However in
practice this may have contributed to breaking the various
abstractions, and the variant console must be coded in the same file as
the parent (because the methods are static).  Another minor symptom of
the incompleteness of the abstraction is the fact that the API for
inheritance of device methods for image formats is named
`IIFORMAT_HAS_SHARED_METHOD', although the semantics are identical.

   One problem encountered in attempting to create an Xft console type
as a derivative of the X console type was that there is no support for
such union types in the consistency-checking code, whether for the
fundamental Lisp consistency checks (the `CHECK_SOMETHING' family of
macros) or for the error-checking variants of many functions.  These
APIs all simply check for the apparent console type, which is a single
symbol (or enumerator).

   To create a new console with mostly new methods, it's probably best
to copy all of the `FUNCTION-CONSOLE-TYPE' files from a similar console
(or several, if the new console type seems like a combination of
several existing console types), renaming the files by substituting
NEW-CONSOLE-TYPE for CONSOLE-TYPE.  Then proceed in the obvious way by
renaming methods from `CONSOLE-TYPE_METHOD-NAME' to
`NEW-CONSOLE-TYPE_METHOD-NAME', and implementing them.

   Once you've done that, then the fun starts.  Insert the
initialization functions (`syms_of_FILE', `vars_of_FILE',
`console_type_create_FILE', _etc._) in `main_1' in `emacs.c'.

   Add a device creation function `make-CONSOLE-TYPE-device' in
`device.el'.  Add the device type to the calls to `Face-frob-property'
in `faces.el', as well as calls to the device initializer functions for
devices and frames, and "additional frobbing" in that file.

   You may wish to add an option to force the initial frame to that
device type to `emacs.c'.  Don't forget to document it in the command
help function in `startup.el'.

   You may need to add support for your console type in
`init_event_stream' in `event-stream.c'.

   If your console has a different UI for fonts or colors, or adds new
capability, you may need to add a `CONSOLE-TYPE-faces.el' file, or add
code to the `PARENT-TYPE-faces.el' file, to support new font or color
capability.  Probably initialization code in `faces.c' will be needed
too.

   A check for the console type is probably needed in `init_redisplay'
in `redisplay.c'.

   Ditto for the `CONSOLE-TYPE-init.el' file.

   Don't forget that Emacs windows are console-dependent, too.  At
least a `WINDOW_CONSOLE-TYPE_P'-checking macro should be added in
`window-impl.h'.

   Note that this project failed; there are probably many other details
to be implemented that I didn't get to.  But don't let that stop you!


File: internals.info,  Node: Modules for the Basic Displayable Lisp Objects,  Prev: Creating a New Console/Device/Frame Type,  Up: Consoles; Devices; Frames; Windows

26.6 Modules for the Basic Displayable Lisp Objects
===================================================

     `console-msw.c'
     `console-msw.h'
     `console-stream.c'
     `console-stream.h'
     `console-tty.c'
     `console-tty.h'
     `console-x.c'
     `console-x.h'
     `console.c'
     `console.h'

   These modules implement the "console" Lisp object type.  A console
contains multiple display devices, but only one keyboard and mouse.
Most of the time, a console will contain exactly one device.

   *This model may no longer suffice.*  The X Window System (at least)
now supports a variety of input devices, including touchscreens and
tablets, as well as the traditional keyboard and mouse, and may even be
able to support multiple instances of a single type of input device
(especially pointing devices) on a single console.

   Consoles are the top of a lisp object inclusion hierarchy.  Consoles
contain devices, which contain frames, which contain windows.

     `device-msw.c'
     `device-tty.c'
     `device-x.c'
     `device.c'
     `device.h'

   These modules implement the "device" Lisp object type.  This
abstracts a particular screen or connection on which frames are
displayed.  As with Lisp objects, event interfaces, and other
subsystems, the device code is separated into a generic component that
contains a standardized interface (in the form of a set of methods) onto
particular device types.

   The device subsystem defines all the methods and provides method
services for not only device operations but also for the frame, window,
menubar, scrollbar, toolbar, and other displayable-object subsystems.
The reason for this is that all of these subsystems have the same
subtypes (X, TTY, NeXTstep, Microsoft Windows, etc.) as devices do.

   *This abstraction is probably broken* (as of late 2004), at least
for X consoles, with the advent of the *Xft* library.  Xft is a
complete break from the traditional approach to text rendering in the
X11 environment, since fonts are composed of glyphs rendered by
_client-side_ code.  These glyphs are then transmitted to the server as
sets of trapezoids, and displayed by the *XRender* extension (where
available; the X11 core protocol can also be used, but this is slow).
The XRender extension is especially attractive because it allows modern
image composition techniques to be used to render antialiased fonts.

   By contrast, the traditional approach renders fonts on the server
side as a collection of bitmaps.  It is also possible use a _font
server_ that knows how to render antialiased fonts, but for some reason
this approach has never caught on.

   The problem that this creates for XEmacs is that the traditional (and
still popular) widget sets, the various Athena variants and the Motif
widget set, are based on the server-side rendering model.  Thus, even if
XEmacs-specific widgets (such as the basic text window, the Lucid
menubar, and the recently added tab control) can be adapted to render
text via Xft, older widgets (such as buttons and labels) and "modern"
widgets derived from traditional widgets (the progress gauge) still
expect their font resources to be converted to server-side fonts.  Then
text is rendered by calls to the core protocol via Xlib, rather than by
calls to the XRender protocol via Xft.

   It's even possible to imagine a situation where a widget is composed
of components which draw their own text (_predefined widgets_) as well
as new components which can draw using more modern methods.  Handling
this will either require reworking the Emacs face mechanism to provide a
way to determine whether this widget can use that font, or ways to give
each of several different widgets, even different components of a given
widget, a different face property.  This is already an issue, as widgets
generally derive their font from the gui-element face, but this might
not be appropriate for widgets embedded in a buffer.

   There seem to be two overall ways to go.

  1. Add face information to the Lucid widget library, and extend the
     coverage of that library to device types that aren't supported by
     it yet (MS Windows, and maybe TTY).

  2. Create a lighter-weight interface, perhaps an extension of the
     device interface, that allows widgets to access face information
     from the device it is implemented on.

     `frame-msw.c'
     `frame-tty.c'
     `frame-x.c'
     `frame.c'
     `frame.h'

   Each device contains one or more frames in which objects (e.g. text)
are displayed.  A frame corresponds to a window in the window system;
usually this is a top-level window but it could potentially be one of a
number of overlapping child windows within a top-level window, using the
MDI (Multiple Document Interface) protocol in Microsoft Windows or a
similar scheme.

   The `frame-*' files implement the "frame" Lisp object type and
provide the generic and device-type-specific operations on frames (e.g.
raising, lowering, resizing, moving, etc.).

     `window.c'
     `window.h'

   Each frame consists of one or more non-overlapping "windows" (better
known as "panes" in standard window-system terminology) in which a
buffer's text can be displayed.  Windows can also have scrollbars
displayed around their edges.

   `window.c' and `window.h' implement the "window" Lisp object type
and provide code to manage windows.  Since windows have no associated
resources in the window system (the window system knows only about the
frame; no child windows or anything are used for XEmacs windows), there
is no device-type-specific code here; all of that code is part of the
redisplay mechanism or the code for particular object types such as
scrollbars.


File: internals.info,  Node: The Redisplay Mechanism,  Next: Extents,  Prev: Consoles; Devices; Frames; Windows,  Up: Top

27 The Redisplay Mechanism
**************************

The redisplay mechanism is one of the most complicated sections of
XEmacs, especially from a conceptual standpoint.  This is doubly so
because, unlike for the basic aspects of the Lisp interpreter, the
computer science theories of how to efficiently handle redisplay are not
well-developed.

   When working with the redisplay mechanism, remember the Golden Rules
of Redisplay:

  1. It Is Better To Be Correct Than Fast.

  2. Thou Shalt Not Run Elisp From Within Redisplay.

  3. It Is Better To Be Fast Than Not To Be.

* Menu:

* Critical Redisplay Sections::
* Line Start Cache::
* Redisplay Piece by Piece::
* Modules for the Redisplay Mechanism::
* Modules for other Display-Related Lisp Objects::


File: internals.info,  Node: Critical Redisplay Sections,  Next: Line Start Cache,  Prev: The Redisplay Mechanism,  Up: The Redisplay Mechanism

27.1 Critical Redisplay Sections
================================

*The following paragraphs are way out-of-date and inaccurate.*

   Within this section, we are defenseless and assume that the
following cannot happen:

  1. garbage collection

  2. Lisp code evaluation

  3. frame size changes

   We ensure (3) by calling `hold_frame_size_changes()', which will
cause any pending frame size changes to get put on hold till after the
end of the critical section.  (1) follows automatically if (2) is met.
#### Unfortunately, there are some places where Lisp code can be called
within this section.  We need to remove them.

   If `Fsignal()' is called during this critical section, we will
`abort()'.

   If garbage collection is called during this critical section, we
simply return. #### We should abort instead.

   #### If a frame-size change does occur we should probably actually
be preempting redisplay.

   *Begin up-to-date stuff*

27.1.1 Nasty Bugs due to Reentrancy in Redisplay Structures handling QUIT
-------------------------------------------------------------------------

These are now fixed as of November 10, 2004.

Crash - reentrant `regenerate_window()'
---------------------------------------

Here is a crash I (ben) just got - November 9, 2004: It can sort of be
reproduced by creating a bunch of frames, opening a bunch of large
files (which may be fontlocking for awhile). and immediately start
Alt-TAB-ing back and forth quickly and constantly scrolling up and down
using the scrolling dial on your mouse.


     Fatal error: assertion failed, file c:\xemacs\build\src\redisplay.c, line 5532,
     !dy->locked

     C backtrace:

     assert_failed(const char * 0x012a4ff0 `string', int 5532, const char * 0x0127bea4 `string') line 3839
     Dynarr_verify_mod_1(void * 0x023ad2b0, const char * 0x012a4ff0 `string', int 5532) line 1306 + 36 bytes
     regenerate_window(window * 0x02f2ca88, long 40372, long 40372, int 2) line 5532 + 25 bytes
     update_line_start_cache(window * 0x02f2ca88, long 40372, long 40401, long 40372, int 0) line 8543 + 19 bytes
     point_in_line_start_cache(window * 0x02f2ca88, long 40372, int 0) line 7850 + 23 bytes
     start_end_of_last_line(window * 0x02f2ca88, long 40372, int 1, int 1) line 8121 + 15 bytes
     end_of_last_line_may_error(window * 0x02f2ca88, long 40372) line 8203 + 17 bytes
     pixel_to_glyph_translation(frame * 0x02f2c900, int 291, int 317, int * 0x0082bb04, int * 0x0082bb00, int * 0x0082bafc, int * 0x0082baf8, window * * 0x0082bae8, long * 0x0082baf4, long * 0x0082baf0, long * 0x0082baec, long * 0x0082bb10, long * 0x0082bb0c) line 9336 + 32 bytes
     mswindows_handle_mousewheel_event(long 49465600, int 0, int -240, tagPOINTS {...}) line 360 + 82 bytes
     mswindows_wnd_proc(HWND__ * 0x00260a42, unsigned int 522, unsigned int 4279238656, long 29885130) line 3561 + 36 bytes
     intercepted_wnd_proc(HWND__ * 0x00260a42, unsigned int 522, unsigned int 4279238656, long 29885130) line 2376
     USER32! 77e11ef0()
     USER32! 77e1204c()
     USER32! 77e121af()
     mswindows_drain_windows_queue(int 0) line 1330 + 9 bytes
     emacs_mswindows_drain_queue() line 1339 + 7 bytes
     event_stream_drain_queue() line 1785
     event_stream_quit_p() line 1893
     check_quit() line 938
     check_what_happened() line 459
     internal_equal(long 22180468, long 22180468, int 0) line 2823 + 14 bytes
     update_image_instance(long 83498640, long 22180468) line 2121 + 18 bytes
     image_instantiate(long 21418616, long 20663624, long 54932896, long 22180468, long 3) line 3403 + 13 bytes
     va_call_trapping_problems_1(void * 0x0082cf94) line 5220 + 221 bytes
     call_trapping_problems_2(long 83160440) line 4867 + 13 bytes
     call_with_condition_handler(long (long, long, long)* 0x010cc4c0 flagged_a_squirmer(long, long, long), long 83160440, long (long)* 0x010cc440 call_trapping_problems_2(long), long 83160440) line 2129 + 7 bytes
     call_trapping_problems_1(long 83160440) line 4874 + 23 bytes
     internal_catch(long 21399864, long (long)* 0x010cc490 call_trapping_problems_1(long), long 83160440, int * volatile 0x0082ce4c, long * volatile 0x0082ce54) line 1527 + 7 bytes
     call_trapping_problems(long 20908160, const char * 0x00000000, int 98315, call_trapping_problems_result * 0x00000000, long (void *)* 0x010cca30 va_call_trapping_problems_1(void *), void * 0x0082cf94) line 5147 + 32 bytes
     call_with_suspended_errors(long (void)* 0x011448c0 image_instantiate(long, long, long, long, long), long 20663624, long 20908160, _error_behavior_struct_ {...}, int 5) line 5314 + 26 bytes
     specifier_instance_from_inst_list(long 21418616, long 20663624, long 54932896, long 21673760, _error_behavior_struct_ {...}, int 1, long 3) line 2501 + 54 bytes
     specifier_instance(long 21418616, long 20663624, long 54932896, _error_behavior_struct_ {...}, int 1, int 0, long 3) line 2614 + 64 bytes
     glyph_image_instance(long 22692176, long 54932896, _error_behavior_struct_ {...}, int 1) line 3955 + 31 bytes
     add_glyph_rune(position_redisplay_data_type * 0x0082d52c, glyph_block * 0x0082d454, int 0, int 0, glyph_cachel * 0x04f4e518) line 1972 + 26 bytes
     create_text_block(window * 0x034635a0, display_line * 0x033bfb28, long 29860, prop_block_dynarr * * 0x0082d7b8, int 2) line 2827 + 30 bytes
     generate_display_line(window * 0x034635a0, display_line * 0x033bfb28, int 1, long 29860, prop_block_dynarr * * 0x0082d7b8, int 2) line 979 + 38 bytes
     regenerate_window(window * 0x034635a0, long 29860, long 25012, int 2) line 5607 + 30 bytes
     update_line_start_cache(window * 0x034635a0, long 25012, long 28767, long 25012, int 0) line 8614 + 19 bytes
     point_in_line_start_cache(window * 0x034635a0, long 25012, int 0) line 7850 + 23 bytes
     start_end_of_last_line(window * 0x034635a0, long 25012, int 1, int 0) line 8121 + 15 bytes
     end_of_last_line(window * 0x034635a0, long 25012) line 8197 + 17 bytes
     Fwindow_end(long 54932896, long 20926544) line 1848 + 13 bytes
     Ffuncall(int 3, long * 0x0082dbb8) line 3841 + 93 bytes
     execute_optimized_program(const unsigned char * 0x032ceee8, int 7, long * 0x03289f40) line 823 + 16 bytes
     funcall_compiled_function(long 52991916, int 1, long * 0x0082dfb0) line 3454 + 85 bytes
     Ffuncall(int 2, long * 0x0082dfac) line 3880 + 17 bytes
     execute_optimized_program(const unsigned char * 0x02f667d8, int 6, long * 0x01558748) line 823 + 16 bytes
     funcall_compiled_function(long 22579576, int 3, long * 0x0082e3ac) line 3454 + 85 bytes
     Ffuncall(int 4, long * 0x0082e3a8) line 3880 + 17 bytes
     execute_optimized_program(const unsigned char * 0x03209c98, int 5, long * 0x03288c68) line 823 + 16 bytes
     funcall_compiled_function(long 51656320, int 1, long * 0x0082e7a4) line 3454 + 85 bytes
     Ffuncall(int 2, long * 0x0082e7a0) line 3880 + 17 bytes
     execute_optimized_program(const unsigned char * 0x0082e9ec, int 4, long * 0x03224990) line 823 + 16 bytes
     Fbyte_code(long 37927380, long 52578688, long 9) line 2564 + 70 bytes
     Feval(long 51505420) line 3601 + 187 bytes
     internal_catch(long 51959412, long (long)* 0x010c6f40 Feval(long), long 51505420, int * volatile 0x00000000, long * volatile 0x00000000) line 1527 + 7 bytes
     execute_rare_opcode(long * 0x0082eee8, const unsigned char * 0x03248365, Opcode Bcatch) line 1380 + 24 bytes
     execute_optimized_program(const unsigned char * 0x03248340, int 2, long * 0x02f3c0a0) line 715 + 17 bytes
     funcall_compiled_function(long 51656276, int 0, long * 0x0082f444) line 3454 + 85 bytes
     Ffuncall(int 1, long * 0x0082f440) line 3880 + 17 bytes
     run_hook_with_args_in_buffer(buffer * 0x04ee9060, int 1, long * 0x0082f440, run_hooks_condition RUN_HOOKS_TO_COMPLETION) line 4361 + 13 bytes
     run_hook_with_args(int 1, long * 0x0082f440, run_hooks_condition RUN_HOOKS_TO_COMPLETION) line 4374 + 23 bytes
     run_hook(long 51959028) line 4443 + 13 bytes
     safe_run_hook_trapping_problems_1(void * 0x013c73c0) line 5517 + 9 bytes
     call_trapping_problems_2(long 83157920) line 4867 + 13 bytes
     call_with_condition_handler(long (long, long, long)* 0x010cc4c0 flagged_a_squirmer(long, long, long), long 83157920, long (long)* 0x010cc440 call_trapping_problems_2(long), long 83157920) line 2129 + 7 bytes
     call_trapping_problems_1(long 83157920) line 4874 + 23 bytes
     internal_catch(long 21399864, long (long)* 0x010cc490 call_trapping_problems_1(long), long 83157920, int * volatile 0x0082f700, long * volatile 0x0082f708) line 1527 + 7 bytes
     call_trapping_problems(long 20925944, const char * 0x00000000, int 131235, call_trapping_problems_result * 0x0082f830, long (void *)* 0x010cd990 safe_run_hook_trapping_problems_1(void *), void * 0x013c73c0) line 5147 + 32 bytes
     safe_run_hook_trapping_problems(long 20741312, long 20739008, int 160) line 5543 + 36 bytes
     run_pre_idle_hook() line 2084 + 24 bytes
     redisplay() line 7224
     Fnext_event(long 37363732, long 20928056) line 2263
     Fcommand_loop_1() line 600 + 15 bytes
     command_loop_1(long 20928056) line 512
     condition_case_1(long 20925944, long (long)* 0x01096a80 command_loop_1(long), long 20928056, long (long, long)* 0x01096630 cmd_error(long, long), long 20928056) line 1918 + 7 bytes
     command_loop_3() line 262 + 35 bytes
     command_loop_2(long 20928056) line 277
     internal_catch(long 20683712, long (long)* 0x010967a0 command_loop_2(long), long 20928056, int * volatile 0x00000000, long * volatile 0x00000000) line 1527 + 7 bytes
     initial_command_loop(long 20928056) line 313 + 28 bytes
     xemacs_21_5_b18_i586_pc_win32(int 1, unsigned short * * 0x0082fed0, unsigned short * * 0x00000000, int 0) line 2551
     main(int 1, char * * 0x00e52610, char * * 0x00e52bb0) line 2992
     mainCRTStartup() line 338 + 17 bytes
     KERNEL32! 7c59893d()

     Lisp backtrace:

       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (catch #<INTERNAL OBJECT (XEmacs bug?) (opaque, size=0) 0x1468938> ...)
       # (unwind-protect ...)
       # bind (inhibit-quit)
       window-end(#<window on "signal.c<2>" 0x5e4a> t)
       # (unwind-protect ...)
       # bind (buffer we-are-screwed check-text-props window)
       lazy-lock-fontify-window(#<window on "signal.c<2>" 0x5e4a>)
       # bind (walk-windows-current walk-windows-start arg which-devices which-frames
      minibuf function)
       walk-windows(lazy-lock-fontify-window no-minibuf #<mswindows-frame "emacs" 0x5
     e49>)
       # (unwind-protect ...)
       # bind (ssf65112 tick frame)
       lazy-lock-maybe-fontify-frame(#<mswindows-frame "emacs" 0x5e49>)
       # bind (frame starting-frame)
       byte-code("..." [starting-frame frame selected-frame frame-visible-p frame-min
     ibuffer-only-p next-frame visible-nomini throw lazy-lock-frame-loop-done t lazy-
     lock-maybe-fontify-frame] 4)
       # (catch lazy-lock-frame-loop-done ...)
       lazy-lock-pre-idle-fontify-windows()
       # (unwind-protect ...)
       # (catch #<INTERNAL OBJECT (XEmacs bug?) (opaque, size=0) 0x1468938> ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # bind (inhibit-quit)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # bind (inhibit-quit)
       (next-event "[internal]")
       # (condition-case ... . error)
       # (catch top-level ...)

Another Lisp trace of a similar situation (C stack trace not available):
........................................................................

     Fatal error: assertion failed, file c:\xemacs\build\src\redisplay.c, line 5532,
     !dy->locked

     Lisp backtrace follows:

       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       scrollbar-page-down((#<window on "*grep*" 0x1a5f9>))
       (dispatch-event "[internal]")
       # (unwind-protect ...)
       # (catch #<INTERNAL OBJECT (XEmacs bug?) (opaque, size=0) 0x1468270> ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (catch #<INTERNAL OBJECT (XEmacs bug?) (opaque, size=0) 0x1468270> ...)
       # (unwind-protect ...)
       # bind (inhibit-quit)
       window-end(#<window on "*grep*" 0x1a5f9> t)
       # (unwind-protect ...)
       # bind (buffer we-are-screwed check-text-props window)
       lazy-lock-fontify-window(#<window on "*grep*" 0x1a5f9>)
       # bind (walk-windows-current walk-windows-start arg which-devices which-frames
      minibuf function)
       walk-windows(lazy-lock-fontify-window no-minibuf #<mswindows-frame "emacs" 0x1
     9f64>)
       # (unwind-protect ...)
       # bind (ssf65112 tick frame)
       lazy-lock-maybe-fontify-frame(#<mswindows-frame "emacs" 0x19f64>)
       # bind (frame starting-frame)
       byte-code("..." [starting-frame frame selected-frame frame-visible-p frame-min
     ibuffer-only-p next-frame visible-nomini throw lazy-lock-frame-loop-done t lazy-
     lock-maybe-fontify-frame] 4)
       # (catch lazy-lock-frame-loop-done ...)
       lazy-lock-pre-idle-fontify-windows()
       # (unwind-protect ...)
       # (catch #<INTERNAL OBJECT (XEmacs bug?) (opaque, size=0) 0x1468270> ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # bind (inhibit-quit)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # bind (inhibit-quit)
       (next-event "[internal]")
       # (condition-case ... . error)
       # (catch top-level ...)

Crash - reentrant `generate_displayable_area()'
-----------------------------------------------

Original code said [Tricky tricky tricky.
`generate_displayable_area()' can (could) be called reentrantly, and
redisplay is not prepared to handle this:].

   assert_failed(const char * 0x0129c8c8 `string', int 5328, const char
* 0x01274068 `string') line 3620 Dynarr_verify_mod_1(void * 0x0250f228,
const char * 0x0129c8c8 `string', int 5328) line 1256 + 36 bytes
generate_displayable_area(window * 0x02480028, long 38776292, int 0,
int 0, int 265, int 169, display_line_dynarr * 0x0250f228, long 0, int
2) line 5328 + 25 bytes output_gutter(frame * 0x0228ad90, gutter_pos
TOP_GUTTER, int 1) line 409 + 69 bytes redraw_exposed_gutter(frame *
0x0228ad90, gutter_pos TOP_GUTTER, int 8, int 23, int 249, int 127)
line 687 + 15 bytes redraw_exposed_gutters(frame * 0x0228ad90, int 8,
int 23, int 249, int 127) line 703 + 29 bytes
mswindows_redraw_exposed_area(frame * 0x0228ad90, int 8, int 23, int
249, int 127) line 862 + 25 bytes mswindows_handle_paint(frame *
0x0228ad90) line 2176 + 25 bytes mswindows_wnd_proc(HWND__ *
0x001003e2, unsigned int 15, unsigned int 0, long 0) line 3233 + 45
bytes intercepted_wnd_proc(HWND__ * 0x001003e2, unsigned int 15,
unsigned int 0, long 0) line 2488 USER32! 77e3a244() USER32! 77e14730()
USER32! 77e1558a() NTDLL! KiUserCallbackDispatcher@12 + 19 bytes
USER32! 77e14680() USER32! 77e1a792() qxeIsDialogMessage(HWND__ *
0x001003e2, tagMSG * 0x0082a93c {msg=0x0000000f wp=0x00000000
lp=0x00000000}) line 2298 + 14 bytes mswindows_is_dialog_msg(tagMSG *
0x0082a93c {msg=0x0000000f wp=0x00000000 lp=0x00000000}) line 165 + 13
bytes mswindows_drain_windows_queue(int 0) line 1282 + 9 bytes
emacs_mswindows_drain_queue() line 1326 + 7 bytes
event_stream_drain_queue() line 1887 event_stream_quit_p() line 1992
check_quit() line 993 unbind_to_hairy(int 35) line 5963 unbind_to_1(int
35, long 20888208) line 5945 + 200 bytes
specifier_instance_from_inst_list(long 21379344, long 38135616, long
36220304, long 20888208, _error_behavior_struct_ {...}, int 1, long 3)
line 2522 + 16 bytes specifier_instance(long 21379344, long 38135616,
long 36220304, _error_behavior_struct_ {...}, int 1, int 0, long 3)
line 2625 + 65 bytes specifier_instance_no_quit(long 21379344, long
38135616, long 36220304, _error_behavior_struct_ {...}, int 0, long 1)
line 2658 + 31 bytes face_property_matching_instance(long 22612340,
long 20860632, long 22530956, long 36220304, _error_behavior_struct_
{...}, int 0, long 1) line 565 + 48 bytes
ensure_face_cachel_contains_charset(face_cachel * 0x0082b014, long
36220304, long 22530956) line 1104 + 35 bytes
update_face_cachel_data(face_cachel * 0x0082b014, long 36220304, long
22612340) line 1304 + 19 bytes query_string_geometry(long 21110576,
long 22612340, int * 0x00000000, int * 0x0082b5b4, int * 0x00000000,
long 38852960) line 2370 + 23 bytes
mswindows_widget_query_string_geometry(long 21110576, long 22612340,
int * 0x0082b5b8, int * 0x0082b5b4, long 38852960) line 2914 + 25 bytes
widget_query_string_geometry(long 21110576, long 22612340, int *
0x0082b5b8, int * 0x0082b5b4, long 38852960) line 514 + 32 bytes
edit_field_query_geometry(long 38857648, int * 0x0082b7b4, int *
0x0082b7b8, image_instance_geometry IMAGE_DESIRED_GEOMETRY, long
38852960) line 920 + 390 bytes widget_query_geometry(long 38857648, int
* 0x0082b7b4, int * 0x0082b7b8, image_instance_geometry
IMAGE_DESIRED_GEOMETRY, long 38852960) line 567 + 26 bytes
image_instance_query_geometry(long 38857648, int * 0x0082b7b4, int *
0x0082b7b8, image_instance_geometry IMAGE_DESIRED_GEOMETRY, long
38852960) line 2015 + 26 bytes glyph_query_geometry(long 38853384, int
* 0x0082b7b4, int * 0x0082b7b8, image_instance_geometry
IMAGE_DESIRED_GEOMETRY, long 38852960) line 4197 + 25 bytes
layout_query_geometry(long 38852960, int * 0x0082b9cc, int *
0x0082b9d0, image_instance_geometry IMAGE_DESIRED_GEOMETRY, long
38404624) line 1351 + 25 bytes widget_query_geometry(long 38852960, int
* 0x0082b9cc, int * 0x0082b9d0, image_instance_geometry
IMAGE_DESIRED_GEOMETRY, long 38404624) line 567 + 26 bytes
image_instance_query_geometry(long 38852960, int * 0x0082b9cc, int *
0x0082b9d0, image_instance_geometry IMAGE_DESIRED_GEOMETRY, long
38404624) line 2015 + 26 bytes glyph_query_geometry(long 38537976, int
* 0x0082b9cc, int * 0x0082b9d0, image_instance_geometry
IMAGE_DESIRED_GEOMETRY, long 38404624) line 4197 + 25 bytes
layout_layout(long 38404624, int 265, int 156, int -2, int -2, long
38273064) line 1468 + 23 bytes widget_layout(long 38404624, int 265,
int 156, int -2, int -2, long 38273064) line 626 + 30 bytes
image_instance_layout(long 38404624, int 265, int 156, int -2, int -2,
long 38273064) line 2102 + 51 bytes glyph_ascent(long 38404624, long
38273064) line 4009 + 21 bytes update_glyph_cachel_data(window *
0x02480028, long 36201168, glyph_cachel * 0x0248c3d8) line 4272 + 13
bytes get_glyph_cachel_index(window * 0x02480028, long 36201168) line
4306 + 17 bytes add_glyph_rune(position_redisplay_data_type *
0x0082bf2c, glyph_block * 0x024bd028, int 0, int 0, glyph_cachel *
0x00000000) line 1800 + 15 bytes
add_glyph_runes(position_redisplay_data_type * 0x0082bf2c, int 0) line
2085 + 31 bytes create_string_text_block(window * 0x02480028, long
38776292, display_line * 0x02514500, long 0, prop_block_dynarr * *
0x0082c13c, int 2) line 4907 + 14 bytes
generate_string_display_line(window * 0x02480028, long 38776292,
display_line * 0x02514500, long 0, prop_block_dynarr * * 0x0082c13c,
int 2) line 5293 + 29 bytes generate_displayable_area(window *
0x02480028, long 38776292, int 0, int 0, int 265, int 169,
display_line_dynarr * 0x0250f228, long 0, int 2) line 5372 + 29 bytes
output_gutter(frame * 0x0228ad90, gutter_pos TOP_GUTTER, int 0) line
409 + 69 bytes update_frame_gutters(frame * 0x0228ad90) line 639 + 15
bytes redisplay_frame(frame * 0x0228ad90, int 1) line 6792 + 9 bytes
redisplay_device(device * 0x0171df00, int 1) line 6911 + 11 bytes
redisplay_without_hooks() line 6957 + 11 bytes
redisplay_no_pre_idle_hook() line 7029 redisplay() line 7011
mswindows_wnd_proc(HWND__ * 0x001003e2, unsigned int 5, unsigned int 0,
long 10223881) line 3424 intercepted_wnd_proc(HWND__ * 0x001003e2,
unsigned int 5, unsigned int 0, long 10223881) line 2488 USER32!
77e3a244() USER32! 77e16362() USER32! 77e14c1a() USER32! 77e1dd30()
mswindows_wnd_proc(HWND__ * 0x001003e2, unsigned int 71, unsigned int
0, long 8578308) line 3926 + 21 bytes intercepted_wnd_proc(HWND__ *
0x001003e2, unsigned int 71, unsigned int 0, long 8578308) line 2488
USER32! 77e3a244() USER32! 77e14730() USER32! 77e174b4() NTDLL!
KiUserCallbackDispatcher@12 + 19 bytes mswindows_set_frame_size(frame *
0x0228ad90, int 265, int 156) line 355 internal_set_frame_size(frame *
0x0228ad90, int 265, int 156, int 0) line 2754 + 24 bytes
Fset_frame_displayable_pixel_size(long 36220304, long 531, long 313,
long 20888208) line 3004 + 32 bytes Ffuncall(int 4, long * 0x0082e778)
line 3844 + 168 bytes execute_optimized_program(const unsigned char *
0x02286e48, int 40, long * 0x01529b80) line 609 + 16 bytes
funcall_compiled_function(long 22433308, int 0, long * 0x0082ec08) line
3452 + 85 bytes Ffuncall(int 1, long * 0x0082ec04) line 3883 + 17 bytes
execute_optimized_program(const unsigned char * 0x02286d40, int 6, long
* 0x01548ddc) line 609 + 16 bytes funcall_compiled_function(long
22505864, int 11, long * 0x0082f00c) line 3452 + 85 bytes Ffuncall(int
12, long * 0x0082f008) line 3883 + 17 bytes
execute_optimized_program(const unsigned char * 0x02503e38, int 47,
long * 0x0152dc48) line 609 + 16 bytes funcall_compiled_function(long
22436784, int 0, long * 0x0082f534) line 3452 + 85 bytes Ffuncall(int
1, long * 0x0082f530) line 3883 + 17 bytes apply1(long 22436784, long
20888208) line 4458 + 11 bytes Fcall_interactively(long 20742816, long
20888208, long 20888208) line 460 + 13 bytes Ffuncall(int 2, long *
0x0082f8ec) line 3844 + 127 bytes call1(long 20854392, long 20742816)
line 4489 + 11 bytes execute_command_event(command_builder *
0x01798f98, long 24439276) line 4198 + 69 bytes Fdispatch_event(long
24439276) line 4569 + 13 bytes Fcommand_loop_1() line 569 + 9 bytes
command_loop_1(long 20888208) line 489 condition_case_1(long 20886024,
long (long)* 0x010955a0 command_loop_1(long), long 20888208, long
(long, long)* 0x01095150 cmd_error(long, long), long 20888208) line
1917 + 7 bytes command_loop_3() line 251 + 35 bytes command_loop_2(long
20888208) line 264 internal_catch(long 20650992, long (long)*
0x010952c0 command_loop_2(long), long 20888208, int * volatile
0x00000000, long * volatile 0x00000000) line 1527 + 7 bytes
initial_command_loop(long 20888208) line 300 + 28 bytes
xemacs_21_5_b10_i586_pc_win32(int 1, char * * 0x00e52620, char * *
0x00e52bb0, int 0) line 2356 main(int 1, char * * 0x00e52620, char * *
0x00e52bb0) line 2733 mainCRTStartup() line 338 + 17 bytes KERNEL32!
77ea847c()


File: internals.info,  Node: Line Start Cache,  Next: Redisplay Piece by Piece,  Prev: Critical Redisplay Sections,  Up: The Redisplay Mechanism

27.2 Line Start Cache
=====================

The traditional scrolling code in Emacs breaks in a variable height
world.  It depends on the key assumption that the number of lines that
can be displayed at any given time is fixed.  This led to a complete
separation of the scrolling code from the redisplay code.  In order to
fully support variable height lines, the scrolling code must actually be
tightly integrated with redisplay.  Only redisplay can determine how
many lines will be displayed on a screen for any given starting point.

   What is ideally wanted is a complete list of the starting buffer
position for every possible display line of a buffer along with the
height of that display line.  Maintaining such a full list would be very
expensive.  We settle for having it include information for all areas
which we happen to generate anyhow (i.e. the region currently being
displayed) and for those areas we need to work with.

   In order to ensure that the cache accurately represents what
redisplay would actually show, it is necessary to invalidate it in many
situations.  If the buffer changes, the starting positions may no longer
be correct.  If a face or an extent has changed then the line heights
may have altered.  These events happen frequently enough that the cache
can end up being constantly disabled.  With this potentially constant
invalidation when is the cache ever useful?

   Even if the cache is invalidated before every single usage, it is
necessary.  Scrolling often requires knowledge about display lines which
are actually above or below the visible region.  The cache provides a
convenient light-weight method of storing this information for multiple
display regions.  This knowledge is necessary for the scrolling code to
always obey the First Golden Rule of Redisplay.

   If the cache already contains all of the information that the
scrolling routines happen to need so that it doesn't have to go
generate it, then we are able to obey the Third Golden Rule of
Redisplay.  The first thing we do to help out the cache is to always
add the displayed region.  This region had to be generated anyway, so
the cache ends up getting the information basically for free.  In those
cases where a user is simply scrolling around viewing a buffer there is
a high probability that this is sufficient to always provide the needed
information.  The second thing we can do is be smart about invalidating
the cache.

   TODO--Be smart about invalidating the cache.  Potential places:

   * Insertions at end-of-line which don't cause line-wraps do not
     alter the starting positions of any display lines.  These types of
     buffer modifications should not invalidate the cache.  This is
     actually a large optimization for redisplay speed as well.

   * Buffer modifications frequently only affect the display of lines
     at and below where they occur.  In these situations we should only
     invalidate the part of the cache starting at where the
     modification occurs.

   In case you're wondering, the Second Golden Rule of Redisplay is not
applicable.


File: internals.info,  Node: Redisplay Piece by Piece,  Next: Modules for the Redisplay Mechanism,  Prev: Line Start Cache,  Up: The Redisplay Mechanism

27.3 Redisplay Piece by Piece
=============================

As you can begin to see redisplay is complex and also not well
documented. Chuck no longer works on XEmacs so this section is my take
on the workings of redisplay.

   Redisplay happens in three phases:

  1. Determine desired display in area that needs redisplay.
     Implemented by `redisplay.c'

  2. Compare desired display with current display Implemented by
     `redisplay-output.c'

  3. Output changes Implemented by `redisplay-output.c',
     `redisplay-x.c', `redisplay-msw.c' and `redisplay-tty.c'

   Steps 1 and 2 are device-independent and relatively complex.  Step 3
is mostly device-dependent.

   Determining the desired display

   Display attributes are stored in `display_line' structures. Each
`display_line' consists of a set of `display_block''s and each
`display_block' contains a number of `rune''s. Generally dynarr's of
`display_line''s are held by each window representing the current
display and the desired display.

   The `display_line' structures are tightly tied to buffers which
presents a problem for redisplay as this connection is bogus for the
modeline. Hence the `display_line' generation routines are duplicated
for generating the modeline. This means that the modeline display code
has many bugs that the standard redisplay code does not.

   The guts of `display_line' generation are in `create_text_block',
which creates a single display line for the desired locale. This
incrementally parses the characters on the current line and generates
redisplay structures for each.

   Gutter redisplay is different. Because the data to display is stored
in a string we cannot use `create_text_block'. Instead we use
`create_text_string_block' which performs the same function as
`create_text_block' but for strings. Many of the complexities of
`create_text_block' to do with cursor handling and selective display
have been removed.


File: internals.info,  Node: Modules for the Redisplay Mechanism,  Next: Modules for other Display-Related Lisp Objects,  Prev: Redisplay Piece by Piece,  Up: The Redisplay Mechanism

27.4 Modules for the Redisplay Mechanism
========================================

     `redisplay-output.c'
     `redisplay-msw.c'
     `redisplay-tty.c'
     `redisplay-x.c'
     `redisplay.c'
     `redisplay.h'

   These files provide the redisplay mechanism.  As with many other
subsystems in XEmacs, there is a clean separation between the general
and device-specific support.

   `redisplay.c' contains the bulk of the redisplay engine.  These
functions update the redisplay structures (which describe how the screen
is to appear) to reflect any changes made to the state of any
displayable objects (buffer, frame, window, etc.) since the last time
that redisplay was called.  These functions are highly optimized to
avoid doing more work than necessary (since redisplay is called
extremely often and is potentially a huge time sink), and depend heavily
on notifications from the objects themselves that changes have occurred,
so that redisplay doesn't explicitly have to check each possible object.
The redisplay mechanism also contains a great deal of caching to further
speed things up; some of this caching is contained within the various
displayable objects.

   `redisplay-output.c' goes through the redisplay structures and
converts them into calls to device-specific methods to actually output
the screen changes.

   `redisplay-x.c' and `redisplay-tty.c' are two implementations of
these redisplay output methods, for X frames and TTY frames,
respectively.

     `indent.c'

   This module contains various functions and Lisp primitives for
converting between buffer positions and screen positions.  These
functions call the redisplay mechanism to do most of the work, and then
examine the redisplay structures to get the necessary information.  This
module needs work.

     `termcap.c'
     `terminfo.c'
     `tparam.c'

   These files contain functions for working with the termcap
(BSD-style) and terminfo (System V style) databases of terminal
capabilities and escape sequences, used when XEmacs is displaying in a
TTY.

     `cm.c'
     `cm.h'

   These files provide some miscellaneous TTY-output functions and
should probably be merged into `redisplay-tty.c'.


File: internals.info,  Node: Modules for other Display-Related Lisp Objects,  Prev: Modules for the Redisplay Mechanism,  Up: The Redisplay Mechanism

27.5 Modules for other Display-Related Lisp Objects
===================================================

     `faces.c'
     `faces.h'

     `bitmaps.h'
     `glyphs-eimage.c'
     `glyphs-msw.c'
     `glyphs-msw.h'
     `glyphs-widget.c'
     `glyphs-x.c'
     `glyphs-x.h'
     `glyphs.c'
     `glyphs.h'

     `objects-msw.c'
     `objects-msw.h'
     `objects-tty.c'
     `objects-tty.h'
     `objects-x.c'
     `objects-x.h'
     `objects.c'
     `objects.h'

     `menubar-msw.c'
     `menubar-msw.h'
     `menubar-x.c'
     `menubar.c'
     `menubar.h'

     `scrollbar-msw.c'
     `scrollbar-msw.h'
     `scrollbar-x.c'
     `scrollbar-x.h'
     `scrollbar.c'
     `scrollbar.h'

     `toolbar-msw.c'
     `toolbar-x.c'
     `toolbar.c'
     `toolbar.h'

     `font-lock.c'

   This file provides C support for syntax highlighting--i.e.
highlighting different syntactic constructs of a source file in
different colors, for easy reading.  The C support is provided so that
this is fast.

     `dgif_lib.c'
     `gif_err.c'
     `gif_lib.h'
     `gifalloc.c'

   These modules decode GIF-format image files, for use with glyphs.
These files were removed due to Unisys patent infringement concerns.


File: internals.info,  Node: Extents,  Next: Faces,  Prev: The Redisplay Mechanism,  Up: Top

28 Extents
**********

* Menu:

* Introduction to Extents::     Extents are ranges over text, with properties.
* Extent Ordering::             How extents are ordered internally.
* Format of the Extent Info::   The extent information in a buffer or string.
* Zero-Length Extents::         A weird special case.
* Mathematics of Extent Ordering::  A rigorous foundation.
* Extent Fragments::            Cached information useful for redisplay.


File: internals.info,  Node: Introduction to Extents,  Next: Extent Ordering,  Prev: Extents,  Up: Extents

28.1 Introduction to Extents
============================

Extents are regions over a buffer, with a start and an end position
denoting the region of the buffer included in the extent.  In addition,
either end can be closed or open, meaning that the endpoint is or is
not logically included in the extent.  Insertion of a character at a
closed endpoint causes the character to go inside the extent; insertion
at an open endpoint causes the character to go outside.

   Extent endpoints are stored using memory indices (see `insdel.c'),
to minimize the amount of adjusting that needs to be done when
characters are inserted or deleted.

   (Formerly, extent endpoints at the gap could be either before or
after the gap, depending on the open/closedness of the endpoint.  The
intent of this was to make it so that insertions would automatically go
inside or out of extents as necessary with no further work needing to
be done.  It didn't work out that way, however, and just ended up
complexifying and buggifying all the rest of the code.)


File: internals.info,  Node: Extent Ordering,  Next: Format of the Extent Info,  Prev: Introduction to Extents,  Up: Extents

28.2 Extent Ordering
====================

Extents are compared using memory indices.  There are two orderings for
extents and both orders are kept current at all times.  The normal or
"display" order is as follows:

     Extent A is ``less than'' extent B,
     that is, earlier in the display order,
       if:    A-start < B-start,
       or if: A-start = B-start, and A-end > B-end

   So if two extents begin at the same position, the larger of them is
the earlier one in the display order (`EXTENT_LESS' is true).

   For the e-order, the same thing holds:

     Extent A is ``less than'' extent B in e-order,
     that is, later in the buffer,
       if:    A-end < B-end,
       or if: A-end = B-end, and A-start > B-start

   So if two extents end at the same position, the smaller of them is
the earlier one in the e-order (`EXTENT_E_LESS' is true).

   The display order and the e-order are complementary orders: any
theorem about the display order also applies to the e-order if you swap
all occurrences of "display order" and "e-order", "less than" and
"greater than", and "extent start" and "extent end".


File: internals.info,  Node: Format of the Extent Info,  Next: Zero-Length Extents,  Prev: Extent Ordering,  Up: Extents

28.3 Format of the Extent Info
==============================

An extent-info structure consists of a list of the buffer or string's
extents and a "stack of extents" that lists all of the extents over a
particular position.  The stack-of-extents info is used for
optimization purposes--it basically caches some info that might be
expensive to compute.  Certain otherwise hard computations are easy
given the stack of extents over a particular position, and if the stack
of extents over a nearby position is known (because it was calculated
at some prior point in time), it's easy to move the stack of extents to
the proper position.

   Given that the stack of extents is an optimization, and given that
it requires memory, a string's stack of extents is wiped out each time
a garbage collection occurs.  Therefore, any time you retrieve the
stack of extents, it might not be there.  If you need it to be there,
use the `_force' version.

   Similarly, a string may or may not have an extent_info structure.
(Generally it won't if there haven't been any extents added to the
string.) So use the `_force' version if you need the extent_info
structure to be there.

   A list of extents is maintained as a double gap array.  One gap array
is ordered by start index (the "display order") and the other is
ordered by end index (the "e-order").  Note that positions in an extent
list should logically be conceived of as referring _to_ a particular
extent (as is the norm in programs) rather than sitting between two
extents.  Note also that callers of these functions should not be aware
of the fact that the extent list is implemented as an array, except for
the fact that positions are integers (this should be generalized to
handle integers and linked list equally well).

   A gap array is the same structure used by buffer text: an array of
elements with a "gap" somewhere in the middle.  Insertion and deletion
happens by moving the gap to the insertion/deletion point, and then
expanding/contracting as necessary.  Gap arrays have a number of useful
properties:

  1. They are space efficient, as there is no need for next/previous
     pointers.

  2. If the items in them are sorted, locating an item is fast - O(log
     N).

  3. Insertion and deletion is very fast (constant time, essentially)
     if the gap is near (which favors localized operations, as will
     usually be the case).  Even if not, it requires only a block move
     of memory, which is generally a highly optimized operation on
     modern processors.

  4. Code to manipulate them is relatively simple to write.

   An alternative would be balanced binary trees, which have guaranteed
O(log N) time for all operations (although the constant factors are not
as good, and repeated localized operations will be slower than for a
gap array).  Such code is quite tricky to write, however.


File: internals.info,  Node: Zero-Length Extents,  Next: Mathematics of Extent Ordering,  Prev: Format of the Extent Info,  Up: Extents

28.4 Zero-Length Extents
========================

Extents can be zero-length, and will end up that way if their endpoints
are explicitly set that way or if their detachable property is `nil'
and all the text in the extent is deleted. (The exception is open-open
zero-length extents, which are barred from existing because there is no
sensible way to define their properties.  Deletion of the text in an
open-open extent causes it to be converted into a closed-open extent.)
Zero-length extents are primarily used to represent annotations, and
behave as follows:

  1. Insertion at the position of a zero-length extent expands the
     extent if both endpoints are closed; goes after the extent if it
     is closed-open; and goes before the extent if it is open-closed.

  2. Deletion of a character on a side of a zero-length extent whose
     corresponding endpoint is closed causes the extent to be detached
     if it is detachable; if the extent is not detachable or the
     corresponding endpoint is open, the extent remains in the buffer,
     moving as necessary.

   Note that closed-open, non-detachable zero-length extents behave
exactly like markers and that open-closed, non-detachable zero-length
extents behave like the "point-type" marker in Mule.


File: internals.info,  Node: Mathematics of Extent Ordering,  Next: Extent Fragments,  Prev: Zero-Length Extents,  Up: Extents

28.5 Mathematics of Extent Ordering
===================================

The extents in a buffer are ordered by "display order" because that is
that order that the redisplay mechanism needs to process them in.  The
e-order is an auxiliary ordering used to facilitate operations over
extents.  The operations that can be performed on the ordered list of
extents in a buffer are

  1. Locate where an extent would go if inserted into the list.

  2. Insert an extent into the list.

  3. Remove an extent from the list.

  4. Map over all the extents that overlap a range.

   (4) requires being able to determine the first and last extents that
overlap a range.

   NOTE: "overlap" is used as follows:

   * two ranges overlap if they have at least one point in common.
     Whether the endpoints are open or closed makes a difference here.

   * a point overlaps a range if the point is contained within the
     range; this is equivalent to treating a point P as the range [P,
     P].

   * In the case of an _extent_ overlapping a point or range, the extent
     is normally treated as having closed endpoints.  This applies
     consistently in the discussion of stacks of extents and such below.
     Note that this definition of overlap is not necessarily consistent
     with the extents that `map-extents' maps over, since `map-extents'
     sometimes pays attention to whether the endpoints of an extents
     are open or closed.  But for our purposes, it greatly simplifies
     things to treat all extents as having closed endpoints.

   First, define >, <, <=, etc. as applied to extents to mean
comparison according to the display order.  Comparison between an
extent E and an index I means comparison between E and the range [I, I].

   Also define e>, e<, e<=, etc. to mean comparison according to the
e-order.

   For any range R, define R(0) to be the starting index of the range
and R(1) to be the ending index of the range.

   For any extent E, define E(next) to be the extent directly following
E, and E(prev) to be the extent directly preceding E.  Assume E(next)
and E(prev) can be determined from E in constant time.  (This is
because we store the extent list as a doubly linked list.)

   Similarly, define E(e-next) and E(e-prev) to be the extents directly
following and preceding E in the e-order.

   Now:

   Let R be a range.  Let F be the first extent overlapping R.  Let L
be the last extent overlapping R.

   Theorem 1: R(1) lies between L and L(next), i.e. L <= R(1) < L(next).

   This follows easily from the definition of display order.  The basic
reason that this theorem applies is that the display order sorts by
increasing starting index.

   Therefore, we can determine L just by looking at where we would
insert R(1) into the list, and if we know F and are moving forward over
extents, we can easily determine when we've hit L by comparing the
extent we're at to R(1).

     Theorem 2: F(e-prev) e< [1, R(0)] e<= F.

   This is the analog of Theorem 1, and applies because the e-order
sorts by increasing ending index.

   Therefore, F can be found in the same amount of time as operation
(1), i.e. the time that it takes to locate where an extent would go if
inserted into the e-order list.  This is O(log N), since we are using
gap arrays to manage extents.

   Define a "stack of extents" (or "SOE") as the set of extents
(ordered in display order and e-order, just like for normal extent
lists) that overlap an index I.

   Now:

   Let I be an index, let S be the stack of extents on I and let F be
the first extent in S.

   Theorem 3: The first extent in S is the first extent that overlaps
any range [I, J].

   Proof: Any extent that overlaps [I, J] but does not include I must
have a start index > I, and thus be greater than any extent in S.

   Therefore, finding the first extent that overlaps a range R is the
same as finding the first extent that overlaps R(0).

   Theorem 4: Let I2 be an index such that I2 > I, and let F2 be the
first extent that overlaps I2.  Then, either F2 is in S or F2 is
greater than any extent in S.

   Proof: If F2 does not include I then its start index is greater than
I and thus it is greater than any extent in S, including F.  Otherwise,
F2 includes I and thus is in S, and thus F2 >= F.


File: internals.info,  Node: Extent Fragments,  Prev: Mathematics of Extent Ordering,  Up: Extents

28.6 Extent Fragments
=====================

Imagine that the buffer is divided up into contiguous, non-overlapping
"runs" of text such that no extent starts or ends within a run (extents
that abut the run don't count).

   An extent fragment is a structure that holds data about the run that
contains a particular buffer position (if the buffer position is at the
junction of two runs, the run after the position is used)--the
beginning and end of the run, a list of all of the extents in that run,
the "merged face" that results from merging all of the faces
corresponding to those extents, the begin and end glyphs at the
beginning of the run, etc.  This is the information that redisplay needs
in order to display this run.

   Extent fragments have to be very quick to update to a new buffer
position when moving linearly through the buffer.  They rely on the
stack-of-extents code, which does the heavy-duty algorithmic work of
determining which extents overly a particular position.


File: internals.info,  Node: Faces,  Next: Glyphs,  Prev: Extents,  Up: Top

29 Faces
********

Not yet documented.


File: internals.info,  Node: Glyphs,  Next: Specifiers,  Prev: Faces,  Up: Top

30 Glyphs
*********

Glyphs are graphical elements that can be displayed in XEmacs buffers or
gutters. We use the term graphical element here in the broadest possible
sense since glyphs can be as mundane as text or as arcane as a native
tab widget.

   In XEmacs, glyphs represent the uninstantiated state of graphical
elements, i.e. they hold all the information necessary to produce an
image on-screen but the image need not exist at this stage, and multiple
screen images can be instantiated from a single glyph.

   Glyphs are lazily instantiated by calling one of the glyph
functions. This usually occurs within redisplay when `Fglyph_height' is
called. Instantiation causes an image-instance to be created and
cached. This cache is on a per-device basis for all glyphs except
widget-glyphs, and on a per-window basis for widgets-glyphs.  The
caching is done by `image_instantiate' and is necessary because it is
generally possible to display an image-instance in multiple domains.
For instance if we create a Pixmap, we can actually display this on
multiple windows - even though we only need a single Pixmap instance to
do this. If caching wasn't done then it would be necessary to create
image-instances for every displayable occurrence of a glyph - and every
usage - and this would be extremely memory and cpu intensive.

   Widget-glyphs (a.k.a native widgets) are not cached in this way.
This is because widget-glyph image-instances on screen are toolkit
windows, and thus cannot be reused in multiple XEmacs domains. Thus
widget-glyphs are cached on an XEmacs window basis.

   Any action on a glyph first consults the cache before actually
instantiating a widget.

30.1 Glyph Instantiation
========================

Glyph instantiation is a hairy topic and requires some explanation. The
guts of glyph instantiation is contained within `image_instantiate'. A
glyph contains an image which is a specifier. When a glyph function -
for instance `Fglyph_height' - asks for a property of the glyph that
can only be determined from its instantiated state, then the glyph
image is instantiated and an image instance created. The instantiation
process is governed by the specifier code and goes through a series of
steps:

   * Validation. Instantiation of image instances happens dynamically -
     often within the guts of redisplay. Thus it is often not feasible
     to catch instantiator errors at instantiation time. Instead the
     instantiator is validated at the time it is added to the image
     specifier. This function is defined by `image_validate' and at a
     simple level validates keyword value pairs.

   * Duplication. The specifier code by default takes a copy of the
     instantiator. This is reasonable for most specifiers but in the
     case of widget-glyphs can be problematic, since some of the
     properties in the instantiator - for instance callbacks - could
     cause infinite recursion in the copying process. Thus the image
     code defines a function - `image_copy_instantiator' - which will
     selectively copy values.  This is controlled by the way that a
     keyword is defined either using `IIFORMAT_VALID_KEYWORD' or
     `IIFORMAT_VALID_NONCOPY_KEYWORD'. Note that the image caching and
     redisplay code relies on instantiator copying to ensure that
     current and new instantiators are actually different rather than
     referring to the same thing.

   * Normalization. Once the instantiator has been copied it must be
     converted into a form that is viable at instantiation time. This
     can involve no changes at all, but typically involves things like
     converting file names to the actual data. This function is defined
     by `image_going_to_add' and `normalize_image_instantiator'.

   * Instantiation. When an image instance is actually required for
     display it is instantiated using `image_instantiate'. This
     involves calling instantiate methods that are specific to the type
     of image being instantiated.

   The final instantiation phase also involves a number of steps. In
order to understand these we need to describe a number of concepts.

   An image is instantiated in a "domain", where a domain can be any
one of a device, frame, window or image-instance. The domain gives the
image-instance context and identity and properties that affect the
appearance of the image-instance may be different for the same glyph
instantiated in different domains. An example is the face used to
display the image-instance.

   Although an image is instantiated in a particular domain the
instantiation domain is not necessarily the domain in which the
image-instance is cached. For example a pixmap can be instantiated in a
window be actually be cached on a per-device basis. The domain in which
the image-instance is actually cached is called the "governing-domain".
A governing-domain is currently either a device or a window.
Widget-glyphs and text-glyphs have a window as a governing-domain, all
other image-instances have a device as the governing-domain. The
governing domain for an image-instance is determined using the
governing_domain image-instance method.

30.2 Widget-Glyphs
==================

30.3 Widget-Glyphs in the MS-Windows Environment
================================================

To Do

30.4 Widget-Glyphs in the X Environment
=======================================

Widget-glyphs under X make heavy use of lwlib (*note Lucid Widget
Library::) for manipulating the native toolkit objects. This is
primarily so that different toolkits can be supported for
widget-glyphs, just as they are supported for features such as menubars
etc.

   Lwlib is extremely poorly documented and quite hairy so here is my
understanding of what goes on.

   Lwlib maintains a set of widget_instances which mirror the
hierarchical state of Xt widgets. I think this is so that widgets can
be updated and manipulated generically by the lwlib library. For
instance update_one_widget_instance can cope with multiple types of
widget and multiple types of toolkit. Each element in the widget
hierarchy is updated from its corresponding widget_instance by walking
the widget_instance tree recursively.

   This has desirable properties such as lw_modify_all_widgets which is
called from `glyphs-x.c' and updates all the properties of a widget
without having to know what the widget is or what toolkit it is from.
Unfortunately this also has hairy properties such as making the lwlib
code quite complex. And of course lwlib has to know at some level what
the widget is and how to set its properties.


File: internals.info,  Node: Specifiers,  Next: Menus,  Prev: Glyphs,  Up: Top

31 Specifiers
*************

Not yet documented.

   Specifiers are documented in depth in the Lisp Reference manual.
*Note Specifiers: (lispref)Specifiers.  The code in `specifier.c' is
pretty straightforward.


File: internals.info,  Node: Menus,  Next: Events and the Event Loop,  Prev: Specifiers,  Up: Top

32 Menus
********

A menu is set by setting the value of the variable `current-menubar'
(which may be buffer-local) and then calling `set-menubar-dirty-flag'
to signal a change.  This will cause the menu to be redrawn at the next
redisplay.  The format of the data in `current-menubar' is described in
`menubar.c'.

   Internally the data in current-menubar is parsed into a tree of
`widget_value's' (defined in `lwlib.h'); this is accomplished by the
recursive function `menu_item_descriptor_to_widget_value()', called by
`compute_menubar_data()'.  Such a tree is deallocated using
`free_widget_value()'.

   `update_screen_menubars()' is one of the external entry points.
This checks to see, for each screen, if that screen's menubar needs to
be updated.  This is the case if

  1. `set-menubar-dirty-flag' was called since the last redisplay.
     (This function sets the C variable menubar_has_changed.)

  2. The buffer displayed in the screen has changed.

  3. The screen has no menubar currently displayed.

   `set_screen_menubar()' is called for each such screen.  This
function calls `compute_menubar_data()' to create the tree of
widget_value's, then calls `lw_create_widget()',
`lw_modify_all_widgets()', and/or `lw_destroy_all_widgets()' to create
the X-Toolkit widget associated with the menu.

   `update_psheets()', the other external entry point, actually changes
the menus being displayed.  It uses the widgets fixed by
`update_screen_menubars()' and calls various X functions to ensure that
the menus are displayed properly.

   The menubar widget is set up so that `pre_activate_callback()' is
called when the menu is first selected (i.e. mouse button goes down),
and `menubar_selection_callback()' is called when an item is selected.
`pre_activate_callback()' calls the function in activate-menubar-hook,
which can change the menubar (this is described in `menubar.c').  If
the menubar is changed, `set_screen_menubars()' is called.
`menubar_selection_callback()' enqueues a menu event, putting in it a
function to call (either `eval' or `call-interactively') and its
argument, which is the callback function or form given in the menu's
description.


File: internals.info,  Node: Events and the Event Loop,  Next: Asynchronous Events; Quit Checking,  Prev: Menus,  Up: Top

33 Events and the Event Loop
****************************

* Menu:

* Introduction to Events::
* Main Loop::
* Specifics of the Event Gathering Mechanism::
* Specifics About the Emacs Event::
* Event Queues::
* Event Stream Callback Routines::
* Other Event Loop Functions::
* Stream Pairs::
* Converting Events::
* Dispatching Events; The Command Builder::
* Focus Handling::
* Editor-Level Control Flow Modules::


File: internals.info,  Node: Introduction to Events,  Next: Main Loop,  Prev: Events and the Event Loop,  Up: Events and the Event Loop

33.1 Introduction to Events
===========================

An event is an object that encapsulates information about an
interesting occurrence in the operating system.  Events are generated
either by user action, direct (e.g. typing on the keyboard or moving
the mouse) or indirect (moving another window, thereby generating an
expose event on an Emacs frame), or as a result of some other typically
asynchronous action happening, such as output from a subprocess being
ready or a timer expiring.  Events come into the system in an
asynchronous fashion (typically through a callback being called) and
are converted into a synchronous event queue (first-in, first-out) in a
process that we will call "collection".

   Note that each application has its own event queue. (It is
immaterial whether the collection process directly puts the events in
the proper application's queue, or puts them into a single system
queue, which is later split up.)

   The most basic level of event collection is done by the operating
system or window system.  Typically, XEmacs does its own event
collection as well.  Often there are multiple layers of collection in
XEmacs, with events from various sources being collected into a queue,
which is then combined with other sources to go into another queue
(i.e. a second level of collection), with perhaps another level on top
of this, etc.

   XEmacs has its own types of events (called "Emacs events"), which
provides an abstract layer on top of the system-dependent nature of the
most basic events that are received.  Part of the complex nature of the
XEmacs event collection process involves converting from the
operating-system events into the proper Emacs events--there may not be
a one-to-one correspondence.

   Emacs events are documented in `events.h'; I'll discuss them later.


File: internals.info,  Node: Main Loop,  Next: Specifics of the Event Gathering Mechanism,  Prev: Introduction to Events,  Up: Events and the Event Loop

33.2 Main Loop
==============

The "command loop" is the top-level loop that the editor is always
running.  It loops endlessly, calling `next-event' to retrieve an event
and `dispatch-event' to execute it. `dispatch-event' does the
appropriate thing with non-user events (process, timeout, magic, eval,
mouse motion); this involves calling a Lisp handler function, redrawing
a newly-exposed part of a frame, reading subprocess output, etc.  For
user events, `dispatch-event' looks up the event in relevant keymaps or
menubars; when a full key sequence or menubar selection is reached, the
appropriate function is executed. `dispatch-event' may have to keep
state across calls; this is done in the "command-builder" structure
associated with each console (remember, there's usually only one
console), and the engine that looks up keystrokes and constructs full
key sequences is called the "command builder".  This is documented
elsewhere.

   The guts of the command loop are in `command_loop_1()'.  This
function doesn't catch errors, though--that's the job of
`command_loop_2()', which is a condition-case (i.e. error-trapping)
wrapper around `command_loop_1()'.  `command_loop_1()' never returns,
but may get thrown out of.

   When an error occurs, `cmd_error()' is called, which usually invokes
the Lisp error handler in `command-error'; however, a default error
handler is provided if `command-error' is `nil' (e.g. during startup).
The purpose of the error handler is simply to display the error message
and do associated cleanup; it does not need to throw anywhere.  When
the error handler finishes, the condition-case in `command_loop_2()'
will finish and `command_loop_2()' will reinvoke `command_loop_1()'.

   `command_loop_2()' is invoked from three places: from
`initial_command_loop()' (called from `main()' at the end of internal
initialization), from the Lisp function `recursive-edit', and from
`call_command_loop()'.

   `call_command_loop()' is called when a macro is started and when the
minibuffer is entered; normal termination of the macro or minibuffer
causes a throw out of the recursive command loop. (To
`execute-kbd-macro' for macros and `exit' for minibuffers.  Note also
that the low-level minibuffer-entering function,
`read-minibuffer-internal', provides its own error handling and does
not need `command_loop_2()''s error encapsulation; so it tells
`call_command_loop()' to invoke `command_loop_1()' directly.)

   Note that both read-minibuffer-internal and recursive-edit set up a
catch for `exit'; this is why `abort-recursive-edit', which throws to
this catch, exits out of either one.

   `initial_command_loop()', called from `main()', sets up a catch for
`top-level' when invoking `command_loop_2()', allowing functions to
throw all the way to the top level if they really need to.  Before
invoking `command_loop_2()', `initial_command_loop()' calls
`top_level_1()', which handles all of the startup stuff (creating the
initial frame, handling the command-line options, loading the user's
`.emacs' file, etc.).  The function that actually does this is in Lisp
and is pointed to by the variable `top-level'; normally this function is
`normal-top-level'.  `top_level_1()' is just an error-handling wrapper
similar to `command_loop_2()'.  Note also that `initial_command_loop()'
sets up a catch for `top-level' when invoking `top_level_1()', just
like when it invokes `command_loop_2()'.


File: internals.info,  Node: Specifics of the Event Gathering Mechanism,  Next: Specifics About the Emacs Event,  Prev: Main Loop,  Up: Events and the Event Loop

33.3 Specifics of the Event Gathering Mechanism
===============================================

Here is an approximate diagram of the collection processes at work in
XEmacs, under TTY's (TTY's are simpler than X so we'll look at this
first):


      asynch.      asynch.    asynch.   asynch.             [Collectors in
     kbd events  kbd events   process   process                the OS]
           |         |         output    output
           |         |           |         |
           |         |           |         |      SIGINT,   [signal handlers
           |         |           |         |      SIGQUIT,     in XEmacs]
           V         V           V         V      SIGWINCH,
          file      file        file      file    SIGALRM
          desc.     desc.       desc.     desc.     |
          (TTY)     (TTY)       (pipe)    (pipe)    |
           |          |          |         |      fake    timeouts
           |          |          |         |      file        |
           |          |          |         |      desc.       |
           |          |          |         |      (pipe)      |
           |          |          |         |        |         |
           |          |          |         |        |         |
           |          |          |         |        |         |
           V          V          V         V        V         V
           ------>-----------<----------------<----------------
                       |
                       |
                       | [collected using `select()' in `emacs_tty_next_event()'
                       |  and converted to the appropriate Emacs event]
                       |
                       |
                       V          (above this line is TTY-specific)
                     Emacs -----------------------------------------------
                     event (below this line is the generic event mechanism)
                       |
                       |
     was there     if not, call
     a SIGINT?  `emacs_tty_next_event()'
         |             |
         |             |
         |             |
         V             V
         --->------<----
                |
                |     [collected in `event_stream_next_event()';
                |      SIGINT is converted using `maybe_read_quit_event()']
                V
              Emacs
              event
                |
                \---->------>----- maybe_kbd_translate() ---->---\
                                                                 |
                                                                 |
                                                                 |
          command event queue                                    |
                                                    if not from command
       (contains events that were                   event queue, call
       read earlier but not processed,              `event_stream_next_event()'
       typically when waiting in a                               |
       sit-for, sleep-for, etc. for                              |
      a particular event to be received)                         |
                    |                                            |
                    |                                            |
                    V                                            V
                    ---->------------------------------------<----
                                                    |
                                                    | [collected in
                                                    |  `next_event_internal()']
                                                    |
      unread-     unread-       event from          |
      command-    command-       keyboard       else, call
      events      event           macro      `next_event_internal()'
        |           |               |               |
        |           |               |               |
        |           |               |               |
        V           V               V               V
        --------->----------------------<------------
                          |
                          |      [collected in `next-event', which may loop
                          |       more than once if the event it gets is on
                          |       a dead frame, device, etc.]
                          |
                          |
                          V
                 feed into top-level event loop,
                 which repeatedly calls `next-event'
                 and then dispatches the event
                 using `dispatch-event'

   Notice the separation between TTY-specific and generic event
mechanism.  When using the Xt-based event loop, the TTY-specific stuff
is replaced but the rest stays the same.

   It's also important to realize that only one different kind of
system-specific event loop can be operating at a time, and must be able
to receive all kinds of events simultaneously.  For the two existing
event loops (implemented in `event-tty.c' and `event-Xt.c',
respectively), the TTY event loop _only_ handles TTY consoles, while
the Xt event loop handles _both_ TTY and X consoles.  This situation is
different from all of the output handlers, where you simply have one
per console type.

   Here's the Xt Event Loop Diagram (notice that below a certain point,
it's the same as the above diagram):

     asynch. asynch. asynch. asynch.                 [Collectors in
      kbd     kbd    process process                    the OS]
     events  events  output  output
       |       |       |       |
       |       |       |       |     asynch. asynch. [Collectors in the
       |       |       |       |       X        X     OS and X Window System]
       |       |       |       |     events  events
       |       |       |       |       |        |
       |       |       |       |       |        |
       |       |       |       |       |        |    SIGINT, [signal handlers
       |       |       |       |       |        |    SIGQUIT,   in XEmacs]
       |       |       |       |       |        |    SIGWINCH,
       |       |       |       |       |        |    SIGALRM
       |       |       |       |       |        |       |
       |       |       |       |       |        |       |
       |       |       |       |       |        |       |      timeouts
       |       |       |       |       |        |       |          |
       |       |       |       |       |        |       |          |
       |       |       |       |       |        |       V          |
       V       V       V       V       V        V      fake        |
      file    file    file    file    file     file    file        |
      desc.   desc.   desc.   desc.   desc.    desc.   desc.       |
      (TTY)   (TTY)   (pipe)  (pipe) (socket) (socket) (pipe)      |
       |       |       |       |       |        |       |          |
       |       |       |       |       |        |       |          |
       |       |       |       |       |        |       |          |
       V       V       V       V       V        V       V          V
       --->----------------------------------------<---------<------
            |              |               |
            |              |               |[collected using `select()' in
            |              |               | `_XtWaitForSomething()', called
            |              |               | from `XtAppProcessEvent()', called
            |              |               | in `emacs_Xt_next_event()';
            |              |               | dispatched to various callbacks]
            |              |               |
            |              |               |
       emacs_Xt_        p_s_callback(),    | [popup_selection_callback]
       event_handler()  x_u_v_s_callback(),| [x_update_vertical_scrollbar_
            |           x_u_h_s_callback(),|  callback]
            |           search_callback()  | [x_update_horizontal_scrollbar_
            |              |               |  callback]
            |              |               |
            |              |               |
       enqueue_Xt_       signal_special_   |
       dispatch_event()  Xt_user_event()   |
       [maybe multiple     |               |
        times, maybe 0     |               |
        times]             |               |
            |            enqueue_Xt_       |
            |            dispatch_event()  |
            |              |               |
            |              |               |
            V              V               |
            -->----------<--               |
                   |                       |
                   |                       |
                dispatch             `Xt_what_callback()'
                event                  sets flags
                queue                      |
                   |                       |
                   |                       |
                   |                       |
                   |                       |
                   ---->-----------<--------
                        |
                        |
                        |     [collected and converted as appropriate in
                        |            `emacs_Xt_next_event()']
                        |
                        |
                        V          (above this line is Xt-specific)
                      Emacs ------------------------------------------------
                      event (below this line is the generic event mechanism)
                        |
                        |
     was there      if not, call
     a SIGINT?   `emacs_Xt_next_event()'
         |              |
         |              |
         |              |
         V              V
         --->-------<----
                |
                |        [collected in `event_stream_next_event()';
                |         SIGINT is converted using `maybe_read_quit_event()']
                V
              Emacs
              event
                |
                \---->------>----- maybe_kbd_translate() -->-----\
                                                                 |
                                                                 |
                                                                 |
          command event queue                                    |
                                                   if not from command
       (contains events that were                  event queue, call
       read earlier but not processed,             `event_stream_next_event()'
       typically when waiting in a                               |
       sit-for, sleep-for, etc. for                              |
      a particular event to be received)                         |
                    |                                            |
                    |                                            |
                    V                                            V
                    ---->----------------------------------<------
                                                    |
                                                    | [collected in
                                                    |  `next_event_internal()']
                                                    |
      unread-     unread-       event from          |
      command-    command-       keyboard       else, call
      events      event           macro      `next_event_internal()'
        |           |               |               |
        |           |               |               |
        |           |               |               |
        V           V               V               V
        --------->----------------------<------------
                          |
                          |      [collected in `next-event', which may loop
                          |       more than once if the event it gets is on
                          |       a dead frame, device, etc.]
                          |
                          |
                          V
                 feed into top-level event loop,
                 which repeatedly calls `next-event'
                 and then dispatches the event
                 using `dispatch-event'


File: internals.info,  Node: Specifics About the Emacs Event,  Next: Event Queues,  Prev: Specifics of the Event Gathering Mechanism,  Up: Events and the Event Loop

33.4 Specifics About the Emacs Event
====================================


File: internals.info,  Node: Event Queues,  Next: Event Stream Callback Routines,  Prev: Specifics About the Emacs Event,  Up: Events and the Event Loop

33.5 Event Queues
=================

There are two event queues here - the command event queue (#### which
should be called "deferred event queue" and is in my glyph ws) and the
dispatch event queue. (MS Windows actually has an extra dispatch queue
for non-user events and uses the generic one only for user events.  This
is because user and non-user events in Windows come through the same
place - the window procedure - but under X, it's possible to
selectively process events such that we take all the user events before
the non-user ones. #### In fact, given the way we now drain the queue,
we might need two separate queues, like under Windows.  Need to think
carefully exactly how this works, and should certainly generalize the
two different queues.

   The dispatch queue (which used to occur duplicated inside of each
event implementation) is used for events that have been read from the
window-system event queue(s) and not yet process by
`next_event_internal()'.  It exists for two reasons: (1) because in many
implementations, events often come from the window system by way of
callbacks, and need to push the event to be returned onto a queue; (2)
in order to handle QUIT in a guaranteed correct fashion without
resorting to weird implementation-specific hacks that may or may not
work well, we need to drain the window-system event queues and then look
through to see if there's an event matching quit-char (usually ^G).  the
drained events need to go onto a queue. (There are other, similar cases
where we need to drain the pending events so we can look ahead - for
example, checking for pending expose events under X to avoid excessive
server activity.)

   The command event queue is used *AFTER* an event has been read from
`next_event_internal()', when it needs to be pushed back.  This
includes, for example, `accept-process-output', `sleep-for' and
`wait_delaying_user_input()'.  Eval events and the like, generated by
`enqueue-eval-event', `enqueue_magic_eval_event()', etc. are also
pushed onto this queue.  Some events generated by callbacks are also
pushed onto this queue, #### although maybe shouldn't be.

   The command queue takes precedence over the dispatch queue.

   #### It is worth investigating to see whether both queues are really
needed, and how exactly they should be used.  `enqueue-eval-event', for
example, could certainly push onto the dispatch queue, and all
callbacks maybe should.  `wait_delaying_user_input()' seems to need
both queues, since it can take events from the dispatch queue and push
them onto the command queue; but it perhaps could be rewritten to avoid
this.  #### In general we need to review the handling of these two
queues, figure out exactly what ought to be happening, and document it.


File: internals.info,  Node: Event Stream Callback Routines,  Next: Other Event Loop Functions,  Prev: Event Queues,  Up: Events and the Event Loop

33.6 Event Stream Callback Routines
===================================

There is one object called an event_stream.  This object contains
callback functions for doing the window-system-dependent operations
that XEmacs requires.

   If XEmacs is compiled with support for X11 and the X Toolkit, then
this event_stream structure will contain functions that can cope with
input on XEmacs windows on multiple displays, as well as input from
dumb tty frames.

   If it is desired to have XEmacs able to open frames on the displays
of multiple heterogeneous machines, X11 and SunView, or X11 and NeXT,
for example, then it will be necessary to construct an event_stream
structure that can cope with the given types.  Currently, the only
implemented event_streams are for dumb-ttys, and for X11 plus dumb-ttys,
and for mswindows.

   To implement this for one window system is relatively simple.  To
implement this for multiple window systems is trickier and may not be
possible in all situations, but it's been done for X and TTY.

   Note that these callbacks are *NOT* console methods; that's because
the routines are not specific to a particular console type but must be
able to simultaneously cope with all allowable console types.

   The slots of the event_stream structure:

`next_event_cb'
     A function which fills in an XEmacs_event structure with the next
     event available.  If there is no event available, then this should
     block.

     IMPORTANT: timer events and especially process events *must not* be
     returned if there are events of other types available; otherwise
     you can end up with an infinite loop in `Fdiscard_input()'.

`event_pending_cb'
     A function which says whether there are events to be read.  If
     called with an argument of 0, then this should say whether calling
     the `next_event_cb' will block.  If called with a non-zero
     argument, then this should say whether there are that many
     user-generated events pending (that is, keypresses, mouse-clicks,
     dialog-box selection events, etc.). (This is used for redisplay
     optimization, among other things.)  The difference is that the
     former includes process events and timer events, but the latter
     doesn't.

     If this function is not sure whether there are events to be read,
     it *must* return 0.  Otherwise various undesirable effects will
     occur, such as redisplay not occurring until the next event occurs.

`handle_magic_event_cb'
     XEmacs calls this with an event structure which contains
     window-system dependent information that XEmacs doesn't need to
     know about, but which must happen in order.  If the
     `next_event_cb' never returns an event of type "magic", this will
     never be used.

`format_magic_event_cb'
     Called with a magic event; print a representation of the innards
     of the event to PSTREAM.

`compare_magic_event_cb'
     Called with two magic events; return non-zero if the innards of
     the two are equal, zero otherwise.

`hash_magic_event_cb'
     Called with a magic event; return a hash of the innards of the
     event.

`add_timeout_cb'
     Called with an EMACS_TIME, the absolute time at which a wakeup
     event should be generated; and a void *, which is an arbitrary
     value that will be returned in the timeout event.  The timeouts
     generated by this function should be one-shots: they fire once and
     then disappear.  This callback should return an int id-number
     which uniquely identifies this wakeup.  If an implementation
     doesn't have microseconds or millisecond granularity, it should
     round up to the closest value it can deal with.

`remove_timeout_cb'
     Called with an int, the id number of a wakeup to discard.  This id
     number must have been returned by the `add_timeout_cb'.  If the
     given wakeup has already expired, this should do nothing.

`select_process_cb'

`unselect_process_cb'
     These callbacks tell the underlying implementation to add or
     remove a file descriptor from the list of fds which are polled for
     inferior-process input.  When input becomes available on the given
     process connection, an event of type "process" should be generated.

`select_console_cb'

`unselect_console_cb'
     These callbacks tell the underlying implementation to add or
     remove a console from the list of consoles which are polled for
     user-input.

`select_device_cb'

`unselect_device_cb'
     These callbacks are used by Unixoid event loops (those that use
     `select()' and file descriptors and have a separate input fd per
     device).

`create_io_streams_cb'

`delete_io_streams_cb'
     These callbacks are called by process code to create the input and
     output lstreams which are used for subprocess I/O.

`quitp_cb'
     A handler function called from the `QUIT' macro which should check
     whether the quit character has been typed.  On systems with SIGIO,
     this will not be called unless the `sigio_happened' flag is true
     (it is set from the SIGIO handler).

   XEmacs has its own event structures, which are distinct from the
event structures used by X or any other window system.  It is the job
of the event_stream layer to translate to this format.


File: internals.info,  Node: Other Event Loop Functions,  Next: Stream Pairs,  Prev: Event Stream Callback Routines,  Up: Events and the Event Loop

33.7 Other Event Loop Functions
===============================

`detect_input_pending()' and `input-pending-p' look for input by
calling `event_stream->event_pending_p' and looking in
`[V]unread-command-event' and the `command_event_queue' (they do not
check for an executing keyboard macro, though).

   `discard-input' cancels any command events pending (and any keyboard
macros currently executing), and puts the others onto the
`command_event_queue'.  There is a comment about a "race condition",
which is not a good sign.

   `next-command-event' and `read-char' are higher-level interfaces to
`next-event'.  `next-command-event' gets the next "command" event (i.e.
keypress, mouse event, menu selection, or scrollbar action), calling
`dispatch-event' on any others.  `read-char' calls `next-command-event'
and uses `event_to_character()' to return the character equivalent.
With the right kind of input method support, it is possible for
(read-char) to return a Kanji character.


File: internals.info,  Node: Stream Pairs,  Next: Converting Events,  Prev: Other Event Loop Functions,  Up: Events and the Event Loop

33.8 Stream Pairs
=================

Since there are many possible processes/event loop combinations, the
event code is responsible for creating an appropriate lstream type. The
process implementation does not care about that implementation.

   The Create stream pair function is passed two void* values, which
identify process-dependent 'handles'. The process implementation uses
these handles to communicate with child processes. The function must be
prepared to receive handle types of any process implementation. Since
only one process implementation exists in a particular XEmacs
configuration, preprocessing is a means of compiling in the support for
the code which deals with particular handle types.

   For example, a unixoid type loop, which relies on file descriptors,
may be asked to create a pair of streams by a unix-style process
implementation.  In this case, the handles passed are unix file
descriptors, and the code may deal with these directly. Although, the
same code may be used on Win32 system with X-Windows. In this case,
Win32 process implementation passes handles of type HANDLE, and the
`create_io_streams' function must call appropriate function to get file
descriptors given HANDLEs, so that these descriptors may be passed to
`XtAddInput'.

   The handle given may have special denying value, in which case the
corresponding lstream should not be created.

   The return value of the function is a unique stream identifier. It
is used by processes implementation, in its  platform-independent part.
There is the get_process_from_usid function, which returns process
object given its USID. The event stream is responsible for converting
its internal handle type into USID.

   Example is the TTY event stream. When a file descriptor signals
input, the event loop must determine process to which the input is
destined. Thus, the implementation uses process input stream file
descriptor as USID, by simply casting the fd value to USID type.

   There are two special USID values. One, `USID_ERROR', indicates that
the stream pair cannot be created. The second, `USID_DONTHASH',
indicates that streams are created, but the event stream does not wish
to be able to find the process by its USID. Specifically, if an event
stream implementation never calls `get_process_from_usid', this value
should always be returned, to prevent accumulating useless information
on USID to process relationship.


File: internals.info,  Node: Converting Events,  Next: Dispatching Events; The Command Builder,  Prev: Stream Pairs,  Up: Events and the Event Loop

33.9 Converting Events
======================

`character_to_event()', `event_to_character()', `event-to-character',
and `character-to-event' convert between characters and keypress events
corresponding to the characters.  If the event was not a keypress,
`event_to_character()' returns -1 and `event-to-character' returns
`nil'.  These functions convert between character representation and
the split-up event representation (keysym plus mod keys).


File: internals.info,  Node: Dispatching Events; The Command Builder,  Next: Focus Handling,  Prev: Converting Events,  Up: Events and the Event Loop

33.10 Dispatching Events; The Command Builder
=============================================

Not yet documented.


File: internals.info,  Node: Focus Handling,  Next: Editor-Level Control Flow Modules,  Prev: Dispatching Events; The Command Builder,  Up: Events and the Event Loop

33.11 Focus Handling
====================

Ben's capsule lecture on focus:

   In GNU Emacs `select-frame' never changes the window-manager frame
focus.  All it does is change the "selected frame".  This is similar to
what happens when we call `select-device' or `select-console'.
Whenever an event comes in (including a keyboard event), its frame is
selected; therefore, evaluating `select-frame' in `*scratch*' won't
cause any effects because the next received event (in the same frame)
will cause a switch back to the frame displaying `*scratch*'.

   Whenever a focus-change event is received from the window manager, it
generates a `switch-frame' event, which causes the Lisp function
`handle-switch-frame' to get run.  This basically just runs
`select-frame' (see below, however).

   In GNU Emacs, if you want to have an operation run when a frame is
selected, you supply an event binding for `switch-frame' (and then
maybe call `handle-switch-frame', or something ...).

   In XEmacs, we *do* change the window-manager frame focus as a result
of `select-frame', but not until the next time an event is received, so
that a function that momentarily changes the selected frame won't cause
WM focus flashing. (#### There's something not quite right here; this
is causing the wrong-cursor-focus problems that you occasionally see.
But the general idea is correct.) This approach is winning for people
who use the explicit-focus model, but is trickier to implement.

   We also don't make the `switch-frame' event visible but instead have
`select-frame-hook', which is a better approach.

   There is the problem of surrogate minibuffers, where when we enter
the minibuffer, you essentially want to temporarily switch the WM focus
to the frame with the minibuffer, and switch it back when you exit the
minibuffer.

   GNU Emacs solves this with the crockish `redirect-frame-focus',
which says "for keyboard events received from FRAME, act like they're
coming from FOCUS-FRAME".  I think what this means is that, when a
keyboard event comes in and the event manager is about to select the
event's frame, if that frame has its focus redirected, the redirected-to
frame is selected instead.  That way, if you're in a minibufferless
frame and enter the minibuffer, then all Lisp functions that run see the
selected frame as the minibuffer's frame rather than the minibufferless
frame you came from, so that (e.g.) your typing actually appears in the
minibuffer's frame and things behave sanely.

   There's also some weird logic that switches the redirected frame
focus from one frame to another if Lisp code explicitly calls
`select-frame' (but not if `handle-switch-frame' is called), and saves
and restores the frame focus in window configurations, etc. etc.  All
of this logic is heavily `#if 0''d, with lots of comments saying "No,
this approach doesn't seem to work, so I'm trying this ...  is it
reasonable?  Well, I'm not sure ..." that are a red flag indicating
crockishness.

   Because of our way of doing things, we can avoid all this crock.
Keyboard events never cause a select-frame (who cares what frame they're
associated with?  They come from a console, only).  We change the actual
WM focus to a surrogate minibuffer frame, so we don't have to do any
internal redirection.  In order to get the focus back, I took the
approach in `minibuf.el' of just checking to see if the frame we moved
to is still the selected frame, and move back to the old one if so.
Conceivably we might have to do the weird "tracking" that GNU Emacs does
when `select-frame' is called, but I don't think so.  If the selected
frame moved from the minibuffer frame, then we just leave it there,
figuring that someone knows what they're doing.  Because we don't have
any redirection recorded anywhere, it's safe to do this, and we don't
end up with unwanted redirection.


File: internals.info,  Node: Editor-Level Control Flow Modules,  Prev: Focus Handling,  Up: Events and the Event Loop

33.12 Editor-Level Control Flow Modules
=======================================

     `event-Xt.c'
     `event-msw.c'
     `event-stream.c'
     `event-tty.c'
     `events-mod.h'
     `gpmevent.c'
     `gpmevent.h'
     `events.c'
     `events.h'

   These implement the handling of events (user input and other system
notifications).

   `events.c' and `events.h' define the "event" Lisp object type and
primitives for manipulating it.

   `event-stream.c' implements the basic functions for working with
event queues, dispatching an event by looking it up in relevant keymaps
and such, and handling timeouts; this includes the primitives
`next-event' and `dispatch-event', as well as related primitives such
as `sit-for', `sleep-for', and `accept-process-output'.
(`event-stream.c' is one of the hairiest and trickiest modules in
XEmacs.  Beware!  You can easily mess things up here.)

   `event-Xt.c' and `event-tty.c' implement the low-level interfaces
onto retrieving events from Xt (the X toolkit) and from TTY's (using
`read()' and `select()'), respectively.  The event interface enforces a
clean separation between the specific code for interfacing with the
operating system and the generic code for working with events, by
defining an API of basic, low-level event methods; `event-Xt.c' and
`event-tty.c' are two different implementations of this API.  To add
support for a new operating system (e.g. NeXTstep), one merely needs to
provide another implementation of those API functions.

   Note that the choice of whether to use `event-Xt.c' or `event-tty.c'
is made at compile time!  Or at the very latest, it is made at startup
time.  `event-Xt.c' handles events for _both_ X and TTY frames;
`event-tty.c' is only used when X support is not compiled into XEmacs.
The reason for this is that there is only one event loop in XEmacs:
thus, it needs to be able to receive events from all different kinds of
frames.

     `keymap.c'
     `keymap.h'

   `keymap.c' and `keymap.h' define the "keymap" Lisp object type and
associated methods and primitives. (Remember that keymaps are objects
that associate event descriptions with functions to be called to
"execute" those events; `dispatch-event' looks up events in the
relevant keymaps.)

     `cmdloop.c'

   `cmdloop.c' contains functions that implement the actual editor
command loop--i.e. the event loop that cyclically retrieves and
dispatches events.  This code is also rather tricky, just like
`event-stream.c'.

     `macros.c'
     `macros.h'

   These two modules contain the basic code for defining keyboard
macros.  These functions don't actually do much; most of the code that
handles keyboard macros is mixed in with the event-handling code in
`event-stream.c'.

     `minibuf.c'

   This contains some miscellaneous code related to the minibuffer
(most of the minibuffer code was moved into Lisp by Richard Mlynarik).
This includes the primitives for completion (although filename
completion is in `dired.c'), the lowest-level interface to the
minibuffer (if the command loop were cleaned up, this too could be in
Lisp), and code for dealing with the echo area (this, too, was mostly
moved into Lisp, and the only code remaining is code to call out to
Lisp or provide simple bootstrapping implementations early in temacs,
before the echo-area Lisp code is loaded).


File: internals.info,  Node: Asynchronous Events; Quit Checking,  Next: Lstreams,  Prev: Events and the Event Loop,  Up: Top

34 Asynchronous Events; Quit Checking
*************************************

* Menu:

* Signal Handling::
* Control-G (Quit) Checking::
* Profiling::
* Asynchronous Timeouts::
* Exiting::


File: internals.info,  Node: Signal Handling,  Next: Control-G (Quit) Checking,  Prev: Asynchronous Events; Quit Checking,  Up: Asynchronous Events; Quit Checking

34.1 Signal Handling
====================


File: internals.info,  Node: Control-G (Quit) Checking,  Next: Profiling,  Prev: Signal Handling,  Up: Asynchronous Events; Quit Checking

34.2 Control-G (Quit) Checking
==============================

_Note_: The code to handle QUIT is divided between `lisp.h' and
`signal.c'.  There is also some special-case code in the async timer
code in `event-stream.c' to notice when the poll-for-quit (and
poll-for-sigchld) timers have gone off.

   Here's an overview of how this convoluted stuff works:

  1.  Scattered throughout the XEmacs core code are calls to the macro
     QUIT; This macro checks to see whether a `C-g' has recently been
     pressed and not yet handled, and if so, it handles the `C-g' by
     calling `signal_quit()', which invokes the standard `Fsignal()'
     code, with the error being `Qquit'.  Lisp code can establish
     handlers for this (using `condition-case'), but normally there is
     no handler, and so execution is thrown back to the innermost
     enclosing event loop. (One of the things that happens when
     entering an event loop is that a `condition-case' is established
     that catches *all* calls to `signal', including this one.)

  2. How does the QUIT macro check to see whether `C-g' has been
     pressed; obviously this needs to be extremely fast.  Now for some
     history.  In early Lemacs as inherited from the FSF going back 15
     years or more, there was a great fondness for using SIGIO (which
     is sent whenever there is I/O available on a given socket, tty,
     etc.).  In fact, in GNU Emacs, perhaps even today, all reading of
     events from the X server occurs inside the SIGIO handler!  This is
     crazy, but not completely relevant.  What is relevant is that
     similar stuff happened inside the SIGIO handler for `C-g': it
     searched through all the pending (i.e. not yet delivered to XEmacs
     yet) X events for one that matched `C-g'.  When it saw a match, it
     set Vquit_flag to Qt.  On TTY's, `C-g' is actually mapped to be the
     interrupt character (i.e. it generates SIGINT), and XEmacs's
     handler for this signal sets Vquit_flag to Qt.  Then, sometime
     later after the signal handlers finished and a QUIT macro was
     called, the macro noticed the setting of `Vquit_flag' and used
     this as an indication to call `signal_quit()'.  What
     `signal_quit()' actually does is set `Vquit_flag' to Qnil (so that
     we won't get repeated interruptions from a single `C-g' press) and
     then calls the equivalent of (signal 'quit nil).

  3. Another complication is introduced in that Vquit_flag is actually
     exported to Lisp as `quit-flag'.  This allows users some level of
     control over whether and when `C-g' is processed as quit, esp. in
     combination with `inhibit-quit'.  This is another Lisp variable,
     and if set to non-nil, it inhibits `signal_quit()' from getting
     called, meaning that the `C-g' gets essentially ignored.  But not
     completely: Because the resetting of `quit-flag' happens only in
     `signal_quit()', which isn't getting called, the `C-g' press is
     still noticed, and as soon as `inhibit-quit' is set back to nil, a
     quit will be signalled at the next QUIT macro.  Thus, what
     `inhibit-quit' really does is defer quits until after the quit-
     inhibitted period.

  4. Another consideration, introduced by XEmacs, is critical quitting.
     If you press `Control-Shift-G' instead of just `C-g', `quit-flag'
     is set to `critical' instead of to t.  When QUIT processes this
     value, it *ignores* the value of `inhibit-quit'.  This allows you
     to quit even out of a quit-inhibitted section of code!
     Furthermore, when `signal_quit()' notices that it was invoked as a
     result of a critical quit, it automatically invokes the debugger
     (which otherwise would only happen when `debug-on-quit' is set to
     t).

  5. Well, I explained above about how `quit-flag' gets set correctly,
     but I began with a disclaimer stating that this was the old way of
     doing things.  What's done now?  Well, first of all, the SIGIO
     handler (which formerly checked all pending events to see if
     there's a `C-g') now does nothing but set a flag - or actually two
     flags, something_happened and quit_check_signal_happened.  There
     are two flags because the QUIT macro is now used for more than
     just handling QUIT; it's also used for running asynchronous
     timeout handlers that have recently expired, and perhaps other
     things.  The idea here is that the QUIT macros occur extremely
     often in the code, but only occur at places that are relatively
     safe - in particular, if an error occurs, nothing will get
     completely trashed.

  6. Now, let's look at QUIT again.

  7.  UNFINISHED.  Note, however, that as of the point when this
     comment got committed to CVS (mid-2001), the interaction between
     reading `C-g' as an event and processing it as QUIT was overhauled
     to (for the first time) be understandable and actually work
     correctly.  Now, the way things work is that if `C-g' is pressed
     while XEmacs is blocking at the top level, waiting for a user
     event, it will be read as an event; otherwise, it will cause QUIT.
     (This includes times when XEmacs is blocking, but not waiting for
     a user event, e.g. `accept-process-output' and
     `wait_delaying_user_events()'.)  Formerly, this was supposed to
     happen, but didn't always due to a bizarre and broken scheme,
     documented in `next_event_internal' like this:

          If we read a `C-g', then set `quit-flag' but do not discard
          the `C-g'.  The callers of `next_event_internal()' will do
          one of two things:

            1. set `Vquit_flag' to Qnil. (`next-event' does this.) This
               will cause the ^G to be treated as a normal keystroke.

            2. not change `Vquit_flag' but attempt to enqueue the ^G,
               at which point it will be discarded.  The next time QUIT
               is called, it will notice that `Vquit_flag' was set.

     This required weirdness in `enqueue_command_event_1' like this:

          put the event on the typeahead queue, unless the event is the
          quit char, in which case the `QUIT' which will occur on the
          next trip through this loop is all the processing we should
          do - leaving it on the queue would cause the quit to be
          processed twice.

     And further weirdness elsewhere, none of which made any sense, and
     didn't work, because (e.g.) it required that QUIT never happen
     anywhere inside `next_event_internal()' or any callers when `C-g'
     should be read as a user event, which was impossible to implement
     in practice.

     Now what we do is fairly simple.  Callers of
     `next_event_internal()' that want `C-g' read as a user event call
     `begin_dont_check_for_quit()'.  `next_event_internal()', when it
     gets a `C-g', simply sets `Vquit_flag' (just as when a `C-g' is
     detected during the operation of `QUIT' or `QUITP'), and then
     tries to `QUIT'.  This will fail if blocked by the previous call,
     at which point `next_event_internal()' will return the `C-g' as an
     event.  To unblock things, first set `Vquit_flag' to nil (it was
     set to t when the `C-g' was read, and if we don't reset it, the
     next call to `QUIT' will quit), and then `unbind_to()' the depth
     returned by `begin_dont_check_for_quit()'.  It makes no difference
     is `QUIT' is called a zillion times in `next_event_internal()' or
     anywhere else, because it's blocked and will never signal.

34.2.1 Reentrancy Problems due to QUIT Checking
-----------------------------------------------

Checking for QUIT can do quite a long of things - since it pumps the
event loop, this may cause arbitrary code to get executed, garbage
collection to happen. etc. (In fact, garbage collection cannot happen
because it is inhibited.) This has led to crashes when functions get
called reentrantly when not expecting it.  Example:

Crash - reentrant `re_match_2()'
--------------------------------

       /* dont_check_for_quit is set in three circumstances:

          (1) when we are in the process of changing the window
          configuration.  The frame might be in an inconsistent state,
          which will cause assertion failures if we check for QUIT.

          (2) when we are reading events, and want to read the C-g
          as an event.  The normal check for quit will discard the C-g,
          which would be bad.

          (3) when we're going down with a fatal error.  we're most likely
          in an inconsistent state, and we definitely don't want to be
          interrupted. */

       /* We should *not* conditionalize on Vinhibit_quit, or
          critical-quit (Control-Shift-G) won't work right. */

       /* WARNING: Even calling check_quit(), without actually dispatching
          a quit signal, can result in arbitrary Lisp code getting executed
          -- at least under Windows. (Not to mention obvious Lisp
          invocations like asynchronous timer callbacks.) Here's a sample
          stack trace to demonstrate:

      NTDLL! DbgBreakPoint@0 address 0x77f9eea9
     assert_failed(const char * 0x012d036c, int 4596, const char * 0x012d0354) line 3478
     re_match_2_internal(re_pattern_buffer * 0x012d6780, const unsigned char * 0x00000000, int 0, const unsigned char * 0x022f9328, int 34, int 0, re_registers * 0x012d53d0 search_regs, int 34) line 4596 + 41 bytes
     re_search_2(re_pattern_buffer * 0x012d6780, const char * 0x00000000, int 0, const char * 0x022f9328, int 34, int 0, int 34, re_registers * 0x012d53d0 search_regs, int 34) line 4269 + 37 bytes
     re_search(re_pattern_buffer * 0x012d6780, const char * 0x022f9328, int 34, int 0, int 34, re_registers * 0x012d53d0 search_regs) line 4031 + 37 bytes
     string_match_1(long 31222628, long 30282164, long 28377092, buffer * 0x022fde00, int 0) line 413 + 69 bytes
     Fstring_match(long 31222628, long 30282164, long 28377092, long 28377092) line 436 + 34 bytes
     Ffuncall(int 3, long * 0x008297f8) line 3488 + 168 bytes
     execute_optimized_program(const unsigned char * 0x020ddc50, int 6, long * 0x020ddf50) line 744 + 16 bytes
     funcall_compiled_function(long 34407748, int 1, long * 0x00829aec) line 516 + 53 bytes
     Ffuncall(int 2, long * 0x00829ae8) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x020ddc90, int 4, long * 0x020ddf90) line 744 + 16 bytes
     funcall_compiled_function(long 34407720, int 1, long * 0x00829e28) line 516 + 53 bytes
     Ffuncall(int 2, long * 0x00829e24) line 3523 + 17 bytes
     mapcar1(long 15, long * 0x00829e48, long 34447820, long 34187868) line 2929 + 11 bytes
     Fmapcar(long 34447820, long 34187868) line 3035 + 21 bytes
     Ffuncall(int 3, long * 0x00829f20) line 3488 + 93 bytes
     execute_optimized_program(const unsigned char * 0x020c2b70, int 7, long * 0x020dd010) line 744 + 16 bytes
     funcall_compiled_function(long 34407580, int 2, long * 0x0082a210) line 516 + 53 bytes
     Ffuncall(int 3, long * 0x0082a20c) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x020cf810, int 6, long * 0x020cfb10) line 744 + 16 bytes
     funcall_compiled_function(long 34407524, int 0, long * 0x0082a580) line 516 + 53 bytes
     Ffuncall(int 1, long * 0x0082a57c) line 3523 + 17 bytes
     run_hook_with_args_in_buffer(buffer * 0x022fde00, int 1, long * 0x0082a57c, int 0) line 3980 + 13 bytes
     run_hook_with_args(int 1, long * 0x0082a57c, int 0) line 3993 + 23 bytes
     Frun_hooks(int 1, long * 0x0082a57c) line 3847 + 19 bytes
     run_hook(long 34447484) line 4094 + 11 bytes
     unsafe_handle_wm_initmenu_1(frame * 0x01dbb000) line 736 + 11 bytes
     unsafe_handle_wm_initmenu(long 28377092) line 807 + 11 bytes
     condition_case_1(long 28377116, long (long)* 0x0101c827 unsafe_handle_wm_initmenu(long), long 28377092, long (long, long)* 0x01005fa4 mswindows_modal_loop_error_handler(long, long), long 28377092) line 1692 + 7 bytes
     mswindows_protect_modal_loop(long (long)* 0x0101c827 unsafe_handle_wm_initmenu(long), long 28377092) line 1194 + 32 bytes
     mswindows_handle_wm_initmenu(HMENU__ * 0x00010199, frame * 0x01dbb000) line 826 + 17 bytes
     mswindows_wnd_proc(HWND__ * 0x000501da, unsigned int 278, unsigned int 65945, long 0) line 3089 + 31 bytes
     USER32! UserCallWinProc@20 + 24 bytes
     USER32! DispatchClientMessage@20 + 47 bytes
     USER32! __fnDWORD@4 + 34 bytes
     NTDLL! KiUserCallbackDispatcher@12 + 19 bytes
     USER32! DispatchClientMessage@20 address 0x77e163cc
     USER32! DefWindowProcW@16 + 34 bytes
     qxeDefWindowProc(HWND__ * 0x000501da, unsigned int 274, unsigned int 61696, long 98) line 1188 + 22 bytes
     mswindows_wnd_proc(HWND__ * 0x000501da, unsigned int 274, unsigned int 61696, long 98) line 3362 + 21 bytes
     USER32! UserCallWinProc@20 + 24 bytes
     USER32! DispatchClientMessage@20 + 47 bytes
     USER32! __fnDWORD@4 + 34 bytes
     NTDLL! KiUserCallbackDispatcher@12 + 19 bytes
     USER32! DispatchClientMessage@20 address 0x77e163cc
     USER32! DefWindowProcW@16 + 34 bytes
     qxeDefWindowProc(HWND__ * 0x000501da, unsigned int 262, unsigned int 98, long 540016641) line 1188 + 22 bytes
     mswindows_wnd_proc(HWND__ * 0x000501da, unsigned int 262, unsigned int 98, long 540016641) line 3362 + 21 bytes
     USER32! UserCallWinProc@20 + 24 bytes
     USER32! DispatchMessageWorker@8 + 244 bytes
     USER32! DispatchMessageW@4 + 11 bytes
     qxeDispatchMessage(const tagMSG * 0x0082c684 {msg=0x00000106 wp=0x00000062 lp=0x20300001}) line 989 + 10 bytes
     mswindows_drain_windows_queue() line 1345 + 9 bytes
     emacs_mswindows_quit_p() line 3947
     event_stream_quit_p() line 666
     check_quit() line 686
     check_what_happened() line 437
     re_match_2_internal(re_pattern_buffer * 0x012d5a18, const unsigned char * 0x00000000, int 0, const unsigned char * 0x02235000, int 23486, int 14645, re_registers * 0x012d53d0 search_regs, int 23486) line 4717 + 14 bytes
     re_search_2(re_pattern_buffer * 0x012d5a18, const char * 0x02235000, int 23486, const char * 0x0223b38e, int 0, int 14645, int 8841, re_registers * 0x012d53d0 search_regs, int 23486) line 4269 + 37 bytes
     search_buffer(buffer * 0x022fde00, long 29077572, long 13789, long 23487, long 1, int 1, long 28377092, long 28377092, int 0) line 1224 + 89 bytes
     search_command(long 29077572, long 46975, long 28377116, long 28377092, long 28377092, int 1, int 1, int 0) line 1054 + 151 bytes
     Fre_search_forward(long 29077572, long 46975, long 28377116, long 28377092, long 28377092) line 2147 + 31 bytes
     Ffuncall(int 4, long * 0x0082ceb0) line 3488 + 216 bytes
     execute_optimized_program(const unsigned char * 0x02047810, int 13, long * 0x02080c10) line 744 + 16 bytes
     funcall_compiled_function(long 34187208, int 3, long * 0x0082d1b8) line 516 + 53 bytes
     Ffuncall(int 4, long * 0x0082d1b4) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x01e96a10, int 6, long * 0x020ae510) line 744 + 16 bytes
     funcall_compiled_function(long 34186676, int 3, long * 0x0082d4a0) line 516 + 53 bytes
     Ffuncall(int 4, long * 0x0082d49c) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x02156b50, int 4, long * 0x020c2db0) line 744 + 16 bytes
     funcall_compiled_function(long 34186564, int 2, long * 0x0082d780) line 516 + 53 bytes
     Ffuncall(int 3, long * 0x0082d77c) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x0082d964, int 3, long * 0x020c2d70) line 744 + 16 bytes
     Fbyte_code(long 29405156, long 34352480, long 7) line 2392 + 38 bytes
     Feval(long 34354440) line 3290 + 187 bytes
     condition_case_1(long 34354572, long (long)* 0x01087232 Feval(long), long 34354440, long (long, long)* 0x01084764 run_condition_case_handlers(long, long), long 28377092) line 1692 + 7 bytes
     condition_case_3(long 34354440, long 28377092, long 34354572) line 1779 + 27 bytes
     execute_rare_opcode(long * 0x0082dc7c, const unsigned char * 0x01b090af, int 143) line 1269 + 19 bytes
     execute_optimized_program(const unsigned char * 0x01b09090, int 6, long * 0x020ae590) line 654 + 17 bytes
     funcall_compiled_function(long 34186620, int 0, long * 0x0082df68) line 516 + 53 bytes
     Ffuncall(int 1, long * 0x0082df64) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x02195470, int 1, long * 0x020c2df0) line 744 + 16 bytes
     funcall_compiled_function(long 34186508, int 0, long * 0x0082e23c) line 516 + 53 bytes
     Ffuncall(int 1, long * 0x0082e238) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x01e5d410, int 6, long * 0x0207d410) line 744 + 16 bytes
     funcall_compiled_function(long 34186312, int 1, long * 0x0082e524) line 516 + 53 bytes
     Ffuncall(int 2, long * 0x0082e520) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x02108fb0, int 2, long * 0x020c2e30) line 744 + 16 bytes
     funcall_compiled_function(long 34186340, int 0, long * 0x0082e7fc) line 516 + 53 bytes
     Ffuncall(int 1, long * 0x0082e7f8) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x020fe150, int 2, long * 0x01e6f510) line 744 + 16 bytes
     funcall_compiled_function(long 31008124, int 0, long * 0x0082ebd8) line 516 + 53 bytes
     Ffuncall(int 1, long * 0x0082ebd4) line 3523 + 17 bytes
     run_hook_with_args_in_buffer(buffer * 0x022fde00, int 1, long * 0x0082ebd4, int 0) line 3980 + 13 bytes
     run_hook_with_args(int 1, long * 0x0082ebd4, int 0) line 3993 + 23 bytes
     Frun_hooks(int 1, long * 0x0082ebd4) line 3847 + 19 bytes
     Ffuncall(int 2, long * 0x0082ebd0) line 3509 + 14 bytes
     execute_optimized_program(const unsigned char * 0x01ef2210, int 5, long * 0x01da8e10) line 744 + 16 bytes
     funcall_compiled_function(long 31020440, int 2, long * 0x0082eeb8) line 516 + 53 bytes
     Ffuncall(int 3, long * 0x0082eeb4) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x0082f09c, int 3, long * 0x01d89390) line 744 + 16 bytes
     Fbyte_code(long 31102388, long 30970752, long 7) line 2392 + 38 bytes
     Feval(long 31087568) line 3290 + 187 bytes
     condition_case_1(long 30961240, long (long)* 0x01087232 Feval(long), long 31087568, long (long, long)* 0x01084764 run_condition_case_handlers(long, long), long 28510180) line 1692 + 7 bytes
     condition_case_3(long 31087568, long 28510180, long 30961240) line 1779 + 27 bytes
     execute_rare_opcode(long * 0x0082f450, const unsigned char * 0x01ef23ec, int 143) line 1269 + 19 bytes
     execute_optimized_program(const unsigned char * 0x01ef2310, int 6, long * 0x01da8f10) line 654 + 17 bytes
     funcall_compiled_function(long 31020412, int 1, long * 0x0082f740) line 516 + 53 bytes
     Ffuncall(int 2, long * 0x0082f73c) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x020fe650, int 3, long * 0x01d8c490) line 744 + 16 bytes
     funcall_compiled_function(long 31020020, int 2, long * 0x0082fa14) line 516 + 53 bytes
     Ffuncall(int 3, long * 0x0082fa10) line 3523 + 17 bytes
     Fcall_interactively(long 29685180, long 28377092, long 28377092) line 1008 + 22 bytes
     Fcommand_execute(long 29685180, long 28377092, long 28377092) line 2929 + 17 bytes
     execute_command_event(command_builder * 0x01be1900, long 36626492) line 4048 + 25 bytes
     Fdispatch_event(long 36626492) line 4341 + 70 bytes
     Fcommand_loop_1() line 582 + 9 bytes
     command_loop_1(long 28377092) line 495
     condition_case_1(long 28377188, long (long)* 0x01064fb9 command_loop_1(long), long 28377092, long (long, long)* 0x010649d0 cmd_error(long, long), long 28377092) line 1692 + 7 bytes
     command_loop_3() line 256 + 35 bytes
     command_loop_2(long 28377092) line 269
     internal_catch(long 28457612, long (long)* 0x01064b20 command_loop_2(long), long 28377092, int * volatile 0x00000000) line 1317 + 7 bytes
     initial_command_loop(long 28377092) line 305 + 25 bytes
     STACK_TRACE_EYE_CATCHER(int 1, char * * 0x01b63ff0, char * * 0x01ca5300, int 0) line 2501
     main(int 1, char * * 0x01b63ff0, char * * 0x01ca5300) line 2938
     XEMACS! mainCRTStartup + 180 bytes
     _start() line 171
     KERNEL32! BaseProcessStart@4 + 115547 bytes

   [explain dont_check_for_quit() et al]


File: internals.info,  Node: Profiling,  Next: Asynchronous Timeouts,  Prev: Control-G (Quit) Checking,  Up: Asynchronous Events; Quit Checking

34.3 Profiling
==============

We implement our own profiling scheme so that we can determine things
like which Lisp functions are occupying the most time.  Any standard
OS-provided profiling works on C functions, which is not always that
useful - and inconvenient, since it requires compiling with profile
info and can't be retrieved dynamically, as XEmacs is running.

   The basic idea is simple.  We set a profiling timer using setitimer
(ITIMER_PROF), which generates a SIGPROF every so often.  (This runs not
in real time but rather when the process is executing or the system is
running on behalf of the process - at least, that is the case under
Unix.  Under MS Windows and Cygwin, there is no `setitimer()', so we
simulate it using multimedia timers, which run in real time.  To make
the results a bit more realistic, we ignore ticks that go off while
blocking on an event wait.  Note that Cygwin does provide a simulation
of `setitimer()', but it's in real time anyway, since Windows doesn't
provide a way to have process-time timers, and furthermore, it's broken,
so we don't use it.) When the signal goes off, we see what we're in, and
add 1 to the count associated with that function.

   It would be nice to use the Lisp allocation mechanism etc. to keep
track of the profiling information (i.e. to use Lisp hash tables), but
we can't because that's not safe - updating the timing information
happens inside of a signal handler, so we can't rely on not being in
the middle of Lisp allocation, garbage collection, `malloc()', etc.
Trying to make it work would be much more work than it's worth.
Instead we use a basic (non-Lisp) hash table, which will not conflict
with garbage collection or anything else as long as it doesn't try to
resize itself.  Resizing itself, however (which happens as a result of
a `puthash()'), could be deadly.  To avoid this, we make sure, at
points where it's safe (e.g. `profile_record_about_to_call()' -
recording the entry into a function call), that the table always has
some breathing room in it so that no resizes will occur until at least
that many items are added.  This is safe because any new item to be
added in the sigprof would likely have the
`profile_record_about_to_call()' called just before it, and the
breathing room is checked.

   In general: any entry that the sigprof handler puts into the table
comes from a backtrace frame (except "Processing Events at Top Level",
and there's only one of those).  Either that backtrace frame was added
when profiling was on (in which case `profile_record_about_to_call()'
was called and the breathing space updated), or when it was off - and in
this case, no such frames can have been added since the last time
`start-profile' was called, so when `start-profile' is called we make
sure there is sufficient breathing room to account for all entries
currently on the stack.

   Jan 1998: In addition to timing info, I have added code to remember
call counts of Lisp funcalls.  The `profile_increase_call_count()'
function is called from `Ffuncall()', and serves to add data to
Vcall_count_profile_table.  This mechanism is much simpler and
independent of the SIGPROF-driven one.  It uses the Lisp allocation
mechanism normally, since it is not called from a handler.  It may even
be useful to provide a way to turn on only one profiling mechanism, but
I haven't done so yet.  -hniksic

   Dec 2002: Total overhaul of the interface, making it sane and easier
to use. -ben

   Feb 2003: Lots of rewriting of the internal code.  Add
GC-consing-usage, total GC usage, and total timing to the information
tracked.  Track profiling overhead and allow the ability to have
internal sections (e.g. internal-external conversion, byte-char
conversion) that are treated like Lisp functions for the purpose of
profiling.  -ben

   BEWARE: If you are modifying this file, be *very* careful.  Correctly
implementing the "total" values is very tricky due to the possibility of
recursion and of functions already on the stack when starting to
profile/still on the stack when stopping.


File: internals.info,  Node: Asynchronous Timeouts,  Next: Exiting,  Prev: Profiling,  Up: Asynchronous Events; Quit Checking

34.4 Asynchronous Timeouts
==========================


File: internals.info,  Node: Exiting,  Prev: Asynchronous Timeouts,  Up: Asynchronous Events; Quit Checking

34.5 Exiting
============

Ben's capsule summary about expected and unexpected exits from XEmacs.

   Expected exits occur when the user directs XEmacs to exit, for
example by pressing the close button on the only frame in XEmacs, or by
typing `C-x C-c'.  This runs `save-buffers-kill-emacs', which saves any
necessary buffers, and then exits using the primitive `kill-emacs'.

   However, unexpected exits occur in a few different ways:

   * A memory access violation or other hardware-generated exception
     occurs.  This is the worst possible problem to deal with, because
     the fault can occur while XEmacs is in any state whatsoever, even
     quite unstable ones.  As a result, we need to be *extremely*
     careful what we do.

   * We are using one X display (or if we've used more, we've closed the
     others already), and some hardware or other problem happens and
     suddenly we've lost our connection to the display.  In this
     situation, things are not so dire as in the last one; our code
     itself isn't trashed, so we can continue execution as normal,
     after having set things up so that we can exit at the appropriate
     time.  Our exit still needs to be of the emergency nature; we have
     no displays, so any attempts to use them will fail.  We simply
     want to auto-save (the single most important thing to do during
     shut-down), do minimal cleanup of stuff that has an independent
     existence outside of XEmacs, and exit.

   Currently, both unexpected exit scenarios described above set
`preparing_for_armageddon' to indicate that nonessential and possibly
dangerous things should not be done, specifically:

   - no garbage collection.

   - no hooks are run.

   - no messages of any sort from autosaving.

   - autosaving tries harder, ignoring certain failures.

   - existing frames are not deleted.

   (Also, all places that set `preparing_for_armageddon' also set
`dont_check_for_quit'.  This happens separately because it's also
necessary to set other variables to make absolutely sure no quitting
happens.)

   In the first scenario above (the access violation), we also set
`fatal_error_in_progress'.  This causes more things to not happen:

   - assertion failures do not abort.

   - printing code does not do code conversion or gettext when printing
     to stdout/stderr.


File: internals.info,  Node: Lstreams,  Next: Subprocesses,  Prev: Asynchronous Events; Quit Checking,  Up: Top

35 Lstreams
***********

An "lstream" is an internal Lisp object that provides a generic
buffering stream implementation.  Conceptually, you send data to the
stream or read data from the stream, not caring what's on the other end
of the stream.  The other end could be another stream, a file
descriptor, a stdio stream, a fixed block of memory, a reallocating
block of memory, etc.  The main purpose of the stream is to provide a
standard interface and to do buffering.  Macros are defined to read or
write characters, so the calling functions do not have to worry about
blocking data together in order to achieve efficiency.

* Menu:

* Creating an Lstream::         Creating an lstream object.
* Lstream Types::               Different sorts of things that are streamed.
* Lstream Functions::           Functions for working with lstreams.
* Lstream Methods::             Creating new lstream types.


File: internals.info,  Node: Creating an Lstream,  Next: Lstream Types,  Prev: Lstreams,  Up: Lstreams

35.1 Creating an Lstream
========================

Lstreams come in different types, depending on what is being interfaced
to.  Although the primitive for creating new lstreams is
`Lstream_new()', generally you do not call this directly.  Instead, you
call some type-specific creation function, which creates the lstream
and initializes it as appropriate for the particular type.

   All lstream creation functions take a MODE argument, specifying what
mode the lstream should be opened as.  This controls whether the
lstream is for input and output, and optionally whether data should be
blocked up in units of MULE characters.  Note that some types of
lstreams can only be opened for input; others only for output; and
others can be opened either way.  #### Richard Mlynarik thinks that
there should be a strict separation between input and output streams,
and he's probably right.

   MODE is a string, one of

`"r"'
     Open for reading.

`"w"'
     Open for writing.

`"rc"'
     Open for reading, but "read" never returns partial MULE characters.

`"wc"'
     Open for writing, but never writes partial MULE characters.


File: internals.info,  Node: Lstream Types,  Next: Lstream Functions,  Prev: Creating an Lstream,  Up: Lstreams

35.2 Lstream Types
==================

stdio

filedesc

lisp-string

fixed-buffer

resizing-buffer

dynarr

lisp-buffer

print

decoding

encoding


File: internals.info,  Node: Lstream Functions,  Next: Lstream Methods,  Prev: Lstream Types,  Up: Lstreams

35.3 Lstream Functions
======================

 -- Function: Lstream * Lstream_new (Lstream_implementation *IMP, const
          char *MODE)
     Allocate and return a new Lstream.  This function is not really
     meant to be called directly; rather, each stream type should
     provide its own stream creation function, which creates the stream
     and does any other necessary creation stuff (e.g. opening a file).

 -- Function: void Lstream_set_buffering (Lstream *LSTR,
          Lstream_buffering BUFFERING, int BUFFERING_SIZE)
     Change the buffering of a stream.  See `lstream.h'.  By default the
     buffering is `STREAM_BLOCK_BUFFERED'.

 -- Function: int Lstream_flush (Lstream *LSTR)
     Flush out any pending unwritten data in the stream.  Clear any
     buffered input data.  Returns 0 on success, -1 on error.

 -- Macro: int Lstream_putc (Lstream *STREAM, int C)
     Write out one byte to the stream.  This is a macro and so it is
     very efficient.  The C argument is only evaluated once but the
     STREAM argument is evaluated more than once.  Returns 0 on
     success, -1 on error.

 -- Macro: int Lstream_getc (Lstream *STREAM)
     Read one byte from the stream.  This is a macro and so it is very
     efficient.  The STREAM argument is evaluated more than once.
     Return value is -1 for EOF or error.

 -- Macro: void Lstream_ungetc (Lstream *STREAM, int C)
     Push one byte back onto the input queue.  This will be the next
     byte read from the stream.  Any number of bytes can be pushed back
     and will be read in the reverse order they were pushed back--most
     recent first. (This is necessary for consistency--if there are a
     number of bytes that have been unread and I read and unread a
     byte, it needs to be the first to be read again.) This is a macro
     and so it is very efficient.  The C argument is only evaluated
     once but the STREAM argument is evaluated more than once.

 -- Function: int Lstream_fputc (Lstream *STREAM, int C)
 -- Function: int Lstream_fgetc (Lstream *STREAM)
 -- Function: void Lstream_fungetc (Lstream *STREAM, int C)
     Function equivalents of the above macros.

 -- Function: Bytecount Lstream_read (Lstream *STREAM, void *DATA,
          Bytecount SIZE)
     Read SIZE bytes of DATA from the stream.  Return the number of
     bytes read.  0 means EOF. -1 means an error occurred and no bytes
     were read.

 -- Function: Bytecount Lstream_write (Lstream *STREAM, void *DATA,
          Bytecount SIZE)
     Write SIZE bytes of DATA to the stream.  Return the number of
     bytes written.  -1 means an error occurred and no bytes were
     written.

 -- Function: void Lstream_unread (Lstream *STREAM, void *DATA,
          Bytecount SIZE)
     Push back SIZE bytes of DATA onto the input queue.  The next call
     to `Lstream_read()' with the same size will read the same bytes
     back.  Note that this will be the case even if there is other
     pending unread data.

 -- Function: int Lstream_close (Lstream *STREAM)
     Close the stream.  All data will be flushed out.

 -- Function: void Lstream_reopen (Lstream *STREAM)
     Reopen a closed stream.  This enables I/O on it again.  This is not
     meant to be called except from a wrapper routine that reinitializes
     variables and such--the close routine may well have freed some
     necessary storage structures, for example.

 -- Function: void Lstream_rewind (Lstream *STREAM)
     Rewind the stream to the beginning.


File: internals.info,  Node: Lstream Methods,  Prev: Lstream Functions,  Up: Lstreams

35.4 Lstream Methods
====================

 -- Lstream Method: Bytecount reader (Lstream *STREAM, unsigned char
          *DATA, Bytecount SIZE)
     Read some data from the stream's end and store it into DATA, which
     can hold SIZE bytes.  Return the number of bytes read.  A return
     value of 0 means no bytes can be read at this time.  This may be
     because of an EOF, or because there is a granularity greater than
     one byte that the stream imposes on the returned data, and SIZE is
     less than this granularity. (This will happen frequently for
     streams that need to return whole characters, because
     `Lstream_read()' calls the reader function repeatedly until it has
     the number of bytes it wants or until 0 is returned.)  The lstream
     functions do not treat a 0 return as EOF or do anything special;
     however, the calling function will interpret any 0 it gets back as
     EOF.  This will normally not happen unless the caller calls
     `Lstream_read()' with a very small size.

     This function can be `NULL' if the stream is output-only.

 -- Lstream Method: Bytecount writer (Lstream *STREAM, const unsigned
          char *DATA, Bytecount SIZE)
     Send some data to the stream's end.  Data to be sent is in DATA
     and is SIZE bytes.  Return the number of bytes sent.  This
     function can send and return fewer bytes than is passed in; in that
     case, the function will just be called again until there is no
     data left or 0 is returned.  A return value of 0 means that no
     more data can be currently stored, but there is no error; the data
     will be squirreled away until the writer can accept data. (This is
     useful, e.g., if you're dealing with a non-blocking file
     descriptor and are getting `EWOULDBLOCK' errors.)  This function
     can be `NULL' if the stream is input-only.

 -- Lstream Method: int rewinder (Lstream *STREAM)
     Rewind the stream.  If this is `NULL', the stream is not seekable.

 -- Lstream Method: int seekable_p (Lstream *STREAM)
     Indicate whether this stream is seekable--i.e. it can be rewound.
     This method is ignored if the stream does not have a rewind
     method.  If this method is not present, the result is determined
     by whether a rewind method is present.

 -- Lstream Method: int flusher (Lstream *STREAM)
     Perform any additional operations necessary to flush the data in
     this stream.

 -- Lstream Method: int pseudo_closer (Lstream *STREAM)

 -- Lstream Method: int closer (Lstream *STREAM)
     Perform any additional operations necessary to close this stream
     down.  May be `NULL'.  This function is called when
     `Lstream_close()' is called or when the stream is
     garbage-collected.  When this function is called, all pending data
     in the stream will already have been written out.

 -- Lstream Method: Lisp_Object marker (Lisp_Object LSTREAM, void
          (*MARKFUN) (Lisp_Object))
     Mark this object for garbage collection.  Same semantics as a
     standard `Lisp_Object' marker.  This function can be `NULL'.


File: internals.info,  Node: Subprocesses,  Next: Interface to MS Windows,  Prev: Lstreams,  Up: Top

36 Subprocesses
***************

The fields of a process are:

`name'
     A string, the name of the process.

`command'
     A list containing the command arguments that were used to start
     this process.

`filter'
     A function used to accept output from the process instead of a
     buffer, or `nil'.

`sentinel'
     A function called whenever the process receives a signal, or `nil'.

`buffer'
     The associated buffer of the process.

`pid'
     An integer, the Unix process ID.

`childp'
     A flag, non-`nil' if this is really a child process.  It is `nil'
     for a network connection.

`mark'
     A marker indicating the position of the end of the last output
     from this process inserted into the buffer.  This is often but not
     always the end of the buffer.

`kill_without_query'
     If this is non-`nil', killing XEmacs while this process is still
     running does not ask for confirmation about killing the process.

`raw_status_low'
`raw_status_high'
     These two fields record 16 bits each of the process status
     returned by the `wait' system call.

`status'
     The process status, as `process-status' should return it.

`tick'
`update_tick'
     If these two fields are not equal, a change in the status of the
     process needs to be reported, either by running the sentinel or by
     inserting a message in the process buffer.

`pty_flag'
     Non-`nil' if communication with the subprocess uses a PTY; `nil'
     if it uses a pipe.

`infd'
     The file descriptor for input from the process.

`outfd'
     The file descriptor for output to the process.

`subtty'
     The file descriptor for the terminal that the subprocess is using.
     (On some systems, there is no need to record this, so the value is
     `-1'.)

`tty_name'
     The name of the terminal that the subprocess is using, or `nil' if
     it is using pipes.

* Menu:

* Ben's separate stderr notes::  Probably obsolete.


File: internals.info,  Node: Ben's separate stderr notes,  Prev: Subprocesses,  Up: Subprocesses

36.0.1 Ben's separate stderr notes (probably obsolete)
------------------------------------------------------

This node contains some notes that Ben kept on his separate subprocess
workspace.  These notes probably describe changes and features that have
already been included in XEmacs 21.5; somebody should check and/or ask
Ben.

ben-separate-stderr-improved-error-trapping
===========================================

this is an old workspace, very close to being done, containing

   * subprocess stderr output can be read separately; needed to fully
     implement call-process with asynch. subprocesses.

   * huge improvements to the internal error-trapping routines (i.e. the
     routines that call Lisp code and trap errors); Lisp code can now be
     called from within redisplay.

   * cleanup and simplification of C-g handling; some things work now
     that never used to.

   * see the ChangeLogs in the workspace.


File: internals.info,  Node: Interface to MS Windows,  Next: Interface to the X Window System,  Prev: Subprocesses,  Up: Top

37 Interface to MS Windows
**************************

* Menu:

* Different kinds of Windows environments::
* Windows Build Flags::
* Windows I18N Introduction::
* Modules for Interfacing with MS Windows::
* CHANGES from 21.4-windows branch::  Probably obsolete.


File: internals.info,  Node: Different kinds of Windows environments,  Next: Windows Build Flags,  Prev: Interface to MS Windows,  Up: Interface to MS Windows

37.1 Different kinds of Windows environments
============================================

(a) operating system (OS) vs. window system vs. Win32 API vs. C runtime library (CRT) vs. and compiler
......................................................................................................

There are various Windows operating systems (Windows NT, 2000, XP, 95,
98, ME, etc.), which come in two basic classes: Windows NT (NT, 2000,
XP, and all future versions) and 9x (95, 98, ME).  9x-class operating
systems are a kind of hodgepodge of a 32-bit upper layer on top of a
16-bit MS-DOS-compatible lower layer.  NT-class operating systems are
written from the ground up as 32-bit (there are also 64-bit versions
available now), and provide many more features and much greater
stability, since there is full memory protection between all processes
and the between processes and the system.  NT-class operating systems
also provide emulation for DOS programs inside of a "sandbox" (i.e. a
walled-off environment in which one DOS program can screw up another
one, but there is theoretically no way for a DOS program to screw up the
OS itself).  From the perspective of XEmacs, the different between NT
and 9x is very important in Unicode support (not really provided under
9x - see `intl-win32.c') and subprocess creation, among other things.

   The operating system provides the framework for accessing files and
devices and running programs.  From the perspective of a program, the
operating system provides a set of services.  At the lowest level, the
way to call these services is dependent on the processor the OS is
running on, but a portable interface is provided to C programs through
functions called "system calls".  Under Windows, this interface is
called the Win32 API, and includes file-manipulation calls such as
`CreateFile()' and `ReadFile()', process-creation calls such as
`CreateProcess()', etc.

   This concept of system calls goes back to Unix, where similar
services are available but through routines with different, simpler
names, such as `open()', `read()', `fork()', `execve()', etc.  In
addition, Unix provides a higher layer of routines, called the C
Runtime Library (CRT), which provide higher-level, more convenient
versions of the same services (e.g.  "stream-oriented" file routines
such as `fopen()' and `fread()') as well as various other utility
functions, such as string-manipulation routines (e.g. `strcpy()' and
`strcmp()').

   For compatibility, a C Runtime Library (CRT) is also provided under
Windows, which provides a partial implementation of both the Unix CRT
and the Unix system-call API, implemented using the Win32 API.  The CRT
sources come with Visual C++ (VC++).  For example, under VC++ 6, look in
the CRT/SRC directory, e.g. for me (ben): /Program Files/Microsoft
Visual Studio/VC98/CRT/SRC. The CRT is provided using either MSVCRT
(dynamically linked) or `LIBC.LIB' (statically linked).

   The window system provides the framework for creating overlapped
windows and unifying signals provided by various devices (input devices
such as the keyboard and mouse, timers, etc.) into a single event queue
(or "message queue", under Windows).  Like the operating system, the
window system can be viewed from the perspective of a program as a set
of services provided by an API of function calls.  Under Windows,
window-system services are also available through the Win32 API, while
under UNIX the window system is typically a separate component (e.g. the
X Windowing System, aka X Windows or X11).  The term "GUI" ("graphical
user interface") is often used to refer to the services provided by the
window system, or to a windowing interface provided by a program.

   The Win32 API is implemented by various dynamic libraries, or DLL's.
The most important are KERNEL32, USER32, and GDI32.  KERNEL32 implements
the basic file-system and process services.  USER32 implements the
fundamental window-system services such as creating windows and handling
messages.  GDI32 implements higher-level drawing capabilities - fonts,
colors, lines, etc.

   C programs are compiled into executables using a compiler.  Under
Unix, a compiler usually comes as part of the operating system, but not
under Windows, where the compiler is a separate product.  Even under
Unix, people often install their own compilers, such as gcc.  Under
Windows, the Microsoft-standard compiler is Visual C++ (VC++).

   It is possible to provide an emulation of any API using any other, as
long as the underlying API provides the suitable functionality.  This is
what Cygwin (www.cygwin.com) does.  It provides a fairly complete POSIX
emulation layer (POSIX is a government standard for Unix behavior) on
top of MS Windows - in particular, providing the file-system, process,
tty, and signal semantics that are part of a modern, standard Unix
operating system.  Cygwin does this using its own DLL, `cygwin1.dll',
which makes calls to the Win32 API services in `kernel32.dll'.  Cygwin
also provides its own implementation of the C runtime library, called
`newlib' (`libcygwin.a'; `libc.a' and `libm.a' are symlinked to it),
which is implemented on top of the Unix system calls provided in
`cygwin1.dll'.  In addition, Cygwin provides static import libraries
that give you direct access to the Win32 API - XEmacs uses this to
provide GUI support under Cygwin.  Cygwin provides a version of GCC
(the GNU Project C compiler) that is set up to automatically link with
the appropriate Cygwin libraries.  Cygwin also provides, as optional
components, pre-compiled binaries for a great number of open-source
programs compiled under the Cygwin environment.  This includes all of
the standard Unix file-system, text-manipulation, development,
networking, database, etc. utilities, a version of X Windows that uses
the Win32 API underlyingly (see below), and compilations of nearly all
other common open-source packages (Apache, TeX, [X]Emacs, Ghostscript,
GTK, ImageMagick, etc.).

   Similarly, you can emulate the functionality of X Windows using the
Win32 component of the Win32 API.  Cygwin provides a package to do this,
from the XFree86 project.  Other versions of X under Windows also exist,
such as the MicroImages MI/X server.  Each version potentially can come
comes with its own header and library files, allowing you to compile
X-Windows programs.

   All of these different operating system and emulation layers can make
for a fair amount of confusion, so:

(b) CRT is not the same as VC++
...............................

Note that the CRT is *NOT* (completely) part of VC++.  True, if you link
statically, the CRT (in the form of `LIBC.LIB', which comes with VC++)
will be inserted into the executable (.EXE), but otherwise the CRT will
be separate.  The dynamic version of the CRT is provided by `MSVCRT.DLL'
(or `MSVCRTD.DLL', for debugging), which comes with Windows.  Hence,
it's possible to use a different compiler and still link with MSVCRT -
which is exactly what MinGW does.

(c) CRT is not the same as the Win32 API
........................................

Note also that the CRT is totally separate from the Win32 API.  They
provide different functions and are implemented in different DLL's.
They are also different levels - the CRT is implemented on top of
Win32.  Sometimes the CRT and Win32 both have their own versions of
similar concepts, such as locales.  These are typically maintained
separately, and can get out of sync.  Do not assume that changing a
setting in the CRT will have any effect on Win32 API routines using a
similar concept unless the CRT docs specifically say so.  Do not assume
that behavior described for CRT functions applies to Win32 API or
vice-versa.  Note also that the CRT knows about and is implemented on
top of the Win32 API, while the Win32 API knows nothing about the CRT.

(d) MinGW is not the same as Cygwin
...................................

As described in (b), Microsoft's version of the CRT (`MSVCRT.DLL') is
provided as part of Windows, separate from VC++, which must be
purchased.  Hence, it is possible to write MSVCRT to provide CRT
services without using VC++.  This is what MinGW (www.mingw.org) does -
it is a port of GCC that will use MSVCRT.  The reason one might want to
do this is (a) it is free, and (b) it does not require a separately
installed DLL, as Cygwin does. (#### Maybe MinGW targets CRTDLL, not
MSVCRT?  If so, what is CRTDLL, and how does it differ from MSVCRT and
`LIBC.LIB'?) Primarily, what MinGW provides is patches to GCC (now
integrated into the standard distribution) and its own header files and
import libraries that are compatible with MSVCRT.  The best way to think
of MinGW is as simply another Windows compiler, like how there used to
be Microsoft and Borland compilers.  Because MinGW programs use all the
same libraries as VC++ programs, and hence the same services are
available, programs that compile under VC++ should compile under MinGW
with very little change, whereas programs that compile under Cygwin will
look quite different.

   The confusion between MinGW and Cygwin is the confusion between the
environment that a compiler runs under and the target environment of a
program, i.e. the environment that a program is compiled to run under.
It's theoretically possible, for example, to compile a program under
Windows and generate a binary that can only be run under Linux, or
vice-versa - or, for that matter, to use Windows, running on an Intel
machine to write and a compile a program that will run on the Mac OS,
running on a PowerPC machine.  This is called cross-compiling, and while
it may seem rather esoteric, it is quite normal when you want to
generate a program for a machine that you cannot develop on - for
example, a program that will run on a Palm Pilot.  Originally, this is
how MinGW worked - you needed to run GCC under a Cygwin environment and
give it appropriate flags, telling it to use the MinGW headers and
target `MSVCRT.DLL' rather than `CYGWIN1.DLL'. (In fact, Cygwin
standardly comes with MinGW's header files.) This was because GCC was
written with Unix in mind and relied on a large amount of Unix-specific
functionality.  To port GCC to Windows without using a POSIX emulation
layer would mean a lot of rewriting of GCC.  Eventually, however, this
was done, and it GCC was itself compiled using MinGW.  The result is
that currently you can develop MinGW applications either under Cygwin
or under native Windows.

(e) Operating system is not the same as window system
.....................................................

As per the above discussion, we can use either Native Windows (the OS
part of Win32 provided by `KERNEL32.DLL' and the Windows CRT as
provided by MSVCRT or CLL) or Cygwin to provide operating-system
functionality, and we can use either Native Windows (the windowing part
of Win32 as provided by `USER32.DLL' and `GDI32.DLL') or X11 to provide
window-system functionality.  This gives us four possible build
environments.  It's currently possible to build XEmacs with at least
three of these combinations - as far as I know native + X11 is no
longer supported, although it used to be (support used to exist in
`xemacs.mak' for linking with some X11 libraries available from
somewhere, but it was bit-rotting and you could always use Cygwin; ####
what happens if we try to compile with MinGW, native OS + X11?).  This
may still seem confusing, so:

Native OS + native windowing
     We call `CreateProcess()' to run subprocesses (`process-nt.c'),
     and `CreateWindowEx()' to create a top-level window
     (`frame-msw.c').  We use `nt/xemacs.mak' to compile with VC++,
     linking with the Windows CRT (`MSVCRT.DLL' or `LIBC.LIB') and with
     the various Win32 DLL's (`KERNEL32.DLL', `USER32.DLL',
     `GDI32.DLL'); or we use `src/Makefile[.in.in]' to compile with
     GCC, telling it (e.g. -mno-cygwin, see `s/mingw32.h') to use MinGW
     (which will end up linking with `MSVCRT.DLL'), and linking GCC
     with -lshell32 -lgdi32 -luser32 etc. (see `configure.in').

Cygwin + native windowing
     We call `fork()'/`execve()' to run subprocesses
     (`process-unix.c'), and `CreateWindowEx()' to create a top-level
     window (`frame-msw.c').  We use `src/Makefile[in.in]' to compile
     with GCC (it will end up linking with `CYGWIN1.DLL') and link GCC
     with -lshell32 -lgdi32 -luser32 etc. (see `configure.in').

Cygwin + X11
     We call `fork()'/`execve()' to run subprocesses
     (`process-unix.c'), and `XtCreatePopupShell()' to create a
     top-level window (`frame-x.c').  We use `src/Makefile[.in.in]' to
     compile with GCC (it will end up linking with `CYGWIN1.DLL') and
     link GCC with -lXt, -lX11, etc. (see `configure.in').

     Finally, if native OS + X11 were possible, it might look something
     like

[Native OS + X11]
     We call `CreateProcess()' to run subprocesses (`process-nt.c'),
     and `XtCreatePopupShell()' to create a top-level window
     (`frame-x.c').  We use `nt/xemacs.mak' to compile with VC++,
     linking with the Windows CRT (`MSVCRT.DLL' or `LIBC.LIB') and with
     the various X11 DLL's (`XT.DLL', `XLIB.DLL', etc.); or we use
     `src/Makefile[.in.in]' to compile with GCC, telling it (e.g.
     -mno-cygwin, see `s/mingw32.h') to use MinGW (which will end up
     linking with `MSVCRT.DLL'), and linking GCC with -lXt, -lX11, etc.
     (see `configure.in').

   One of the reasons that we maintain the ability to build under Cygwin
and X11 on Windows, when we have native support, is that it allows
Windows compilers to test under a Unix-like environment.


File: internals.info,  Node: Windows Build Flags,  Next: Windows I18N Introduction,  Prev: Different kinds of Windows environments,  Up: Interface to MS Windows

37.2 Windows Build Flags
========================

`CYGWIN'
     for Cygwin-only stuff.

`WIN32_NATIVE'
     Win32 native OS-level stuff (files, process, etc.).  Applies
     whenever linking against the native C libraries - i.e.  all
     compilations with VC++ and with MINGW, but never Cygwin.

`HAVE_X_WINDOWS'
     for X Windows (regardless of whether under MS Win)

`HAVE_MS_WINDOWS'
     MS Windows native windowing system (anything related to the
     appearance of the graphical screen).  May or may not apply to any
     of VC++, MINGW, Cygwin.

   Finally, there's also the MINGW build environment, which uses GCC
(similar to Cygwin), but native MS Windows libraries rather than a
POSIX emulation layer (the Cygwin approach).  This environment defines
WIN32_NATIVE, but also defines MINGW, which is used mostly because uses
its own include files (related to Cygwin), which have a few things
messed up.

   Formerly, we had a whole host of flags.  Here's the conversion, for
porting code from GNU Emacs and such:

Old Constant       New Constant
---------------------------------------------------------------
`WINDOWSNT'        `WIN32_NATIVE'
`WIN32'            `WIN32_NATIVE'
`_WIN32'           `WIN32_NATIVE'
`HAVE_WIN32'       `WIN32_NATIVE'
`DOS_NT'           `WIN32_NATIVE'
`HAVE_NTGUI'       `WIN32_NATIVE', unless it ends up already bracketed
                   by this
`HAVE_FACES'       always true
`MSDOS'            determine whether this code is really specific to
                   MS-DOS (and not Windows - e.g. DJGPP code); if so,
                   delete the code; otherwise, convert to `WIN32_NATIVE'
                   (we do not support MS-DOS w/DOS Extender under XEmacs)
`__CYGWIN__'       `CYGWIN'
`__CYGWIN32__'     `CYGWIN'
`__MINGW32__'      `MINGW'

