Copyright © 2003 jsd

1  Compatibly Encoding Fancy Symbols in HTML Documents

The objective is to be able to publish documents on the web and have them look right to as many readers as possible. This is not so easy, because the various browsers are mutually incompatible.

If we confine ourselves to a certain set of “basic” characters (including the Latin alphabet, decimal digits, and basic punctuation), then there is pretty much a consensus on how a computer should represent them for purposes of computation, transmission, and storage.

But often there are reasons to include a larger cast of characters in our documents. We want to do this in a way that works correctly on all the major browsers,1 and alas it does not suffice to make something that looks good just on the machine where it was generated.

The logical way to proceed is by a two-step process. The first step is to figure out what characters (also called glyphs) you want to deal with. At this level of abstraction, the glyphs are identified by (and only by) their names. The second step is to choose an encoding scheme for the glyphs, and of course the matching decoding scheme whereby the text in your document is decoded into a displayable pattern of glyphs.

2  Named Entities

2.1  PostScript

The PostScript language implements the aforementioned two-step process quite explicitly. The setup process (1) loads a set of named glyphs and (2) creates a lookup table that defines how the “show” command maps bytes to glyphs.

Adobe, the originator of PostScript, is also a major font vendor, so they play a big role in deciding what names go with what glyphs.

Since we are primarily discussing web documents we won’t go into details about PostScript.

2.2  STIX

The STIX consortium is busy making and collecting glyphs, and assigning names to them. See reference 1.

2.3  HTML

HTML of course supports the plain Latin alphabet, digits, and basic punctuation. In addition, the HTML powers-that-be have selected some additional entities, including some accented characters and some mathematical operators. These are “officially” part of HTML, so any browser than can render these symbols really ought to do so.

For example “lambda” is the name of the Greek letter λ, while “prop” is the name of the mathematical proportional-to operator.

As for the second step, HTML defines several different high-level encodings for these entities. The simplest high-level encoding refers to the characters by name: If you utter the character-sequence ∝ in an HTML document, the glyph for the proportional-to operator will be produced.

In addition, HTML assigns numbers to all the glyphs. You can use these numbers to form high-level encodings, using either decimal or hexadecimal numerals. For example, lambda can be represented by λ (decimal) or equivalently by λ (hex).

Perhaps paradoxically, the most-common characters such as “A” and “!” are not assigned names by the HTML spec. You can refer to them by numeric encoding, but not by name. For example, “A” can be represented by A or by A but not by &A; or anything like that. Of course it can also be represented at the low level by the bare character.

The situation is just the reverse for a couple of special symbols, namely ampersand and less-than. If you want to display the glyph for the less-than symbol, you must refer to it by one of the high-level encodings &lt; or &#60; or &#x3c; since the bare character “<” is reserved by HTML at the syntactic level.

It is convenient that the number assigned to each and every HTML entity (such as &#38; for ampersand) agrees with the Unicode numbering (section 3.3). The HTML entities are a small subset of the Unicode entities.

Similarly, the ISO-8859-1 characters are a subset of the HTML entities, and are numbered the same.

3  Document Encodings

In this section we discuss low-level encodings, that is, things that happen below the HTML level. A document encoding is also known as the document’s charset.

3.1  Seven-Bit Encoding

The venerable 7-bit ASCII encoding is extremely popular. It contains the basic Latin alphabet, digits, and basic punctuation. It shows up as the first half (i.e. the first 128 codes) of all the ISO-8859-* encodings, and many other encodings as well.

Also note that the HTML high-level encoding (such as &65; for “A”) uses this same venerable numbering scheme.

3.2  Eight-Bit and Sixteen-Bit Document Encodings

Far and away the majority of web documents use the ISO-8859-1 encoding, which ISO calls the “Western” encoding. Its first half reproduces the 7-bit ASCII encoding. The second half contains some Latin characters with diacritical marks, some currency symbols, and a bit more punctuation. It is reasonably useful for the major languages found on the western edge of Eurasia, but is not complete even in that context. This encoding uses eight bits per character.

There are many other ISO encodings. For example, there is ISO-8859-5. Again its first half is 7-bit ASCII. The second half of contains the Cyrillic letters. Unlike ISO-8859-1, it cannot encode the Latin letters with diacritical marks. It had to get rid of them to make room for the Cyrillic letters without using more than 256 codes.

There are obvious computational advantages to an encoding where each character fits into eight bits. But nowadays the trend is to move away from that.

3.3  Unicode

Encodings that are specialized for one language are usually not very good for other languages. This leads us to consider unicode. It can represent pretty much everything, including Chinese, Cyrillic, math symbols, and more.

Let’s analyze how Unicode fits into the two-step scheme.

(1) As far as I can tell, Unicode does not provide the glyphs nor assign names to glyphs. It merely describes them, in terms like “left-pointing angle bracket”. If you want the glyphs and/or glyph-names, you must find them elsewhere.

(2a) Unicode defines a numbering scheme for the glyphs. Every glyph you can think of has a number. This is pretty much the de-facto standard for numbering things.

(2b) Unicode defines various encoding schemes which encode the aforementioned numbers as bytes in a document. And conversely there are corresponding ways to decode the bytes of your document to produce the desired glyphs.

The most common encoding scheme is called utf-8. It has the interesting property of being a variable-length encoding. The basic Latin characters can be represented in a single 8-bit byte. Greek, Cyrillic, and Chinese characters can be represented in two bytes. Most math symbols can be represented in three bytes.

The principal alternative to utf-8 is utf-16. It represents each and every glyph in two bytes, which is no problem since fewer than 65,000 glyphs have been defined. For instance, the utf-16 code for “A” is just a 16-bit word representing the integer 65. There are two variants of utf-16, depending on which byte of the 16-bit word comes first in your document.

Unicode already exists, and is moderately well supported by the major browsers and operating systems. You can write and publish documents using unicode today, and they will pretty much just work, so long as you stick to the major alphabets of the world, and stick to the most-common math operators. Limitations will be discussed in section 5.6.

3.4  Switching the Codepage (Misnamed “Font”)

A font is a complete collection of type (or glyphs) in a consistent size and style.

For years, Microsoft has had something they call the “symbol font” … but it’s not a font. It’s an encoding; it’s a codepage.

The conjecture is that they use the same mechanism for changing fonts and for changing encodings, and that the “symbol font” was named by somebody too clueless to know the difference between a font and a codepage.

An 8-bit codepage can address at most 256 glyphs. Any normal font contains far, far more than 256 glyphs.

Choosing a font means choosing a style. People don’t switch to the so-called “symbol font” because they want to change the style. Rather they switch because they want to change the encoding.

To say the same thing in different words: If you wish to make a collection of 256 glyphs and point to them using an 8-bit codepage, that’s fine … but please don’t call it a font. Such a collection is an encoding issue, not a font issue.

Some nasty vestiges of the font-versus-encoding confusion still remain. For instance, MSIE (version 6 and below) can represent math operators (such as proportional-to), but does not reliably recognize the unicode or HTML-entity representations. The only way to make sure such symbols look OK under MSIE is to switch to the so-called “symbol font”. That is, you are forced to consider the proportional-to character to be some non-mathematical character, written in a “style” that makes it “look like” a proprtional-to symbol. Obviously such kludges wreak havoc on search utilities and other things that want to pay attention to the meaning of the characters without regard to “style”.

Similar remarks apply to many other characters, as may be able to see from the bottom half of the table in reference 2. Characters encoded in the symbol font are shown in red, side-by-side with the corresponding HTML entity is shown in black. In theory these should be the same (in any browser that implements the symbol font at all), but if you view that table using MSIE you will discover that MSIE does not reliably render the HTML code correctly.

MSIE knows how to render them if they are represented using the symbol font kludge, but does not reliably render the corresponding HTML entity codes. Apparently under some rare circumstances MSIE does recognize the entity codes, but experts are unable to explain when it works and when it doesn’t.

The “symbol-font” situation creates terrible interoperability problems, since standards-compliant browsers, quite understandably, don’t process the font-switching directives the way MSIE does. So if you want to write a document that can be displayed on a range of browsers, you can’t just use the MSIE representation and you can’t just use any standard representation. You must resort to a bizarrely complicated workaround. Specifically, here is the simplest quasi-portable representation I know for the proportional-to operator:

<!--[if gte IE 3]><font face=Symbol>&#181;</font><![endif]
--><![if lt IE 3]>&#8733;<![endif]>

As demonstrated in reference 3, by using this workaround you can produce documents that look OK on both MSIE and Mozilla. This covers most but not all of the official HTML entities. Exceptions will be discussed in section 5.

4  Substitute Fonts

Naive logic suggests that when a browser wants to display a character, it should find the encoding of that character and then look up that codepoint in the “current” font.

That leaves us with the question of what to do if the “current” font doesn’t have a usable glyph at this codepoint. Well, the Firefox browser does something clever: It looks through some list of fonts to see if there is some other font that does have a usable glyph at this codepoint, and if so, it uses that.

This can be confusing, because it means that if you see a glyph on your screen, you don’t know whether it came from your “current” font … and indeed you have no way (as far as I know) to tell what font it did come from.

5  Browser Bugs

5.1  Font Bugs

Don’t blame the browser for problems inherent in the font itself. For instance, the Times New Roman font from Abisource doesn’t contain the glyphs for N-ary Summation or N-ary Product, so if you choose such a font some browsers will have no way of displaying the desired glyph. If you switch to the Times font from Adobe the problem goes away.

There is however some tricky business here: If the “current” font does not provide a usable glyph at a given codepoint, Firefox (and its siblings) will search all fonts on the system (in who-knows-what order) until it finds a usable glyph at that codepoint. This increases the chance that the browser will be able to display something useful, but it means that you don’t necessarily know which font the displayed glyph came from.

5.2  MSIE

MSIE does not render most of the entities classified as “NEW” in reference 4.

MSIE renders the HTML code &#8593; or &uarr; as an upward arrow, as it should. However, one would also expect an upward arrow in position 173 of the so-called symbol font ... but it’s not there. This is a pain because the symbol-font arrows are much better looking than the HTML arrows, and it would have been nice to generate them whenever possible.

There are weird context-dependent bugs in MSIE. Suppose you want to produce following string of symbols:
It is symmetric; characters to the right of the product symbol should be identical to the corresponding characters on the left. But if you use MSIE if you code the sequence in the obvious way, it will sometimes be rendered wrongly, as you can see by viewing this:
using MSIE. It may switch to the wrong font to render the product symbol, and then stay in the wrong font for some hard-to-predict length of time. This only happens under some circumstances, notably if the document uses utf-8 encoding and uses the default font selection (i.e. the document doesn’t specify its own fonts via a style sheet or whatever).

5.3  Opera

Opera appears to be remarkably well-behaved and standards-compliant.

5.4  Mozilla

Mozilla is mostly well-behaved and standards-compliant. One slight drawback is that it apparently it coerces all right quotes and left quotes to be symmetrical vertical tick-marks. I can live with that, although it’s not maximally beautiful or elegant.

5.5  Netscape

Netscape version 4.8 (and presumably earlier versions) imposes a horrible dilemma:

I’ve given up on symbol compatibility for Netscape. It has horrible non-standard behavior in non-utf mode, and markedly different yet still horrible non-standard behavior in utf mode. I’m not going to lose sleep over this, because Netscape is a pretty old browser and only a few people are still using it.

5.6  Recommendations, plus Pros and Cons

If you are writing in one particular language, there is almost certainly an encoding optimized for you. For example, there are at least 20 different Cyrillic encodings, each using one byte per character, each optimizing certain details at the expense of others.

If you are writing in a non-alphabetic language such as Chinese, there are encodings optimized for that, too, using two bytes per character.

Things get much more interesting if you need to handle more than one language at once. Imagine for instance a browser that wants to show you a synopsis of each file in a directory, when the files are in many different languages. In such a situation unicode looks pretty attractive. Computer infrastucture (operating systems, editors, browsers, search engines) appear to be evolving towards relying on unicode.

If you are writing HTML documents that are mostly in one language, with occasional snippets of math or other languages, a reasonable option is to use your whatever encoding is “normal” for your locale, supplemented by the HTML high-level encoding if/when there’s something your low-level encoding can’t handle. You must stick in the kludgey workarounds when necessary to deal with the MSIE issue.

If you find that no matter what 8-bit encoding you choose, a significant fraction (more than a few percent) of the characters in your document are outside that encoding, you have nothing to lose and everything to gain by switching to unicode.

For documents written mainly in English you can easily confine yourself to using the 7-bit ASCII low level encoding, and represent everything else using the HTML high-level encoding. Having done that, you can (in theory at least) make an arbitrary choice as to what you say is “the” low-level encoding of your document: you could call it utf-8, ISO-8859-1, ISO-8859-5, etc., and it shouldn’t make any difference.

Sometimes the practice differs from the theory. For example, Browsers such as Mozilla and Netscape allow (indeed require) the user to set certain preferences separately for each encoding. These preferences include things like default font-family and font-size. This is relevant because the ISO-8859-1 is so exceedingly common that many users have no idea that other encodings exist, and have not set up suitable preferences for them. The result is that when they visit a non-ISO-8859-1 page, it looks “funny” but they don’t know why. So this is an argument against declaring your document to be anything other than ISO-8859-1 if/when it could just as well be ISO-8859-1.

For this reason alone, if you have a choice between ISO-8859-1 or something else, I suggest sticking with ISO-8859-1, just to avoid rocking the boat.

You can observe the encoding-dependent behavior of Netscape and/or Mozilla by comparing reference 5 against reference 2. Those two documents are identical except for their charset declarations, and they do not contain any bytes outside the 7-bit ASCII range, where utf-8 is byte-for-byte identical to ISO-8859-1.

The standard default encoding on the world-wide-web is ISO-8859-1.

If you decide you want some other encoding, there are two things you need to do. First, the HTML file ought to specify its content-encoding in its “head” section. For example, if you want the utf-8 encoding, you say:

<META http-equiv="Content-Type" content="text/html; charset=utf-8">

Secondly, you should teach whatever server is serving that file to send the correct content-type in the HTTP header. Note that the HTTP header is conceptually different from the HTML head section. Also note that web servers do not generally read the HTML document to discover META directives. One correct approach, assuming you have an Apache server, is to put a line like this in the .htaccess file:

AddType text/html;charset=UTF-8 html htm

That tells the server that all .html and .htm files in the directory should be treated as utf-8. (If you work harder you can select which files to get which treatment.)

Very weird things happen with some browsers if the encoding in the HTTP header is contradicting by a META directive in the HTML head.

5.7  Tools

The most widely-available HTML editor is word-for-windows (aka winword). It can read in an HTML file, edit it, and write it back out again. Alas it doesn’t do a very nice job. If it reads in a file with utf-8 encoding, it will write it back out in windows-1252 encoding. That destroys any hope of compatibility or portability, because there are lots of people who don’t run windows and the windows-1252 encoding means nothing to them.

Winword also makes heavy use of the abominable symbol-font encoding. It does this even for symbols such as the copyright symbol for which widely-compatible simple representations are available. In fact winword uses 323 bytes to represent even ultra-simple things like the copyright symbol.

I write most of my documents in the LaTeX language. That means I can convert them to .pdf format using pdflatex, and/or convert them to HTML using HeVeA.

I have a postprocessor that I apply to the HTML generated by HeVeA. It catches such things as

... and converts them to the HTML-decimal encoding. It emits the workaround for MSIE gradient and proportional-to operators.

5.8  Compatibility

It is relatively easy to get a list of symbols; see e.g. reference 6.

What we need is a list that says what symbols are well handled by which browsers. Reference 2 and especially reference 7 are a start in that direction, but more needs to be done. It’s a real crock to start using a symbol and then discover it can’t be displayed on some systems.

6  References and Resources

http://www.stixfonts.org/ The Scientific and Technical Information Exchange (STIX) font creation project.

../iso/font-table1.htm This document’s charset is declared to be utf-8, in contrast to reference~5. There are many symbols for which neither MSIE nor Netscape renders the HTML entities correctly. That is, MSIE behaves the same as it did with the ISO encoding, while Netscape has diametrically reversed its behavior, substituting one form of intolerance for the other.

../iso/font-table.html A table using a workaround so that symbols can be displayed on a wider range of browsers.

http://www.w3.org/TR/html401/sgml/entities.html Authoritative list of HTML entities, including names and numbers.

../utf/font-table1.htm A font table displaying the MS symbol font and the corresponding HTML entity codes. This document’s charset is declared to be utf-8, in contrast to reference~2. There are many symbols for which Netscape renders the HTML codes (but not the symbol-font codes), while MSIE does just the reverse, i.e. rendering the symbol-font codes (but not the HTML codes).

http://www.alanwood.net/unicode/sitemap.html Alan Wood’s Unicode Resources contains a nicely-organized list of alphabetical and mathematical symbols, including names, HTML-decimal encodings, and unicode encodings.

../sym-kludge A snippet of code that could be used as a starting point for building a database of mappings between the various incompatible representations. The BGS (bogus) attribute indicates that MSIE doesn’t reliably render the HTML entity code.

http://www.unicode.org/Public/UNIDATA/UnicodeData.txt Authoritative list of Unicode characters and their properties.

../utf/font-chart.html A chart showing a goodly number of characters and the various ways of encoding them.

http://www.ams.org/STIX/bnb/stix-tbl.asc26MR98 Names, numbering, and description of glyphs for mathematical symbols.

Command to obtain an approximate list of postscript fonts on your system:
locate .afm | xargs -d "\n" grep -h FontName | less

Postscript code to show all the glyphs in a postscript font: http://home.att.net/~tom.brodhead/showfont.ps

Command to obtain an approximate list of X fonts on your system:

Command to show all the glyphs in an X font:

Relationship between Adobe glyph names and unicode numbers: http://unicode.org/Public/MAPPINGS/VENDORS/ADOBE/symbol.txt

Information on individual unicode characters: http://www.fileformat.info/info/unicode/char/23a1/fontsupport.htm

Information about free unicode fonts: http://en.wikipedia.org/wiki/Free_software_Unicode_fonts

A shareware unicode font with good coverage of math symbols and brackets: http://www.code2000.net/

This discussion applies to the browser versions that were current as of late 2003, namely Microsoft Internet Explorer 6.0, Mozilla 5.0, Opera 7.23, and Netscape 4.8.

Copyright © 2003 jsd