Imagelib: The Image Library

The imagelib, or image library, is responsible for the decoding and display of images. From a high-level perspective, the imagelib generates an in-memory representation of a decoded image using the stream of data that results from reading a supplied URL. The core imagelib, consisting of purely cross-platform code, provides the following services:

URL data stream management
Image decoding, including decompression
Caching of decoded images
Color-space conversion
Progress and update notification

Note that the list of core imagelib functionality does not include the actual display of images, i.e. painting pixels on the monitor or printer. Logically, this code should be part of the imagelib, but as a result of historical decisions, the image display code is comingled with other platform-specific display code in each of the front-ends (See Where It's Headed, below.)

History

2.x	Added rate-adaptive progressive JPEG (PJPEG) and animated GIF support. Also, custom palette support added for Windows, in which the 8-bit hardware palette was matched to the displayed image.
3.x	Small changes to allow JavaScript-controlled loading of images from JavaScript.
4.x	Imagelib partially rewritten for cleaner API. As a consequence of schedule pressure, custom palette support was disabled.

How It Works

Layout issues a request to get/display an image URL as a client for the image library. The request could come from an HTML tag <img src=foo.gif> or a "View Image" request, for example.

The image library doesn't care how the image request was originally generated. It accepts data from whatever data stream is specified, buffering it until it has enough to process.

The main entry to the image library is the function IL_GetImage(). Layout calls IL_GetImage through lo_GetImage() in ns/lib/layout/layimage.c. IL_GetImage() returns a data structure called IL_ImageReq. Although multiple requests for the same image may be made on the same html page, a unique IL_ImageReq handle is generated for every request.

This does not mean the image must be decoded for each request. A single image URL may be used countless times on a page. A cache of previously decoded and sized images is kept and each image's information is kept in a data structure called an image_container. When an image request is issued, the imagelib searches through the cache to find a matching image_container. If it finds a matching container, it uses the previously decoded data. If not, it decodes the image data and creates a new image container which is added to the image cache for future use.

An image observer is created for the image request. The observer is a mechanism designed to monitor the image_request status and states like the following:
        - the image's progress
        - the image pixmap was updated
        - the image finished decoding
        - a frame of an image/animation finished decoding
        - a cached image completed decoding
        - the image request was destroyed and the observer list is ready for cleanup.

After the image's header has been read and the image's natural dimensions and target dimensions are understood, the image library decodes until it has a line's worth of data. The decoded data is resized to the target image size. If a transparency mask exists, the mask is also resized to the target image size. Any transformations needed to match the target color depth also occurs here.

The image libary deals with the image by line, rather than by blocks. The line ready for front end display is sent to the front end, by way of the function IL_DisplaySubImage().

When the observer realizes the image finished decoding, clean up can occur. Clean up, however, does not occur until the page is unloaded. First the image request is destroyed. The image container is destroyed. A image group container is destroyed last and insures all image structures are freed.

Primary Data Structures

For each image, an image container is created for each decompressed/sized image:
il_container_struct (in ns/modules/libimg/src/if.h)

Each time the client(layout) asks the image library to display an image, an new image request handle is generated by IL_GetImage():

IL_ImageReq (in ns/modules/libimg/src/if.h)

image_struct :
IL_Image_struct (in ns/modules/libimg/src/il.h)

How to Add a New Image Format

The image library uses function tables, or vtables, to provide an interface for image decoders. Five functions provide the interface for each image format decoder to the rest of the image library. These functions are:

il_xxx_init : allocates any structures required for the decoding.

il_xxx_write_ready: Not needed on all formats. Determine the max number of bytes to read ahead while waiting for timer. This number is limited to WIN16 malloc limitations.

il_xxx_write: Most of the real work is done in il_xxx_write. All decompression, color space transformations, dithering functions are called by this function. Called by il_write() in if.c.

il_xxx_complete: notifies the observer of the complete status. Calls il_xxx_abort() for clean up.

il_xxx_abort: Cleans up and frees any lingering structures

The function il_first_write() in file if.c sets up the vtable and attaches it to the the image_container for the new image. First it associates the newly created image container with the data stream. After determining the image format type, it assigns the five functions for that image type to the virtual table.

/* example: */
        switch (ic->type)
        {
                case IL_GIF:
                        init = il_gif_init;
                        ic->write = il_gif_write;
                        ic->complete = il_gif_complete;
                        ic->write_ready = il_gif_write_ready;
                        ic->abort = il_gif_abort;
                        break;

                case IL_JPEG:
                        init = il_jpeg_init;
                        ic->write = il_jpeg_write;
                        ic->abort = il_jpeg_abort;
                        ic->complete = il_jpeg_complete;
                        break;
        .
        .
        .
                default:
                        ILTRACE(1,("il: ignoring unknown image type (%d)", ic->type));
                        return MK_IMAGE_LOSSAGE;
        }
/* end example */

Where It's Headed

This section describes directions for future devlopment of the image library.

Modularization

Right now, it isn't possible to build a simple, standalone image viewer using the imagelib without a lot of excess baggage (inclusion of unrelated header files, unused components and stubs for missing functions). Although the imagelib makes greater use of function tables (vtables in C++ parlance) in order to define abstract interfaces to external components than almost any other Navigator library, this work is incomplete; There are still a few functions that are external to the imagelib that are called using hard-coded references, e.g. NET_GetURL(). All such naked references to external functions should be removed so that the imagelib can be used as a component, e.g. to be included either within a standalone viewer program or within the Navigator.

The vtable-like interfaces that the imagelib uses are currently implemented using JMC, a componentization scheme that was used internally at Netscape for a short while, but which was abandoned shortly after its introduction. Any use of JMC should be expunged, as it adds a good deal of unnecessary complexity to the build process and the JMC tools won't be supported in the future. Ideally, the new imagelib interfaces should be migrated to the COM-like componentization scheme that is being introduced by the next generation plugin code in 5.0.

Aside from the problem of modularizing the core, cross-platform imagelib functionality, there is the separate issue that the platform-specific code that actually draws the image pixels on the screen currently resides within each platform's front-end, e.g. the code for Windows is in the ns/cmd/winfe directory. (There is also a PostScript "front-end" for printing mages on PostScript printers.). This code tends to be somewhat gratuitously entangled with each platforms's native widgets and classes. This image-drawing code should be isolated and placed in an "md" directory within the imagelib ("md" stands for machine-dependent). Again, the final goal is to build a standalone image viewer for each of the three platforms, using only the imagelib and a little glue code, so as to streamline imagelib development.

Image plugins

The interface between the generic imagelib code and decoder for a particular image format is a relatively simple one. It should be possible to modularize image decoders and dynamically load them in much the same way that browser plugins are handled today. This would allow 3rd parties to add new image formats that could be handled by <IMG> tags. These image plugins could take advantage of the imagelib services such as color-space conversion, caching and display, so that a plugin need only supply the raw image decoding code. Unlike existing plugins that use platform-specific APIs for their display, the image plugin API would be cross-platform so, assuming reasonable coding practices, these plugins would merely need to be recompiled for each platform.

Eliminating special handling of icons

The use of certain well-known URLs as image icons was established as a de-facto standard by the earliest web browsers, e.g. for directory listings. For some unknown reason, these icons were implemented in the Navigator by code that was completely separate from the imagelib, using platform-specific icon storage and display code within each front-end. There's no reason that these icons couldn't be stored within cross-platform files using some standard compressed image forma and decoded by the imagelib in exactly the same manner as any external image. This would eliminate a lot of special-case both code within the imagelib and particularly within the individual front-ends. It would also simplify porting the Navigator to new platforms and reduce memory consumption because each platform currently stores the icons in a (platform-specific) uncompressed format.

Better palette control

On X11, it should be possible to adapt to any existing colors in the palette. In existing version, nav attempts to allocate a regular lattice of RGB colors (a "color cube"). However, the eight "corner" colors of the cube, combined with any well-distributed set of colors in the cube interior should provide reasonable levels of dithering.

Ideally, should allocate 216-color cube, but if that's not possible, ...

Reduced memory use

It is a limitation of the current imagelib that all the images on an HTML page must be stored in memory in decoded form, whether or not the image is visible on the screen. For example, if a page is displayed with ten 512x512 pixel images on a system with a 24-bit display, 7.5 megabytes of memory are required to store those decoded images in memory, even though only one at a time might be displayed. This limitation can be especially painful on systems that tend to have severe memory constraints, usually Macs and Win16 machines.

A much better system would expose an entry-point into the imagelib that would be called whenever a low-memory situation is encountered. This code would attempt to purge the pixmap memory for images that haven't been recently displayed or which aren't visible on the screen. Even though the image pixmap would be discarded, all the information necessary to reconstruct it could be maintained, i.e. the image's source URL, image dimensions, etc. When a request is made to display a purged image, the imagelib would refetch it and redecode it.

Gamma-correction

Identical RGB values are reproduced differently on different display devices. Due to differences in hardware/software CLUTs, DAC nonlinearity, differences in monitor phosphors, monitor gamma, etc., the same image might look very different on a Mac than on a Windows box or an SGI workstation. The greatest culprit in accounting for these differences is system gamma.