Francis Robichaud

Optimizing Mozilla and Pixmap Management in X

Posted on July 8th, 2008 by Skaber.
Categories: Programming, Linux, mozilla.

The lastest Firefox release, version 3.0, relays on the Gecko as a layout engine and libpr0n for image decompression. The underlying Cairo framework greatly improves the code portability and increases the rendering on supported platforms. Cairo is used in several projects and sounds very promising with the eventual support of glitz to benefit of 3d hardware acceleration.Thebes is the C++ code used to wrap the Cairo framework in Mozilla. It uses ImageSurfaces which get decompressed image data from libpr0n. The current design of these surfaces is to send a full pixmap (decompressed images) to X without any limitation. Then, the underlying Cairo surfaces can be painted and the Mozilla application’s memory freed. The actual image data is kept as a pixmap in X’s memory. This choice of storing the pixmaps in X instead of in the application’s memory can increase the performances when rendering image: the new visible data does not need to transfered on every scrolling event.

On the other hand, storing pixmaps in X might not be the best solution to optimize the speed of image rendering. Considering that the full pixmap needs to be transfered between the application and X, rendering the first visible frame will be slower. Also, using unlimited memory in X may steal the available ressources for other applications.

Firefox has a long history of known bugs related to pixmap storage in X. I’ve focused on bugs 296818 and 395260 during the last weeks with the unique goal of changing way Mozilla handles pixmaps. The most affected users of pixmap storage are those who use Firefox on thin clients. These usually have low available memory and can’t be used to store pixmaps. This simple page on a 128 Mbs RAM thin clients takes a while to load a few images and then causes a quick OOM kill before page rendering ends. One might say that showing multiple 5000×1220 images resized to 50×50 might not be common on the web but isn’t the web full of bad html coders ?

Frederico Mena Quintero has been very concerned about this behavior and integrated some modifications in Firefox to increase the quality of pixmap management. The infamous MOZ_DISABLE_IMAGE_OPTIMIZE environment variable has also been integrated in Mozilla’s sources to reduce memory used by pixmap in X. Unfortunately, none of the integrated patches changes the behavior whitout any quality issue in the overall browing experience.

While comparing Firefox to Opera and Konqueror, I noticed that only Opera seemed to effectively manage it’s memory and pixmap caching. These graphics roughly represent both application memory and X memory when rendering this simple test page.

	X Memory	App Memory
Firefox 3.0
Opera 9.27
Konqueror 3.5.9

Note : these graphics have been generated using these basic homemade scripts to give an overview of the total memory usage. The y scale represents the memory usage in bytes while the x scale is the sample number. appmem.py - memx.py (generate application data with appmem.sh)

From these graphics, we can conclude that there’s is a clear possibility to optimize memory consumption by pre-manipulating images before we send them to X. At least, this is what Opera seems to do and that this results in better memory management. Why wouldn’t Mozilla applications keep the uncompressed image data locally instead of pushing it to X ?

There are actually two situations to take into consideration to achieve memory optimization. First, Mozilla should send only visible portions of images to X, this forces to recompute the visible portion on every user operation, such as scrolling or window resizing, and send the new sub-images to X. Image scaling also needs to be taken into consideration. For example, why would we transfer the full data of a 1920×1200 image to X if it has been resized to 1024×768 ? The pixmap would simply be using memory ignored when rendering. As the Cairo developer Carl Worth pointed out, the advantage of transferring a full pixmap to X and let it resize an image would be to use the rendering extension that could be available in X. Doing so still requires to transfer a full pixmap between the X application client and the X server. The transfer itself is very bandwidth consuming when using a remote X connection, even on a 100Mbit network. For example, a 5000×1200 - 24 bits image represents around 46 Mbs of data to transfer and around 4.6 seconds of delay before we can render the first image. There is probably also an overhead when transferring images using local sockets but this would need to be verified by performing additional tests.

At first, I had implemented a basic downscaling algorithm and sub-image creation from an image’s original data. Since there are references to the GDK library in Thebes, I took the freedom to use available functions to manipulate raw image data. As this page explains it, GDK uses the RGBA format to represent a GdkPixbuf while Cairo respects the X server’s ARGB format. On a second thought, this results in the red and blue channel being inverted while manipulating images. There’s no reason why this would have consequences on the resulting image. Furthermore, GDK offers flexibility on the algorithms to scale images while the X server uses the nearest-neighbor interpolation algorithm. Hence, the gdk_pixbuf_new_subpixbuf() and gdk_pixbuf_scale_simple() functions offer the quickest way to easily pre-manipulate images in Thebes before the pixmaps are sent to X through the Cairo library. Moreover, the quality of the downscaled images is WAY better when using an algorithm different to the nearest-neighbor interpolation. The difference in Firefox is shown here :

Downscaled images in Firefox 3.0 using X’s nearest-neighbor interpolation: (current behavior)

Downscaled images using GDK’s bilinear interpolation:

Mozilla’s QA team has built the Talos performance testing project to verify that code modifications do not decrease the product quality.
FF3 built from source output
FF3 patched sources output

Update : After testing the patch in a few environments, I discovered that the scrolling quality could be sluggish when Firefox was used on a thin client with an application server having low resources. My first thought was that the transfer of raw data to the X server added an important overhead to the quickness of rendering but that hasn’t been the case in other environments. Since we can assume that a thin client’s backend should have the minimum ressources to support mallocs without using the swap area, there should be no visible difference of usage on a thin client. To give users control over image manipulation, I added the “browser.gdk_interpolation_threshold_percent” preference variable that allows values between 1-100 and has for effect to either use or bypass GDK image manipulations. The default value is 50% which only affects images being scaled by a factor of 50% or 200% and images with a visible portion under 50%. Lowering this value to it’s minimum (1%) would turn the feature off while using 100% will premanipulate any image overflowing it’s container. I’ve generated new Talos reports here run on a thin client. Finally, as Joe from #gfx did mention, I added support for images upscaling with GDK to increase the image quality by also using the bilinear interpolation algorithm. Here is the difference between an original 309×329 px image :
Upscaled image in Firefox 3.0 using X’s nearest-neighbor interpolation: (current behavior)

Upscale image using GDK’s bilinear interpolation:

Finally, here’s the complete patch that I hope will be integrated in Firefox’s and included in a future 3.0.x update. It currently only changes the behavior on UNIX systems since the main goal is to reduce memory consumption in X and that I am not aware of that kind of problems for GDI+ or Quartz backends.

diff : x-memory-optimization.diff
cvs diff : x-memory-optimization.cvs.diff

1 comment.