Hello

In today's rant, I look at a few different text document formats, and compare the file size of each, I then question what we trying to achieve with these formats anyway?

If the guts of file formats are not your cup of tea, then move along and come back tomorrow when we will talk about something else.

Method

The process was the same for each application. I opened the program, typed 'Hello World' and then saved the file. I did nothing else in the program so as not to generate any more metadata. The text sample was literally Hello World, i.e. eleven Characters (ten letters and a space), no newlines, no selected fonts or styles.

Results

Format Application File Size (bytes) .txt Emacs 21.4.1 11 .abw Abiword 2.4.6 2517 .odt OpenOffice Writer 2.20 6674 .doc Microsoft Word 2003 SP2 24064

I don't have Office 2007 and OOXML, it might be interesting if someone could tell us the filesize of Hello World in that.

What is this space being used for?

Besides Emacs, all the other applications are storing a lot of style information and metadata about things that I did not specify. They are storing the default font type and size, paper type and size, as well as information about features that I have not used: footnotes, embedded images, macros and so on.

Abiword is slightly more modular and smarter than the other two formats, it does include all the default fonts and so on, but it does not store information about unused features. The size difference between the Abiword format on the one hand and ODF and Microsoft on the other is greater than shown here because I am not actually comparing like-to-like.

The Abiword file is in a human readable format already. The .doc format is binary, so it would take up a lot more space if expressed in a human readable format. Likewise, ODF files are archive files. I unzipped the .odt file into human readable files and there was a whopping 88K of data inside it - remember this is just for 'Hello World'.

Do I want default styles?

The first question that arises for this is when I do not explicitly select any styles, such as paper type or font size and so on, should the application choose styles for me and write it into the file? Or should it respect the default style of whatever computer and application the file might be opened with?

Let's look at this by taking an example, the font type. Now the question is whether the application should include font type information within the file, even if I have not explicitly selected a font?

Let's imagine that I create my file on Linux and then share my file with someone on Windows using Word that is set up to understand the format of the file (either by default or through a plugin).

The first option would be to not include any style information. So 'Hello World' would open on their screen in Word's default font, which I am assuming is still 'Times New Roman'.

The second option would be to include the style information. So it saves in the metadata that font should be 'DejaVu Sans' (this is the default format on OpenOffice on my version of Linux). Word opens the file but Windows does not have DejaVu Sans, so 'Hello World' opens in Times New Roman anyway, looking exactly the same as in the previous example.

If I really cared that the other person could see the document in DejaVu Sans, i.e. that the document on their screen has to look the same as the document on my screen, then we would embed the font in the file, i.e. we would end up with something that looks rather like PDF. The filesize may end up as several megabytes (just to say 'Hello World').

There is a difference of course between the file format and the behaviour of the application that implements the format. For the fun of it, I stripped out the style defaults from the Abiword file, and got it just below 1000 bytes, and it still opened without error.

I don't know exactly how much style information I could strip out of my .odt file before it became a invalid ODF document according to the standards, but it would be an interesting exercise to try to create the smallest valid ODF file.

Dumb and Dumber?

The second question that arises is about modularity, or to put it another way, about 'stretchiness'. I personally like the Abiword approach, i.e. it only creates image metadata when you embed an image, it only includes information about footnotes when you include a footnote, it only has metadata about macros when you have macros and so on. The file starts as simply as possible and only becomes complicated when you need it to.

The ODF format has lots of scaffolding just in case you use these extra features. My Hello World ODF file archive contained over half-a-dozen XML files and over a dozen folders.

I'm sure there are lots of reasons that could be given to justify for all this bloat, but I am not sure that the file format should be optimised for a very small percentage of the files actually being created in the real world, i.e. a file that uses every single feature and so all the metadata and all the subdirectories. So in other words, ODF text and .doc format are optimised for people writing large complex documents that contain lots of macros.

The problem is that for these tasks, you would be better off with a specialist tool. For example, for typesetting you are better off with a tool that generates LaTeX, for complex corporate tasks, it is almost always more efficient to have proper corporate applications with servers and database- backends and the like than building everything upon word processor macros.

So these file formats are optimised for people using office tools beyond their natural capabilities, i.e. people doing a sloppy job because they do not have adequate technical support or budgets for anything else.

Let me be a bit clearer, what people often think of as an 'advanced' or 'complex' document, often is not that complicated from a technical perspective. A document with some tables and some embedded images is not a complex document. While most extremely complex documents ('monster documents) are extremely complex because people are tying too hard with the wrong approach and tools.

Back to the future?

So a few people are quietly discussing putting both OOXML and ODF to one side and instead looking at the format commonly called XHTML+, which is summarised in RFC 3236 (the MIME type is application/xhtml+xml), to see how far it can go.

If you have a XHTML+ document and a stylesheet in CSS, you can cover 90% or more of the documents that are actually created, PDF or more specialist formats can mop up the rest. Tables, embedded images, complex styles and so on are all fine. Being an XML-based format, it is machine readable; being an HTML-based document, it is already web-ready.

This would be a return to some of the 'rich text' approaches of the past, or at least the next generation equivalent of them. These approaches were using classic (non XML) HTML for documents and emails and so on. The difference now is that such a XHTML+ based document format would be native XML and the style would be separated from the content.

This document format would not need many changes to existing office applications, every word processor released in the last half-decade should be able to at least open an XHTML file with a stylesheet, Microsoft Word included; with minimal help, these applications should also be able to write the files in the particular way required by the format.

The idea of a light, fast and easy to use XHTML+XML based document format (that simply ignores the dead-end monster documents) is a great idea, but I doubt this will get anywhere, the world is too exasperated for a new format right now, but it is interesting none the less for how it reflects on ODF and OOXML.

Which part is the document anyway?

A lot of discussion about document formats especially in the media, this post included, is somewhat dubious because the terminology is ambiguous.

Ambiguous terminology normally arises because either the whole picture is not seen and/or because the actual specifics are not understood, i.e. there are breath and/or depth issues hiding under the discussion.

So far we have been discussing what we should do when taking a document from my computer to your computer, i.e. what should be transferred? But to have the discussion properly, we need to agree on what is a document anyway? Why am I trying to share bits and bytes with you in the first place?

This I think is where we need to use a different word altogether, let's use the word 'information'. No one wants to share documents for the sake of it. People want to share information, and people want to do this easily, accurately and efficiently.

What do I mean by accurately? By this, we mean that the information that people receive is as faithful as possible to the information sent. Faithfulness is the older English word, the french import is fidelity, (of course, both originally derive from the Latin word 'fides', the Roman goddess of trust).

So when people talk about that the results of some document converter are 'high-fidelity' or 'low-fidelity', the question is fidelity to what? Fidelity to the information itself, fidelity to the visual appearance, fidelity to the file structure, fidelity to the macros?

Information not Macros

You might have guessed already, that my bias is towards fidelity to the information itself. Some companies, organisations and governments are asking themselves the following question:

*Can ODF include all the styles and macros that have piled up inside our documents that have piled up on our hard-drives and shared drives over the last decade? *

This is the wrong question. The better question is 'can we get all our information out of these binary formats so we can move to a more efficient way of working?'

This data should not be put into new XML-based documents, just to pile up again on people's hard-drives and shares, it should be in server driven tools, databases, web pages and so on.

I am not a fan of office software macros because macros are crutches. By all means have plugins for your word processors and other office software, but software logic should be shared separately from information. Likewise, corporate styles should not be combined with data until the last possible moment, i.e. when the data is actually leaving the organisation, i.e. in web sites, PDFs and so on.

Over to you

That's how I feel about it today anyway! This might well be one of those posts that you can find holes and edge cases that get me to change or refine my opinions. So leave a comment and let me know.