Author’s note: This is a revised version of my previous blog to make it more international.
Author’s note: Link to a Romanian translation of this article (by Alexander Ovsov)
Data versus Information
As part of open government initiatives, government agencies are creating guidelines and goals for making more of the information held by them more accessible to the public. See for example, the Open data initiative in the UK and the EU directive on Public Sector Information.
Of course, we advocate the judicious use of PDF for disseminating that government information. Well, in most situations.
We use the terms data and information to distinguish at least two ways government agencies are being asked to provide information to the public in electronic form. Data is just the raw numbers, names, places, etc. The stuff one might pull out of a database. But data can be shaped into information by providing interpretations, drawing conclusions, pointing out inconsistencies and then packaging it in an attractive form. And then there is the most extreme view that the medium is the message!
When we go to government websites we usually want information not data. When an analyst goes to a government website she might want just the raw data so that it can be interpreted, shaped and analyzed, and turned into a specific document as information not provided by the government website.
Hammers
There is the saying that if you have a hammer, then everything looks like a nail. And we have to confess that, from our view, almost all needs are best addressed using PDF. PDF is our metaphorical hammer. We have some justification for this position as the most significant PDF software provider in the world.
Government agencies can, and do, use PDF for effective information distribution. Information distributed as a PDF can be downloaded, read in its electronic form, saved for later reference, shared and printed. Everyone has a PDF reader. PDF documents can also be infused with, what we at Adobe like to call, “rich document” features. The final representation of the information can be very important. As noted earlier, the medium is the message, or it certainly can make an important contribution.
However, for the person who wants raw data, PDF isn’t the right choice. See, we are willing to refrain from hitting everything in sight with our PDF hammer! But raw data isn’t of much use unless you have metadata (data about data) describing, how to access the data elements, the ranges and meanings for the individual data items, where the data comes from, how old the data is, how authentic, and so many other important properties. So we will argue later that, in fact, PDF can play a very important role in distributing raw data, offering a clever means to accompany that data with meaningful and accurate metadata. (We are using the term metadata here in a very general way as data about data, and maybe a little differently from what you are used to.)
XML
The government initiatives have surfaced people who have XML hammers. Some XML enthusiasts, but certainly not all, go overboard. We think it hurts their cause. As I have blogged earlier (XML for ••• ), XML is one of the most misunderstood, yet useful, technologies we have in today’s toolkit. We need to drop our hammers and consider the facts.
Using XML for raw data is the kernel of a good idea. But there are some limitations of XML that need to be addressed when considering it as the only option for data distribution:
1. Those who are not familiar with XML, need to realize that XML isn’t a single markup language for a single use, but it is a method for defining and using specialized markup languages. That is why we have to say XML for business cards, XML for invoices, XML for classifying political action committees and so on. There are thousands of such XML markup languages and there will be thousands more to cover all those government datasets where XML is appropriate. I also have a blog on this topic.
2. Large raw datasets can be prohibitively large when expressed in an XML markup language. Unnecessarily large, from an information theoretic view. For example, here is an XML data file that can be found on www.data.gov at this webpage. Note that, when you download this file, it is a ZIP file whose size is 11,903,362 bytes. When you unzip it though, you get a 220,655,917-byte XML file. In this case, the EPA personnel know that XML files can be very large and have packaged it in a ZIP for downloading to reduce the transmission time by a multiple of over 18. In other words, if it takes a minute to download the ZIP’ed version, it will take over 18 minutes to download a raw XML version. After unzipping, the XML file is identical to the original. So any advocacy for XML, should always be accompanied with a discussion on file size and considerations of using something like ZIP. To do otherwise would be irresponsible.
3. XML files need additional metadata in order to make use of the data that is found within them. If you are given three numbers (007, 56, 00010) you might not guess that they represent a birthday, which is more commonly supplied as 7-10-1956. But even if it is provided in that form, it isn’t representative of how it is often expressed in other countries as 10/07/56. Unless given additional information, there is an ambiguity between the 7 and the 10 as to which is the month and which the day. And this is a trivial example. We need extra information, e.g., metadata, in order to be able to interpret data accurately and make proper use of it. The basic syntactic rules used for a markup language can and should be provided by offering an XML Schema (.xsd file) but even that does not go far enough. We also need to explain the semantics; that usually requires a technical document.
4. There are other raw data formats that might be more suited to particular needs such as standard spreadsheet files (.csv — character separated values) or the Microsoft and Open Office spreadsheet formats (.xlsx and .ods, respectively). The latter can also include the formulae and rules needed to compute dependent and summary data. In addition there may be very application specific files, not in XML format, appropriate for specific needs. For example, shapefiles (.shp) for holding spatial feature information as in the http://www.data.gov/raw/6 webpage reference earlier.
Note: There can be an argument made that compressing/decompressing files is so time consuming that the time lost there is not made up in reduced transmission time. With today’s lighting fast CPU’s the compression and decompression times are relatively minor, but the transmission times can be a problem if you don’t have the latest and greatest Internet connections. So nearly always, the choice to used compressed data is the right one for data expressed in an XML markup language.
PDF versus ZIP
In another InsidePDF blog (PDF File Attachments), the file attachment features of standard PDF (ISO 32000) are described. To summarize, any number of file attachments, in any format, can be embedded into any PDF file. They can be extracted for use by anyone receiving the PDF file. In addition, when files are attached/embedded into the PDF, they will be compressed using the same compression method that is used in ZIP files: deflate/flate. For the purposes of distributing government data, this is nearly ideal. The PDF file can carry the .xml or other raw data files as compressed attachments and that base PDF document, itself, can provide all the additional semantic information that would be needed in order to make use of the data — the metadata. If the raw data is in XML form, then a compressed XML Schema file (.xsd) can also be attached to the PDF document. So when using PDF, the points made above are addressed: file size, necessary metadata to define the XML markup language used, and formats other than XML.
Sample PDF envelope containing XML data
We have created a sample PDF envelope starting from this government dataset. You might want to look it over. Note that both the XML dataset and the associated Schema file are attachments to the PDF that helps to define the XML markup language used for this file. We took the general introduction from the government web page and made up a brief description for each of the XML elements found in the file. Make sure to use a PDF reader that can display the attachment annotations and that can extract the attachments. Adobe Reader can do that.
Programmatic extraction of any data out of PDF
PDF attachments can be of any format and can also be organized hierarchically, just as you can with a ZIP file. And like ZIP, there are numerous open source projects devoted to the creation, modification and viewing of PDF. One very popular one is iText by Bruno Lowagie and iText Software. They also have an excellent example demonstrating how to use iText to create a PDF containing various data files. In addition, it also shows how someone could programmatically extract those contents. This means that the PDF is not only usable for pure human consumption but also for pure machine consumption or a combination thereof.
Authenticating PDF envelopes by PDF Digital Signature
Another important benefit that just works out when using PDF envelopes, is that the digital signature technology available for PDF files also can cover the attachments automatically since they are an official part of the PDF file. Government agencies can send digitally certified PDF files containing data files and their customers can authenticate that the PDF, and all the attachments, came from that agency and have not been tampered with. See the InsidePDF blog about Authenticated PDF Documents.
Other ways to use PDF attachments for government information delivery
So we described how we use PDF to provide a complete package for raw data downloading. But we can also augment richly formatted PDF information files with the raw data files that were used to create the PDFs. For example, if a document contains, pie charts, bar charts or tabular information, the raw data that was used to create that formatted information can be attached to the PDF file. An annotation can be placed on the chart or table that allows the appropriate attachment to be extracted. This is covered a bit more in an InsidePDF blog: PDF file attachments.
A third use for PDF attachments is to create hybrid files. These are PDF files that attach the original editable source document (e.g., .odf or .docx file) that was used to create the PDF. In a sense, this make PDF editable, something that people have asked for. Quoting from the OpenOffice.org website: “A hybrid PDF/ODF file is a PDF file that contains an embedded ODF source file. Hybrid PDF/ODF files will be opened in OpenOffice.org as an ODF file without any layout changes. Users without this extension can open the PDF part of the hybrid file with their PDF viewer.” Adobe Acrobat’s Microsoft Office tools can also create PDF files with the Office file that created them as an attachment.
So hammer away, you PDF enthusiasts.