The Question of Standards

1.1 Introduction

The flow of incoming new astronomical catalogues and data sets is ever increasing; and since 1993, large tables published in some astronomical journals are also available as electronic files, due to the agreement with the A&A Editors (See Editorial note of A&A 266, E1), and to the development of the electronic publishing for the AAS journals.

The question of a uniform description is raised as soon as we expect to use any of the catalogues in a transparent way, e.g. to ingest the data into a data-base, to load a table of a published paper into our image processing system, etc...

1.2 FITS: why

FITS format (Flexible Image Transport Format, see e.g. the FITS Support Office Home Page) is apparently the most popular data format for the exchange of computer data in our discipline, and is endorsed by IAU. FITS for coding images is widely available (e.g. in the popular xv image displayer), and the FITS way of specifying the position and the geometry of the field of view on the sky (the World Coordinate System or WDS) allows to combine images of the sky and data coming from catalogues in tools like Aladin. FITS defines also a format for tabular data.

1.3 FITS: why not

FITS is well accepted as a transport format; it apparently is also being commonly used for data archiving (e.g. ESO, OGIP) of image or spectral data. However, as far as tabular data is concerned, we prefer not to store the data in any of the two FITS formats defined for tabular data, mainly because

the data stored at CDS are data published in the astronomical literature, and we want to be able to compare by eye the computer files with the published material; the FITS ascii table files have no carriage-return, and such tables are therefore not immediately printable.
The description in FITS of the table structure is well adapted for computer usage; however, an overview of the contents of a table — which parameters, and their meaning — is not pleasant to read by a human eye;
due to the fixed-length record structure, FITS files are generally larger than standard ascii files, even for FITS binary tables;
due to this structure of bulk files, no standard Unix tool like grep, sort, join, paste and other awk utility which contribute to make Unix so powerful can be applied to FITS files — even an editor cannot be used.

1.4 Proposed solution

We do not want to keep several versions of the same file: not only it would eat up a lot of disk space, but mainly the existence of astronomical catalogues in several data format standards would inevitably generate conflicts and incompatibilities.

In order to be able to use the various tools, we therefore chose to keep the astronomical catalogues as plain ascii files, and to store the description of the catalogue as a separate ascii file. This description file includes all the necessary information related to the catalogue: author(s), reference(s) of the related published papers, brief summary, scientific keys, caption and accurate description of each table of the catalogue.

This description file — the key of the catalogue — is especially important: it must be easily readable by a human eye, and simultaneously contain all the information required to achieve the conversion into other formats if required: for instance, the transformation of tabular data files into FITS can be done automatically from the CDS FTP server; and a client/server prototype, described in the cdsclient package, has been developed for remote copies of the astronomical catalogues in a few formats, including FITS. This description file is also used for checking purposes: a standalone program named anafile, is used in this context.

The standard discussed in this document mainly addresses the tabular data, but some catalogs may include data in a non-tabular form, like maps, images, video sequences, etc... Such non-tabular data should be described in the description file on the file level.

The standards proposed here are developed in the following chapters. Some points in the proposed standards are not completely fixed and possible alternatives are indicated by a note.

1.5 History of this document

The first version of this document was published as the paper Adopted standards for catalogues at CDS published in the Bulletin d'Information du Centre de Données astronomiques de Strasbourg (BICDS), (Ochsenbein F., 1994, BICDS 44, 19).

Version 1.4 of the document was dated 12 September 1994, and resulted from discussion with ADC colleagues, mainly N. Paul Kuin. Common conventions concerned the standard filename conventions — ReadMe as the description file, extensions .dat for the data files — and the basic label definitions.

Version 1.5 is dated 12 June 1996, and contained a few more conventions on file names and label definitions.

Version 2.0 is dated February 2000, and resulted from discussions with G. Schwarz (AAS, Tucson)

1.6 Related Documents

The following related documents are available on the network:

ftp://cds.unistra.fr/pub/cats/doc.ps: this document, in Postscript format
http://cds.unistra.fr/doc/submit.htx: short overview and tips for preparing a standardized catalogue
http://cds.unistra.fr/Cats.html: list of catalogues and tables available at CDS
http://vizier.cds.uinistra.fr/: home page of the VizieR service, with links to the existing clones. VizieR is a direct application of the usage of the standardized description of catalogues for the ingestion of astronomical data into a database.
http://vizier.cds.uinistra.fr/doc/astrores.htx: a possibility of getting the results in XML, an emerging standard which basically allows to have the data and their meta-data (accurate description of the data) in a single document; XML allows to display the data, and offers a possibility of using the data in new applications.

/srv/httpd/Pages/doc/catstd-1.htx