5.3 Hierarchical Grouping Proposal

 (Jennings et al. 1996) have proposed this convention, which provides a mechanism for logically associating different HDUs that are physically separated into groups that may have a hierarchical structure. It arose in part from efforts to convert between FITS and the Hierarchical Data Format (HDF). HDF is under continuous development at the National Center for Supercomputing Applications (NCSA), and current information is available from the NCSA World Wide Web site at http://hdf.ncsa.uiuc.edu/. The discussion in this Guide is taken from that paper. Whether this convention needs to be submitted for IAUFWG endorsement or may be considered a recognized convention without being part of the formal FITS rules is still under discussion.

The grouping proposal provides several capabilities:

• It provides an alternative to EXTLEVEL for creating a hierarchical structure, one that has an explicitly defined tree structure identifying the higher level groups to which each lower level group belongs.
• It creates a FITS structure analogous to the Vgroup structure of HDF.
• It provides a means for logically including a particular HDU in more than one group without repeating the HDU.
• It provides a means for associating HDUs that are in different files, even those in different physical locations, into logical groups.

The last feature is becoming particularly useful as the fraction of data available on-line has increased and access from one computer to data on another has become routine. Related data may now be stored in separate locations and accessed electronically, instead of putting separate copies of all the related data on several machines.

Jennings et al. provide an illustration of where such a convention would be useful. Consider a set of observations, each of which has associated with it an observation log, event list, derived image, and calibration data, and for which many observations share the same calibration data. It would be possible to logically associate each (log, event list, image, calibration) set into a separate group; in addition, the same calibration data HDU could belong to many groups but would only need to be stored once. Another possible group would be one of observations of the same object at different times, and these observations could be combined into a group even though they might reside on multiple FITS files.

An extension HDU called a group table defines a group and its members. This table may be either an ASCII table extension (XTENSION= 'TABLE   ') or a binary table (XTENSION= 'BINTABLE'). Group tables are required to contain the following keywords in the header:

• EXTNAME (character) must have the value GROUPING
• EXTVER (integer) has a value that serves as a group identification number. Every HDU with EXTNAME = 'GROUPING' in a given FITS file must have a different value of EXTVER. This number distinguishes the group from any other defined in the file.

The following keyword is not required but is strongly recommended:

GRPNAME (character) provides a name for the group table. This name should contain only letters, digits, and underscore (hexadecimal 5F, decimal 95); in particular, it should not contain any embedded ASCII blanks (hexadecimal 20, decimal 32). The value is case-insensitive.

A group can contain any number of members. The members may be a mix of HDU types, in any order. They may be primary HDUs, extensions of types endorsed by the IAUFWG (currently TABLE, IMAGE, and BINTABLE), and other extensions that conform to the general rules for extensions provided in section 3.3 but have not been endorsed specifically by the IAUFWG. Member tables may be group tables, thus identifying subordinate groups. Group members can be identified by the group table in two ways: by reference or by position. Either or both can be used. Both have potential drawbacks. For identification by reference, the HDU is identified using the values of the XTENSION, EXTNAME, and EXTVER keywords. A possible difficulty is that there may be more than one HDU with these keywords in a previously created data set, making it impossible to identify the HDU uniquely. For identification by position, the numerical order of the HDU in the data set is specified. Caution is required in the use of this method because the order of HDUs in a FITS file may be changed when the file is copied. The location of group members not in the same file as the group table is specified by the World Wide Web Uniform Resource Identifier (URI). The table must specify the kind of URI being used: Uniform Resource Locator (URL), Uniform Resource Name (URN), or any future form of URI. The identification information is provided in columns of the group table with prescribed names, given as values of the TTYPEn keyword.

The following keywords are used to define columns of the group table if all or some group members are identified by reference:

• TTYPEn = 'MEMBER_XTENSION' is the heading for a column whose entries are character strings with the values of the XTENSION keyword, for extensions that belong to the group. For primary HDUs that are members, the string 'PRIMARY ' is be used. If no value is given, as when the group table contains a mix of identifications by reference and by position and the HDU is identified by position, an ASCII NULL (hexadecimal 00, decimal 000) may be used.
• TTYPEn = 'MEMBER_NAME' is the heading for a column whose entries are character strings with the values of the EXTNAME keyword for group member extensions. The treatment of extensions with no value of EXTNAME is the same as that for the case where no value of XTENSION is available.
• TTYPEn = 'MEMBER_VERSION is the heading for a column whose entries are integers with the values of the EXTVER keyword for group member extensions. For primary headers or where the EXTVER keyword is absent, the entry should be 1.

The following keyword is used to define a column needed if all or some members are identified by position:

• TTYPEn = 'MEMBER_POSITION' is the heading for a column of group member positions given as integers, with 0 (zero) representing the primary HDU, 1 (one) the first extension, etc. If the value is invalid or undefined, the entry should contain the null value specified by the TNULLn keyword for that column.

The following keywords are required if some or all of the group members are not in the same file as the group table:

• TTYPEn = 'MEMBER_URI_TYPE' is the heading for a column containing the acronym for the URI type used to identify the member location. Entries permitted as this is written are URN and URL; other entries will become acceptable as new URI types are defined.
• TTYPEn = 'MEMBER_LOCATION' is the heading for a column containing the URIs for the group members. For group members in the same file as the group table, the field may be filled with blanks; for members on the same machine but in a different file, the relative URI may be used instead of the absolute URI.

If either of the above fields is undefined, the field should contain an ASCII NULL.

A group table may contain any number of additional, user-defined columns. Group table columns may appear in any order. The values of TTYPEn are not case-sensitive.

To permit inclusion of HDUs from existing FITS files in groups, the grouping proposal does not require any new keywords for group members. Use of the EXTNAME and EXTVAL keywords defined in section 3.3.2, the rules for generalized extensions, permitting identification by reference, is strongly recommended. Every member of a group located in a particular file should have a unique combination of XTENSION, EXTNAME, and EXTVER keywords. Group tables that are members should have a unique combination of EXTNAME and EXTVER keywords, and there should not be an ASCII group table and a binary group table with the same (EXTNAME, EXTVER) in the same file. In addition, the use of EXTNAME allows HDUs of a given extension type and structure to be identified by a common name.

Use of the following two keywords also is strongly recommended:

• GRPIDn (integer) gives the EXTVER value of the nth group table of which the HDU is a member if the member is in the same file as the group table and is the negative of the EXTVER value if the member and group table are in different files. The presence of this keyword also warns software that is changing the position of the HDU that the HDU is a member of a group, naming the group table where entries used for identification by position will need to be changed.
• GRPLCn (character) gives the URI for group table n. It is not necessary if the value of GRPIDn is positive.

An HDU may be a member of many groups. If the HDU is a member of m groups, then there will be GRPID1 $\ldots$ GRPIDm keywords, each identifying one of these groups. If the value of a particular GRPIDn is negative, it is the negative of the group identification number (the value of EXTVER in the group table), the group table is not in the same file as the member, and there will be a GRPLCn keyword with the URI where the group table is located.

The Jennings et al. proposal presents a number of illustrations. As of this writing, it is available at http://fits.gsfc.nasa.gov/group.html.