Non-HEP data format/transport packages

Most of these packages are based around "scientific" data, that is multidimensional arrays. Several go beyond that. Most have utilities, viewers, and some have the ability share data with commercial viewing tools. It offers some perspective that these methods are meant in part to allow sharing of data BETWEEN experiments, not just between collaborators. These were culled from

Software for Portable Scientific Data Management, Computers In Phyisics, May/June 1993 (Includes a nice comparison chart)
Creating and Using PDB Files, Computers In Physics, March/April 95
MathHDF: MathLink-based Distributed Visualization Between Mathematica and HDF Files Computers In Physics, May/June 93

The Computers in Physics articles may be out of date for HDF/CDF/netCDF.

The CDF (Common Data Format) from NASA
The netCDF outgrowth of the CDF from Unidata (NCAR), with extensions aimed at networking.
The HDF Hierarchical Data Format from NCSA incorporates many features of CDF and allows user definitions of new data types. The CDF (High Energy Physics CDF) group is quite interested in this package.
The PDB Portable BinaryData Format form Livermore allows portable storage of arbitrary C structures or Fortran Common Blocks. It is part of the larger PACT project. Local copies (as of 31 July 1995) of documentation include:
- PACT Frequently Asked Questions
- The PACT Postscript manual.
- PDB Frequently Asked Questions
- The rather large PDB Postscript manual.
- PDBview Postscript manual
- PDBdiff Postscript manual
Unfortunately, the postscript copies seem to only allow sequential viewing ("next screen" in ghostview, perhaps because of its Framemaker parentage. I also have notes from a phone conversation with Stewart Brown, one of the authors; I'm responsible for the transcription, not him.

Mostly on the basis of the first of the articles above, I had felt that

only HDF and PDB were high-level enough to directly support complex data
HDF requires central registration of new data types, which to me seemed too restrictive and beaurocratic to use for each new object we define
that leaves PDB to investigate; I made a phone call but have done nothing more.


URL's seen in sci.data.formats via Ed May from:
 
                G R E E N    E G G S    R E P O R T
                                by
                        mailto:rick@ar.com
 
 
        The Green Eggs Report is a monthy post of URLs
        spotted within news:sci.data.formats
 
These discuss self-describing portable data formats used by scientific
communities outside of HEP.
 
ANNOUNCING: WWW-based CDF data syst
http://nssdc.gsfc.nasa.gov/omniweb/ow.html
http://coney.gsfc.nasa.gov/Mathews/jason.html
 
Scientific Data Format Information
http://info.mcc.ac.uk
wais://host.name.domain
http://star-www.rl.ac.uk/
wais://wais.unidata.ucar.edu
ftp://crusty.er.usgs.gov/pub/netcdf.asce.ps
http://www.unidata.ucar.edu/packages/netcdf/
http://stardust.jpl.nasa.gov/stdref/stdref.htm
http://www-mipl.jpl.nasa.gov/vic_file_fmt.html
http://www.sara.nl/Consumer.Report/Report.html
ftp://ftp.unidata.ucar.edu/pub/netcdf/guide.ps.Z
ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf.tar.Z
http://www.wimsey.com/~infosafe/saif/saifHome.html
http://www.msi.umn.edu/SciVis/Packages/packages.html
http://www.unidata.ucar.edu/packages/netcdf/faq.html
http://www.ncsa.uiuc.edu/SDG/Software/HDF/HDFIntro.html
ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/ncview.tar.Z
http://www.unidata.ucar.edu/packages/netcdf/utilities.html
http://www.unidata.ucar.edu/packages/netcdf/guide.txn_toc.html

James T. Linnemann