Subject: Re: migrating versions of data

Hi Jim,

> 
>                                                  MSU 517-355-3328, 20-JUL-1995
> 
>     I am trying to understand how data version migration is handled in DSPACK.
>  Suppose I write out version 3 of an electron object.  Now I upgrade my
>  software and make version 4, which has in my structure a new slot for TRD
>  information.  I read the old data with my new program.  What happens?  I
>  presume if I try to read it at the structure level, with or without
>  pointers, I'm in  trouble. 

Yes - right now this is true.
We are working on it - see below. 

> 
>     Is there any provision for calling say dsdef to inform the system
>  of the current version of the object?  Then an error message could be
>  generated when the full structure was requested with dsget etc, although
>  components could be returned safely.
>

If you define the new object before reading the old one from old tape/file
- you will get an error message when reading the file header.
But - I don't think you could access data safely...

Yes - I see what you mean - the idea that we could still use the data...
There is a idea around which is very close - this is that you just tell
DSPACK what you think the size of the object is. If it is not what you
think it is - we could turn some of the routines off - this is not
too complicated and could be implemented as a safety feature.

>     It appears that your data access method assumes full storage association,
> and is thus incompatible with Fortran 90 and perhaps with C++.  Is that your
> viewpoint?
> 

Not necessarily.
As it happens - this is a convenient way of working - people call DSPACK
routines to get a pointer to data and from then on assumme that their
program knows what the data are like.

There are tools to make it easier - like header file generation from
DSPACK files - you can read the header of the file and produce include
files for your fortran, c (and C++) - you can then compile your programs
with it and you have a guarantee that the pointer mapping makes sense.

But - this does not solve the problem of 'version migration'

At present - you may choose not to use full storage association.
Calls like IDSGET can access individual elements of the objects
by name - they will work independently of the version. IDSGET
also returns the size of the structure in DSPACK - so you can reach
the same element in the next entry by offsetting the pointer returned
by IDSGET (in this case you would not use C pointer arithmetic
because it relies on the structure definitions in your program)

However - there is a grand plan to allow full storage association
with version handling.
It has two stages:

1) Simple mapping by name:
   This will work in the following way:
   a) You have a DSPACK file with an object defined in some way.
   b) Now you run a DSPACK program, define the same object differently
   c) You read the old tape.
      The data will end up in your new object in the right places
      by copying elements of the old object to the new object.
      The mapping scheme will be derived from names. All elements
      in the new object which are not found in the old object will
      be 0, all elements in the old object missing from the new
      object will be lost.
      Optionally - the old object may be renamed and kept in memory
      as well.

2) Full version handling:
   a) Each object will have a version number
   b) Each object may contain a full prescription of conversion from
      previous version. This would be user-defined, although some
      tools may be provided to produce startup conversion table
      based on method 1) above - then the user could modify it.
      These conversion tables would be kept in object repositories
      together with object definitions and would be stored as
      object resources when the objects are defined.
      In the most sophisticated case the conversion could involve
      more than just mapping - in principle the elements of the new
      structure could be calculated from the elements of the old
      structure - or even from elements of some other structure.
   c) If the object found on input had different version from one
      already defined in the program - a search for conversion
      tables would start - hopefully resulting in generating
      an effective conversion table from the old to the new one.
      This would be done once while reading the file/tape header
      - from then on the generated table would be used to convert
      data.

I hope these two methods should satisfy most of the requirements.
(Although the second one is obviously much more complicated and
is a bit lower on the priority list - the first one is in fact very
high on the job list - the pressure is growing and we will have to
have it quite soon)

Best regards

Ryszard Zybert
    zybert@rhi.ph.bham.ac.uk