Column-Wise Ntuple I/O Optimization

Why Ntuple I/O is so important

There are several factors that would affact your data analysis time:
In D0, almost all the data in Challenge machine are stored in the form of Column-Wise Ntuples. When people do data analysis using PIAF server and system load is not high, I found that Challenge CPUs often have to wait for I/O, e.g. reading data from disks. I/O is the bottle neck. Ntuple I/O is directly related to this.

For each CWN, data and control information are stored in physical records. Each physical records contains logical records corresponding to variable buffers. The relation between logical record and variable buffer is 

	logical record = variable buffer + 15 words of ZEBRA header
Each physical record may contain more than one logical records depending on the size of record length and buffer size you set when you make a Ntuple. Because each logical record corresponding to one variable, one physical record may contain more than one variables. In CWN, data are retrieved by variable, which means each time a physical record is read into memory, only a logical record related to a variable is used. For example, if you use 4096 as your record length, and 1009 as your buffer size, then logical record length is 1009+15=1024, so a physical record may contain 4 logical records, each time 4096 words are read from disk to memory, only 1024 words are used, of which 1009 words are real data.

The highest I/O efficiency is achieved when physical record length is equal to logical record length. Then each time a physical record is read, it contains the variable data exactly what we want. In this case, it means we need to set buffer size equal to physical record length minus 15 words of ZEBRA header.

On the other side, use large physical record length reduces the total number of records in a CWN file. It reduces the number of I/O access times.

When you make a new CWN, you can control the record length and variable buffer size. You give the record length when you call HBOOK routine HROPEN, and you can call HBOOK routine HBSET to decide the buffer size, the default buffer size is 1024-15=1009 in which 15 is the ZEBRA header size. One thing worth notice is that in order for calling HBSET effective, you have to call it before calling routine HBNT which is used to book a new CWN. Otherwise, buffer size still use the default value 1009.

Now, you probably make a conclusion that increase buffer size and record length will give your CWN data better performance. But it also brings some side effects. One of the side effect is that it increases the file size a bit. The file size it increases depends on how many variables in your N-tuple. For example, if you have 300 variables in your CWN, and the original record length is 1024, we will increase the record length to 4096, then AT MOST it will increase a file size by: 

	(4096 - 1024) X 300 X 4 = 3.6 MB
multiply 4 converts words to bytes in 32 bits machine.

For a file which has size around 50MB, this increase is fine, but for a file which has size 3 MB, it is not worth increasing the record length and buffer size to a large value like 4096. You can decide if a file needs a larger record length and buffer size by simple calculation. The goal is to reduce the number of r/w times of I/O when you analyze the data.

The other side effect of increasing record length is that it requires larger memory both in making the new N-tuple and analyzing it. Fortunatly, when you analyze it, the memory increase is minor, only 12KB/variable in the above example. PAW and PIAF can afford this without any problem.

CONCLUSION:

Choose proper record length and buffer size according to the size of a N-tuple file.

Statistics

I used a QCD Ntuple file as proto-type, this Ntuple has 220K events, 70 variables, 115 MB file size, 8191 words of record length and 1009 words of buffer size. I made three other Ntuples using this one. The buffer size for these three Ntuples have been set to "record length - 15", the record lengths for these three Ntuples are 1024, 4096, 8191 respectively. These three Ntuples diff with the original one only on buffer size and record length. Table bolow gives time to plot one vector variable with index for diffrent Ntuples using PAW.

Ntuple
record length (words)
buffer size (words)
seek time (s)
read time (s)
key and cycle decode time (s)
other time (s)
Total time (s)
1
8191
1009
15
14
22
15
66
2
1024
1009
14
16
17
17
64
3
4096
4081
1
6
2
21
30
4
8191
8176
1
4
2
21
28

From the table, we can see for this perticular Ntuple, change record length to 4096 and buffer size to 4096-15 has reduced record seek time, read time and RZ key and cycle decoding time a lot. Ntuple I/O has been almost optimized. Increase buffer size further to 8191-15 wouldn't reduced I/O time to NONE.

Ntuple I/O optimization tools

Some tools in Challenge machine and d0sgi clusters:
info_nt
tells you what the record length is and how many records have been used (Maximum 65000) in a CWN file. Also tells you buffer size of a Ntuple.
chlen_nt
changes an old CWN file to a new one with specified record length, buffer size is set to optimizing value "record length - 15". The new Ntuple file differs old file only in record length and buffer size.
merge_nt
Merges several similar small CWNs to a big one, record length and buffer size will be changed for the new CWN too.

Notice:

dongzhao@fnal.gov