RDI work

This page is for changes in handling RDI format, occasioned by issue 777

Here, 'doc' means OS_TM_Apr14.pdf from Teledyne/RDI

Task 1. find the data chunks

• Fixed Leader Data format the documented length changes across firmware versions. What a mess. The 2014 doc says this is 50 bytes long (firmware 23.17; doc figure 46, page 145) but Gonzalez' file (firmware 23.19) has 60 bytes. I think the only answer is to ignore the length given in the tables and work with the data-offset information ... if I can figure that out.

Browse[1]> header$dataOffset
 [1]   28   88  148  790 1112 1434 1756 2078 2159 2193 2243
Browse[2]> LEN<-diff(c(0,header$dataOffset))
Browse[2]> LEN
 [1]  28  60  60 642 322 322 322 322  81  34  50
Browse[2]> LEN[1]==6+2*header$numberOfDataTypes # header OK
[1] TRUE
Browse[2]> LEN[2]==50 # FLD wrong length
[1] FALSE
Browse[2]> LEN[3]==60 # FLD right length
[1] TRUE
Browse[2]> LEN[4]==2+8*header$numberOfCells ## velo is ok
[1] TRUE
Browse[2]> LEN[5]==2+4*header$numberOfCells ## corr is ok
[1] TRUE
Browse[2]> LEN[6]==2+4*header$numberOfCells ## intensity is ok
[1] TRUE
Browse[2]> LEN[7]==2+4*header$numberOfCells ## percent-good is ok
[1] TRUE
Browse[2]> LEN[8]==2+4*header$numberOfCells ## status is ok
[1] TRUE
Browse[2]> LEN[9]==81 ## bottom track is ok
[1] TRUE

Then we have something 34 bytes long (meaning?) and something 50 bytes long (meaning?)

Hm. Let's look at codes, since this is how I now plan to scan the file; I'm writing my interpretation to the right of the codes (based on table 33, page 146 of doc)

Browse[12]> cbind(buf[1+c(0,header$dataOffset)],buf[2+c(0,header$dataOffset)])
      [,1] [,2]
 [1,]   7f   7f Header
 [2,]   00   00 Fixed leader
 [3,]   80   00 variable leader
 [4,]   00   01 velocity profile data
 [5,]   00   02 correlation profile data
 [6,]   00   03 echo intensity profile data
 [7,]   00   04 percent good profile data
 [8,]   00   05 status profile data
 [9,]   00   06 bottom track data
[10,]   00   30 binary fixed attitude
[11,]   d8   30 binary variable attitude (since d8 is between 0x40 and 0xF0)
[12,]   00   20 navigation

so that looks like this is going to work ... the plan therefore is to work through the file, using the data offsets and IGNORING the old method I had, which skipped based on what I had previously thought were known lengths of data chunks.

Hm... we have ensembleStart as a vector of pointers to the 7F7F flags. So that's what we need already ... I had thought I would need to recode something. Below is a test on profile 1; it works fine with other profiles

Browse[12]> P<-1; cbind(buf[ensembleStart[P]+c(0,header$dataOffset)], buf[1+ensembleStart[P]+c(0,header$dataOffset)])
      [,1] [,2]
 [1,]   7f   7f
 [2,]   00   00
 [3,]   80   00
 [4,]   00   01
 [5,]   00   02
 [6,]   00   03
 [7,]   00   04
 [8,]   00   05
 [9,]   00   06
[10,]   00   30
[11,]   d8   30
[12,]   00   20

STATUS I think this is working now in the rdi branch. At least, issue test 777b.R works fine, reproducing the matlab results and also having some sort of navigation info (not checked yet).