KeithM At Home.

on the web since 2010.not retro.just old-fashioned

Inside monoVeon. Decompressing Data Format 30.

Inside the X3F file.

The x3f-raw-format document details the compression schemes for the SD series camera, but excludes the DP series camera. The schemes used for the SD camera do however provide an insight into the DP series. Having decoded the Directory of a sample image and found the start of the raw image block at 00165B68 we start with the following data:

00165B60: 53 45 43 69 00 00 02 00 r.a.l... SECi....
00165B70: 03 00 00 00 1E 00 00 00 80 0A 00 00 00 07 00 00 ........ �.......
00165B80: 30 E9 96 00 0��..... .....�.@

Interpreting this SECi block, we get:

4 bytes: 53 45 43 69 = SECi : Section Identifier
4 bytes: 00 00 02 00 = 2.0 : Version ( 2 bytes minor, 2 bytes major )
4 bytes: 03 00 00 00 = d3 : image data type : 3 = other
4 bytes: 1E 00 00 00 = d30 : Data format : 30 = Unknown RAW image compression in Sigma DP1 camera ( and DP2 it seems )
4 bytes: 80 0A 00 00 = d2688 : width of image ( or image columns )
4 bytes: 00 07 00 00 = d1792 : height of image ( or image rows )
4 bytes: 30 E9 96 00 = d9890096 : RowSize : value suggests block length...

On comparing a couple of files, the next few bytes always had the same values:

00165B80: 00 02 00 02 00 02 00 00 03 80 03 40 0��..... .....�.@
00165B90: 03 20 04 C0 04 60 04 E0 04 70 04 F0 04 10 04 A0 . .+.`.� .p.�...�
00165BA0: 05 D0 06 D8 06 DC 00 00 .�.�._.. .!1.��2.

This block starts with four 16bit numbers:

00 02 = d512
00 02 = d512
00 02 = d512
00 00 = d0

which are the starting values for differential Huffman decoding for R, G and B. In all the DP2 files I examined, these values were always the same. I still read and use these values though when decoding the file. Next we have fourteen pairs of bytes, which are thirteen Huffman table entries followed by terminating zeros:

03 80 00000011 10000000
03 40 00000011 01000000
03 20 00000011 00100000

04 c0 00000100 11000000
04 60 00000100 01100000
04 e0 00000100 11100000
04 70 00000100 01110000
04 f0 00000100 11110000
04 10 00000100 00010000
04 a0 00000100 10100000

05 d0 00000101 11010000

06 d8 00000110 11011000
06 dc 00000110 11011100
00 00 00000000 00000000

As with the documented compression schemes, the first byte gives the number of bits and the second byte gives the bit pattern. The leaf values are inferred from the order of the table so the first entry is leafvalue = 0, the last entry is leafvalue = 12. Hence:

Bit pattern Leaf Value
100 0
010 1
001 2
1100 3
0110 4
1110 5
0111 6
1111 7
0001 8
1010 9
11010 10
110110 11
110111 12

We are now here in the image block:

00165BA0: 14 21 31 00 EE CC 32 00 .�.�._.. .!1.��2.
00165BB0: E9 FA 32 00 A0 06 9F F4 3A A2 14 F2 42 43 29 51 ڷ2.�.�� :�.=BC)Q
00165BC0: 62 C4 32 14 C6 29 64 92 2C 48 A7 42 49 24 86 52 b-2.�)d� ,H�BI$�R

We next read three 32bit values:

h00312114 = d3219732
h0032ccee = d3329262
h0032fae9 = d3341033

These values are the number of bytes to be read for three blocks of data for red, green and blue respectively. The blocks however always start on a 16 byte boundary, so are zero padded. Looking at the start and end of the three blocks:

00165BB0: A0 06 9F F4 3A A2 14 F2 42 43 29 51 ڷ2.�.�� :�.=BC)Q
00165BC0: 62 C4 32 14 C6 29 64 92 2C 48 A7 42 49 24 86 52 b-2.�)d� ,H�BI$�R

First byte is at h165BB4. So if we now read h312114 bytes, that takes us to h477CC7, which is here:

00477CC0: 75 22 49 24 8B 12 49 24 00 00 00 00 00 00 00 00 u"I$�.I$ ........
00477CD0: 00 00 00 00 D3 FF A7 FD 62 49 24 5C 89 8E 1D 48 ....ˠ�� bI$\��.H
00477CE0: 8B 12 1E 11 62 49 0F 16 12 43 22 44 34 2C 99 E6 �...bI.. .C"D4,ֵ

We now have 12 bytes of '00' which aligns us to a 16 byte boundary from where we started. Restart from h477CD4, and now read h32CCEE bytes, that takes us to h7A49C1, which is here:

007A49C0: 64 93 00 00 A0 06 9F F4 58 B0 B2 1D 48 91 62 48 d�..�.�� X��.H�bH
007A49D0: 78 B3 1C 58 8B 12 49 22 C5 31 52 48 B0 B2 45 88 x�.X�.I" +1RH��E�

We now have 2 bytes of '00' to get to the boundary. Restart at h7A49C4, read h32FAE9 bytes, which takes us to hAD44AC, which is here:

00AD44A0: 90 D0 A6 85 92 1A 16 49 16 22 C4 92 49 00 00 00 ����..I ."-�I...
00AD44B0: 00 00 00 00 53 45 43 69 00 00 02 00 02 00 00 00 ....SECi ........

and we have 7 bytes to get us to the boundary, which is where the next SECi block starts. Phew !

Decoding the image block.

In outline, the program flow to decode the image block is:

Read Header Block

Read RGB starting values

Read Huffman table and build decoder tree

For each colour ( R, G, B )

For each pixel in the raw image

Read bits to get next Huffman entry

Leafvalue gives number of bits to read next

Read bits to get differential value

Decoded value = Previous Value + Differential Value

There is, however, a twist. With the SD compression schemes, the first value in a row is a decoded value and subsequent values are differential. The scheme used for the DP series is subtly different - which is where the ackowledgement to Dave Coffin's DCRaw comes in.

First, for SD cameras, the differential values are simply added to the previous value within the bit-length of the image ( 8 or 16bit ) - that is 8bit or 16bit addition. The scheme used for DP cameras closely follows that used for Nikon cameras. When reading bits for the differential value, the following code is used to retrieve the number of bits to be read, and then the value:

Private Function GetDiffIndirectValue() As Integer

Dim i As Integer

Dim bFlip As Boolean

Dim nNode, nNumBits, nValue As Integer

'start from top node in tree. It has no value ( -1 ), so we will move from here...
nNode = 0
nNumBits = HuffmanTree(nNode).nLeafValue
While (nNumBits = -1) 'while we're sitting on a leaf that has no value ( -1 )...
nNode = HuffmanTree(nNode).nBranches(GetBit()) 'traverse tree to next node down the indicated branch...
If (nNode = -1) Then 'not a valid branch - error reading data so bail out...
szGlobalErrMsg &= "GetDiffIndirectValue: Dead-end Forwarder..." & vbCrLf
Return (Integer.MinValue)
End If
nNumBits = HuffmanTree(nNode).nLeafValue 'get the leaf value at this node...
End While

If (nNumBits = 0) Then Return (0)

nValue = GetBit()
bFlip = Not CBool(nValue) 'if first bit is '0', then we need to flip value...
For i = 1 To nNumBits - 1
nValue = GetBit() + (nValue << 1)
Next

If (bFlip) Then nValue -= (1 << nNumBits) - 1

Return (nValue)

End Function

And for the final turn of the twist... with the SD cameras, the previous decoded value used is always from the previous pixel. Not so the DP scheme. Again, this code has it's roots in DCRaw. Having taken the easy option of declaring a couple of public variables:

Public vpred(3), hpred(1) As Integer

before decoding each block of R, G and B, load the respective starting value into an array:

'load vpred array with respective predictor value..
For i = 0 To 3
vpred(i) = PredictorValue
Next i

where PredictorValue is the required R, G or B starting value for the colour you are decoding. The loop for decoding the image values is:

For y = 0 To nImageHeight - 1
For x = 0 To nImageWidth - 1

nValue = GetDiffIndirectValue()
If (nValue = Integer.MinValue) Then
szGlobalErrMsg &= "DecodeSigma0330Image: Decode error..." & vbCrLf
GoTo ohooh
End If

If (x < 2) Then
i = 2 * (y And 1) + (x And 1)
vpred(i) += nValue
hpred(x) = vpred(i)
Else
hpred(x And 1) += nValue
End If

ImageOut(x, y).nR = (hpred(x And 1))

Next x
Next y

this being the particular loop for Red.

Since this code was originally written to decode compressed images from Bayer pattern sensors, this kind of implies that the compression engine used in the True II processor originally started life in a Bayer camera !!

Hopefully I've entered all the above data correctly. My apologies if I've typo'd anything - I don't usually spot spelling mistakes until the precise moment after hitting the Send button.

Page Last Updated: 28th February 2010, Maintained by KeithM.