How can I read 10-bit Raw image? which contain RGB-IR data

Question

I want to know how extract the rgb image from my 10-bit raw (it has rgb-ir imagedata) data?

How can I read in Python or MATLAB?

The camera resolution at the time of capture were 1280x720: indoor photo Image for download outdoor photo Image 2 for download

camera Model: e-CAM40_CUMI4682_MOD

Thanks a lot

Answer 1

I used the following image processing stages:

• Bayer mosaic color channel separation.
• Linear stretching each color channel.
• Simple white balance.
• Replace IR color channel with green (convert image to standard Bayer format).
• Restore Bayer mosaic.
• Simple gamma correction.
• Demosaic

Instead of processing the IR color channel, I replaced it with green color channel.

According to the RGB image you added, I found the CFA order.
The CFA (color filter array) order is:

B | G
-- --
IR| R

The following Matlab code process the image to RGB:

srcN = 1280;
srcM = 720;

f = fopen('image_raw.raw', 'r');

%Read as transposed matrix dimensions, and transpose the matrix.
%The reason for that, is that Matlab memory oreder is column major, and
%raw image is stored in row major (like C arrays).
I = fread(f, [srcN, srcM], 'uint16');
fclose(f);
I = I';

%Convert from range [0, 1023] range [0, 1] (working in double image format).
I = I/(2^10-1);

%Bayer mosaic color channel separation
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Assume input format is GBRG Bayer mosaic format.
%Separate to color components.
B = I(1:2:end, 1:2:end);
G = I(1:2:end, 2:2:end);
IR = I(2:2:end, 1:2:end);
R = I(2:2:end, 2:2:end);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%Linear stretching each color channel.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Linear streatch blue color channel.
B = imadjust(B, stretchlim(B, [0.02 0.98]),[]);

%Linear streatch green channel.
G = imadjust(G, stretchlim(G, [0.02 0.98]),[]);

%Linear streatch red color channel.
R = imadjust(R, stretchlim(R, [0.02 0.98]),[]);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%Simple white balance
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Median or R, G and B.
rgb_med = [median(R(:)), median(G(:)), median(B(:))];
rgb_scale = max(rgb_med)./rgb_med;

%Scale each color channel, to have the same median.
R = R*rgb_scale(1);
G = G*rgb_scale(2);
B = B*rgb_scale(3);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%Restore Bayer mosaic.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Insert streached color channnels back into I.
I(1:2:end, 1:2:end) = B;
I(1:2:end, 2:2:end) = G;
%I(2:2:end, 1:2:end) = G; %Replace IR with Green.
I(2:2:end, 2:2:end) = R;
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%Replace IR with green - resize green to full size of image first.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
T = imresize(G, [srcM, srcN]); %T - temporary green, size 1280x720
I(2:2:end, 1:2:end) = T(2:2:end, 1:2:end); %Replace IR with Green.
I = max(min(I, 1), 0); %Limit I to range [0, 1].
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%Simple gamma correction
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
gamma = 0.45;
I = I.^gamma;
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%Demosaic
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Convert to uint8 (range [0, 255]).
I = uint8(round(I*255));
RGB = demosaic(I, 'bggr');
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

imshow(RGB);

Result:

Now the colors are normal...

---

Outdoor image processing :

Applying "indoor" processing on the outdoor image, gets the following result:

White trees is a sign for Near-IR spectrum penetration to R, G and B pixels (not only to IR pixels).
The chlorophyll of the vegetation has high reflection in the Near-IR spectrum. See: http://missionscience.nasa.gov/ems/08_nearinfraredwaves.html‌​, and search it on Google.
Subtracting IR from Red, Green and Blue color channels is required.

---

I used the following image processing stages:

• Bayer mosaic color channel separation.
• Subtract IR "surplus" from Red, Green and Blue color channels.
• Linear stretching each color channel.
• Simple white balance.
• Restore Bayer mosaic.
• Simple gamma correction.
• Demosaic.
• Resize RGB image to lower resolution.

The following Matlab code process the outdoor image to RGB:

srcN = 1280;
srcM = 720;

f = fopen('ir_6.raw', 'r');

%Read as transposed matrix dimensions, and transpose the matrix.
%The reason for that, is that Matlab memory oreder is column major, and
%raw image is stored in row major (like C arrays).
I = fread(f, [srcN, srcM], 'uint16');
fclose(f);
I = I';

%Convert from range [0, 1023] range [0, 1] (working in double image format).
I = I/(2^10-1);

%Bayer mosaic color channel separation
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Assume input format is GBRG Bayer mosaic format.
%Separate to color components.
B = I(1:2:end, 1:2:end);
G = I(1:2:end, 2:2:end);
IR = I(2:2:end, 1:2:end);
R = I(2:2:end, 2:2:end);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%Subtract IR "surplus" from R, G and B.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%The coefficients were tuned by trial and error...
ir_r = 1.3;  % 130% of IR radiation is absorbed by red pixels??? 
ir_g = 0.35; % 35% of IR radiation is absorbed by green pixels.
ir_b = 0.3;  % 30% of IR radiation is absorbed by blue pixels.

IR = imresize(IR, size(I)); %Resize IR to the size of I.
IR = max(min(IR, 1), 0); %Limit IR to range [0, 1] (because imresize values slightly outside the range of input).

R = R - IR(2:2:end, 2:2:end)*ir_r; %Subtract IR for R (IR scale coefficient is ir_r).
G = G - IR(1:2:end, 2:2:end)*ir_g; %Subtract IR for G (IR scale coefficient is ir_g).
B = B - IR(1:2:end, 1:2:end)*ir_b; %Subtract IR for B (IR scale coefficient is ir_b).

R = max(min(R, 1), 0); %Limit IR to range [0, 1] 
G = max(min(G, 1), 0);
B = max(min(B, 1), 0);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%Linear stretching each color channel.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Linear streatch blue color channel.
B = imadjust(B, stretchlim(B, [0.02 0.98]),[]);

%Linear streatch green channel.
G = imadjust(G, stretchlim(G, [0.02 0.98]),[]);

%Linear streatch red color channel.
R = imadjust(R, stretchlim(R, [0.02 0.98]),[]);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%Simple white balance
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Median or R, G and B.
rgb_med = [median(R(:)), median(G(:)), median(B(:))];
rgb_scale = max(rgb_med)./rgb_med;

%Scale each color channel, to have the same median.
R = R*rgb_scale(1);
G = G*rgb_scale(2);
B = B*rgb_scale(3);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%Restore Bayer mosaic.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Insert streached color channnels back into I.
I(1:2:end, 1:2:end) = B;
I(1:2:end, 2:2:end) = G;
I(2:2:end, 2:2:end) = R;
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%Replace IR with green - resize green to full size of image first.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
T = imresize(G, [srcM, srcN]); %T - temporary green, size 1280x720
I(2:2:end, 1:2:end) = T(2:2:end, 1:2:end); %Replace IR with Green.
I = max(min(I, 1), 0); %Limit I to range [0, 1].
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%Simple gamma correction
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
gamma = 0.45;
I = I.^gamma;
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%Demosaic
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Convert to uint8 (range [0, 255]).
I = uint8(round(I*255));
RGB = demosaic(I, 'bggr');
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

RGB = imresize(RGB, size(I)/2); %Shrink size of RGB image for reducing demosaic artifacts.

imshow(RGB);

Result is not so good, but it demonstrates the concept that IR channel can be subtracted from the Red Green and Blue channels.
There is still work to be done...
Result image:

Reason for the "false color" green patches:
Saturated pixels in the Red color channel (saturated in the raw input), are not handled properly.
Problem can by solved by reducing exposure (taking the shot with lower exposure time).