Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Announcements and Chat > General Discussion

Reply
 
Thread Tools Search this Thread Display Modes
Old 26th February 2016, 08:44   #1  |  Link
rudyb
Registered User
 
Join Date: Aug 2014
Posts: 29
LZW compression

Hi,
I have a question about LZW compression.

I know that mostly if people apply LZW compression it will be applied on uncompressed TIFF image. And I believe it makes sense to do it on TIFF format, and that is because for a TIFF image (with 3 RGB channel per pixel), there is a high chance that neighboring pixels have similar channel information. For example, it is highly likely that two pixels next to each other are very close in colors, so they will have similar red channels, and similar blue and green channels. So, LZW compression will work well.

My question is whether it makes sense to apply LZW on RAW data (before Bayer interpolation)?
Because for a Bayer pattern, there is only one channel information per pixel. So, adjacent pixels will have information from different channels. For example, if a pixel is green, then the next pixel will be blue, and the next pixel will be green again (prior to de-mosaicing).....
So, I am thinking there would be less correlation between neighboring values, which, I am thinking, this will result in LZW to be less efficient !!!

So, do people use LZW on Raw data? Does anyone know if applying LZW in RAW format will be less effective than applying it in TIFF format?
rudyb is offline   Reply With Quote
Old 26th February 2016, 09:44   #2  |  Link
wonkey_monkey
Formerly davidh*****
 
wonkey_monkey's Avatar
 
Join Date: Jan 2004
Posts: 1,822
Quote:
I know that mostly if people apply LZW compression it will be applied on uncompressed TIFF image.
That may be because TIFF is one of the few containers (along with GIF) where LZW has near-universal support.

Strictly speaking the compression is applied to raw image data, and results in a TIFF file.

As for Bayer data, Canon (and probably several others) uses a form of lossless JPEG compression on their raw files. At a guess, they presumably consolidate all the green bits, all the red bits, and all the blue bits somehow rather than trying to treat them all together as related data.
__________________
My AviSynth filters / I'm the Doctor
wonkey_monkey is offline   Reply With Quote
Old 26th February 2016, 11:02   #3  |  Link
filler56789
SuperVirus
 
filler56789's Avatar
 
Join Date: Jun 2012
Location: Antarctic Japan
Posts: 941
Quote:
Originally Posted by rudyb View Post
.........

So, do people use LZW on Raw data? Does anyone know if applying LZW in RAW format will be less effective than applying it in TIFF format?
TIFF is just a container. Besides LZW and ZIP (Deflate), it also supports JPG compression.
filler56789 is offline   Reply With Quote
Old 26th February 2016, 16:37   #4  |  Link
rudyb
Registered User
 
Join Date: Aug 2014
Posts: 29
I am not sure if I agree that RAW and TIFF are similar. They are different. (http://photo.net/learn/raw/)
For example, RAW data there is only one channel per pixel, which is the bayer pattern. Because it hasn't yet been interpolated to have three channels (RGB) per pixel.
Where as IFF has already passed the interpolation stage, and it is already de-mosaiced, and each pixel has three channels (RGB).

Another explanation is the following statement from (http://www.poundslabs.com/documents/digitaldemys.pdf):

"Even though the TIFF file only retains 8 bits per channel of information, it will
take up twice the storage space than a RAW file because it has three 8 bit
color channels versus one 12 bit RAW channel."

So, applying LZW on TIFF is different thatn applying LZW on RAW. And I was wondering if we can apply LZW on RAW images? And is it worse or better than applying it on TIFF?
rudyb is offline   Reply With Quote
Old 26th February 2016, 17:37   #5  |  Link
filler56789
SuperVirus
 
filler56789's Avatar
 
Join Date: Jun 2012
Location: Antarctic Japan
Posts: 941
^ Thanks for the links, I too was thinking that

RAW = 'uncompressed 24-bit RGB'.

But again: TIFF is not the same as 'uncompressed 24-bit RGB'.

Quote:
And I was wondering if we can apply LZW on RAW images?
Yes.

Quote:
And is it worse or better than applying it on TIFF?
I really don't know Assuming you have a RAW file and its equivalent 24-bit RGB version, you could apply the old UNIX program 'compress' on them, and then compare the resulting filesizes.

Last edited by filler56789; 26th February 2016 at 22:13.
filler56789 is offline   Reply With Quote
Old 26th February 2016, 20:30   #6  |  Link
pandy
Registered User
 
Join Date: Mar 2006
Posts: 1,044
You can use old concept: bitplane - RGB= 3 channels, 8 bits in channel so in total 24 bitplanes - they may be correlated together and this should be quite nice to do multi thread nowadays.
And usually they apply first RLL and later LZW.
pandy is offline   Reply With Quote
Old 27th February 2016, 19:08   #7  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,416
It makes sense to use LZW, or your favorite lossless compression, which is why they all do. I've never seen a RAW that wasn't compressed, usually by huffman or ljpeg (which is just predictor + huffman, really), although I'm sure there are some older formats that don't. DNG can use deflate... and lossy jpeg, but that's just the exact same blocky 8-bit crap that a regular jpeg is.

A better question is whether it's worth using JPEG-XR, H.264, H.265/BPG, or some other high-bit still image format, whether lossless or lossy. Even lossless, the predictions, transforms, and use of CABAC makes for a significant gain over current RAW formats, but there's a potential for much more minimally lossy reduction while retaining deep-color fidelity. Right now it's mostly a matter of them not caring, since capacity keeps going up faster than they can make larger sensors.
__________________
There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order.
foxyshadis is offline   Reply With Quote
Old 5th March 2016, 06:23   #8  |  Link
hcrs
Linux & embedded dude
 
Join Date: Feb 2016
Posts: 3
Quote:
Originally Posted by rudyb View Post
So, do people use LZW on Raw data? Does anyone know if applying LZW in RAW format will be less effective than applying it in TIFF format?
In principle it could work. Though to get better results, you can try the following tricks...
1) Split channels and compress them separately. I.e. RED channel is more similar to RED, GREEN is more similar to GREEN, and BLUE is more similar to BLUE. So it could make sense to compress each channel separately.

2) This said, bayer data format haves only half of green pixels. Conversion into "real" RGB does interpolarion, faking half of missing pixels to something approximated. So under equal condirions you have only half of green pixel data to chew on. Which is nice reduction of amount of work, right from the very start, hence smaller file is to be expected. In "real" RGB you have twice as much green pixels to chew on, half of them are fake/interpolated but it is not to be taken as granted you'll be able to detect this redundancy and compress green channel by factor of 2. Downside is: to view this file you'll still would need to "debayer" it to display it on RGB disaplays, etc.

3) There is another fancy thing to try for LZW-like algo. Simplest is "delta transform". Real-world images from camera rarely have exactly matching pixels due to noise and nature of picture. This means LZ-based algos can't compress them well. However, storing only difference from previous pixel could make your data far more compressible, since many real world pictures change slowly and hence "deltas" are going to be quite similar, and LZ-based algos can chew on it far better. If you take a look on e.g. well known "Lena.bmp", most LZ compressors can compress it by 2-3% on their own, but if you delta-compress it, they can chop off about 20% of size instead. There're more tricks to try, take a look on PNG filters to get more ideas. Hih-compression PNG optimizers are trying various pre-filters and choosing one which works best on particular picture. Even slower but more efficient trick is to try various zlib modes vs various pre-filters, choosing best of the best. Though it makes compression quite slow - there're many combos to try.

4) It could be better idea to use "LZ+Entropy coding" schemes, like zlib, zstd, lzma and somesuch, depending on what compression and decompression speeds and ratios you target. Since data nature does not assumes exact matches, pure LZ-based things aren't best for real-world graphics. But LZ+Entropy would yeld you at least some compression due to entropy coding phase. Entropy coding will notice values distribution isn't uniform most of time on most pictures. Hehce it could gain some margins. In fact some picture-oriented compressors omitting LZ part at all and go for huffman-only. But it would perform poorly on "computer-generated" pictures with large areas of identical color, where LZ could kick in.

5) There is usually inherent correlation between values of R, G and B channels. I guess it could be possible to "predict" pixels using neighbors and emit only "prediction error" as actual description of next pixels. Probably best idea could be to use both inter-channel correlations and the fact real world pictures are quite slow to change.

P.s. I'm not graphics compressing expert, visiting forums like "encode.ru" could be much better idea, experts here could probably invent some fancy way to compress bayer kind of data in better way.

@foxyshadis, RAWs are usually being used for ADVANCED PHOTO PROCESSING. That's where LOSSLESS MATTERS. Even small value errors could easily develop into crappy artifacts after image would pass like 10 different filters in processing pipeline of RAW development software. RAWs are supposed to keep source data unchanged, storing as much data as one could yeld from matrix right as you've got it from matrix. Then you do all post-processing yourself, instead of camera's DSP. Since RAW processing software isn't pressed by real time, you can do much better processing, correcting far more issues and yelding much better visual result.

That's where EXACT pixel values MATTER. Lossy compression of RAW camera images is a pointless thing. You'll get better ratios, sure. But you'll lose whole point ot RAW, so one can just use JPEG, WebP or somesuch. But lossy images are very bad for advanced processing since lossy errors tend to confuse RAW processing algos and develop into nasty looking artifacts, killing ability to do decent post-processing. So it seems this guy attempts to invent what I can descrive as "FLAC for bayer pixels". You offer him mp3. Or vorbis, or aac. But he is likely up for "studio-grade processing", Where lossy hurts. As ready example, I can denoise RAW images in a "visually lossless" ways much better than it would be possible for even highest quality JPEG. Even "dabayer" is semi-lossy and there're more than one way to do it, so it usually configurable in RAW development software how to do it.
hcrs is offline   Reply With Quote
Reply

Tags
compression, lossless compression, lzw, video camera

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 17:50.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.