What amount of skew distortion is tolerable (there should be animated gif):
Left picture is sequence shot while knocking the camera to get some rolling shutter skew effect, most noticeable on frame 3 (bottom part skewed left).
Right picture is the attempt to align frames as much as possible, eliminating both shake and skew. The analysis by Deshaker failed - I matched frames manually.
Btw the filter was setup to output result in original (distorted, fisheye) space, to ease comparison.
Not perfect but this result convinced me of two things:
1) it is very hard to ruin material while camera is handheld - the skew is quite soft over all frames I have inspected - I never got a frame where camera is changing direction (the shots are 48 fps).
2) the sufficient de-skew algorithm is very simple and fast (the part which does pixel projection, analysis is whole different story).