When a scanned document is slightly rotated anti-clockwise, this causes a problem for OCR because the 2nd half of every line appears before the 1st half of every line. The incorrect order is a disaster for language translations.

The FOSS tool to deskew is unpaper. But it often fails to deskew. There are --pre-rotate and --post-rotate options which you might expect to give some manual control, but strangely you can only rotate by 90°. The unpaper options are not well explained.

What are folks doing in this situation?

I recall from ~20 years ago Windows paperless tools had a deskewing function that was manual. You draw a line under a line of text on the page and that input is used for rotation. It was limited to PDFs but at least it worked. Is there anything like that in the free world?

  • A_norny_mousse@piefed.zip
    link
    fedilink
    English
    arrow-up
    2
    ·
    edit-2
    5 days ago

    You deskew it in GIMP? Sure, you want automation, but…

    How is unpaper “The FOSS tool to deskew” when it “often” fails to do just that?

    The unpaper options are not well explained.

    There’s https://manpages.org/unpaper and https://github.com/unpaper/unpaper/tree/main/doc

    The various --deskew-* options seem well explained to me.


    apt search deskew returns ocrmypdf and pagetools, no idea if that all goes back to unpaper or not.

    • evenwichtOP
      link
      fedilink
      arrow-up
      1
      arrow-down
      1
      ·
      edit-2
      4 days ago

      GIMP

      You deskew it in GIMP? Sure, you want automation, but…

      GIMP is for image processing not document manipulation. When loading a 4-page PDF, GIMP tries a hack of importing each page as a separate layer to a single image. To change pages, in principle that should be a matter of changing layers (page-up and page-down according to the menu, but it had no effect). GIMP automatically presented page-4, which is a bit odd. And I could not change layers.

      Luckily page 4 happened to be the page I needed to deskew. But the rotate tool has no mechanism to fix a line that underlines a line of text followed by a mechanism that rotates as needed to make the line horizontal. We must resort to grabbing and dragging until it looks right. Eyeballing that is much more prone to human error than drawing a straight line under a line of text on the page.

      When exporting the result as PDF, we can specify that each layer in the image become a page in the PDF output. And it seems to work though I did not get the impression that the page I deskewed remained in the A4 geometry that it started with because there was some kind of extra gap between the edges and the frame. Since GIMP is not designed for documents, it would not likely have a concept of A4 paper which would then need to be preserved.

      ImageMagick – perhaps

      The other tool I can think of would be ImageMagick. It’s also not meant for documents but it looks like there is potential for this to be a solution. I ran display $image_of_a_page, selected the rotate transformation, and was able to draw a straight line which then became the baseline for rotation. This may be the answer but I have to fiddle with it more. ImageMagick often fails the principle of least astonishment. E.g. if you feed it some 600dpi images and simply instruct it to montage them, the result is extremely blurry. Makes no sense. The fix is that users must determine the resolution of the input images, do the math, and then explicitly specify the expected output resolution in order to have non-lossy processing. It’s bizarre how ImageMagick arbitrarily picks a fixed default resolution regardless of the input unless a user micro-manages that. So I’m not sure what kind of side-effects ImageMagick might have in this case.

      Unpaper

      How is unpaper “The FOSS tool to deskew” when it “often” fails to do just that?

      Because it’s the only FOSS tool that even attempts to offer a deskewing function for documents (AFAIK). It seems to be the one (and only) tool for this. Everything else is hacks intended for loose images – with functionality called “rotate” because it has no expectation of textual content. (Note that PDF is not an image format, it’s actually just a container of images and other things which can be quite dicey when working with tools not made for PDF manipulation).

      The various --deskew-* options seem well explained to me.

      Glad to hear you understand it! So, here are my questions:

       -dn { left \| top \| right \| bottom },...; --deskew-scan-direction { left \| top \| right \| bottom },...
               Edges from which to scan for rotation. Each edge of a mask can be used to detect the mask's rotation. If multiple edges are specified,  the
               average value will be used, unless the statistical deviation exceeds --deskew-scan-deviation. Use left for scanning from the left edge, top
               for scanning from the top edge, right for scanning from the right edge, bottom for scanning from the bottom.  Multiple  directions  can  be
               separated by commas. (default: left,right)
      

      The opening sentence is ambiguous to readers (who don’t know how the algorithm works). What is the meaningful difference between scanning from the left and scanning from the right? Why would the scan direction make a difference? What is a mask in this context? A mask is generally used when you want to limit a function to operation within (or outside of) a specified area. But that would not seem to make sense here. Why would the default be left,right and not left,down,right,top, for example? Is top and bottom meant for landscape documents that are not rotated upright?

       -ds pixels; --deskew-scan-size pixels
               Size of virtual line for rotation detection. (default: 1500)
      

      What is the “virtual line”? I might guess that it is the width of a line of text, but I can only guess because it is not explained. What are the algorithmic consquences of this value being too long, or too short? When deskew fails to rotate a page, how would I determine what the virtual line size should be?

        -dd ratio; --deskew-scan-depth ratio
               Amount of dark pixels to accumulate until scanning is stopped, relative to scan-bar size. (default: 0.5)
      

      An accumulation is typically measured as a total quantity not a ratio. We can expect input docs to have an all-white left margin which has nearly zero dark pixels. So this dd ratio seems to be incomplete. We might expect there to be a certain concentration of dark pixels before we start counting the vector length. And likely a certain concentration to continue scanning, and a certain concentration to stop scanning, which I suppose would be the same as that of starting the scan. Without knowing what the scan bar size is, how can we determine what this value needs to be? And if we know the scan bar size, how is the ratio calculated? If it’s the qty of black pixels over the qty of whitespace starting from the left edge of the paper, I don’t suppose a ratio of 50% (the default) would generally be reached.

       -dv deviation; --deskew-scan-deviation deviation
               Maximum statistical deviation allowed among the results from detected edges. No rotation if exceeded. (default: 1.0)
      

      What is a “detected edge”? Is that a line of text or an edge of the paper? I can’t tell from the man page how many edges there might be as we don’t know how the algorithm works. If It scans with a --deskew-scan-range of 5° and a --deskew-scan-step of 0.1, this seems to imply that it scans 100 times to find a single edge. Or is it finding multiple candidate edges from the same starting point and counting each of those as an edge that would then be compared for this deviation? From there, it seems unclear how many starting points it would attempt. I see no parameter for the distance between each starting point. If each starting point yields an edge, are all the edges for the whole page being considered for this deviation value? Or is it a deviation of candidate edges for just one line of text? If it’s the latter, how many lines of text are being processed?

      apt search deskew returns ocrmypdf and pagetools, no idea if that all goes back to unpaper or not.

      No results on oldstable but maybe Trixie has more. I often use ocrmypdf after deskewing because I thought it could not deskew. But your finding prompted me to look at the man page for ocrmypdf which reveals a --deskew option. So I will have to experiment with that.