You know, I’ve really looked hard at what’s out there and haven’t been able to find anything else that’s totally free/open, that runs well on CPU, and which has better quality output than Tesseract. I found a couple Chinese projects but had trouble getting them to work and the documentation wasn’t great. If you have any leads on others to try I’d love to hear about them.
One of the benefits of this project is that it doesn’t seem to matter that much that there are mistakes in the OCR output as long as you’re dealing with words, where the meaning would be clear to a smart human trying to make sense of it and knowing that there are probable OCR errors. For numbers it’s another story, though.
> You know, I’ve really looked hard at what’s out there and haven’t been able to find anything else that’s totally free/open, that runs well on CPU, and which has better quality output than Tesseract. I found a couple Chinese projects but had trouble getting them to work and the documentation wasn’t great. If you have any leads on others to try I’d love to hear about them.
I did more or less the same, trying to solve the same problem. I ended up biting the bullet and using Amazon Textract. The OCR is much better than Tesseract, and the layout tool is quite reliable to get linear text out of 2-columns documents (which is critical for my use case).
I would be very happy to find something as reliable that would work on a workstation without relying on anyone’s cloud.
Yes, I imagine it's using the same OCR model as the iPhone, which is really incredibly good. In fact, it's so good that I made a little app for fun just to be able to use it for OCRing whole PDF books:
Interesting! I’ll give it a try, I have a couple of large books to OCR (to be honest, the name in all caps with underscores is not really encouraging).
From your experience, how does the OCR engine work with multiple-columns documents?
The iOS app would likely not handle two-column text very well. I really made the iOS app on a lark for personal use, the whole thing took like 2 hours, and I'd never even made a Swift or iOS app before. It actually took longer to submit it to the App Store than it did to create it from scratch, because all the hard stuff in the app uses built-in iOS APIs for file loading, PDF reading, screenshot extraction, OCR, NLP for sentence splitting, and sharing the output.
I think the project I submitted here would do that better, particularly if you revised the first prompt to include an instruction about handling two column text (like "Attempt to determine if the extracted text actually came from two columns of original text; if so, reformat accordingly.")
The beauty of this kind of prompt engineering code is that you can literally change how the program works just by editing the text in the prompt templates!
They're just a new Swift-only interface to the same underlying behaviors, no apparent improvement. I was hoping for more given the visionOS launch but alas
What I'm trying now is combining ML Kit v2 with Live Text - Apple's for the accurate paragraphs of text, and then custom indexing that against the ML Kit v2 output to add bounding rects and guessing corrections for missing/misidentified parts from ML Kit (using it only for bounding rects and expecting it will make mistakes on the text recognition)
I also investigated private APIs for extracting rects from Live Text. It looks possible, the APIs are there (it has methods or properties which give bounding rects as is obviously required for Live Text functionality), but I can't wrap my head around accessing them yet.
I feel like text detection is much better covered by the various ML models discussed elsewhere in the comments. Maybe you can combine those with Live Text. I found Tesseract pretty ok for text detection as well but I don’t know if any of the models are good for vertical text.
Huh, I tried with the version from pip (instead of my package manager) and it completes in 22s. Output on the only page I tested is considerably worse than tesseract, particularly with punctuation. The paragraph detection seemed to not work at all, rendering the entire thing on a single line.
Even worse for my uses, Tesseract had two mistakes on this page (part of why I picked it), and neither of them were correctly read by EasyOCR.
Partial list of mistakes:
1. Missed several full-stops at the end of sentences
2. Rendered two full-stops as colons
3. Rendered two commas as semicolons
4. Misrendered every single em-dash in various ways (e.g. "\_~")
5. Missed 4 double-quotes
6. Missed 3 apostrophes, including rendering "I'll" as "Il"
7. All 5 exclamation points were rendered as a lowercase-ell ("l"). Tesseract got 4 correct and missed one.
Yes, that's one of the ones I tried. It seemed to be more designed for things like receipts and menus rather than books. But in any case, I found it hard to set up and use (and it's likely slow on the CPU compared to Tesseract, which despite its low accuracy, is at least very fast on CPU).
One of the benefits of this project is that it doesn’t seem to matter that much that there are mistakes in the OCR output as long as you’re dealing with words, where the meaning would be clear to a smart human trying to make sense of it and knowing that there are probable OCR errors. For numbers it’s another story, though.