It is hard to avoid references to artificial intelligence in articles and on social media, and easy to assume it is a new and emerging technology. You would be forgiven for thinking so, but AI has been around for a long time — dating back to the 1950s.
Optical character recognition (OCR) is another branch of the same field, commercialised in the 1970s by Ray Kurzweil, who made it possible to convert printed text in almost any font into machine-readable content. The same principles apply when a scanned document is turned into an image with printed text in it.
What interests me is text in natural scenes. That is best explained with a picture.
An example of text in a natural scene
Consider this photo, taken on one of my holiday trips:
If you are from Victoria, you will almost certainly recognise the landmark — the entrance to the Great Ocean Road, a well-known tourist stop with platforms where you can pose with the grand archway behind you.
Below is the same photo after analysis, with the detected text outlined in red. The text reads: GREAT OCEAN ROAD.
If your reaction is “that is just OCR applied to an everyday photo”, you are right. That is precisely what text in natural scenes means.
Why it is harder than it looks
Many researchers consider the classical OCR problem solved. Text in natural scenes is nowhere near solved, because the real world is far less cooperative than a scanned page:
- The text is unstructured. You have to interpret it in the context of where it sits in the scene. A photo of a person might contain a slogan on a T-shirt, a brand name on a cap, and a tattoo reading “BELIEVE” on a forearm. Deciphering the characters is only half the job; understanding what each piece of text is matters just as much.
- The text is rarely tidy. It is skewed, non-horizontal, partially hidden behind other objects, and set in wildly varying fonts.
- Visibility works against you. Poor contrast between text and background, glare, reflections and natural lighting all interfere.
- The image itself may be imperfect — blur from camera limitations, motion, or other aberrations.
How those challenges are being tackled
There are real difficulties remaining, but there is also steady success using AI to:
- Localise the text — find the regions of the scene where text is present.
- Pass those localised regions through further processing to improve OCR accuracy.
- Segment the image to classify where the text sits, which supplies the context needed to turn raw characters into something meaningful.
Applied to the example above, the result can be structured as something like: a signboard reading “GREAT OCEAN ROAD”, overhanging a road, with trees and mountains behind.
Deeper insight, or privacy concern?
This is the question I keep coming back to, and it genuinely cuts both ways.
Photos and videos may contain text that gives further clues and context under deeper analysis. Often it sits in the background, or somewhere not remotely obvious. If you shared something you would not have shared had you known what it revealed, that is a privacy concern.
On the other hand, that same insight is a genuine accessibility gain. Someone who is vision impaired or legally blind cannot read the information in an image directly. The extra clues and context can help enormously.
Both statements are true at once, which is what makes this interesting rather than simple. The same capability that describes a photo to someone who cannot see it will also tell an analyst which road you were standing on.
You can see what text a photo of yours gives up by running it through snapWONDERS.