What we learned measuring our Sudoku photo reader

Updated November 5, 2026

What counts as read right

A puzzle is read right only if all 81 squares are right: one wrong digit makes a different puzzle. So we count whole puzzles, not squares. The two can tell very different stories, as this page shows: on printed photos the reader now gets 98.9% of the squares right, yet the share of whole puzzles barely moved, because most wrong squares sit in a few hard pictures.

We also count the most dangerous mistake: a puzzle read wrong that still has exactly one answer. A real Sudoku has one answer, so a reading with none or with many is caught at once. A wrong puzzle with one answer looks fine, which is why the solver marks every square it was unsure of and asks you to compare them with your puzzle before solving.

With 200 pictures, a result is only so precise. For the main results we give a 95% range (a Wilson interval): 89.5% of 200 pictures means the true rate is most likely between 84.5% and 93.0%.

The pictures we tested on

We started with two open datasets. Lexski's Sudoku Image Recognition dataset (CC0) has 1,400 pictures, most of them screenshots and photos people shared online: apps in every style, dark themes, pencil marks, puzzles half solved on paper. The dataset of Baptiste Wicht and Jean Hennebert (CC BY 4.0) has newspaper Sudoku photographed with phones, some of them finished by hand in pen.

Before changing anything, we set three test sets aside: 200 Lexski pictures, 40 newspaper photos and 40 newspaper puzzles finished by hand. We did not train on them or look at them while working. Every decision was made on 200 other Lexski pictures, the tuning set, and the test sets were read once, at the end.

Where it started

The first version, from 31 October, read 101 of the 200 Lexski pictures exactly, 34 of the 40 newspaper photos and none of the 40 finished puzzles. Most of its trouble came before the digits. It took the biggest square-ish shape of ink as the board, and on a screenshot that shape often takes in a header or the number pad, while the thin grey lines of a dark theme are missed altogether. On the tuning set it put the grid in the right place only 118 times out of 200.

What we changed, step by step

Each row is one change, and the number is how many of the 200 tuning pictures were then read exactly. The last three rows are the second round.

Most of the gain came from three changes. The grid is now found by its lines: every guess at the board is moved onto ten evenly spaced lines in each direction and judged side by side, and a small network that maps where a board is gets a vote. Each square is cut out whole, so a pencil mark stays small in its corner and a light digit on a dark tile looks like ink on paper. And the digit networks learned from real squares cut from Lexski's training pictures, with pencil marks as a class of their own. Three small networks, trained apart, now read each square and their answers are averaged, because they make their mistakes on different squares.

A finished puzzle gets extra help. When a board holds 79 digits or more, the reader looks for the likeliest complete grid in which every row, column and box holds 1 to 9. Read square by square, only 7 of the 40 handwritten test puzzles came out exactly; read as finished Sudoku, all 40 did.

The second round added three smaller changes. A full-size phone photo is halved step by step before it is shrunk, so thin lines and pencil strokes are averaged in rather than skipped. A square whose ink is both much shorter and much fainter than the board's sure digits is treated as a likely pencil mark. And the turn of a sideways photo is judged by all three networks instead of one.

What did not help

We measured these on the tuning set too, and left them out:

• A bigger digit network: 59,000 weights instead of 42,000 read one more tuning picture and ran 3.8 times slower in the browser.
• 3,000 generated app screenshots with pencil marks and dark themes: the same score with or without them.
• Reading colour instead of grey, so that a red pen is as dark as black ink: two more tuning pictures, but handwriting was read worse.
• Distorting the training digits to look more like handwriting: handwriting was read worse.
• Letting the one-answer rule change more squares: more puzzles read right, but also more of the dangerous wrong puzzles with one answer.
• Retraining the three networks on fresh data, or using four or six of them: within the noise, so the first ones stayed.
• Treating the edge of the picture as the edge of the board, for boards cut off by the frame: as many pictures lost as gained.

Results on the first test sets

Puzzles read exactly in each test set, by the first version and by the reader as it is now:

On the Lexski pictures the reader went from 50.5% to 89.5% (84.5% to 93.0%), and puzzles read wrong yet with one answer went from 7 to 4. In three of those four, the one extra digit is the digit that belongs there, so the puzzle still leads to the right solution; the fourth is a clearly printed 1 that the dataset leaves empty. 177 of the 179 came at the end of the first round. The last two came in the second round, after we had studied what failed, so this set is no longer untouched.

Forty out of forty sounds perfect, but 40 pictures can only show that the true rate is above about 91%. And a check we should have run first: four of the 40 newspaper test photos are also in that dataset's training set, two of them pixel for pixel. Without them, 36 of 36.

A new test: 200 photos no version had seen

By then the first test sets had been read and their failures studied, so they could no longer tell us how the reader does on pictures it has never met. We searched Hugging Face, Kaggle, GitHub, Zenodo, Figshare and AIcrowd for more labelled Sudoku photos. Most sets were generated boards, single digits or copies of Wicht's photos. Three GitHub collections had real labelled photos, all of printed puzzles: 100 photos of an open puzzle book (NatalijaPetric/SudokuDataset), 80 photos mostly of newspapers (mineshpatel1/sudoku-solver, after dropping 49 that are Wicht's own photos and one copy), and 20 printed sheets from a University of Bucharest computer vision course project (danielsociu/SudokuRecognition).

None of the three states a licence, so we used them only to measure: never to train on, and they are not shown here. The list of pictures was fixed and saved before any change, chosen by rule from file hashes and labels, and the pictures were not opened until the second round's last change. Then each version of the reader read them once. The table counts the puzzles read exactly.

On printed puzzles the reader had never seen, all the work of both rounds changed little. The first version read 82.0% of them exactly, the current one 83.5% (77.7% to 88.0%): it gained 14 puzzles and lost 11. What did improve is the squares: 98.9% are read right instead of 94.2%, and the grid is found in every photo instead of 195 of 200. Puzzles read wrong yet with one answer: 5 then, 6 now.

The losses have a clear cause. To read pencil marks and dark themes, each square is now cut out whole and faint ink counts. On thin newspaper the reverse page shows through, faint and mirrored, and the new reader reads those shadows as digits; the first version looked only at the biggest shape of ink in a square and never saw them. On curled book pages, the board's outer line or a box line can fall inside a square and be read as a 1.

Two of the misses are mistakes in the labels: the photo shows a 3 and a 9 where the label says 9 and 8, and a 4 where it says 2. Counted right, the current reader reads 84.5%.

What still goes wrong

Of the 200 new photos, 33 were not read exactly. 22 are pages of the open puzzle book: near the spine the page curls more than the reader follows, so a line ends up inside a square and is read as a 1, or a corner square is cut across a line. 7 are newspapers with the reverse page showing through. The rest are a photo of a screen with moiré, a board edge read as a 1, and the two label mistakes.

Of the 200 Lexski pictures, 21 were not read exactly: pencil marks read as digits, including erased pencil that left faint digits behind (7); one extra digit, mostly a lone pencil mark read as an entry (5); the grid in the wrong place or not found, on app themes with no lines between squares or a grid drawn by hand (5); and unusual tiles and colours (4).

Why not 97%, and what comes next

Everything runs in your browser: a model of 224 KB and about 850 lines of JavaScript, a median of about 300 milliseconds per photo on a laptop, and about 1.2 seconds with the processor slowed four times, as on a slow phone. A bigger network did not help within that budget. What is left is a long tail: curled pages, show-through, pencil marks and unusual app styles. Every public labelled set we found is now used or seen, so the next round needs photos we take and label ourselves, with the faint ink and curled page problems fixed and measured on them.

For the best scan

• Hold the phone straight above the puzzle, with the whole grid in the frame.
• Flatten a book or a newspaper: curled pages cause most misses on printed puzzles.
• On thin newspaper, put a plain sheet of paper behind the page so the other side does not show through.
• Scan before you write pencil marks in the puzzle.
• Before solving, compare the squares the solver marks with your puzzle.

Data and sources

• Lexski, Sudoku Image Recognition (Hugging Face), CC0 1.0
• Wicht, B. and Hennebert, J., Sudoku dataset (GitHub), CC BY 4.0; "Camera-based Sudoku recognition with deep belief network", SoCPaR 2014
• NatalijaPetric, SudokuDataset (GitHub), no licence stated, used only to measure
• mineshpatel1, sudoku-solver (GitHub), no licence stated, used only to measure
• danielsociu, SudokuRecognition (GitHub), University of Bucharest course project, no licence stated, used only to measure
• How the solver reads a photo, step by step

Try the solver →