---
title: "How a Sudoku Photo Solver Works in a Browser | Sudoku Flow"
description: "A Sudoku photo reader that runs in the browser: shadows, finding the grid, a 57 KB network and the one-answer rule, tested on real newspaper photos."
url: https://sudokuflow.com/tools/sudoku-solver/how-it-works/
language: en
last_modified: 2026-10-31
---

# How the Sudoku solver reads a photo, with no server

Our [Sudoku solver](https://sudokuflow.com/tools/sudoku-solver/) can read a puzzle from a photo. Everything happens on your phone or computer: the photo is never uploaded, and no server does any of the work. This page walks through each step, shows the pictures each step makes, and says how often it reads a real newspaper photo right.

[Try the solver →](https://sudokuflow.com/tools/sudoku-solver/)

## The rule: no server

A photo of a puzzle on your kitchen table is nobody's business but yours, so we decided early that it would never leave the device. That also means there is nothing to scale and nothing to pay for per photo: a thousand people scanning at once cost the same as one. The price is size and speed. Everything the browser needs has to be small enough to download and fast enough to run on a phone.

The reader is two files. [scan.js](https://sudokuflow.com/assets/scan.js) finds the grid and cuts it up: about 400 lines, 20 KB (7 KB compressed), served unminified so you can read it. A digit network of 57 KB (38 KB compressed) reads the squares, and it is only downloaded the first time you scan. The solving is done by the hint engine of our app, written in Dart and compiled to JavaScript (299 KB, 90 KB compressed): the same code that gives hints in the app.

## 1. Find the ink, not the shadows

The first job is to tell ink from paper. The usual trick compares each pixel with the average of its neighbourhood, and it works until a shadow falls across the page: along the shadow's edge, the dark side looks darker than its surroundings, and a line of false ink appears right through the grid.

Instead, we estimate the paper's own brightness at every pixel with a grey closing: take the brightest value in a small window around each pixel, then the darkest of those values in the same window. Lines and digits are narrower than the window, so they vanish from the estimate. A shadow is wider, so its edge stays exactly where it is. Ink is then whatever is clearly darker than the paper it sits on. Below, we added a hard shadow to a newspaper photo: it is in the paper estimate, and not in the ink.

![A newspaper Sudoku with a hard diagonal shadow, the paper light estimated from it with the shadow in it, and the ink found, with no trace of the shadow](https://sudokuflow.com/assets/tech/scan-ink.jpg)

*Left to right: the photo with a shadow we added, the paper light our code estimates, and the ink it finds. Photo: Sudoku dataset by Baptiste Wicht and Jean Hennebert, CC BY 4.0, shadow added.*

## 2. Find the grid

A Sudoku grid is one connected shape: its outline and all its lines touch. So every large, roughly square ink shape that is mostly empty inside is a candidate. Its four corners are the pixels with the smallest and largest values of x+y and x-y, and those four points give a homography: the projective transform that maps the slanted grid in the photo onto a flat square.

Our first version took the biggest candidate. On 400 test photos, 251 lost more than half of their digits, because the biggest candidate was often the edge of the page. Now every candidate is straightened and scored: how much of the eight inner lines in each direction is ink, at the places where a 9 by 9 grid puts them. A real grid scores close to 1, the edge of a page close to 0. The best score wins, and under 0.35 we say there is no grid. When the best is weak, the picture is also tried with its colours inverted, which is how a dark-mode screenshot gets read.

![A newspaper page with three outlines: the frame around the puzzle with a score of 0.34, another puzzle cut off at the top with 0.20, and the puzzle's grid itself in green with 0.98](https://sudokuflow.com/assets/tech/scan-grid.jpg)

*Three candidates on one page: the frame around the puzzle scores 0.34, another puzzle cut off at the top 0.20, the grid itself 0.98. Photo: Wicht and Hennebert, CC BY 4.0, outlines and scores drawn by our code.*

## 3. Straighten it and cut out the squares

The winning grid is straightened onto a board of 324 by 324 pixels, 36 to a square, and the ink step runs again on it. In each square we keep the biggest ink shape near the middle, as long as it is tall enough to be a digit (pencil notes are too small) and does not fill the whole square (a leftover piece of grid line). The shape is scaled to fit a 20 by 20 box and centred in 28 by 28, the way the classic MNIST digits are. An empty square has no shape and is not read at all.

![The grid straightened onto a square board, the squares cut into small white-on-black digit pictures, and the digits read, shown in blue on the board](https://sudokuflow.com/assets/tech/scan-board.jpg)

*The straightened board, the boxes the network reads, and what it read. Photo: Wicht and Hennebert, CC BY 4.0.*

## 4. Read the digits

A small convolutional network reads each box: a 5 by 5 convolution with 12 filters, max pooling, a 3 by 3 convolution with 24 filters, pooling again, then layers of 64 and 10. It answers 1 to 9, or 0 for a shape that is not a digit. The weights are stored as 8-bit integers, which is why the model is 57 KB, and the browser runs it with about sixty lines of plain JavaScript, no library. If its answers are unsure on average, the board is tried a quarter turn at a time, for a photo taken with the phone sideways.

It learned from two kinds of photos. The first are 6,000 we generated: real puzzles from our app's bank, drawn in 385 font faces with different lines, colours, pencil notes and newspaper text around them, then laid on a table at a slant, with light falling off, sometimes a shadow, blur, noise and JPEG artefacts. Some are screenshots instead, in light and dark mode. 15% of the font faces were kept out of training, so the test measures fonts the network has never seen. The second are 160 real photos of newspaper Sudoku taken with phones, from the [Sudoku dataset of Baptiste Wicht and Jean Hennebert](https://github.com/wichtounet/sudoku_dataset) (CC BY 4.0). Their other 40 photos were never used for training: they are the real-world test below.

One detail mattered more than any other. The boxes the network learned from were cut by the same scan.js that runs on this site, in a headless browser, so whatever quirks the cutting has, the network has already seen them. On generated photos in fonts it had never seen, it reads 99.1% of the boxes right.

## 5. A Sudoku has one answer

A reading that is one digit off usually gives a puzzle with no solution, or with many. That is a free check, and a free fix: when the reading does not have exactly one answer, the squares the network was least sure of try its next guesses, one square and then two at a time, the likeliest first, until the puzzle has exactly one answer. On the 40 real test photos this turned 31 exact readings into 34. Every square the network was unsure of, or that the rule changed, is marked on the board, and the solver asks you to compare the numbers with your puzzle before solving.

## 6. Solve it, step by step

Once the numbers are in, the solver hands the puzzle to our app's hint engine. On each step it looks for the easiest technique that places a number, the way a person would, and explains it on the board. The puzzle gets the level of its hardest step, on the app's six levels from Easy to Extreme. All 60,000 puzzles in our app's bank solve this way without a forcing chain. A few famous "hardest" puzzles need one, or a step the engine cannot explain, and the solver says so.

## How well it works

| Test set | Photos | Grid found | Read exactly | Wrong, yet one answer |
|---|---|---|---|---|
| Real newspaper photos (Wicht and Hennebert test set) | 40 | 37 | 34 | 0 |
| Generated photos and screenshots, unseen fonts | 400 | 400 | 365 | 4 |

34 of the 40 real photos came out exactly right, and none was read wrong yet still looked like a valid puzzle. On the generated set, 4 of 400 were, which is why the solver always asks you to check before solving. For comparison, the dataset's page reports a 17.5% error rate on the same 40 photos for the method in Wicht's PhD thesis; we may not count errors the same way, so take that loosely. A photo takes a median of 90 to 210 milliseconds in Chrome on a laptop; a phone takes longer.

## What still goes wrong

- Part of the grid outside the frame: two of the three real photos with no grid found have the top edge cut off.
- Blur and faint print: the third is out of focus, with pale lines.
- Thin digits: in one photo, three thin 1s were too faint to cut out. The puzzle then had more than one answer, so the solver said so instead of solving the wrong puzzle.
- Handwriting: the network learned printed digits, and we have not tested it on handwritten ones. Scan the puzzle before you write in it.

For the best result, take the photo straight on, with the whole grid in the frame and in good light.

## Read the code

scan.js is served as written, comments included: [/assets/scan.js](https://sudokuflow.com/assets/scan.js). The model is [a JSON file](https://sudokuflow.com/assets/data/scan-model.json) of base64 int8 weights. The pictures on this page were drawn from what that same code computed for these photos.

The photos on this page and the real test set are from Wicht, B. and Hennebert, J., "Camera-based Sudoku recognition with deep belief network", 6th International Conference of Soft Computing and Pattern Recognition (SoCPaR), 2014, released under CC BY 4.0.

[Try the solver →](https://sudokuflow.com/tools/sudoku-solver/)

![Sudoku app icon](https://sudokuflow.com/assets/app-icon.png)

## Want more boards like this?

We also make a Sudoku app for iPhone, iPad, Mac and Android. There are no ads in it at all, it works offline, and when you ask for a hint it shows you the technique instead of just filling in a number.

[App Store iPhone, iPad, Mac](https://apps.apple.com/app/apple-store/id6502922368?pt=125379530&ct=sudokuflow_web&mt=8) [Google Play Android](https://play.google.com/store/apps/details?id=com.rubigogames.sudoku&referrer=utm_source%3Dsudokuflow%26utm_medium%3Dweb%26utm_campaign%3Dsite)

Sudoku Flow is made by Rubigo Games, the developer of the Sudoku app recommended on this site.