How I Trained a Captcha Solver on 4,000 Labelled Images

October 27, 2026 • Matthew Duong • Software Engineering;Machine Learning;Startups • 14 min read

Three small models now solve 96.7% of test captchas unattended

A product I was working on now reads a form’s start-page captcha with three small neural networks I trained on 4,000 hand-labelled images. The agent submits an answer only when all three agree. On a fixed test set of 150 images:

All three agree on 146. 145 of those are right, and 1 is submitted wrong.
On the other 4, the agent falls back to asking a person.

So 96.7% of captchas are solved correctly with nobody involved, 2.7% go to a person, and 0.7% cost one wrong attempt. The 99.3% figure later in this post is accuracy when the models agree (145 of 146), not overall accuracy.

Before this, every run needed a person. The product fills in a long form on the form’s official website, and its first page shows a security code: a 250 × 50 image with four to six distorted characters. Until October 2026, a filing run stopped there and asked a person to type it in. Everything after it was automatic, but a run could not start unattended.

The second captcha, on the Sign and Submit page, stays human on purpose. Typing it is the applicant’s act of signing. Solving it automatically would sign on their behalf.

I trained my own model instead of calling a vision API

A hosted vision model was the obvious first option. I went with a model of my own for three reasons:

The images never leave my cluster.
There is no cost per solve.
It is fast. One model reads an image in about 9 ms on one thread of a Ryzen AI 9 HX 370, so all three take under 30 ms. That excludes decoding the PNG, and it is the warm time after the model has loaded.

The model is a small CRNN: four convolution blocks turn the image into 62 columns, a two-layer bidirectional GRU reads them left to right and right to left, and CTC decodes four to six characters. CTC matters because the code length varies. A fixed six-slot classifier would need to learn where a four-character code ends. The model has 1.4 million parameters and exports to a 5.7 MB ONNX file.

Captcha image 3 × 50 × 250 RGB, 0 to 1, transparency flattened onto white
Four convolution blocks 160 × 3 × 62 Pooling halves the height four times and the width twice
Columns as a sequence 62 × 256 Each column's 480 features projected to 256
Bidirectional GRU, two layers 62 × 256 Reads the columns left to right and right to left
Per-column classifier 62 × 26 25 characters plus the CTC blank
CTC greedy decode 4 to 6 characters Best class per column, collapse repeats, drop blanks

The model, one image at a time. Shapes are channels × height × width, then steps × features.

The input stays in colour. Some captcha styles separate the text from the background only by hue, and grayscale would throw that away.

It is trained in PyTorch 2.14 and runs in Node with onnxruntime-node, because the agent is a TypeScript Playwright job.

The training setup is small enough for a laptop CPU

AdamW at a peak learning rate of 0.002 with a one-cycle schedule, weight decay 0.0001, batch size 32, gradients clipped at 5.
Up to 150 epochs. In the second half, a model stops once validation has not improved for 25 epochs. The sixth run stopped at epochs 117, 75 and 98.
An exponential moving average of the weights (decay 0.995) is what gets scored and saved.
Augmentation is small rotations, shifts, scaling and shear, plus heavy colour changes: hue jitter, shuffled colour channels, occasional inversion and grayscale. A style seen a few times should not be learned as “orange text”. There are no flips, because a mirrored character is a different character.

The training step is the standard CTC loop. Targets are the codes concatenated, with their lengths alongside, and every input has 62 steps:

ctc = nn.CTCLoss(blank=0, zero_infinity=True)
for batch, targets, target_lengths in loader:
    batch = batch.to(device, non_blocking=True)
    log_probs = model(batch).permute(1, 0, 2)  # CTC wants [T, N, C]
    input_lengths = torch.full((batch.shape[0],), log_probs.shape[0], dtype=torch.long)
    loss = ctc(log_probs, targets.to(device), input_lengths.to(device), target_lengths.to(device))
    optimiser.zero_grad()
    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 5.0)
    optimiser.step()
    schedule.step()
    ema.update_parameters(model)

Three models took 55 minutes at 800 images and 205 minutes at 3,400, on the CPU of a ThinkPad P14s.

Collecting 6,000 images was harder than training on them

I wrote a harvester that loads the start page, saves the captcha, clicks reload and saves the next one.

My cluster’s outbound address is the same as my office network. Harvesting thousands of captchas from it risked getting the address flagged, and that address is the one real filings come from. The harvester refuses to run if its public IP matches the office, so I harvested from other networks.

The harvester uploads to an admin page in the product, where I label images one at a time. After the first few hundred labels the alphabet was clear: 25 characters, 345689ABCDEHJKMNPRSTUVWXY. The captcha leaves out the confusable ones (0 and O, 1 and I and L, 2 and Z, 7, G, Q, F). Lengths are four, five and six, about a third each.

6,000 captchas became 3,385 training images

I harvested about 6,000 captchas and labelled 3,997 of them. The other 2,000 or so are still unlabelled. I froze 150 labelled images as the test set on day one and have never changed which images are in it. Images labelled later only ever go to training or validation.

The remaining 3,847 are shuffled and split 88/12: 3,385 for training and 462 for validation. Validation picks each model’s best epoch and decides when to stop. The test set is only scored at the end. Every number in this post is measured on those 150.

One training run cannot tell you whether a change helped

Run	Training images	Test exact match	All three agree	Right when they agree
first	743	85.3% (one model)
third	827	mean 89.1% (85.3, 90.7, 91.3)	85%	94.5%
fourth	1,618	mean 94.2% (92.7, 94.0, 96.0)	93%	96.4%
fifth	2,506	mean 95.1% (94.7, 96.0, 94.7)	95.3%	97.2%
sixth	3,385	mean 98.4% (96.7, 99.3, 99.3)	97.3%	99.3%

The first two runs used nearly the same data and differed by 1.3 points. They also missed mostly different images: of 22 and 24 misses, only 10 were shared. With 150 test images, one image is 0.7 points, so a single run is mostly noise.

From the third run on, every run trains three models from different random seeds and I compare the means. The third run also added averaged weights (an exponential moving average of the model, scored and saved instead of the last step) and early stopping. Same data as the second run, 5 points better on average.

After that, the settings stayed the same and the gains came from labels. Accuracy kept climbing at 4,000, so more labelled data was still the best thing I could add.

Test exact match by training images, mean of three seeds

Each whisker spans the worst and best of the three seeds. The settings were the same in all four runs. Only the labelled data grew, and a few labels were corrected along the way.

Three models agreeing is a better signal than one model’s confidence

My first plan was to submit a reading when the model’s confidence was above a threshold. The confidence turned out to be badly calibrated. A model that dropped a character could still score 0.99.

Models trained from different seeds make different mistakes, but not entirely different ones. They saw the same training images, so they share some blind spots, and JJS4EA below is one: all three read the same wrong answer. Agreement lowers the error rate, it does not remove it. So the agent reads each image with all three and submits only when they give the same answer. Otherwise it reloads the captcha or hands it to a person. A wrong answer costs one of the site’s limited attempts, so a cautious rule pays for itself.

These are the five test images the shipped models do not all read correctly. All four disagreements come from the weakest of the three, and three of them are a dropped character: the kind of mistake that could still score 0.99 confidence. The one wrong agreed answer is JJS4EA, where the first J is cut off at the top and all three read a T.

A captcha reading JJS4EA — Each model's reading, in seed order. The first image is submitted wrong. The other four go to a person.

A captcha reading MVDR9H — Each model's reading, in seed order. The first image is submitted wrong. The other four go to a person.

At 3,000 labels this was still under the 98% I wanted, and I shipped it anyway: a person still gets the last attempt, so being wrong 3% of the time cost a little time, not a filing. At 4,000 labels it cleared 98%.

150 test images cannot prove 99.3%

The 99.3% is one miss in 146. With a sample that small, the 95% confidence interval for accuracy when the models agree runs from 96.2% to 99.98%. The result is good, but it is not yet a reliability claim.

The test set is also not as clean as “frozen” suggests. It never trains a model, but I looked at its scores after every run to decide what to try next and which run to ship. After six runs it has partly become a second validation set. The fix is a larger, fresh test set once there are enough labels, scored rarely. The numbers from before and after it will not be comparable, which is why I have not swapped it yet.

The models were most useful for labelling and checking labels

Each round, I export the dataset, train, run the new models over every image, and import their readings back into admin. The labeller then pre-fills each unlabelled image with the models’ guess, least confident first. Most images become one keystroke, and the hard ones, which teach the model most, come first.

A captcha reading BYYAX in dark letters over a grey wavy background — The labeller, re-created. The guess is pre-filled and selected, so Enter accepts it and typing replaces it.

The risk is accepting a wrong guess and teaching the model its own mistake. The labeller marks low-confidence guesses, and the rule is to read the image, not the suggestion.

The same readings find bad labels. Where all three models disagree with a saved label, either the label or the models are wrong. On the 3,000-label export that was 12 images. Some were my typos (TRBSN was JRBSN). Some were the models’ mistakes. Early on I assumed two test-set disagreements were typos, checked them by eye, and found the labels were right. A model disagreeing is a reason to look, not a verdict. Admin now has a mode that walks through saved labels one at a time, likeliest mistakes first.

A captcha reading JRBSN in striped green and purple letters — Check labelled, re-created. The saved label starts in the box, and the models' reading sits underneath it, never in it.

Python and Node must prepare the image identically

The model trains in Python and runs in Node. If Node flattens transparency or scales pixels slightly differently, nothing errors. The model just gets quietly worse in production.

In Python, the dataset loader flattens transparency onto white with Pillow, keeps the three colour channels and scales each byte to a float between 0 and 1:

def load_image(path: Path) -> torch.Tensor:
    image = Image.open(path).convert("RGBA")
    if image.size != (WIDTH, HEIGHT):
        image = image.resize((WIDTH, HEIGHT), Image.BILINEAR)
    background = Image.new("RGBA", image.size, (255, 255, 255, 255))
    rgb = Image.alpha_composite(background, image).convert("RGB")
    array = np.asarray(rgb, dtype=np.float32) / 255.0
    return torch.from_numpy(array).permute(2, 0, 1).contiguous()  # [3, 50, 250]

Inference in Node is short. The preprocessing has to match that exactly, including the rounding when transparency is flattened onto white:

const ALPHABET = "345689ABCDEHJKMNPRSTUVWXY";
const { width, height, data } = PNG.sync.read(png); // always RGBA
const plane = width * height;
const input = new Float32Array(3 * plane);
for (let pixel = 0; pixel < plane; pixel++) {
    const alpha = data[pixel * 4 + 3];
    for (let channel = 0; channel < 3; channel++) {
        // Over white, rounded to a byte as Pillow's alpha_composite does.
        const value = Math.round((data[pixel * 4 + channel] * alpha + 255 * (255 - alpha)) / 255);
        input[channel * plane + pixel] = value / 255;
    }
}

const session = await InferenceSession.create(modelBytes, { executionProviders: ["cpu"], intraOpNumThreads: 1 });
const { log_probs } = await session.run({ image: new Tensor("float32", input, [1, 3, 50, 250]) });

// Greedy CTC: likeliest class per column, collapse repeats, drop blanks (class 0).
let text = "";
let previous = 0;
for (let step = 0; step < 62; step++) {
    const scores = log_probs.data.slice(step * 26, step * 26 + 26) as Float32Array;
    const best = scores.indexOf(Math.max(...scores));
    if (best !== 0 && best !== previous) text += ALPHABET[best - 1];
    previous = best;
}

Before shipping any model I run every image in the export through both and compare. For the current models, all 6,000 images get the same text from Node and Python, with confidence within 0.00005.

The model ships like code

The ONNX files are committed to the repo and built into the agent’s Docker image. A new model is a pull request with its test numbers in the description. It goes to staging first, then production, and rolling back is a revert. The dataset itself never goes in git.

Two things are left. I want an alert if the agreement rate drops in production, which would mean the site changed its captcha style. And the best training data is captchas from real runs, where the site’s accept or reject is a free label. Keeping those needed a privacy policy change first, so collecting them waits for that.

Next time I would freeze 500 test images and train three seeds from day one

Most of what I would change is the order I did things in, not the model.

A bigger test set from the start. The original plan said 500 images. I froze 150 from the first export of about 1,000 labels. Every number since has a wide interval, and replacing the set now breaks comparison with every earlier run.
Three seeds from the first run. The original plan was one model and a confidence threshold. The first two runs showed only that one run is noise, and the threshold never worked.
Label checks before the labels. The rule that refuses characters the captcha never uses arrived at about 2,000 labels. The checker that found my typos arrived at 3,000. Both are small, and both would have cleaned the data earlier.
The blind relabel I skipped. The original plan said to relabel 200 random images a day later and compare. I never did, so I do not know how often I misread these captchas myself. So I do not know how much noise is in the labels, or whether cleaning them would have been worth more than adding to them.
Less pipeline up front. The original plan had versioned dataset snapshots and a training job in CI. I built neither. An export zip and a laptop were enough for six runs.

I would keep the CRNN with CTC. Mean accuracy went from 89% to 98% with no change to the model or its settings. Every gain after the third run came from labels, so a bigger or newer architecture would have been the wrong place to spend time.

TL;DR

The product now solves the start-page captcha with three small CRNN models I trained on 4,000 hand-labelled images. The agent submits only when all three agree. On 150 test images that solves 145 correctly, sends 4 to a person and gets 1 wrong. Training three seeds per run was what made results comparable. More labels did the rest, and the models’ readings made labelling faster and found label mistakes. The test set is now too small and too often consulted to prove much beyond 96%, so a larger fresh one is next.