Academic / deep learning · 2023-11-10
HMER: Image to LaTeX Converter
A spatial CNN–Transformer that translates rendered mathematical expressions into LaTeX, reported from autoregressive decode rather than teacher-forced token accuracy.
- Role
- ML project developer
- Contribution
- Replaced a teacher-forced LSTM baseline with a spatial CNN–Transformer that maps rendered formula images to LaTeX.
- Timeframe
- 2023, revised 2026
- Status
- Academic
Academic image-to-text project. Headline generation numbers are from the epoch-14 Transformer checkpoint under autoregressive greedy decode on 256 validation samples. They are not a test-split score and not a handwriting-OCR result.
View the project website · Read the source
Problem
Turning a formula image into editable LaTeX is a structured image-to-sequence problem: the model has to see symbols in 2D layout and emit a 1D token string. The repository is named HMER, but the training data is IM2LaTeX-100k — rendered printed expressions, not handwriting. The cover grid is that dataset.
Dataset
The local split lists contain 75,275 training, 8,370 validation, and 10,355 test expressions. The formula list has 103,559 normalized strings. Images are scaled to fit a 64×512 canvas and never cropped; white right-padding is masked out of cross-attention so the decoder cannot attend to empty canvas. Sequences truncate at 141 tokens. The tokenizer is fit on the training split only with min_freq=5.
Approach
The original CNN/ResNet encoder with an LSTM decoder collapsed each image to a single vector, so attention over the encoder was a no-op. Validation scored teacher-forced next-token accuracy, not generated text. Those legacy figures — token accuracy 62.6%, BLEU-4 0.154, Levenshtein similarity 0.283 after 25 epochs — are an upper bound on generation quality. No old checkpoint remained in the repo, so they could not be re-scored under greedy or beam decode.
The current stack keeps spatial layout. A three-block CNN (Conv-BN-ReLU-MaxPool) downsamples by 8 and projects to d_model=256, then adds 2D sinusoidal positional encoding. The encoder emits an 8×64 feature grid (512 memory tokens) plus a padding mask. A 4-layer, 8-head pre-norm Transformer decoder with tied embeddings generates the token sequence. Training uses AdamW (lr 3e-4, weight decay 1e-4), linear warmup then cosine decay, label-smoothed cross-entropy (0.1), and gradient clipping at 1.0, in fp32 on Apple Silicon MPS.
Checkpoints are selected on validation BLEU, not validation loss. Reported generation metrics use batched greedy decode. Beam search (size 5, length penalty 0.7) exists for test and predict; it is not the protocol behind the numbers on this page.
Evaluation
The headline generation numbers are from the best Transformer checkpoint (epoch 14) under autoregressive greedy decode on 256 validation samples: BLEU-4 0.790, exact match 21.9%, and normalized edit distance 0.178. BLEU is smoothed corpus BLEU-4 (NLTK method 3), not a per-sentence geometric mean that zeros out short formulas. Exact match requires the generated token sequence to equal the reference after stripping START/END/PAD. Edit distance is mean Levenshtein / max(len); lower is better.
The 91.5% token accuracy is a teacher-forced diagnostic (next-token accuracy under gold prefixes on the full 8,370-expression validation split). It is not a generation metric. The LSTM baseline numbers below are also teacher-forced and are not the same protocol.
| Metric | Transformer | Legacy LSTM |
|---|---|---|
| BLEU-4 | 0.790 | 0.154 |
| Exact match | 21.9% | — |
| Edit distance | 0.178 | — |
| Token acc | 91.5% | 62.6% |
Training logged 21 epochs. Epoch 14 is the BLEU-selected checkpoint; later epochs did not beat it. BLEU, exact match, and edit distance below use the same 256-sample greedy protocol.
| Epoch | BLEU | Exact match | Edit dist |
|---|---|---|---|
| 1 | 0.117 | 0.0% | 0.730 |
| 5 | 0.638 | 5.5% | 0.285 |
| 10 | 0.751 | 15.2% | 0.209 |
| 14 | 0.790 | 21.9% | 0.178 |
| 21 | 0.784 | 18.8% | 0.192 |
The public source includes the model, the evaluation workflow, and outputs/img2latex_transformer_v1/metrics/metrics.json. Useful checks include expression-level exact match and inspecting failures on unusual notation and layout. A full test-split generation score is not in the saved artifacts.
Outcome
The epoch-14 Transformer checkpoint is the reported result: generated BLEU-4 0.790, exact match 21.9%, and normalized edit distance 0.178 under autoregressive greedy decode. The useful change is the protocol as much as the architecture: spatial features plus generated-text metrics, instead of a single-vector encoder scored under teacher forcing.
Limitations
These figures are a 256-sample greedy-decode snapshot on the validation split of rendered IM2LaTeX-100k expressions. They do not measure handwriting, a full test split, or beam-search quality. Generated LaTeX can be incorrect and should be reviewed. Exact match at 21.9% means most expressions are still not token-for-token correct even when BLEU is high. The project is academic work, not a production OCR reliability estimate.
Evidence
- Project website
- Source repository
- Transformer run metrics:
outputs/img2latex_transformer_v1/metrics/metrics.json - Legacy protocol note:
outputs/baseline_honest.json
Plates