src.dackar.text_processing.VocabSpellChecker

Closed-vocabulary spell corrector.

Unlike the dictionary- and model-based SpellChecker, this corrector only ever substitutes a token with a word that is already present in a caller-supplied vocabulary. It never proposes a correction outside that set, so it is safe for specialised corpora whose terms are absent from general English dictionaries (domain jargon, part names, un-expanded abbreviations): an unknown token with no close vocabulary match is left untouched rather than mangled. It has no dependencies beyond the standard library.

Algorithm

For each whitespace-separated token in the (lower-cased) text:

1. Pass through unchanged if the token is shorter than min_token_len or is already in the vocabulary.

2. Otherwise take the single nearest vocabulary word via difflib.get_close_matches() at cutoff (the SequenceMatcher ratio threshold).

3. Substitute when a match is found; otherwise keep the original token.

The corrector is deterministic.

Attributes

logger

Classes

VocabSpellChecker

Correct tokens against a fixed vocabulary using difflib ratios.

Module Contents

src.dackar.text_processing.VocabSpellChecker.logger

class src.dackar.text_processing.VocabSpellChecker.VocabSpellChecker(vocabulary, cutoff=0.85, min_token_len=5)

Correct tokens against a fixed vocabulary using difflib ratios.

The corrector is intentionally conservative: it only accepts a correction whose SequenceMatcher ratio meets cutoff and only acts on tokens longer than min_token_len. Out-of-vocabulary tokens with no close match are left unchanged.

Parameters:

• vocabulary (Iterable[str]) – The valid words. Corrections are only ever drawn from this set; entries are lower-cased on construction.

• cutoff (float) – Minimum SequenceMatcher ratio for a correction to be accepted. Default 0.85. Lower values raise recall but also the risk of false corrections.

• min_token_len (int) – Tokens shorter than this are never corrected (too short to tell a typo from a valid short word). Default 5.

_cutoff = 0.85

_min_len = 5

_vocab: frozenset[str]

_vocab_list: list[str]

transform(text)

Return text with out-of-vocabulary tokens replaced by their nearest vocabulary match. Tokens already in the vocabulary or shorter than min_token_len are passed through unchanged.

The text is lower-cased before tokenising, so every returned token is lower-case — including tokens that were not corrected. Callers that surface the result to users (e.g. a cleaned_description field) should expect this normalisation.

Parameters:

text (str)

Return type:

str

_correct(token)

Return the corrected form of a single token.

Parameters:

token (str)

Return type:

str

static _is_affix_variant(token, candidate)

Return True when candidate is token with a prefix/suffix dropped.

A closed-vocabulary match that is a strict substring of the token (two or more characters shorter) is almost never a typo correction — it is the token with an affix removed. For prefixed domain verbs that often inverts the meaning (uninstall -> install, unisolate -> isolate, reenergize -> energize, retorque -> torque). Refusing these keeps the corrector from turning an operation into its opposite; a genuine typo (transposition, single substitution) is not a substring of the original and is unaffected.

Parameters:

• token (str)

• candidate (str)

Return type:

bool