src.dackar.RCA.ner.augment_chunks

Second-pass augmentation of mdParser chunks with structured JSON summaries.

Goal

Read the mdParser output *_chunks.jsonl, and for each chunk that is intended for vector indexing, generate two structured JSON views using Ollama:

  1. retrieval_summary_json

  2. rca_frame_json

Then write an enriched JSONL file: *_chunks_enriched.jsonl

This module is designed for Option A: - Do NOT modify your existing pdfParser.py or mdParser.py pipelines. - Run this as a separate step from a notebook or script.

Inputs

  • chunks_jsonl_path: path to <doc_id>_chunks.jsonl produced by mdParser.py

  • document_index: dict loaded from index/document_index.json produced by pdfParser.py

  • structured_output (optional): dict loaded from <doc_id>_structured_output.json

  • ner_provider (optional): callable that returns NERSeed per chunk

Outputs

  • <doc_id>_chunks_enriched.jsonl file (same directory as input chunks file)

  • returns output path and summary stats

Attributes

_CAUSAL_KEYWORD_PATTERNS_SEED

_EQUIP_TAG_DEFAULT_RE

Classes

AugmentStats

Statistics returned by augmentation.

Functions

_make_ner_seed_cs_factory(ner_seed)

Build a causal_sentence_factory from NERSeed mechanisms/outcomes.

_load_json(path)

_iter_jsonl(path)

_write_jsonl(path, records)

_safe_int(x[, default])

_safe_optional_int(x)

_extract_equipment_ids_quick(text[, limit])

Minimal equipment tag extraction for seeding. (You can swap in your full helper.)

default_ner_seed_from_chunk(chunk)

Build an NERSeed using only fields already present in mdParser output.

augment_chunks_with_structured_summaries(...[, model, ...])

Enrich an mdParser chunks.jsonl file with structured summaries (JSON) using Ollama.

_validate_stage5_alias_consistency(record)

Ensure compatibility aliases remain aligned with enrichment payload.

_extract_causal_spans(stage5_payload[, min_confidence])

Return (cause_texts, effect_texts) from Stage 5 statements at or above min_confidence.

build_embedding_text(chunk_text, ner_seed, ...[, ...])

build_chunk_metadata(chunk, ctx, ner_seed, ...[, ...])

_uniq(items)

build_processed_text_record(*, doc_id, doc_type, ...)

validate_processed_text_record(record)

to_chroma_payload(record)

Module Contents

src.dackar.RCA.ner.augment_chunks._CAUSAL_KEYWORD_PATTERNS_SEED: List[Dict[str, Any]][source]
src.dackar.RCA.ner.augment_chunks._make_ner_seed_cs_factory(ner_seed)[source]

Build a causal_sentence_factory from NERSeed mechanisms/outcomes.

Parallel to adapter.py _make_ner_cs_factory, but operates on NERSeed string lists rather than ResolvedSpan objects. Returns None when the seed has no mechanism/outcome texts so the caller falls back to dep_fallback unchanged.

Parameters:

ner_seed (src.dackar.RCA.summarizers.reliability_summarizer.NERSeed)

Return type:

Optional[Any]

src.dackar.RCA.ner.augment_chunks._EQUIP_TAG_DEFAULT_RE[source]
src.dackar.RCA.ner.augment_chunks._load_json(path)[source]
Parameters:

path (pathlib.Path)

Return type:

Dict[str, Any]

src.dackar.RCA.ner.augment_chunks._iter_jsonl(path)[source]
Parameters:

path (pathlib.Path)

Return type:

Iterable[Dict[str, Any]]

src.dackar.RCA.ner.augment_chunks._write_jsonl(path, records)[source]
Parameters:

  • path (pathlib.Path)

  • records (Iterable[Dict[str, Any]])

Return type:

None

src.dackar.RCA.ner.augment_chunks._safe_int(x, default=0)[source]
Parameters:

  • x (Any)

  • default (int)

Return type:

int

src.dackar.RCA.ner.augment_chunks._safe_optional_int(x)[source]
Parameters:

x (Any)

Return type:

Optional[int]

src.dackar.RCA.ner.augment_chunks._extract_equipment_ids_quick(text, limit=50)[source]

Minimal equipment tag extraction for seeding. (You can swap in your full helper.)

Input: free text Output: list of tags like P-101A, MOV-204A, PT-1102…

Parameters:

  • text (str)

  • limit (int)

Return type:

List[str]

src.dackar.RCA.ner.augment_chunks.default_ner_seed_from_chunk(chunk)[source]

Build an NERSeed using only fields already present in mdParser output.

This is intentionally conservative and works even if you haven’t wired your full nuclear NER pipeline into mdParser chunks yet.

Inputs

chunk: dict from chunks.jsonl. Typical keys from mdParser include:

  • text: str

  • keywords: list[str]

  • mentions_component_ids: list[str] (if present)

  • standards_refs: dict (if present)

  • type, granularity, etc.

Output

NERSeed with:

  • equipment_ids: derived via regex

  • components: from mentions_component_ids if present

  • everything else empty (you can fill later using your NER outputs)

Parameters:

chunk (Dict[str, Any])

Return type:

src.dackar.RCA.summarizers.reliability_summarizer.NERSeed

class src.dackar.RCA.ner.augment_chunks.AugmentStats[source]

Statistics returned by augmentation.

total_chunks: int[source]
eligible_chunks: int[source]
summarized_chunks: int[source]
skipped_already_present: int[source]
failed_chunks: int[source]
output_path: str[source]
src.dackar.RCA.ner.augment_chunks.augment_chunks_with_structured_summaries(chunks_jsonl_path, *, model=None, timeout=90, max_tries=3, output_suffix='_enriched', overwrite=False, summarize_granularities=('section', 'paragraph'), only_indexable=True, doc_type_override=None, authority_override=None, ner_seed_provider=None, stage5_nlp=None, stage5_llm_cfg=None)[source]

Enrich an mdParser chunks.jsonl file with structured summaries (JSON) using Ollama.

Inputs

chunks_jsonl_path:

Path to <doc_id>_chunks.jsonl produced by mdParser.py.

model:

Ollama model name. If None, reliability_summarizer uses env OLLAMA_MODEL or default.

timeout:

Request timeout seconds per summary call.

max_tries:

Retry ladder attempts per summary.

output_suffix:

Output file suffix. If input is abc_chunks.jsonl, output becomes abc_chunks_enriched.jsonl.

overwrite:

If False and output exists, raises an error.

summarize_granularities:

Which chunk granularities to summarize. mdParser uses “section” and “paragraph” for TextChunk.

only_indexable:

If True, only summarize chunks where index_in_vector_store == True.

doc_type_override:

Force doc_type for all chunks (“SOP”, “CR”, “WO”, “ECA”, “OTHER”). If None, auto-detect using early chunk text.

authority_override:

Force authority_level: “mandatory”,”guidance”,”informational”,”unknown”. If None, SOP->mandatory else informational.

ner_seed_provider:

Optional function chunk->NERSeed. If not provided, uses default_ner_seed_from_chunk().

stage5_nlp:

Optional initialized NLP pipeline to pass explicitly into Stage 5 causal/condition extraction.

Output

AugmentStats including output_path. Writes a new JSONL alongside input.

Output record format (per chunk)

Adds (when summarized):

  • retrieval_summary_json: dict

  • rca_frame_json: dict

  • retrieval_summary_text: str (flattened for embeddings)

  • rca_frame_text: str (flattened for embeddings)

  • augmentation: { “status”: “ok”|”error”, “error”: str|None }

Chunks that are not summarized are written unchanged (plus minimal augmentation status if desired).

Parameters:

  • chunks_jsonl_path (str | pathlib.Path)

  • model (Optional[str])

  • timeout (int)

  • max_tries (int)

  • output_suffix (str)

  • overwrite (bool)

  • summarize_granularities (Tuple[str, ...])

  • only_indexable (bool)

  • doc_type_override (Optional[str])

  • authority_override (Optional[str])

  • ner_seed_provider (Optional[Callable[[Dict[str, Any]], src.dackar.RCA.summarizers.reliability_summarizer.NERSeed]])

  • stage5_nlp (Any)

  • stage5_llm_cfg (Optional[Dict[str, Any]])

Return type:

AugmentStats

src.dackar.RCA.ner.augment_chunks._validate_stage5_alias_consistency(record)[source]

Ensure compatibility aliases remain aligned with enrichment payload.

Parameters:

record (Dict[str, Any])

Return type:

List[str]

src.dackar.RCA.ner.augment_chunks._extract_causal_spans(stage5_payload, min_confidence=0.35)[source]

Return (cause_texts, effect_texts) from Stage 5 statements at or above min_confidence.

cause_texts — causal precursor spans; routed to mechanisms in NERSeed backfill. effect_texts — failure/outcome spans; routed to outcomes in NERSeed backfill.

min_confidence=0.35 corresponds to at least one filled field (connector OR a cause/effect span) in _score_causal_statement, filtering out the emptiest extractions.

Parameters:

  • stage5_payload (Optional[Dict[str, Any]])

  • min_confidence (float)

Return type:

Tuple[List[str], List[str]]

src.dackar.RCA.ner.augment_chunks.build_embedding_text(chunk_text, ner_seed, retrieval_summary, rca_frame, stage5_payload=None, max_chars=3500)[source]
Parameters:

  • chunk_text (str)

  • ner_seed (src.dackar.RCA.summarizers.reliability_summarizer.NERSeed)

  • retrieval_summary (Dict[str, Any])

  • rca_frame (Dict[str, Any])

  • stage5_payload (Optional[Dict[str, Any]])

  • max_chars (int)

Return type:

str

src.dackar.RCA.ner.augment_chunks.build_chunk_metadata(chunk, ctx, ner_seed, retrieval_summary, rca_frame, stage5_payload=None)[source]
Parameters:

  • chunk (Dict[str, Any])

  • ctx (src.dackar.RCA.summarizers.reliability_summarizer.ChunkContext)

  • ner_seed (src.dackar.RCA.summarizers.reliability_summarizer.NERSeed)

  • retrieval_summary (Dict[str, Any])

  • rca_frame (Dict[str, Any])

  • stage5_payload (Optional[Dict[str, Any]])

Return type:

Dict[str, Any]

src.dackar.RCA.ner.augment_chunks._uniq(items)[source]
Parameters:

items (List[str])

Return type:

List[str]

src.dackar.RCA.ner.augment_chunks.build_processed_text_record(*, doc_id, doc_type, chunk_index, chunk, ctx, ner_seed, retrieval_summary, rca_frame, metadata, embedding_text, stage5_payload)[source]
Parameters:

  • doc_id (str)

  • doc_type (str)

  • chunk_index (int)

  • chunk (Dict[str, Any])

  • ctx (src.dackar.RCA.summarizers.reliability_summarizer.ChunkContext)

  • ner_seed (src.dackar.RCA.summarizers.reliability_summarizer.NERSeed)

  • retrieval_summary (Dict[str, Any])

  • rca_frame (Dict[str, Any])

  • metadata (Dict[str, Any])

  • embedding_text (str)

  • stage5_payload (Dict[str, Any])

Return type:

Dict[str, Any]

src.dackar.RCA.ner.augment_chunks.validate_processed_text_record(record)[source]
Parameters:

record (Dict[str, Any])

Return type:

List[str]

src.dackar.RCA.ner.augment_chunks.to_chroma_payload(record)[source]
Parameters:

record (Dict[str, Any])

Return type:

Dict[str, Any]