src.dackar.RCA.ner.augment_chunks ================================= .. py:module:: src.dackar.RCA.ner.augment_chunks .. autoapi-nested-parse:: augment_chunks.py ================= Second-pass augmentation of mdParser chunks with structured JSON summaries. Goal ---- Read the mdParser output `*_chunks.jsonl`, and for each chunk that is intended for vector indexing, generate two structured JSON views using Ollama: 1) retrieval_summary_json 2) rca_frame_json Then write an enriched JSONL file: `*_chunks_enriched.jsonl` This module is designed for Option A: - Do NOT modify your existing pdfParser.py or mdParser.py pipelines. - Run this as a separate step from a notebook or script. Inputs ------ - chunks_jsonl_path: path to `_chunks.jsonl` produced by mdParser.py - document_index: dict loaded from `index/document_index.json` produced by pdfParser.py - structured_output (optional): dict loaded from `_structured_output.json` - ner_provider (optional): callable that returns NERSeed per chunk Outputs ------- - `_chunks_enriched.jsonl` file (same directory as input chunks file) - returns output path and summary stats Attributes ---------- .. autoapisummary:: src.dackar.RCA.ner.augment_chunks._CAUSAL_KEYWORD_PATTERNS_SEED src.dackar.RCA.ner.augment_chunks._EQUIP_TAG_DEFAULT_RE Classes ------- .. autoapisummary:: src.dackar.RCA.ner.augment_chunks.AugmentStats Functions --------- .. autoapisummary:: src.dackar.RCA.ner.augment_chunks._make_ner_seed_cs_factory src.dackar.RCA.ner.augment_chunks._load_json src.dackar.RCA.ner.augment_chunks._iter_jsonl src.dackar.RCA.ner.augment_chunks._write_jsonl src.dackar.RCA.ner.augment_chunks._safe_int src.dackar.RCA.ner.augment_chunks._safe_optional_int src.dackar.RCA.ner.augment_chunks._extract_equipment_ids_quick src.dackar.RCA.ner.augment_chunks.default_ner_seed_from_chunk src.dackar.RCA.ner.augment_chunks.augment_chunks_with_structured_summaries src.dackar.RCA.ner.augment_chunks._validate_stage5_alias_consistency src.dackar.RCA.ner.augment_chunks._extract_causal_spans src.dackar.RCA.ner.augment_chunks.build_embedding_text src.dackar.RCA.ner.augment_chunks.build_chunk_metadata src.dackar.RCA.ner.augment_chunks._uniq src.dackar.RCA.ner.augment_chunks.build_processed_text_record src.dackar.RCA.ner.augment_chunks.validate_processed_text_record src.dackar.RCA.ner.augment_chunks.to_chroma_payload Module Contents --------------- .. py:data:: _CAUSAL_KEYWORD_PATTERNS_SEED :type: List[Dict[str, Any]] .. py:function:: _make_ner_seed_cs_factory(ner_seed) Build a causal_sentence_factory from NERSeed mechanisms/outcomes. Parallel to adapter.py _make_ner_cs_factory, but operates on NERSeed string lists rather than ResolvedSpan objects. Returns None when the seed has no mechanism/outcome texts so the caller falls back to dep_fallback unchanged. .. py:data:: _EQUIP_TAG_DEFAULT_RE .. py:function:: _load_json(path) .. py:function:: _iter_jsonl(path) .. py:function:: _write_jsonl(path, records) .. py:function:: _safe_int(x, default = 0) .. py:function:: _safe_optional_int(x) .. py:function:: _extract_equipment_ids_quick(text, limit = 50) Minimal equipment tag extraction for seeding. (You can swap in your full helper.) Input: free text Output: list of tags like P-101A, MOV-204A, PT-1102... .. py:function:: default_ner_seed_from_chunk(chunk) Build an NERSeed using only fields already present in mdParser output. This is intentionally conservative and works even if you haven't wired your full nuclear NER pipeline into mdParser chunks yet. Inputs ------ chunk: dict from chunks.jsonl. Typical keys from mdParser include: - text: str - keywords: list[str] - mentions_component_ids: list[str] (if present) - standards_refs: dict (if present) - type, granularity, etc. Output ------ NERSeed with: - equipment_ids: derived via regex - components: from mentions_component_ids if present - everything else empty (you can fill later using your NER outputs) .. py:class:: AugmentStats Statistics returned by augmentation. .. py:attribute:: total_chunks :type: int .. py:attribute:: eligible_chunks :type: int .. py:attribute:: summarized_chunks :type: int .. py:attribute:: skipped_already_present :type: int .. py:attribute:: failed_chunks :type: int .. py:attribute:: output_path :type: str .. py:function:: augment_chunks_with_structured_summaries(chunks_jsonl_path, *, model = None, timeout = 90, max_tries = 3, output_suffix = '_enriched', overwrite = False, summarize_granularities = ('section', 'paragraph'), only_indexable = True, doc_type_override = None, authority_override = None, ner_seed_provider = None, stage5_nlp = None, stage5_llm_cfg = None) Enrich an mdParser chunks.jsonl file with structured summaries (JSON) using Ollama. Inputs ------ chunks_jsonl_path: Path to `_chunks.jsonl` produced by mdParser.py. model: Ollama model name. If None, reliability_summarizer uses env OLLAMA_MODEL or default. timeout: Request timeout seconds per summary call. max_tries: Retry ladder attempts per summary. output_suffix: Output file suffix. If input is `abc_chunks.jsonl`, output becomes `abc_chunks_enriched.jsonl`. overwrite: If False and output exists, raises an error. summarize_granularities: Which chunk granularities to summarize. mdParser uses "section" and "paragraph" for TextChunk. only_indexable: If True, only summarize chunks where `index_in_vector_store == True`. doc_type_override: Force doc_type for all chunks ("SOP", "CR", "WO", "ECA", "OTHER"). If None, auto-detect using early chunk text. authority_override: Force authority_level: "mandatory","guidance","informational","unknown". If None, SOP->mandatory else informational. ner_seed_provider: Optional function chunk->NERSeed. If not provided, uses default_ner_seed_from_chunk(). stage5_nlp: Optional initialized NLP pipeline to pass explicitly into Stage 5 causal/condition extraction. Output ------ AugmentStats including output_path. Writes a new JSONL alongside input. Output record format (per chunk) -------------------------------- Adds (when summarized): - retrieval_summary_json: dict - rca_frame_json: dict - retrieval_summary_text: str (flattened for embeddings) - rca_frame_text: str (flattened for embeddings) - augmentation: { "status": "ok"|"error", "error": str|None } Chunks that are not summarized are written unchanged (plus minimal augmentation status if desired). .. py:function:: _validate_stage5_alias_consistency(record) Ensure compatibility aliases remain aligned with enrichment payload. .. py:function:: _extract_causal_spans(stage5_payload, min_confidence = 0.35) Return (cause_texts, effect_texts) from Stage 5 statements at or above min_confidence. cause_texts — causal precursor spans; routed to mechanisms in NERSeed backfill. effect_texts — failure/outcome spans; routed to outcomes in NERSeed backfill. min_confidence=0.35 corresponds to at least one filled field (connector OR a cause/effect span) in _score_causal_statement, filtering out the emptiest extractions. .. py:function:: build_embedding_text(chunk_text, ner_seed, retrieval_summary, rca_frame, stage5_payload = None, max_chars = 3500) .. py:function:: build_chunk_metadata(chunk, ctx, ner_seed, retrieval_summary, rca_frame, stage5_payload = None) .. py:function:: _uniq(items) .. py:function:: build_processed_text_record(*, doc_id, doc_type, chunk_index, chunk, ctx, ner_seed, retrieval_summary, rca_frame, metadata, embedding_text, stage5_payload) .. py:function:: validate_processed_text_record(record) .. py:function:: to_chroma_payload(record)