src.dackar.RCA.doc_extraction ============================= .. py:module:: src.dackar.RCA.doc_extraction Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/src/dackar/RCA/doc_extraction/adapter/index /autoapi/src/dackar/RCA/doc_extraction/epistemics/index /autoapi/src/dackar/RCA/doc_extraction/schema/index /autoapi/src/dackar/RCA/doc_extraction/store/index Attributes ---------- .. autoapisummary:: src.dackar.RCA.doc_extraction.EXTRACTABLE_DOC_TYPES Exceptions ---------- .. autoapisummary:: src.dackar.RCA.doc_extraction.DocExtractionStoreError src.dackar.RCA.doc_extraction.EmbeddingModelVersionError Classes ------- .. autoapisummary:: src.dackar.RCA.doc_extraction.ConfidenceLevel src.dackar.RCA.doc_extraction.DocExtractionRecord src.dackar.RCA.doc_extraction.DocExtractionAdapter src.dackar.RCA.doc_extraction.DocExtractionStore src.dackar.RCA.doc_extraction.SemanticMatch src.dackar.RCA.doc_extraction.EpistemicClassifier src.dackar.RCA.doc_extraction.EpistemicsRoutingConfig Functions --------- .. autoapisummary:: src.dackar.RCA.doc_extraction.build_epistemics_manifest_summary Package Contents ---------------- .. py:class:: ConfidenceLevel Bases: :py:obj:`str`, :py:obj:`enum.Enum` str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'. .. py:attribute:: HIGH :value: 'high' .. py:attribute:: MEDIUM :value: 'medium' .. py:attribute:: LOW :value: 'low' .. py:class:: DocExtractionRecord One extraction record per identified causal chain within a source document. A single CR/WO may produce multiple records (one per causal chain). Records with all semantic fields null are stored but flagged for human review. `fm_id_candidate` is null at ingestion time; resolved via batch KG lookup at RCA run time (§3.3 Step C). `embedding_model_version` is null until the record is embedded in Stage 2. .. py:attribute:: doc_id :type: str .. py:attribute:: chain_index :type: int .. py:attribute:: identified_effect :type: Optional[str] .. py:attribute:: assessed_cause :type: Optional[str] .. py:attribute:: inferred_fm_label :type: Optional[str] .. py:attribute:: fm_id_candidate :type: Optional[str] .. py:attribute:: fm_id_candidate_alt :type: Optional[str] .. py:attribute:: confidence :type: ConfidenceLevel .. py:attribute:: cause_is_symptom :type: bool .. py:attribute:: as_found :type: Optional[str] .. py:attribute:: as_left :type: Optional[str] .. py:attribute:: procedural_deviation_score :type: float .. py:attribute:: extraction_version :type: str .. py:attribute:: embedding_model_version :type: Optional[str] .. py:attribute:: needs_human_review :type: bool :value: False .. py:attribute:: ruled_out_mechanisms :type: List[str] :value: [] .. py:attribute:: event_time_start :type: Optional[datetime.datetime] :value: None .. py:attribute:: event_time_end :type: Optional[datetime.datetime] :value: None .. py:attribute:: event_time_confidence :type: Optional[str] :value: None .. py:attribute:: source_cr_id :type: Optional[str] :value: None .. py:attribute:: source_wo_id :type: Optional[str] :value: None .. py:attribute:: source_event_id :type: Optional[str] :value: None .. py:attribute:: fm_resolution_status :type: Optional[str] :value: None .. py:attribute:: fm_resolution_score :type: Optional[float] :value: None .. py:attribute:: doc_type :type: str :value: '' .. py:attribute:: finding_status :type: Optional[str] :value: None .. py:attribute:: authority_level :type: Optional[str] :value: None .. py:attribute:: epistemic_class :type: Optional[str] :value: None .. py:attribute:: classification_resolution_level :type: Optional[str] :value: None .. py:attribute:: degraded_classification :type: bool :value: False .. py:method:: embed_text() Concatenation of semantic fields used for embedding (null-safe). .. py:method:: is_null_record() Return True when no semantic content was extracted (all three core fields empty). Null records are the sentinel produced for a document with no extractable causal language; they are stored and retrievable by metadata but never surface in similarity queries (their embed_text is a single space). .. py:method:: is_recurrence_eligible() Return True when this record may contribute to recurrence counting. Records with fm_resolution_status == "ambiguous" require analyst promotion before being eligible; all others (auto_resolved, unresolved, or not yet resolved) are treated as eligible by default. Phase C will add an additional gate: records whose epistemic_class is not "analyzes_past_degradation" will be ineligible regardless of fm_resolution_status. That gate is intentionally off here to avoid a scoring change before calibration. .. py:method:: as_chroma_metadata() Flat metadata dict for Chroma upsert (all values must be str/int/float/bool). .. py:class:: DocExtractionAdapter(ner_pipeline, nlp = None, llm_cfg = None, extraction_version = 'ner-v1.0_gaz-v1_llm-none', _causal_extractor = None) Wraps HybridNERPipeline + causal_condition_adapter to produce DocExtractionRecords. Produces one record per identified causal chain in the document. A document with no extractable causal language produces one null record (confidence=low, needs_human_review=True). fm_id_candidate is always None at extraction time; resolved via batch KG lookup at RCA run time (see DocExtractionStore.resolve_fm_candidates). .. py:attribute:: _ner .. py:attribute:: _nlp :value: None .. py:attribute:: _llm_cfg :value: None .. py:attribute:: extraction_version :value: 'ner-v1.0_gaz-v1_llm-none' .. py:attribute:: __causal_extractor :value: None .. py:method:: _get_causal_extractor() .. py:method:: extract(doc_id, text, doc_type, section_role = 'body') Extract all causal chains from a single document text. :param doc_id: Source document identifier (e.g. "CR-2026-00123"). :param text: Full document text (single chunk; multi-chunk handling is a future extension). :param doc_type: Must be one of EXTRACTABLE_DOC_TYPES. :param section_role: Hint for condition-state extraction ("body", "as_found", "as_left", etc.). :returns: List of DocExtractionRecord, one per causal chain. Never empty — a document with no causal language returns a single null record. :raises ValueError: if doc_type is not in EXTRACTABLE_DOC_TYPES. .. py:method:: _null_record(doc_id, as_found, as_left, proc_score) .. py:data:: EXTRACTABLE_DOC_TYPES .. py:class:: DocExtractionStore(persist_directory, embed_model = 'nomic-embed-text-v1.5', ollama_base_url = None, fm_resolution_threshold = 0.88, epistemics_classifier = None) Chroma-backed store for DocExtractionRecord objects. One collection (``"doc_extractions"``) stores all extraction records across all document types. Each record's embed_text (§4.1) is computed and stored at upsert time; similarity queries run against these pre-computed vectors. Key guarantees: - Embedding model version is written into every record's metadata. A mismatch between query-time model and stored model raises EmbeddingModelVersionError. - fm_id_candidate is always null at ingestion; resolve_fm_candidates() writes it back in batch at the start of an RCA run (§3.3 Step C). - query() deduplicates by doc_id, returning only the best-scoring chain per document. .. py:attribute:: COLLECTION_NAME :value: 'doc_extractions' .. py:attribute:: _HNSW_SPACE :value: 'cosine' .. py:attribute:: persist_directory .. py:attribute:: embed_model :value: 'nomic-embed-text-v1.5' .. py:attribute:: ollama_base_url .. py:attribute:: fm_resolution_threshold :value: 0.88 .. py:attribute:: epistemics_classifier :value: None .. py:attribute:: _collection :value: None .. py:attribute:: _embedder :value: None .. py:attribute:: _degraded_operations :type: List[Dict[str, Any]] :value: [] .. py:property:: embedding_model_version :type: str .. py:method:: _get_embedder() .. py:method:: _get_collection() .. py:method:: _embed_texts(texts) .. py:method:: _embed_query(text) .. py:method:: _chroma_collection() .. py:method:: upsert(record) Embed and store one extraction record. Returns the Chroma record_id. Records with no embed_text are stored with a single-space document to avoid Chroma rejecting empty strings; they are retrievable by metadata but will not surface in similarity queries. .. py:method:: upsert_batch(records) Embed and store multiple extraction records in one batch call. .. py:method:: query(query_text, *, top_k = 5, similarity_threshold = 0.75, near_match_window = 0.1, filter_meta = None, exact_doc_ids = None) Query for semantically similar extraction records. The embedding model used at query time must match the collection's stored ``embedding_model_version``. A mismatch raises EmbeddingModelVersionError. Deduplication: only the highest-scoring chain per doc_id is returned. :param query_text: The query string (e.g. ``fm.name | fm.expected_symptoms | event.symptom_description``). :param top_k: Maximum number of doc_id-deduplicated results to return. :param similarity_threshold: Minimum cosine similarity for inclusion in the main result set. :param near_match_window: Width of the soft zone below threshold that populates near_matches. :param filter_meta: Optional Chroma metadata pre-filter (e.g. ``{"doc_type": "CR"}``). :param exact_doc_ids: Set of doc_ids already counted via exact-match recurrence (kg_context.past_events). Matching records are excluded from both matches and near_matches to prevent double-counting. None or empty set disables the guard. :returns: - matches: similarity >= similarity_threshold, deduplicated, top_k max, exact_doc_ids excluded - near_matches: similarity in [similarity_threshold - near_match_window, similarity_threshold) :rtype: (matches, near_matches) where .. py:method:: _assert_model_version() Raise EmbeddingModelVersionError if stored records use a different model. .. py:method:: resolve_fm_candidates(fm_list, *, resolution_threshold = None) Resolve fm_id_candidate for unresolved extraction records. Called once per RCA run before Step 3 / Step 2d. For each record with fm_id_candidate == "" (unresolved), embeds the stored inferred_fm_label and compares against the KG FM list for the current asset neighborhood. Writes fm_id_candidate (and fm_id_candidate_alt) back to the collection if cosine similarity >= resolution_threshold. :param fm_list: List of (fm_id, fm_label) tuples from the KG. :param resolution_threshold: Override default; defaults to self.fm_resolution_threshold. :returns: Number of records updated. .. py:method:: count() Return total number of extraction records in the collection. .. py:method:: delete_by_doc_id(doc_id) Delete all extraction records for a given source document (re-ingestion path). .. py:method:: _record_degradation(operation, exc) Record a degraded (error-swallowed) operation for run-manifest surfacing. query() and resolve_fm_candidates() degrade to empty/zero results on backend failure rather than crashing a batch run; each such event is captured here so store_health_summary() can report it and the failure is not misread downstream as a legitimate "no semantic recurrence". .. py:property:: degraded :type: bool True when any query/resolve operation swallowed a backend error this run. .. py:method:: store_health_summary() Degradation summary for run_manifest (store_health section). Returns ``{"degraded": bool, "degraded_operation_count": int, "events": [...]}``. The orchestrator stamps this onto the run manifest so a swallowed Chroma/embedding error surfaces as an explicit degraded-run signal instead of a silent under-count. .. py:class:: SemanticMatch One deduplicated result from a DocExtractionStore.query() call. .. py:attribute:: record_id :type: str .. py:attribute:: doc_id :type: str .. py:attribute:: chain_index :type: int .. py:attribute:: identified_effect :type: Optional[str] .. py:attribute:: assessed_cause :type: Optional[str] .. py:attribute:: inferred_fm_label :type: Optional[str] .. py:attribute:: fm_id_candidate :type: Optional[str] .. py:attribute:: confidence :type: src.dackar.RCA.doc_extraction.schema.ConfidenceLevel .. py:attribute:: cause_is_symptom :type: bool .. py:attribute:: similarity_score :type: float .. py:attribute:: fm_resolution_status :type: Optional[str] :value: None .. py:attribute:: doc_type :type: str :value: '' .. py:attribute:: finding_status :type: Optional[str] :value: None .. py:attribute:: authority_level :type: Optional[str] :value: None .. py:attribute:: epistemic_class :type: Optional[str] :value: None .. py:attribute:: classification_resolution_level :type: Optional[str] :value: None .. py:attribute:: degraded_classification :type: bool :value: False .. py:property:: confidence_weight :type: float Numeric weight for this match's confidence level (HIGH=1.0, MEDIUM=0.7, LOW=0.3). Used as a multiplier in semantic_contribution (§4.3) so lower-confidence extractions contribute proportionally less to effective_recurrence_count. .. py:property:: cause_is_symptom_factor :type: float Down-weight factor when the assessed cause is itself a symptom (0.5) vs. a mechanism (1.0). A symptom-as-cause is a weaker recurrence signal than a true failure mechanism, so it halves this match's semantic_contribution. .. py:property:: semantic_contribution :type: float Fractional recurrence contribution for effective_recurrence_count (§4.3). .. py:exception:: DocExtractionStoreError Bases: :py:obj:`RuntimeError` Base error for DocExtractionStore backend failures (Chroma / embedding backend). query() and resolve_fm_candidates() deliberately degrade to empty/zero results on backend failure rather than raising, recording each event via _record_degradation() so it surfaces in the run manifest (see store_health_summary()). Callers that prefer fail-loud semantics can inspect store_health_summary()["degraded"] and escalate. .. py:exception:: EmbeddingModelVersionError Bases: :py:obj:`DocExtractionStoreError` Raised when the query-time embedding model does not match the collection's stored model. .. py:class:: EpistemicClassifier(config = None) Applies the four-way epistemic classification to a document record. Usage ----- config = EpistemicsRoutingConfig(policy_version="epistemics-v1.0") classifier = EpistemicClassifier(config) annotation = classifier.classify(meta) ``meta`` may be a dict (Chroma metadata), a DocExtractionRecord, or a SemanticMatch — any object that exposes the relevant fields via attribute or dict access. Priority chain (§3.3) ---------------------- 1. finding_status — semantic; not degraded 2. authority_level — semantic; not degraded 3. doc_type — syntactic proxy; degraded_classification = True 4. default — no metadata; degraded_classification = True .. py:attribute:: config .. py:attribute:: _doc_type_table .. py:method:: classify(record) Classify one record and return an EpistemicAnnotation. ``record`` may be: - a dict (Chroma metadata dict) - a DocExtractionRecord - a SemanticMatch - any object with attribute access for the relevant fields .. py:method:: annotate_record(record) Classify ``record`` in-place, writing annotation fields back to it. Supports DocExtractionRecord and SemanticMatch (both have the three annotation fields as attributes). No-ops silently on other types. .. py:method:: _resolve_finding_status(finding_status, doc_type) Look up (finding_status, doc_type) in the routing table. Tries exact (finding_status, doc_type) first, then (finding_status, "") as the doc_type-agnostic fallback. Returns None if not found. .. py:class:: EpistemicsRoutingConfig Versioned configuration artifact for the epistemic routing table. policy_version is stamped on run_manifest.pipeline_config and on every EpistemicAnnotation so that routing decisions are reproducible. The routing tables are embedded in epistemics.py and versioned via policy_version; changing any table entry requires a version bump. .. py:attribute:: policy_version :type: str :value: 'epistemics-v1.0' .. py:attribute:: doc_type_overrides :type: Dict[str, str] .. py:function:: build_epistemics_manifest_summary(cross_pattern_evidence, policy_version) Build the epistemics section of run_manifest.artifacts (Phase A). Counts epistemic_class distribution, degraded_classification counts by doc_type, and classification_resolution_level distribution across all doc extractions visible to the pipeline. Sourced from cross_pattern_evidence all_links provenance when available; otherwise returns a minimal stub. This function is called by the orchestrator's _stage_g_finalize_manifest(). It is defined here (not in the orchestrator) so that it can be unit-tested independently without the orchestrator's heavy kg dependencies.