SCIO Matrix — Data Pipeline

Run each phase in order. Phases 1–2 take seconds. Phase 4 extracts layered semantic knowledge. Phase 4b refines confidence. Phase 5 clusters tag vocabulary. Phase 6 reviews clusters. Phase 7 applies canonical labels.

1

Load Matrix Data

Idle

Decrypt synapse.dat, extract sheet 2, and store all matrix items in the database.

2

Parse Config Groups

Idle

Read dll.config and map all matrix IDs to their source groups (Cardio, Allersode, Spinal, etc.).

4

LLM Semantic Enrichment

Idle

Extract layered semantic knowledge from each item (body/mind/spiritual/environmental/remedy) and store as typed graph edges.

Model:

Sample:

4b

Confidence Refinement

Idle

Resubmit low-confidence tags to stronger models for CONFIRM / RAISE / LOWER / REASSIGN / REMOVE decisions. Full audit trail stored in tag_refinement_history.

Pass 1:

Pass 2:

Pass 3:

Confidence threshold:

Max passes:

Sample:

Community filter:

5

Embed & Cluster

Idle

Embed unique LLM tag values with text-embedding-3-small, then cluster with HDBSCAN per layer to discover canonical vocabulary.

Min cluster size:

5b

LLM Cluster Naming

Idle

Send each HDBSCAN cluster to the LLM to generate a precise canonical label, 1-sentence description, coherence score, and keep/rename/split/reject suggestion. Run this before Phase 6 so the review table shows meaningful names.

Model:

5c

Semantic Deduplication

Idle

Embedding pre-filter skips clearly distinct pairs before touching the LLM. Only pairs with cosine similarity ≥ embed threshold AND LLM confidence ≥ merge threshold are merged. Resume-safe — already-compared pairs are automatically skipped.

Model:

Merge threshold:

Embedding pre-filter:

0.78

cosine-similarity — pairs below this skip the LLM entirely

5d

Vocab Grouping

Idle

Embed Phase 5b cluster descriptions with text-embedding-3-small, then HDBSCAN per layer to build a second-level vocabulary hierarchy — grouping related clusters (e.g. leaky_gut + gut_permeability) into shared vocab themes.

Min group size:

6

Review Clusters

Approve, rename, or reject each discovered cluster. Approved canonical labels will be applied in Phase 7.

Pending: 0Approved: 0Rejected: 0

No pending clusters — run Phase 5 first, or all clusters already reviewed.

7

Apply Normalization

Idle

Push all approved canonical labels back to matrix_item_tag and mark is_normalized = TRUE. A DB backup is taken first; rollback resets only this run's rows.

No approved clusters yet.

8

Community Detection

Idle

Run Leiden graph algorithm on the item co-occurrence network. Items sharing similar tag profiles cluster into thematic communities (liver · hepatic, grief · trauma, digestive · gut, etc.).

Resolution:

Min shared tags:

Stopword cutoff:

max tag freq

9

Community Naming

Idle

Send each Leiden community's items + their enriched tags to the LLM. Generates a clinical name, tagline, main themes, and practitioner insights for every community. Makes the knowledge graph queryable by Corvus.

Model:

9b

Cross-Item Relationships

Idle

Mine directed semantic edges between items within the same community: CAUSES · TREATS · SUPPORTS · DISRUPTS · INDICATES · LEADS_TO · etc. Transforms the knowledge graph from named clusters into a traversable network Corvus can walk to answer 'what causes X?' or 'what treats Y?'

Model:

Min confidence:

9c

Protocol Pattern Mining

Idle

Two-step: (1) association rule mining on the Phase 9b graph — finds statistically significant multi-item co-occurrence patterns ({heavy_metal} → DISRUPTS → {vitamin, mineral}) at zero API cost. (2) LLM names each pattern and explains the clinical mechanism (~$0.10 on Flash).

Model:

9d

Inter-Community Relationships

Idle

Mine directed semantic edges between communities (not just within them). Gives Corvus a chapter-level navigation map: Heavy Metal cluster → DISRUPTS → Methylation cluster, Gut Dysbiosis cluster → CAUSES → Immune Hypersensitivity cluster, etc. ~78 community pairs, ~$0.05 on Flash.

Model:

9e

Emergent Relation Archetypes

Idle

Ask LLM to describe each relationship in free-form mechanism language, then cluster those descriptions with HDBSCAN. The data teaches you its own relationship vocabulary — archetypes like 'biochemical_conversion', 'drainage_cascade', or 'energetic_resonance_bridge' emerge from the graph itself, not from your predefined list.

Model:

9f

Archetype Deduplication

Idle

Phase 9e's emergent archetypes are frequently rediscovered under slightly different names — 'homeopathic_cardiac_frequency', '..._synergy', '..._resonance' are the same mechanism. This bucket-compares archetypes by layer_affinity, pre-filters with embedding cosine similarity, then asks the LLM to confirm true mechanism equivalents vs. genuinely distinct mechanisms that share domain vocabulary. Merges remap item_relation.pattern_id and are fully reversible per run_id.

Model:

Merge threshold:

Embedding pre-filter:

0.8

cosine-similarity — pairs below this skip the LLM entirely

10

Item Knowledge Base

Idle

Generate deep, multi-channel knowledge records for every enriched matrix item using DeepSeek v4 Pro. Pass 1 produces mechanism-level knowledge per category (enzymes, cofactors, depletion chains, crystal/homeopathic/flower-essence fields). Pass 2 adds cross-domain connections, convergence patterns, and quality scores for Corvus pattern-recognition and evidence-convergence reasoning.

Model:

Pass 1

Core knowledge: mechanisms, interactions, syndromes, category-specific fields

Pass 2

Deep connections: cross-domain bridges, convergence patterns, quality scores

Pass 3

Targeted remediation: re-enrich items below quality threshold using their own gap list from Pass 2

Threshold:

— items below this completeness or coherence score are re-enriched

12

Body Map Data Prep

Idle

Two-step: (1) deterministic vertebra pass — regex-extracts spine level (C1-C7/T1-T12/L1-L5/sacrum) from matrix items already literally named that way, zero API cost. (2) LLM classifies every remaining item's organ/body_region/chakra against a closed vocabulary (reuses the existing ORGANS/BODY_REGIONS taxonomy, adds new CHAKRAS/VERTEBRAE). Prepares matrix_item_tag for the future interactive body-map visualization — see docs/BODY_MAP_DATA_PREP.md.

Model:

4c

Anatomy Tag Consistency Check

Idle

Cross-references Phase 12's organ tags against Phase 4's existing body_system classification. Free deterministic pre-filter finds mismatches, then two INDEPENDENT LLM stages (Stage 1: deepseek-v4-pro with max reasoning effort; Stage 2: deepseek-v4-pro standard, not shown Stage 1's verdict) each judge confirm/reassign/remove. Only applied when both stages agree — disagreement is flagged for manual review, never guessed. See docs/BODY_MAP_DATA_PREP.md.

Model:

Live Data Preview

Run a phase to preview data

No data yet.

Enrichment Run History

Each run is isolated by run_id. Rollback deletes only that run's LLM tags and resets those items so they can be re-enriched. Source-confirmed tags (dll_config) and manual overrides are never touched.

No runs yet. Click Refresh after enrichment.

Activity Log