Transformation Probes

Experimental. API may change.

Two probes for measuring transformation effects:

• compare() - Measure differences between before/after datasets
• profile_relationship() - Measure JOIN structure between two datasets

Probes return structured measurements. They do not interpret results or suggest fixes.

---

compare(before, after, key)

Measures what changed between two datasets.

import kontra

result = kontra.compare(
    before=raw_df,
    after=transformed_df,
    key="order_id",  # or ["col1", "col2"] for composite
)

Same-named or different-named keys

Use key= when the identifying column has the same name on both sides. When the two sides name it differently — the common FK→PK case — use before_key= and after_key= instead:

# before uses "organization_id", after uses "id"
result = kontra.compare(
    tickets, orgs,
    before_key="organization_id",
    after_key="id",
)

Provide exactly one of key or the before_key/after_key pair; passing both raises ValueError. For composite keys, pass lists — before_key and after_key are paired positionally and must have the same number of columns.

Any source vs any source

before and after are resolved through the same connectors the validation engine uses, so each side can be any supported source and the two sides can be different kinds — compare a database table to a Parquet file, a file to a DataFrame, a named datasource to a live connection, and so on:

# Database table  vs  file
kontra.compare("postgres:///public.orders", "s3://lake/orders.parquet", key="id")

# Named datasource  vs  DataFrame
kontra.compare("prod_db.users", staging_df, key="user_id")

# Bring-your-own connection  vs  file  (pass the table for the connection side)
kontra.compare(conn, "./orders.csv", key="id", before_table="public.orders")

Accepted sources: Polars/pandas DataFrame, list-of-dicts, file/cloud path (.parquet, .csv, s3://, abfss://), database URI (postgres://…/schema.table, mssql://…/schema.table, clickhouse://…/db/table), named datasource (prod_db.users), or a live database connection object. Both sides are fully materialized before comparison; database extras (kontra[postgres] / kontra[sqlserver]) are required for the database sources.

Parameters

Parameter	Type	Description
before	any source	Dataset before transformation
after	any source	Dataset after transformation
key	str or list[str]	Column(s) identifying rows (same name on both sides)
before_key	str or list[str]	Key column(s) on before (use with after_key for different-named keys)
after_key	str or list[str]	Key column(s) on after (use with before_key)
before_table	str	Table ref when before is a DB connection object
after_table	str	Table ref when after is a DB connection object
sample_limit	int	Max samples per category (default: 5)

Output Schema

result.to_dict()  # Returns:
{
  "meta": {
    "before_rows": 1000,
    "after_rows": 1200,
    "key": ["order_id"],
    "execution_tier": "polars"
  },
  "row_stats": {
    "delta": 200,      # after_rows - before_rows
    "ratio": 1.2       # after_rows / before_rows
  },
  "key_stats": {
    "unique_before": 1000,
    "unique_after": 1000,
    "preserved": 1000,     # keys in both
    "dropped": 0,          # keys in before only
    "added": 0,            # keys in after only
    "duplicated_after": 50 # keys appearing >1x in after
  },
  "change_stats": {
    "unchanged_rows": 800,
    "changed_rows": 200
  },
  "column_stats": {
    "added": ["new_col"],
    "removed": [],
    "modified": ["amount"],
    "modified_fraction": {"amount": 0.15},
    "nullability_delta": {
      "amount": {"before": 0.0, "after": 0.12}
    }
  },
  "samples": {
    "duplicated_keys": ["A123", "B456"],
    "dropped_keys": [],
    "changed_rows": [
      {"key": "A123", "before": {"amount": 100}, "after": {"amount": 200}}
    ]
  }
}

Key Fields

Field	Meaning
row_stats.delta	Change in row count
row_stats.ratio	Ratio of after/before rows
key_stats.preserved	Keys present in both datasets
key_stats.dropped	Keys lost in transformation
key_stats.added	New keys in after
key_stats.duplicated_after	Count of keys appearing more than once in after
change_stats.changed_rows	Rows where non-key columns differ
column_stats.modified_fraction	Per-column: fraction of rows where value changed

Property Access

result = kontra.compare(before, after, key="user_id")

# Direct attributes
result.before_rows           # 1000
result.after_rows            # 1200
result.row_delta             # 200
result.row_ratio             # 1.2
result.preserved             # 1000
result.dropped               # 0
result.added                 # 0
result.duplicated_after      # 50
result.changed_rows          # 200
result.unchanged_rows        # 800
result.columns_added         # ["new_col"]
result.columns_removed       # []
result.columns_modified      # ["amount"]
result.modified_fraction     # {"amount": 0.15}

# Samples
result.samples_duplicated_keys   # ["A123", "B456"]
result.samples_dropped_keys      # []
result.samples_changed_rows      # [{"key": ..., "before": ..., "after": ...}]

# Output formats
result.to_dict()   # Nested dict
result.to_json()   # JSON string
result.to_llm()    # Compact text for LLM context

---

profile_relationship(left, right, on)

Measures the structural relationship between two datasets on a join key.

import kontra

profile = kontra.profile_relationship(
    left=orders,
    right=customers,
    on="customer_id",  # or ["col1", "col2"] for composite
)

Use on= when the join key has the same name on both sides. When the sides name it differently — the common FK→PK case — use left_on= and right_on= instead (mirroring pandas' merge naming):

# left uses "organization_id", right uses "id"
profile = kontra.profile_relationship(
    tickets, orgs,
    left_on="organization_id",
    right_on="id",
)

Provide exactly one of on or the left_on/right_on pair; passing both raises ValueError. Composite keys are paired positionally and must match in arity.

left and right accept any source compare() does (DataFrame, file/cloud path, database URI, named datasource, or a live connection with left_table/right_table), mixed freely.

Parameters

Parameter	Type	Description
left	any source	Left dataset
right	any source	Right dataset
on	str or list[str]	Column(s) to join on (same name on both sides)
left_on	str or list[str]	Join key column(s) on left (use with right_on for different-named keys)
right_on	str or list[str]	Join key column(s) on right (use with left_on)
left_table	str	Table ref when left is a DB connection object
right_table	str	Table ref when right is a DB connection object
sample_limit	int	Max samples per category (default: 5)

Output Schema

profile.to_dict()  # Returns:
{
  "meta": {
    "on": ["customer_id"],
    "left_rows": 10000,
    "right_rows": 500,
    "execution_tier": "polars"
  },
  "key_stats": {
    "left": {
      "null_rate": 0.0,
      "unique_keys": 10000,
      "duplicate_keys": 0,
      "rows": 10000
    },
    "right": {
      "null_rate": 0.02,
      "unique_keys": 450,
      "duplicate_keys": 50,
      "rows": 500
    }
  },
  "cardinality": {
    "left_key_multiplicity": {"min": 1, "max": 1},
    "right_key_multiplicity": {"min": 1, "max": 3}
  },
  "coverage": {
    "left_keys_with_match": 9800,
    "left_keys_without_match": 200,
    "right_keys_with_match": 450,
    "right_keys_without_match": 0
  },
  "samples": {
    "left_keys_without_match": ["C991", "C882"],
    "right_keys_without_match": [],
    "right_keys_with_multiple_rows": ["C123", "C456"]
  }
}

Key Fields

Field	Meaning
key_stats.left.unique_keys	Distinct key values in left
key_stats.right.duplicate_keys	Keys appearing >1x in right
cardinality.left_key_multiplicity.max	Maximum rows per key in left
cardinality.right_key_multiplicity.max	Maximum rows per key in right
coverage.left_keys_with_match	Left keys that exist in right
coverage.left_keys_without_match	Left keys not in right

Property Access

profile = kontra.profile_relationship(orders, customers, on="customer_id")

# Direct attributes
profile.left_rows              # 10000
profile.right_rows             # 500
profile.left_unique_keys       # 10000
profile.right_unique_keys      # 450
profile.left_duplicate_keys    # 0
profile.right_duplicate_keys   # 50
profile.left_null_rate         # 0.0
profile.right_null_rate        # 0.02

# Cardinality
profile.left_key_multiplicity_min    # 1
profile.left_key_multiplicity_max    # 1
profile.right_key_multiplicity_min   # 1
profile.right_key_multiplicity_max   # 3

# Coverage
profile.left_keys_with_match      # 9800
profile.left_keys_without_match   # 200
profile.right_keys_with_match     # 450
profile.right_keys_without_match  # 0

# Samples
profile.samples_left_unmatched     # ["C991", "C882"]
profile.samples_right_unmatched    # []
profile.samples_right_duplicates   # ["C123", "C456"]

# Output formats
profile.to_dict()   # Nested dict
profile.to_json()   # JSON string
profile.to_llm()    # Compact text for LLM context

---

Notes

Database probes materialize data

The Python compare() and profile_relationship() implementations do not currently push probe computation into the database. They materialize both inputs into Polars; compare() loads every column, and duplicate keys can expand intermediate joins substantially. Samples may contain raw key or row values. Use sample_limit=0 for sensitive data and treat these probes as bounded investigations, not unrestricted scans of very large tables.

The official MCP narrows this surface: sources must be configured names, samples are forced off, keys are capped at eight columns, and a metadata row-count preflight guards materialization. Catalog estimates can be stale, so that ceiling is a cost guardrail rather than a security boundary.

• Probes measure structure. They do not interpret correctness.
• duplicated_after counts keys (not rows) appearing more than once.
• modified_fraction is computed only for preserved keys.
• NULL handling: NULLs in join keys are excluded from unique counts.
• Samples are bounded and explanatory only. They do not affect counts.