Types and configuration¶

Dataclasses and helpers matching the clean/suggest request and response payloads. Implementation module: cleancsv.types.

class cleancsv.types.MojibakeMapping(from_, to)[source]¶

Single replacement rule for fixing mojibake in EncodingConfig.

from_¶

Source byte/character sequence to replace (JSON key from).

Type:: str

to¶

Replacement string sent to the API as to.

Type:: str

to_api()[source]¶

Serialize to the API JSON object shape (from / to keys).

Returns:: Dict with keys "from" and "to" suitable for encoding.custom_mojibake_mappings.
Return type:: dict[str, str]

class cleancsv.types.DedupConfig(columns=None, strategy='first', fuzzy=False, threshold=0.85, normalize=False, preview=False, nulls_equal=True)[source]¶

Deduplication options under dedup in CleanConfig.

columns¶

Column names to use for duplicate detection; None means API default.

Type:: list[str] | None

strategy¶

How to pick the surviving row among duplicates.

Type:: Literal[‘first’, ‘last’, ‘most_complete’, ‘merge’]

fuzzy¶

Whether to use fuzzy matching.

Type:: bool

threshold¶

Fuzzy similarity threshold (e.g. 0.85).

Type:: float

normalize¶

Whether to normalize values before comparing.

Type:: bool

preview¶

If true, API may return preview rows without full output.

Type:: bool

nulls_equal¶

Whether null/empty values match each other for dedup.

Type:: bool

to_api()[source]¶

Convert to a JSON-serializable dict for the dedup section of clean config.

Returns:: Mapping ready for inclusion in CleanConfig.to_api_dict().
Return type:: dict[str, Any]

class cleancsv.types.EncodingConfig(encoding='auto', fallback=None, fix_mojibake=True, custom_mojibake_mappings=None, normalize_unicode=True, strip_control_chars=True, strip_invisible=True, fix_bom=True, line_endings='unix')[source]¶

Encoding and text-normalization options under encoding in CleanConfig.

encoding¶

Source encoding name or "auto".

Type:: str

fallback¶

Optional list of encodings to try when detection is ambiguous.

Type:: list[str] | None

fix_mojibake¶

Enable mojibake repair heuristics.

Type:: bool

custom_mojibake_mappings¶

Optional list of MojibakeMapping rules.

Type:: list[cleancsv.types.MojibakeMapping] | None

normalize_unicode¶

If True or a Unicode form name, apply normalization.

Type:: bool | Literal[‘NFC’, ‘NFD’, ‘NFKC’, ‘NFKD’]

strip_control_chars¶

Strip control characters from cell text.

Type:: bool

strip_invisible¶

Strip invisible Unicode characters.

Type:: bool

fix_bom¶

Handle byte-order mark.

Type:: bool

line_endings¶

Normalized line ending style: unix, windows, or auto.

Type:: Literal[‘unix’, ‘windows’, ‘auto’]

to_api()[source]¶

Convert to JSON-serializable dict for the encoding section.

Returns:: Mapping for CleanConfig.to_api_dict().
Return type:: dict[str, Any]

class cleancsv.types.NullColumnConfig(treat_empty_as_null=None, null_values=None, fill=None, cast=None)[source]¶

Per-column null handling under null_handling.per_column.

treat_empty_as_null¶

Whether empty strings count as null for this column.

Type:: bool | None

null_values¶

Extra string values to treat as null.

Type:: list[str] | None

fill¶

Fill strategy (e.g. leave, drop_row, or API-specific tokens).

Type:: str | None

cast¶

Target type for casting: string, int, float, boolean, auto.

Type:: Literal[‘string’, ‘int’, ‘float’, ‘boolean’, ‘auto’] | None

to_api()[source]¶

Emit only keys that are set (omits None fields).

Returns:: Dict fragment for one column in per_column.
Return type:: dict[str, Any]

class cleancsv.types.NullHandlingConfig(null_values=None, treat_empty_as_null=True, type_inference=True, fill='leave', per_column=None)[source]¶

Global and per-column null handling under null_handling in CleanConfig.

null_values¶

Strings treated as null globally.

Type:: list[str] | None

treat_empty_as_null¶

Whether empty cells are null by default.

Type:: bool

type_inference¶

Whether to infer column types.

Type:: bool

fill¶

Default fill strategy for nulls.

Type:: str

per_column¶

Optional map of column name to NullColumnConfig.

Type:: dict[str, cleancsv.types.NullColumnConfig] | None

to_api()[source]¶

Convert to JSON-serializable dict for the null_handling section.

Returns:: Mapping for CleanConfig.to_api_dict().
Return type:: dict[str, Any]

class cleancsv.types.CleanConfig(format='json', dedup=None, encoding=None, null_handling=None, null_representation=None, filter_invalid_emails=False, email_columns=None)[source]¶

Cleaning options sent as the multipart form field config (JSON string).

Corresponds to the CleanCSV upload/config schema: output format, optional dedup, encoding, null_handling, email filters, etc.

format¶

Response format for the clean endpoint: json or csv.

Type:: Literal[‘json’, ‘csv’]

dedup¶

Optional DedupConfig.

Type:: cleancsv.types.DedupConfig | None

encoding¶

Optional EncodingConfig.

Type:: cleancsv.types.EncodingConfig | None

null_handling¶

Optional NullHandlingConfig.

Type:: cleancsv.types.NullHandlingConfig | None

null_representation¶

How nulls appear in output (API-specific string).

Type:: str | None

filter_invalid_emails¶

Drop or flag rows with invalid emails when enabled.

Type:: bool

email_columns¶

Column names to validate as email when filtering is on.

Type:: list[str] | None

to_api_dict()[source]¶

Build the full config object as a plain dict for JSON encoding.

Returns:: Nested dict matching the API’s expected config JSON structure.
Return type:: dict[str, Any]

to_json()[source]¶

Serialize to_api_dict() to a compact JSON string (no extra whitespace).

Returns:: JSON text suitable for the multipart config field.
Return type:: str

classmethod from_suggested_rule(rule, *, format='json')[source]¶

Build a minimal clean config from one SuggestedRule (e.g. after suggest).

Parameters:

rule (SuggestedRule) – A suggested rule from SuggestResponse.
format (Literal['json', 'csv']) – Desired clean output format (json or csv).

Returns:

CleanConfig with dedup populated from the rule’s columns and flags.

Return type:

CleanConfig

class cleancsv.types.SuggestedRule(dedup_by, fuzzy, normalize, reason, threshold=None)[source]¶

One deduplication suggestion from the /suggest API.

dedup_by¶

Column names the API recommends for duplicate detection.

Type:: list[str]

fuzzy¶

Whether fuzzy matching was suggested.

Type:: bool

normalize¶

Whether normalization was suggested.

Type:: bool

reason¶

Short explanation from the API.

Type:: str

threshold¶

Suggested fuzzy threshold, if any.

Type:: float | None

classmethod from_api(raw)[source]¶

Parse a single rule object from API JSON (camelCase keys).

Parameters:: raw (Mapping[str, Any]) – One element from the suggested_rules array.
Returns:: SuggestedRule instance.
Return type:: SuggestedRule

class cleancsv.types.SuggestResponse(row_count, column_count, columns, suggested_rules, null_profiles=None, warnings=None)[source]¶

Structured JSON body from POST .../suggest.

row_count¶

Number of data rows analyzed.

Type:: int

column_count¶

Number of columns.

Type:: int

columns¶

Column names in order.

Type:: list[str]

suggested_rules¶

List of SuggestedRule entries.

Type:: list[cleancsv.types.SuggestedRule]

null_profiles¶

Optional per-column null statistics from the API.

Type:: list[dict[str, Any]] | None

warnings¶

Optional list of warning objects from the API.

Type:: list[dict[str, Any]] | None

classmethod from_api(raw)[source]¶

Parse the top-level suggest response JSON.

Parameters:: raw (Mapping[str, Any]) – Parsed JSON object from the suggest endpoint.
Returns:: SuggestResponse with nested rules converted to dataclasses.
Return type:: SuggestResponse

class cleancsv.types.CleanResult(status_code, content_type, body, headers=<factory>)[source]¶

Raw HTTP result from POST .../clean before parsing structured JSON/CSV.

status_code¶

HTTP status (typically 200 on success).

Type:: int

content_type¶

Normalized primary Content-Type (without parameters).

Type:: str

body¶

Raw response bytes (JSON or CSV depending on config).

Type:: bytes

headers¶

Response headers as a flat mapping.

Type:: Mapping[str, str]

property text: str¶

Decode body as UTF-8 with replacement for invalid sequences.

Returns:: Unicode text of the response body.

parse_json()[source]¶

Parse the body as JSON into CleanResponse when format is JSON.

Returns:: Structured clean response with meta, data, and optional preview fields.
Raises:: ValueError – If Content-Type does not indicate JSON or JSON is invalid.
Return type:: CleanResponse

class cleancsv.types.CleanResponse(meta, data, duplicate_groups=None, preview_source=None, preview_cleaned=None, preview_row_in_output=None, preview_row_removal_reason=None)[source]¶

Parsed JSON body from a successful JSON-format clean operation.

meta¶

Opaque metadata dict from the API (e.g. stats, timing).

Type:: dict[str, Any]

data¶

Cleaned rows as list of column-name to value dicts.

Type:: list[dict[str, Any]]

duplicate_groups¶

Optional duplicate grouping information.

Type:: list[dict[str, Any]] | None

preview_source¶

Optional preview of source rows.

Type:: list[dict[str, Any]] | None

preview_cleaned¶

Optional preview of cleaned rows.

Type:: list[dict[str, Any]] | None

preview_row_in_output¶

Optional flags per preview row.

Type:: list[bool] | None

preview_row_removal_reason¶

Optional reasons for excluded preview rows.

Type:: list[str | None] | None

classmethod from_api(raw)[source]¶

Parse the top-level clean JSON response.

Parameters:: raw (Mapping[str, Any]) – Parsed JSON object from the clean endpoint (JSON format).
Returns:: CleanResponse with optional preview fields set when present.
Return type:: CleanResponse

cleancsv.types.config_from_mapping(m)[source]¶

Build CleanConfig from a plain dict (e.g. loaded from JSON on disk).

Recognizes the same keys as the API: format, dedup, encoding, null_handling, null_representation, filter_invalid_emails, email_columns.

Parameters:: m (Mapping[str, Any]) – Configuration mapping in API snake_case shape.
Returns:: Populated CleanConfig with nested dataclasses where sections exist.
Return type:: CleanConfig