corpus-cleaning-pipeline
description (verbatim frontmatter)
Typed models (Pydantic + FilterReason enum) and verified, profiled performance/library-quality reference notes for an offline LLM-pretraining-corpus cleaning pipeline — schema validation, vectorized quality filters, exact-hash dedup, datasketch-backed MinHash+LSH near-dedup, real tiktoken token counting, a composite quality-ranking score, sharded parquet output. Grounded in the real pipeline.py at organizations/workspaces/projects/2026-07-06-xai-interview-prep/ take-home/submission/ (an xAI Data Engineer take-home submission), not a hypothetical framework — this skill's SKILL.md and models.py cite that file's actual stage names, actual measured thresholds, and its actual library choices. `FilterReason`'s enum members are grepped 1:1 from pipeline.py's real filter-name strings (tests/test_models.py enforces this as a live drift check, not a static claim) — if pipeline.py's filters change, this skill's tests fail until the enum is updated to match. Deliberately does NOT add pydantic as a runtime dependency to pipeline.py itself; this skill is the typed documentation/verification layer alongside it, scope-honestly framed as a DELIBERATE broadening of cwc-data-engineering's charter beyond its original Coworkers session-telemetry domain (user-approved explicitly, not an organic extension — see README.md's "Scope" section). Use when working on corpus/document-cleaning batch pipelines, MinHash/LSH near-dedup, tiktoken-based token accounting, or the xAI take-home submission specifically. Do NOT use for Coworkers board/session telemetry (sibling `session-telemetry-erd` skill's territory) or Cloudflare cache/CDN snapshots (cwc-analytics-engineering).
summary
Typed models (Pydantic + FilterReason enum) and verified, profiled performance/library-quality reference notes for an offline LLM-pretraining-corpus cleaning pipeline — schema validation, vectorized quality filters, exact-hash dedup, datasketch-backed MinHash+LSH near-dedup, real tiktoken token c...
source
plugins/cwc-data-engineering/skills/corpus-cleaning-pipeline/SKILL.md
provenance
created 2026-07-10 18:13:13 · JSON