Data Engineer for Text Transformation

Overview We require a Python Data Engineer to complete a technical data transformation and enrichment pipeline. Phase 1 (Raw Data Extraction) is already complete. The raw asset pool is hosted in a Google Drive directory and consists of: Raw German Text Logs: Unified conversational message threads grouped by system category. Binary Attachments: Accompanying technical PDFs, circuit schematics, engineering drawings, and reference images. Your task is to build the secondary processing pipeline to clean, translate, format, and package these raw assets into a structured, English-normalized, production-ready knowledge base optimized for ingestion into downstream RAG (Retrieval-Augmented Generation) architectures. Scope of Work & Deliverables Text Sanitization & Noise Filtering: Write Python processing utilities (Regex/parsers) to systematically strip platform clutter (user signatures, navigation links, layout symbols) and low-value transactional chat, while strictly preserving dense engineering data, supplier/fabricator contact details, and technical references. Technical Translation Pipeline: Construct an automated pipeline leveraging the DeepL API or a local LLM environment to batch-translate raw German strings into precise technical English. You must implement strict checks to ensure alphanumeric part tracking numbers, torque specifications, and physical measurements are preserved with 100% fidelity. Dual-Track Attachment Ingestion: * Track A (Storage): Automatically rename and catalog downloaded PDF/Image binary assets using a uniform naming convention mapping directly to source thread and post IDs. Track B (Enrichment): Parse readable text/tables from documentation binaries and route visual schematics through a Vision LLM (e.g., GPT-4o / Claude 3.5 Sonnet) to generate descriptive metadata summaries. Append these summaries textually inline directly beneath the matching post log. Markdown Transformation & Metadata Injection: Output the finalized English text strings as standard Markdown (.md) files wrapped in explicit, platform-agnostic YAML front-matter blocks tracking URLs, system sub-domains, and archival metadata flags. Logbook Compaction: Construct a concatenation utility to bundle individual thread documents into single macro-domain master logbooks. The script must employ strict Heading 1 (#) formatting layouts to establish clear, native logical chunk boundaries for downstream vector store splitters. Technical Requirements Strong proficiency in Python (Advanced string manipulation, JSON manipulation, layout-aware data structures). Experience with layout-preserving text extraction tools and PDF libraries (e.g., Docling, Unstructured, OcrMyPdf). Direct experience orchestrating prompt logic or data flows with Vision-Language Models (VLMs) and translation APIs. Solid structural understanding of RAG ingestion design patterns, data chunking strategies, and metadata indexing principles.