Data Engineer (Python, PySpark, SQL, ETL, AWS S3) – Long-Term Opportunity

Data Engineer – Job Support (2 Hours/Day) Position Overview We are seeking an experienced Data Engineer to provide ongoing technical support and guidance for enterprise-level data engineering projects. This is a long-term remote engagement requiring approximately 2 hours of support per day. The ideal candidate will have strong expertise in Python, PySpark, SQL, data modeling, ETL/ELT pipelines, and modern data architecture. The role involves troubleshooting, code reviews, performance optimization, solution design, and mentoring on real-world data engineering challenges. Key Responsibilities Data Development * Design, develop, and maintain scalable, secure, and high-performance data solutions. * Build and optimize data pipelines for large-scale data processing. * Develop robust ELT/ETL workflows to support business and analytical requirements. Task Automation & Workflow Optimization * Automate complex data workflows and business processes. * Improve operational efficiency through workflow optimization and reusable solutions. * Identify opportunities to streamline data processing and reporting activities. Rapid Delivery & Continuous Improvement * Follow an MVP-first approach with iterative enhancements. * Support rapid solution delivery through quick iterations, testing, and validation. * Participate in proof-of-concept (POC) development and technical evaluations. Research & Innovation * Stay updated with emerging technologies and best practices in data engineering. * Explore new frameworks, methodologies, and tools to improve platform capabilities. * Contribute innovative ideas for improving scalability, reliability, and performance. Collaboration & Technical Guidance * Work closely with cross-functional teams, including developers, analysts, and architects. * Provide technical support, troubleshooting, and mentoring. * Participate in design discussions and solution reviews. Documentation & Governance * Create and maintain technical documentation for solutions, processes, and workflows. * Follow organizational standards for security, compliance, and operational stability. * Assess technical risks and ensure adherence to engineering best practices. --- Required Qualifications * 5+ years of hands-on experience in Data Engineering or Software Engineering. * Strong proficiency in Python and PySpark. * Advanced SQL development skills, including Stored Procedures and query optimization. * Strong understanding of Data Modeling concepts and best practices. * Experience building and maintaining ELT/ETL data pipelines. * Hands-on experience optimizing Spark jobs and Spark clusters. * Experience working with AWS S3 and/or NAS storage systems. * Strong understanding of application and data architecture patterns. * Experience with: * Git Version Control * CI/CD Pipelines * Code Reviews * Unit and Integration Testing * Agile Development Methodologies * Excellent analytical, troubleshooting, and problem-solving abilities. * Strong verbal and written communication skills. Preferred Qualifications * Experience in the Banking, Financial Services, or Finance domain. * Hands-on experience with ETL tools such as: * Apache Airflow * IBM DataStage * Talend * SSIS * Experience with BI and Reporting tools such as Tableau. * Experience with containerization technologies: * Docker * Kubernetes * OpenShift * Exposure to AI/ML technologies, Machine Learning frameworks, and Large Language Models (LLMs). Education * Bachelor's Degree in Computer Science, Information Technology, Engineering, or a related field. * Equivalent industry experience will also be considered. Engagement Details * Role Type: Job Support / Technical Mentoring * Duration: Long-Term * Work Mode: Remote * Availability: 2 Hours per Day * Start Date: Immediate * Time Zone: Flexible (with overlap as required) Answer these as well: 1. How many years of experience do you have with Python and PySpark? 2. Have you worked on Spark optimization and performance tuning? 3. Describe a large-scale ETL/ELT pipeline you have built. 4. Which ETL tools have you worked with? 5. Do you have experience with AWS S3 and cloud-based data platforms? 6. Have you worked in Banking or Financial Services projects? 7. What experience do you have with Docker, Kubernetes, or OpenShift? 8. Have you worked with AI/ML or LLM-based solutions? 9. Are you available for 2 hours of daily support?