Hire the Best Pyspark Developers
in Brazil

You probably think clicking "deploy" on Databricks from the cloud marketplace is all it takes to build a modern data stack. Instead, you get unmanageable infrastructure, skyrocketing costs, and pipelines feeding reports nobody trusts. 𝗜 𝗳𝗶𝘅 𝘁𝗵𝗮𝘁. 𝗡𝗼 𝗮𝗴𝗲𝗻𝗰𝗶𝗲𝘀, 𝗻𝗼 𝗯𝗹𝗼𝗮𝘁. Just a multi-certified, 5+ years of experience Cloud Solutions Architect building automated, high-integrity platforms that turn raw data into a competitive advantage. If you shoot me a invitation or message I'll send you a personalized Loom video back on how I may be able to help you; and of course, to prove that I'm the real deal, 𝗻𝗼 𝗔𝗜 𝗶𝗻𝘃𝗼𝗹𝘃𝗲𝗱! Whether you are building a greenfield lakehouse from scratch or migrating legacy systems to the cloud, I architect efficient, cost-effective environments that scale without the overhead. I understand the business bottom line just as well as the underlying code. ✪ 100% Job Success Score | 5.0★ average ✪ Proven experience on multi-cloud architectures 💡 𝗪𝗵𝗮𝘁 𝗜 𝗱𝗼: • 𝗗𝗮𝘁𝗮 𝗣𝗹𝗮𝘁𝗳𝗼𝗿𝗺 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴: I build production-ready environments using Terraform. No manual marketplace or standard deployments that break at scale. • 𝗥𝗲𝗹𝗶𝗮𝗯𝗹𝗲 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴: Raw data becomes actionable. I build resilient Medallion architectures and automated ETL/ELT pipelines so your stakeholders actually trust the numbers. • 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗠𝗟𝗢𝗽𝘀: I bridge the gap between data engineering and machine learning. Using MLflow and Databricks Model Serving, I operationalize models into scalable, real-time REST endpoints and automated streaming inference pipelines. • 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 & 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆: Proper data governance utilizing Unity Catalog (no legacy Hive metastores) to ensure your data is accessible, secure, and future-proof. • 𝗖𝗹𝗼𝘂𝗱 𝗖𝗼𝘀𝘁 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻: Most companies overspend on cloud infrastructure. I architect systems that pay for themselves in weeks by eliminating overhead and inefficiencies with efficient auditing and monitoring features. ✅ 𝗖𝗲𝗿𝘁𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀 (𝘃𝗲𝗿𝗶𝗳𝗶𝗲𝗱): • Databricks Professional Data Engineer • Databricks Associate Data Engineer • Databricks Lakehouse Fundamentals • GCP Professional Data Engineer • GCP Associate Cloud Engineer • GCP Cloud Digital Leader • AWS Associate Solutions Architect • AWS Cloud Practitioner 🔧 𝗘𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲 𝘄𝗶𝘁𝗵 𝗖𝗹𝗼𝘂𝗱 𝗦𝗲𝗿𝘃𝗶𝗰𝗲𝘀: • 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀: Workflows, LDP (Lakeflow Declarative Pipelines), Unity Catalog, Workflows, Databricks SQL, MLFlow. • 𝗔𝗺𝗮𝘇𝗼𝗻 𝗪𝗲𝗯 𝗦𝗲𝗿𝘃𝗶𝗰𝗲 (𝗔𝗪𝗦): EMR, Athena, Redshift, Glue, S3, RDS, Kinesis Data Firehose, Kinesis, and Data Streams. • 𝗚𝗼𝗼𝗴𝗹𝗲 𝗖𝗹𝗼𝘂𝗱 𝗣𝗹𝗮𝘁𝗳𝗼𝗿𝗺 (𝗚𝗖𝗣): Bigquery, Dataform, Composer, Dataflow, Dataproc, Cloud Storage, Pub/Sub, Cloud Functions, and Looker Studio. • 𝗠𝗶𝗰𝗿𝗼𝘀𝗼𝗳𝘁 𝗔𝘇𝘂𝗿𝗲: Data Factory, Synapse, and Storage Account. • 𝗢𝘁𝗵𝗲𝗿𝘀: Terraform, dbt, Airflow, Airbyte, Hadoop, and Hive. ⚙️ 𝗖𝗼𝗿𝗲 𝗲𝘅𝗽𝗲𝗿𝘁𝗶𝘀𝗲: • 𝗥𝗼𝗹𝗲𝘀: Data Architect, Data Engineer, Solutions Architect, Platform Engineer • 𝗣𝗹𝗮𝘁𝗳𝗼𝗿𝗺𝘀: Databricks (Delta Lake, Unity Catalog, Lakeflow, Workflows), BigQuery • 𝗜𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲: Infrastructure as Code (IaC), Terraform, Multi-Cloud (AWS, GCP, Azure) • 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲: Medallion Architecture, Data Lakehouse, Data Governance, Data Quality, Machine Learning • 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴: PySpark, Python, SQL, dbt, Apache Airflow, ETL/ELT, CDC, Batch and Stream Processing

Hi there, thanks for considering me for your project! 👋 My name is Paulo, a Computer Science graduate from Brazil with 2.5 years of experience in Data Engineering using Python. With a strong background in algorithms, mathematics, and logic, I’ll quickly adapt to new tools and technologies to find the best solutions for your needs. 📚 🇧🇷 What I can do for you: • Web Scraping – Extract and process data from websites 🔎 • Develop ETL & Data Pipelines – Automate data collection and transformation 🔄 • API Integrations – Connect and exchange data efficiently 🔌 • Train and Deploy Machine Learning Models – Build intelligent, data-driven solutions 🤖 I’ll break down technical concepts in an intuitive and easy-to-understand way. 💡 My work is driven by commitment, dedication, and professionalism. 👨🏾‍💻 Available in your time zone for smooth collaboration. ⌚️

I have almost 5 years of experience working with data, having started as a data analyst, then data scientist, and for the past 4 years, as a data engineer. I've worked in many projects related to API consumption, web scraping and automation. I work with Python, SQL, PySpark, and AWS services like Glue, S3, Redshift, and IAM. Also have got experience with Azure services like ADLS and Functions. I have hands-on experience building scalable, production-grade pipelines using the medallion architecture, with automation and orchestration through serverless services. Experience with infrastructure-as-code practices using Terraform. I'm passionate about clean engineering, automation, and creating end-to-end data solutions that drive business value. Related to web scraping these are the tools I use depending on the need: requests, selenium, bs4, playwright. To deal with anti-bot we can always use proxies (that we can gather for free and I do have a database for that already), user-agents and cookies to mimic human like behavior. We scrape the data as json, html, xml or plain text and turn them into structured data as an EXCEL file, csv, database etc.

I am a Software Engineer with 8 year of experience in developing applications. I already work with a lot of technologies including: VB.NET, C#, Asp. Net Web Forms, Asp.Net Web Api, Asp.Net MVC, JQuery, Angular JS, Angular, Entity Framework, NHibernate, Delphi. I have experience in both backend and front-end development. I have a lot experience in T-SQL (Sql Server), but I also have some knowledge in Oracle, MySql,Postgre and MongoDB. For control version I already work with Git, Bitbucket and TFS. Recently I've shifted my career to work with Big Data, I'm finishing my thesis on Data Science and Advanced Analytics and I'm currently working as a Big Data Engineer.

I build AI systems that go beyond prototypes — production-ready, well-architected, and designed to scale. With 7+ years of experience across industry and academic research (MSc in NLP, published at ACL & WSDM), I bring both deep technical skills and a practical, results-driven approach to every project. My clients tend to stay. My longest engagement ran 3 years, and most of my UpWork projects have been 1+ year collaborations. I don't just write code — I ask the right questions, suggest better approaches, and make sure the final product actually works in the real world. 🔎 What I build ★ AI Agents & Chatbots — text, voice, multi-language (OpenAI, Anthropic, Azure AI, LiveKit) ★ RAG Pipelines — semantic search, vector databases, MCP servers, knowledge retrieval ★ NLP & LLM Applications — classification, entity recognition, summarization, Q&A ★ Data Extraction — OCR, web scraping, document parsing, transaction processing ★ Cloud-Native Backends — microservices, event-driven architectures, serverless ★ Real-Time Audio/Video Processing — transcription, translation, speaker diarization ★ MLOps & Monitoring — experiment tracking, evaluation, observability 🛠️ Tech I work with daily ★ Python: FastAPI, Flask, Streamlit, Pandas, scikit-learn, TensorFlow/Keras, spaCy, NLTK ★ AI/LLM: OpenAI, Anthropic Claude, DeepSeek, Gemini, LangChain, LlamaIndex, CrewAI, Guardrails AI ★ Cloud: AWS (Lambda, SQS, S3, Batch, EC2, RDS, SageMaker), Azure, GCP, Terraform, Docker ★ Databases: PostgreSQL, Supabase, MongoDB, DynamoDB ★ Vector DBs: Weaviate, Pinecone, Milvus, pgvector ★ MLOps: MLFlow, Langfuse, DeepEval, Weights & Biases, OpenTelemetry ★ Other: Tesseract OCR, Faster-Whisper, Pyannote, FastMCP, Selenium 🚀 Recent projects → 3-year AI platform for a finance company: chatbot (voice + text), OCR pipelines, merchant classification with GPT-4o, web scraping, MLOps setup → Real-time transcription & translation system on AWS for multi-language conferences: GPU-accelerated Whisper, speaker diarization, LangChain summarization → RAG backend with MCP server: semantic search across Slack, ClickUp, and Fireflies using Weaviate, LlamaIndex, and Supabase pgvector 📢 What clients say about me ▸ "Christian is a great developer, and asks relevant questions for the problems we give him, he's not just a 'pair of hands' but a helpful advisor for improving your initial suggestion on how to solve the problem. He's very fast to iterate and delivers code with great quality." ▸ "Went above and beyond to help with code for a large project, and completed tasks quickly and efficiently. Understood exactly what was needed for the job and executed with precision. I will absolutely be working with him again in the future!" Let's build something great — send me a message and let's talk about your project.

Most data teams have infrastructure that looks good on paper but still can't answer basic business questions fast enough. I fix that end to end. With 7+ years in data engineering, I specialize in Snowflake, Databricks, and AWS building pipelines that actually run in production without breaking at 6am. Recent work includes: • CI/CD pipeline for a US healthcare client using Schemachange + Snowflake with automated DQ checks and deployment gating • Databricks-to-Snowflake migration of a 200-view mart package, rebuilt natively in Snowflake SQL • ML-powered matching service on FastAPI with pgvector semantic search and a LightGBM reranker • End-to-end RAG pipeline using Airflow, Pinecone, and OpenAI I work as an independent contractor no middleman, no overhead. I treat your problem as mine to solve, not a ticket to close. If your data is slow, unreliable, or not being used strategically, let's talk.