Big Data Developer Job Description Template

An effective description can help you hire the best fit for your job. Check out our tips to provide details that skilled professionals are looking for.

Trusted by


Tips for Writing a Big Data Engineer Job Description

A big data engineer is a professional who is responsible for the management of data sets that are too big for traditional database systems to handle. They create, design, and implement data processing jobs in order to transform the data into a more usable format. They also ensure that the data is secure and complies with industry standards to protect the company’s information. 

Below, we will cover a sample job description, exploring the daily responsibilities and necessary qualifications for a big data engineer. 

The Job Overview

We are seeking a big data engineer to join our data analytics team. The successful candidate will be responsible for overseeing the creation and maintenance of our database infrastructure, including collecting and maintaining data, ensuring the integrity of our data, and creating and training data models.

Responsibilities

Below are some of the responsibilities of a big data engineer:

  • Design the architecture of our big data platform
  • Perform and oversee tasks such as writing scripts, calling APIs, web scraping, and writing SQL queries
  • Design and implement data stores that support the scalable processing and storage of our high-frequency data
  • Maintain our data pipeline
  • Customize and oversee integration tools, warehouses, databases, and analytical systems
  • Configure and provide availability for data-access tools used by all data scientists
Job Qualifications and Skill Sets

Below are the qualifications expected of a big data engineer:

  • 3 to 5 years of relevant data engineering experience
  • Bachelor’s degree or higher in computer science, data science, or a related field
  • Hands-on experience with data cleaning, visualization, and reporting
  • At least 2 years of relevant experience with real-time data stream platforms such as Kafka and Spark Streaming
  • Experience working in an agile environment
  • Familiarity with the Hadoop ecosystem
  • Experience with platforms such as MapReduce, Apache Cassandra, Hive, Presto, and HBase
  • Excellent analytical and problem-solving skills
  • Excellent communication and interpersonal skills

Big Data Developer Hiring Resources

Explore talent to hire
Learn about cost factors

Big Data Developers you can meet on Upwork

  • $50 hourly
    Mohamed S.
    • 5.0
    • (2 jobs)
    London, ENGLAND
    Featured Skill Big Data
    Data Mining
    Data Science
    Fraud Detection
    Data Analysis
    PySpark
    SAS
    Credit Scoring
    Apache Hadoop
    SQL
    Python
    As a seasoned Data Scientist and Technical Product Manager, I bring extensive experience in Financial Crime Risk and Credit Risk management, coupled with deep proficiency in Python, Spark, SAS (Base, EG, and DI Studio), Hadoop, and SQL. Transitioning into freelancing, I am eager to leverage my skills to contribute to diverse projects. While Upwork's guidelines restrict sharing direct links to external profiles, I am happy to provide a detailed portfolio from my LinkedIn upon request.
  • $40 hourly
    Feras A.
    • 5.0
    • (21 jobs)
    Oakville, ON
    Featured Skill Big Data
    Financial Statement
    Financial Analysis
    Tidyverse
    Data Analysis
    Microsoft Excel PowerPivot
    Data Modeling
    Automation
    R
    Power Query
    Microsoft Power BI
    Data Visualization
    SQL
    Intuit QuickBooks
    With 7+ years of experience in Power BI development, I help businesses transform raw data into actionable insights through scalable dashboards, automated workflows, and financial reporting solutions. I specialize in turning messy row data into interactive, drillable dashboards that save teams hours of manual work while delivering clarity, accuracy, and efficiency. My approach blends technical expertise with a strong focus on finance and process automation—so your data doesn’t just look good, it drives decisions. Core Expertise: 🔹 Power BI Development - Data wrangling (Excel/CSV/TXT/PDF → clean, structured datasets) - Data extraction , automation and integration using R programming , Power Automate and Zapier - Optimized star-schema data models for performance - Advanced DAX (time intelligence, dynamic aggregations, allocations) - Publishing & tenant administration (gateways, security, governance) 🔹 Projects Completed - Direct to customer sales dashboard - Amazon store analysis - RFM Analysis - Cohort Analysis - Profit & Loss (P&L) statements (monthly/quarterly/yearly) - Budget vs. Actuals, YoY, QoQ, MoM comparisons - Multi-entity/cost center breakdowns - Supplier/customer profitability analysis - Multi-currency and fiscal/calendar year support 🔹 Power Platform Solutions -Automating manual tasks (PDF/Excel conversions, file consolidation) -Scheduled data refreshes & seamless integrations Ongoing support & maintenance for reports and datasets Why Work With Me? ✅ Data Clarity: I turn scattered files into clear, interactive dashboards. ✅ Scalable Processes: From inconsistent CSVs to automated reporting pipelines. ✅ Ongoing Partnership: Training, documentation, and long-term support included. 📊 Let’s build reports that save time, improve decisions, and eliminate headaches.
  • $40 hourly
    Rai S.
    • 5.0
    • (8 jobs)
    Lahore, PUNJAB
    Featured Skill Big Data
    Transact-SQL
    Google Cloud Platform
    Git
    Apache Airflow
    Microsoft SQL Server
    Data Analysis
    Business Intelligence
    Machine Learning
    BigQuery
    dbt
    SQL
    PySpark
    Python
    With a strong foundation in Mathematics, Data Engineering, AI, and Cloud Technologies, I specialize in designing and implementing 𝐬𝐜𝐚𝐥𝐚𝐛𝐥𝐞 𝐝𝐚𝐭𝐚 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞𝐬, 𝐦𝐚𝐜𝐡𝐢𝐧𝐞 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐬𝐨𝐥𝐮𝐭𝐢𝐨𝐧𝐬, and 𝐜𝐥𝐨𝐮𝐝-𝐧𝐚𝐭𝐢𝐯𝐞 𝐚𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞𝐬. My expertise lies in SQL, Python, Spark-hadoop architecture, Databricks, GCP, AWS, and MLOps enabling businesses to unlock insights, optimise performance, and drive AI-powered innovation. I led data teams, with agile work management, driving strategic data initiatives through mentorship, stakeholder collaboration, budget optimization, and a strong commitment to Equality, Diversity, and Inclusion (EDI). 🔹 𝐂𝐥𝐨𝐮𝐝 & 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠: Architected end-to-end data solutions, including 𝗦𝗤𝗟 𝗦𝗲𝗿𝘃𝗲𝗿 to 𝗕𝗶𝗴𝗤𝘂𝗲𝗿𝘆 and 𝗧𝗲𝗿𝗮𝗱𝗮𝘁𝗮 to 𝗦𝗽𝗮𝗿𝗸-𝗵𝗮𝗱𝗼𝗼𝗽 architecture migrations, ETL/ELT pipelines, and real-time data processing 🔹 𝐀𝐈 & 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠: Built ML models using AWS Sagemaker, Tensorflow, Vertex AI, Document AI, Jupyter notebooks for fraud detection, predictive analytics and Fair AI ensuring transparency, data compliance and ethical AI adoption in data lifecycle management 🔹 𝐁𝐢𝐠 𝐃𝐚𝐭𝐚 & 𝐀𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐬: Engineered cost-optimised, high-performance data warehouses, leveraging Data Lake, Databricks, dbt, EMR, Dataproc, PySpark, Cloudera, Kafka, Tableau and Looker for BI solutions 🔹 𝐀𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐨𝐧 & 𝐃𝐞𝐯𝐎𝐩𝐬: Streamlined deployments with CI/CD (GitHub Actions, Terraform, Cloud Build), improving infrastructure scalability and security. 🔹 𝐑𝐞𝐬𝐞𝐚𝐫𝐜𝐡 & 𝐈𝐧𝐧𝐨𝐯𝐚𝐭𝐢𝐨𝐧: Published research in 𝐩𝐫𝐞𝐬𝐭𝐢𝐠𝐢𝐨𝐮𝐬 𝐯𝐞𝐧𝐮𝐞𝐬 (𝐀𝐂𝐌, 𝐄𝐥𝐬𝐞𝐯𝐢𝐞𝐫) on AI fairness, fraud detection, and intelligent systems. I thrive at the intersection of 𝐭𝐞𝐜𝐡𝐧𝐨𝐥𝐨𝐠𝐲, 𝐩𝐫𝐨𝐛𝐥𝐞𝐦-𝐬𝐨𝐥𝐯𝐢𝐧𝐠, 𝐚𝐧𝐝 𝐢𝐦𝐩𝐚𝐜𝐭, turning complex data challenges into efficient, scalable, and AI-driven solutions. If you're looking for someone to 𝐨𝐩𝐭𝐢𝐦𝐢𝐳𝐞 𝐲𝐨𝐮𝐫 𝐝𝐚𝐭𝐚 𝐚𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞, 𝐬𝐜𝐚𝐥𝐞 𝐀𝐈 𝐬𝐨𝐥𝐮𝐭𝐢𝐨𝐧𝐬, or 𝐦𝐢𝐠𝐫𝐚𝐭𝐞 𝐭𝐨 𝐭𝐡𝐞 𝐜𝐥𝐨𝐮𝐝—let’s connect!
Want to browse more talent? Sign up

Join the world’s work marketplace

Find Talent

Post a job to interview and hire great talent.

Hire Talent
Find Work

Find work you love with like-minded clients.

Find Work