Description:
We are looking for an experienced Senior Data Engineer to design, develop, and maintain scalable data pipelines, data platforms, and analytics solutions. The ideal candidate should have strong experience with cloud data platforms, data engineering frameworks, ETL/ELT pipelines, distributed data processing, data warehousing, and DevOps practices.
Key Responsibilities
- Design, develop, and maintain scalable and reliable data pipelines for batch and real-time data processing.
- Build and optimize ETL/ELT workflows using modern data engineering tools and frameworks.
- Develop data solutions using Databricks, Apache Spark, Azure Data Factory, Azure Synapse, Snowflake, and BigQuery.
- Implement real-time and event-driven data pipelines using Apache Kafka.
- Orchestrate complex data workflows using Apache Airflow and dbt.
- Work with cloud-based data services across Azure and AWS environments.
- Develop and optimize data models, data warehouses, and data lakes.
- Perform data transformation, cleansing, validation, and quality checks.
- Optimize Spark jobs, SQL queries, pipelines, and data warehouse performance.
- Implement CI/CD, infrastructure automation, and deployment processes using GitHub, Jenkins, Azure DevOps, Terraform, and Docker.
- Work with Kubernetes for container orchestration and deployment of data workloads.
- Collaborate with Data Scientists, BI Developers, Software Engineers, and business stakeholders.
- Support reporting and analytics solutions using Power BI, Tableau, and Qlik.
- Troubleshoot data pipeline failures, performance issues, and production incidents.
- Ensure data security, reliability, scalability, and governance across data platforms.
- Contribute to technical documentation, coding standards, and best practices.
Required Skills & Experience
- 5+ years of professional experience in Data Engineering.
- Strong proficiency in SQL and data modeling.
- Hands-on experience with Apache Spark / PySpark.
- Strong experience with cloud data platforms, particularly Azure and/or AWS.
- Experience with Databricks, Azure Data Factory, Azure Synapse, Snowflake, or BigQuery.
- Strong understanding of ETL/ELT architecture and data pipeline development.
- Experience with Apache Kafka for streaming and event-driven data processing.
- Hands-on experience with Apache Airflow for workflow orchestration.
- Experience with dbt for data transformation and analytics engineering.
- Good understanding of relational databases and SSMS.
- Experience using Git/GitHub and modern software development practices.
- Knowledge of Docker, Kubernetes, Terraform, Jenkins, and Azure DevOps.
- Familiarity with Jupyter Notebook and VS Code.
- Understanding of BI and visualization platforms such as Power BI, Tableau, or Qlik.
- Strong problem-solving, analytical, and communication skills.