Data Engineer

Description:

We are looking for highly skilled Data Engineers to join the our team. The ideal candidates will have strong experience in backend development, automation, big data processing, and cloud platforms, with the ability to design, build, and optimize large-scale data pipelines and workflows.

Key Responsibilities:

• Develop and maintain ETL pipelines and distributed data processing systems.

• Build workflows integrating Rails apps, Java code, Spark jobs, and Iceberg data.

• Work on big data technologies like Apache Spark (batch & streaming jobs, performance tuning).

• Manage and optimize data storage solutions, including Apache Iceberg and SQL (large datasets, query optimization).

• Automate processes and maintain Linux-based systems.

• Deploy and manage solutions on AWS (EC2, S3, EMR/Glue, IAM).

• Collaborate with cross-functional teams to gather requirements, design solutions, and ensure smooth integration.

Required Skills & Experience:

• Strong experience in scripting languages (Python, Ruby, or others).

• Proficiency in Apache Spark and SQL.

• Experience with Apache Iceberg (tables, schema evolution, partitioning).

• Working knowledge of AWS (EC2, S3, EMR/Glue, IAM).

• Strong Linux automation and system administration skills.

• Good understanding of ETL pipelines and distributed data processing.

• Excellent communication, requirement gathering, and cross-team collaboration skills.