Lead Data Engineer

Description:

Are you passionate about data engineering, analytics, and driving insights to support critical business decisions? If you thrive in a dynamic, collaborative, and fast-paced startup environment, we have a fantastic opportunity for you. We are looking for a Data Engineer to lead our data engineering efforts for analytics, handling client interactions, and potentially leading a team in the future.

Responsibilities:

• Lead data engineering projects that foster collaboration across the company to derive insights from diverse data sources, including clickstream, marketing, product funnels, etc., to support pivotal business decisions.

• Develop scalable and extensible data architectural patterns and ETL processes to accommodate our growing user base.

• Contribute to a robust culture of code review and maintain high standards in coding practices

. • Manage resources efficiently in an Azure or AWS environment using tools like Terraform.

• Integrate new data sources, build innovative tools, and generate new metrics that empower our Product, Marketing, and Data Science teams.

• Apply feature engineering and data enrichment techniques for large-scale consumer and B2B products.

• Enhance data pipelines for improved performance, scalability, and reliability.

Background:

• Bachelor’s degree in computer science or equivalent with at least 3 years of professional experience as a Data Engineer or similar role.

• Proficient in distributed processing frameworks like Databricks/Spark, stream processing, and event-driven technologies such as Kafka, including REST and Python APIs.

• Have expertise in Python, Spark, including Spark SQL, DataFrames, Spark Streaming, RDD caching, Spark MLib, and the ability to write, debug, and optimize Spark code.

• Skilled in using Terraform for infrastructure as code.

• A strong DataOps mindset and opinions on innovative warehousing tools and ETL processes.

• Experience in working with cloud platforms such as Azure and AWS.

• Excellent problem-solving and analytical skills.

• Superior written and verbal communication skills in English, capable of translating complex technical topics into clear and concise terms.

• A passion for engineering excellence and tolerance for ambiguity in a fast-paced, startup setting.

• Experience in schema design and data modeling with a focus on creating elegant and intuitive datasets.