Data Engineer

Description:

As a key member of our founding team, you will design, build and own the end-to-end pipelines that ingest, process and structure data from thousands of dynamic web sources. Your work is the foundation that enables our AI to understand the complex world of treatments and clinical trials accurately. You are responsible for both the flow and the fidelity of our most critical asset: our data.

𝗪𝗵𝗮𝘁 𝗬𝗼𝘂’𝗹𝗹 𝗗𝗼

*Design and scale web scraping systems that reliably extract data from thousands of sources using Scrapy, Playwright and Selenium

*Build and maintain Python ETL pipelines that process and transform raw data into structured, analytics-ready formats

*Optimize our Elasticsearch cluster to deliver fast, accurate search results for our AI applications

*Automate data workflows to ensure reliable, scheduled execution of all data processes

*Implement data quality frameworks with automated validation checks that continuously monitor data accuracy and completeness

*Build monitoring and alerting systems that provide real-time visibility into pipeline health and data quality metrics

*Establish and enforce data quality standards to maintain consistent, reliable data structures across all systems

*Proactively identify and resolve data issues by diagnosing root causes and implementing permanent fixes

*Ensure data freshness and reliability across all ingested data to maintain trust in our AI-driven insights

𝗪𝗵𝗮𝘁 𝗬𝗼𝘂 𝗕𝗿𝗶𝗻𝗴

*A bachelor’s or master’s in Computer Science, IT or a related field

*Proven expertise in large-scale web scraping (Scrapy/Playwright/Selenium)

*Strong Python skills for data processing and ETL

*Hands-on experience with Elasticsearch and handling big data

*Experience with job scheduling and automation (cron, Airflow, etc.)