Data Engineer

 

Description:

𝗕𝘂𝗶𝗹𝗱 𝘁𝗵𝗲 𝗧𝗿𝘂𝘀𝘁𝗲𝗱 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲 𝗳𝗼𝗿 𝗔𝗜-𝗗𝗿𝗶𝘃𝗲𝗻 𝗛𝗲𝗮𝗹𝘁𝗵𝗰𝗮𝗿𝗲 At Victreat, we're engineering a real-time data intelligence engine that transforms the unstructured web into a structured, queryable and trusted database. This is the core of our 𝗚𝗿𝗼𝘂𝗻𝗱𝗯𝗿𝗲𝗮𝗸𝗶𝗻𝗴 𝗧𝗿𝗲𝗮𝘁𝗺𝗲𝗻𝘁 𝗡𝗮𝘃𝗶𝗴𝗮𝘁𝗼𝗿, an AI platform that guides patients and professionals to the best care paths. You will not just build the pipeline; you will guarantee the integrity of the data that makes this life-changing AI possible and reliable.

 

𝗧𝗵𝗲 𝗥𝗼𝗹𝗲: 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿, 𝗥𝗲𝗮𝗹-𝗧𝗶𝗺𝗲 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 & 𝗤𝘂𝗮𝗹𝗶𝘁𝘆 As a key member of our founding team, you will design, build and own the end-to-end pipelines that ingest, process and structure data from thousands of dynamic web sources. Your work is the foundation that enables our AI to understand the complex world of treatments and clinical trials accurately. You are responsible for both the flow and the fidelity of our most critical asset: our data.

 

 

𝗪𝗵𝗮𝘁 𝗬𝗼𝘂’𝗹𝗹 𝗗𝗼

*Design and scale web scraping systems that reliably extract data from thousands of sources using Scrapy, Playwright and Selenium

 

*Build and maintain Python ETL pipelines that process and transform raw data into structured, analytics-ready formats

 

*Optimize our Elasticsearch cluster to deliver fast, accurate search results for our AI applications

 

*Automate data workflows to ensure reliable, scheduled execution of all data processes

 

*Implement data quality frameworks with automated validation checks that continuously monitor data accuracy and completeness

 

*Build monitoring and alerting systems that provide real-time visibility into pipeline health and data quality metrics

 

*Establish and enforce data quality standards to maintain consistent, reliable data structures across all systems

 

*Proactively identify and resolve data issues by diagnosing root causes and implementing permanent fixes

*Ensure data freshness and reliability across all ingested data to maintain trust in our AI-driven insights

 

 

𝗪𝗵𝗮𝘁 𝗬𝗼𝘂 𝗕𝗿𝗶𝗻𝗴

*A bachelor’s or master’s in Computer Science, IT or a related field

 

*Proven expertise in large-scale web scraping (Scrapy/Playwright/Selenium)

 

*Strong Python skills for data processing and ETL

 

*Hands-on experience with Elasticsearch and handling big data

 

*Experience with job scheduling and automation (cron, Airflow, etc.)