Data Engineer

Description:

We are developing a real time data intelligence engine that transforms the web into a structured, queryable database, enabling groundbreaking healthcare research. We need an engineer to architect the core data acquisition and automation systems that make this possible.

𝗬𝗼𝘂𝗿 𝗠𝗶𝘀𝘀𝗶𝗼𝗻:

*Architect and scale robust web scraping systems to collect data from thousands of dynamic sources using Scrapy, Playwright and Selenium.

*Build and optimize high performance ETL pipelines that deliver clean, structured data for analysis and AI models.

*Develop resilient data workflows that intelligently adapt to source changes and ensure data quality at scale.

*Apply AI agent frameworks to automate complex data extraction and enrichment tasks.

*Integrate and manage data flow into and out of our Elasticsearch platform.

𝗪𝗲’𝗿𝗲 𝗟𝗼𝗼𝗸𝗶𝗻𝗴 𝗙𝗼𝗿:

*Proven experience building large scale web scraping systems with Scrapy and Playwright/Selenium.

*Strong proficiency in Python and expert knowledge of data engineering best practices

*Hands on experience with Elasticsearch.

*Practical experience with AI agent frameworks (e.g., LangChain, CrewAI) and a clear understanding of how to apply them to automate and intelligentize data workflows.