Data Engineer - Quality

This is a remote position.

Alternative Path is seeking skilled Data Engineer to collaborate on client projects with an asset management firm. In this role, you will collaborate with individuals across various company department to shape and innovate new products and features for our platform, enhancing existing ones. You will have a large degree of independence and trust, but you won't be isolated; the support of the Engineering team leads, the Product team leads, and every other technology team member is behind you. This is an opportunity to join a team-first meritocracy and help grow an entrepreneurial group inside Alternative Path. You will be asked to contribute, given ownership, and will be expected to make your voice heard.

Role Summary Design and implement a Python-based tool to automate the configuration and definition of quality control checks for vendor data sets. The tool will leverage LLM and data set documentation to understand the critical fields and leverage machine learning algorithms to define a threshold for quality checks. The checks are configured in a proprietary data quality engine. The tool will be used to set up new data feeds and review and adjust thresholds for existing checks on a periodic basis.
You will work closely with Data Managers and Software Engineers to understand data sets and their existing checks and develop techniques to improve the coverage of checks across our platform. Location - Remote

Requirements

Role Overview The Data Guardian Team is seeking a creative and detail-oriented developer to contribute to projects. The team develops a data quality engine that runs data quality assertions on a vast array of data and orchestrates operational alerting workflows to our Data Operations team for review and remediation. We are seeking someone who is self-motivated, self-sufficient, with a passion for tinkering and a love for automation.

In your role, you will:
➢ Collaborate with other software engineers to understand and resolve requirements
➢ Design and implement a data quality automation tool using LLMs and machine learning algorithms
➢ Develop dashboards for data quality coverage and compliance

Qualifications Must have ➢ Bachelor’s or Master’s degree in computer science or engineering
➢ 3-5 years of experience in Data Engineering
➢ Strong experience building enterprise solutions in Python
➢ Pandas and data science transformation logic
➢ Exposure to SQL

Preferred
➢ AWS experience including Kubernetes
➢ S3 experience
➢ Trino experience

Originally posted on Himalayas