Join our team as a Senior Data Engineer building the data pipelines and semantic layer for our enterprise data platform. Our current focus is an AI-driven factory that turns business data needs into governed, tested data products on Databricks. You will also work on the team's other data and software projects, including CI/CD automation.
You will own pipeline capabilities end to end, within the standards set by the team's architects. You will decide whether AI-generated pipelines are correct, and review the work of early-career data engineers.
You will own work from design through production support. You must be comfortable with unclear requirements, unfamiliar enterprise data, changing priorities, and strict security constraints.
Responsibilities include:
- Own a pipeline capability end to end, such as incremental loads or data quality evidence.
- Build data pipelines on Databricks, and extend the reusable components that engineers and AI agents build pipelines from.
- Design and build incremental and historical loads, such as MERGE, change data capture, and SCD Type 2.
- Review AI-generated pipelines, mappings, joins, and incremental logic, and sign off on them within the team's standards.
- Write test data, independent expected results, and data quality rules, and add scenarios to the team's benchmark suites.
- Work directly with requesters, domain experts, and data owners to set grain, keys, business rules, and acceptance criteria for each product.
- Build semantic layer definitions, such as Unity Catalog metric views, from the data architect's designs.
- Map data products and source data to the enterprise ontology, following the data architect's design.
- Write data contracts and the quality evidence that shows each product can be trusted.
- Deploy pipelines as code through CI/CD, and support your releases in production, including freshness and quality monitoring.
- Review the work of early-career data engineers and help mentor them.
- Use AI coding assistants daily, and review what they generate before it ships.
- Document pipelines, tests, and decisions.
This is an individual contributor role. You will lead technical work in your area without formal personnel authority.
