Back to Job Search

Senior Data Engineer

Executive Monroe Consulting Group is recruiting on behalf of a #1 global market leader in digital shopper marketing. Our client is looking for a Senior Data Engineer that will be directly reporting to the Deputy General Manager. This job initially offers a remote work set-up and will transition to Hybrid (2-3 days onsite) with a night shift schedule.

Job Summary
We're hiring a hands-on Senior Data Engineer/Data Architect to own our SQL data estate, design scalable pipelines, and lead data enrichment across our Azure-first platform. You'll set the standards for modeling, quality, security, and cost while writing production-grade Python and SQL daily.

Key Job Responsibilities:

SQL Databases:

  • Design and evolve schemas for OLTP/OLAP (Azure SQL, Synapse, Delta Lake), with partitioning, indexing, and RLS for multi-tenant isolation.
  • Establish data contracts and versioning, govern schema evolution, and implement CDC + SCD patterns.
  • Performance engineering: query tuning, resource classes, caching strategies, and cost guardrails.

Data Pipelines:

  • Architect ELT/ETL across batch & streaming using Azure Data Factory/Synapse/Databricks, Event Hubs/Service Bus, Functions, and Container Apps/AKS.
  • Build reliable, observable pipelines (idempotent, retryable, lineage-aware) with SLAs/SLOs and runbooks.
  • Implement CI/CD for data (dbt/SQL projects, PySpark jobs, tests) using GitHub Actions and IaC (Terraform/Bicep).

Data Enrichment

  • Define and operate enrichment layers: UPC/GS1, OCR/EXIF metadata, taxonomies, embeddings, and third-party data joins.
  • Curate gold/semantic models for analytics & product APIs; manage feature/metric definitions and documentation.
  • Partner with DS/ML to operationalize feature stores, model outputs, drift signals, and evaluation tables.

Azure Architecture & Governance:

  • Own reference architecture across ADLS Gen2, Synapse/Databricks, Azure SQL/SQL Server, Cosmos DB (incl. vector), Azure AI Search, Key Vault, Purview.
  • Security & compliance by default: encryption, secret management, RBAC/ABAC, data retention and GDPR/SOC 2 controls.
  • Observability: OpenTelemetry + Azure Monitor/App Insights, data quality tests, freshness SLAs, and lineage in Purview.

What you'll build (examples):

  • A durable image ingestion & enrichment pipeline: validate assets, extract OCR/UPC, compute embeddings, store lineage, publish search-ready views.
  • A hybrid retrieval layer (vector + filters) across Cosmos DB/Azure AI Search for similarity and recommendations.

Key Job Qualifications:

  • Extremely strong Python & SQL (you can diagnose complex query plans, write PySpark and pandas with equal ease).
  • 7+ years in data engineering/architecture with production ownership of SQL databases and pipelines.
  • Deep Azure experience: ADLS Gen2, Data Factory/Synapse/Databricks, Azure SQL/SQL Server, Functions, Event Hubs/Service Bus, Key Vault.
  • Proven design skills in data modeling (star/snowflake, Data Vault/Lakehouse), CDC/SCD, and semantics (dbt or equivalent).
  • Track record implementing data quality frameworks, lineage, and cost/performance guardrails at scale.
  • Strong understanding of multi-tenant SaaS, security, and privacy (GDPR basics).

Nice to have:

  • Cosmos DB (incl. vector) and Azure AI Search; embedding pipelines for images/text.
  • Feature stores, MLflow/registries, real-time inference plumbing.
  • SQL Server internals, PolyBase/Serverless SQL; Postgres familiarity.
  • Purview rollouts, governance programs, and data product operating models.

Our Tech Stack:

  • Azure (ADLS Gen2, Data Factory, Synapse, Databricks, Functions, Event Hubs, Key Vault, Monitor)
  • Delta/Parquet, Azure SQL/SQL Server, Cosmos DB (vector), Azure AI Search
  • Python (pandas, PySpark, FastAPI for data services), dbt (or equivalent), GitHub Actions, Terraform/Bicep
  • Observability: OpenTelemetry, Azure Monitor/App Insights, Sentry/Datadog (where applicable)