Data Engineer

This profile represents the data-engineering shape most companies need once dashboards start disagreeing with each other: someone who makes the pipelines reliable and the warehouse trustworthy.

Background

Seven years building data infrastructure, first in AdTech where event volumes are unforgiving, then in retail where the questions are commercial: inventory, margin, and demand. Writes pipelines in Python, uses Spark when the data is genuinely big and skips it when it is not, orchestrates everything with Airflow, and lands it in Snowflake modeled for the people who will query it. The typical arc for this archetype starts with maintaining scripts someone else wrote on cron, and ends with owning the platform those scripts should have been.

What the work actually covers

Ingestion and ETL. Pulling from the sources companies actually have: transactional databases via change data capture, third-party APIs with rate limits and flaky auth, event streams, and the occasional SFTP drop of CSV files that a partner refuses to modernize. The craft is making each of these idempotent, so a rerun never double-counts, and incremental, so a growing source does not mean a growing bill.

Orchestration. Airflow DAGs with explicit dependencies, ownership, and SLAs, where a failure pages someone who can act on it. Backfills are designed in from the start: reprocessing last March should be one command, not a week of manual state surgery. This archetype has usually inherited at least one tangle of interdependent cron jobs and knows exactly why that pattern fails.

Warehouse and lakehouse. Layered modeling in Snowflake or an equivalent: raw data as it arrived, a cleaned and conformed layer, and business-ready marts, typically built with dbt style transformations so logic is versioned and tested. When history is large and queried rarely, offloading cold data to object storage in open table formats keeps the warehouse fast and the bill sane.

Data quality. Checks that run inside the pipeline, not after complaints: row counts against expectations, null and uniqueness constraints, freshness monitors, and schema-change alarms on sources. The senior habit is publishing data with contracts, so downstream teams find out about a breaking change before it hits their dashboard.

Cost. Snowflake credits and Spark cluster hours are watched like a budget line. Typical levers: right-sizing warehouses, killing zombie schedules, pruning full-table scans, and moving from always-on clusters to per-job compute. In AdTech-scale environments, pipeline cost per processed event is a design input, not an afterthought.

Seniority signals

Seven years shows up as restraint. Signals worth probing: they can defend running a job in plain Python and Postgres instead of Spark, and explain the threshold where that flips. They can describe how they made a pipeline idempotent and what breaking that cost them once. They have handled a source schema change that silently corrupted data downstream and can walk through how they detected and rebuilt it. They talk about who consumes the data as readily as they talk about tools. Juniors optimize queries; seniors reduce the number of pipelines that need to exist.

What to look for if you are hiring this archetype

In interviews, ask the candidate to design a daily pipeline from a transactional database into a warehouse, then ask what happens when Tuesday’s run fails and nobody notices until Friday. The answer reveals whether they think in backfills and idempotency or just in happy paths. Ask how they would detect that yesterday’s numbers are wrong before a stakeholder does. Ask what they would do with a dashboard nobody has opened in six months. Red flags: proposing a streaming architecture for data that is consumed once a day, no opinion on modeling conventions, and treating data quality as the analysts’ problem.

Engagement fit

Best fit is a company whose reporting has outgrown its scripts: a first proper warehouse to stand up, a pipeline estate to untangle, or a data team of one to three that needs senior reinforcement. Works well full-time as the founding data platform owner or part-time as the senior layer over a junior team. Fixed-scope audits of an existing pipeline setup are a common low-risk starting point. Less of a fit if the need is mostly analysis and dashboarding, which is analyst work, or heavy ML model development, which is the adjacent Python data and ML archetype in this pool. Rates for this archetype typically land in the band shown on this card and move with seniority and region.

Typical engagement

Often starts by untangling an existing pipeline estate or standing up the first proper warehouse, then continues as the team’s data platform owner. Brazil hours, which line up almost fully with the US working day. Available now.

This is a representative sample of the vetted pool. Request it and we confirm real, current availability for a developer with this background.