Role Overview
The Data Engineer is responsible for building and operating Inventeer’s data foundation: reliable pipelines, clean models, optimized data structures, and trusted datasets that power analytics, reporting, and data-driven product/operational decisions.
This is a delivery-focused, systems role. Success is measured by reliable data flows, trusted tables, efficient query performance, repeatable migrations, clear ownership, and low-maintenance infrastructure—not by ad-hoc data pulls.
Key Responsibilities
Data Architecture, Modeling & Sources of Truth
- Design and evolve Inventeer’s data architecture: source systems → pipelines → modeled datasets → downstream consumers.
- Define and maintain sources of truth and canonical datasets for product and operational reporting.
- Build data models that balance correctness, scalability, performance, and ease of use (analytics-friendly schemas).
Data Structures, Performance & Optimization
- Plan and review database structures: schemas, indexes, partitioning strategies, and query patterns.
- Diagnose and improve performance bottlenecks (slow queries, inefficient joins, missing indexes, poor cardinality assumptions).
- Establish standards for data modeling and performance (naming, normalization/denormalization strategy, retention/archiving).
Pipeline Development (ETL/ELT)
- Build and maintain ETL/ELT pipelines that ingest data from internal databases, third-party tools, and external APIs.
- Implement incremental loads, backfills, deduplication strategies, and schema evolution handling.
- Ensure pipelines are efficient, secure, and resilient to upstream changes.
System Migration & Data Transformation
- Plan and execute migrations between systems and environments, including cases where schemas differ significantly.
- Design transformation logic to map old structures into new data models while preserving correctness and traceability.
- Execute cutover strategies: backfills, incremental sync, validation, reconciliation, and rollback planning.
- Maintain migration documentation and repeatable migration playbooks for future use.
Orchestration, Reliability & Observability
- Operate orchestration workflows (Airflow/Prefect/Dagster or equivalent): scheduling, retries, alerting, and dependency management.
- Implement pipeline observability: monitoring, logging, lineage awareness, and failure alerts.
- Define SLAs/SLOs for critical datasets and ensure stakeholders know what’s reliable and when.
Data Quality & Governance
- Implement data quality checks (freshness, completeness, validity, referential integrity, anomaly detection where applicable).
- Build automated validation and reconciliation to prevent silent failures—especially during migrations and schema changes.
- Maintain documentation for datasets, pipelines, and contracts (schemas, definitions, ownership).
Enablement & Collaboration
- Partner with Data Analysts and Engineering teams to ensure datasets support KPI definitions and reporting needs.
- Support instrumentation design from an engineering perspective (events, identifiers, tenant boundaries, consistency).
- Reduce time-to-insight by making data discoverable, consistent, and easy to query.
Required Technical Skills
- Strong experience building and operating data pipelines end-to-end (ingestion → modeling → serving).
- Strong SQL skills and hands-on PostgreSQL experience (required).
- Experience planning and executing data migrations across systems, including complex transformations and validation.
- Experience with at least one orchestration framework (Airflow/Prefect/Dagster or equivalent).
- Strong ability to build resilient integrations with external APIs and third-party tools.
- Experience with data modeling for analytics (star schemas, wide tables, event models, dimensional thinking).
- Practical experience with performance optimization (indexes, query plans, schema design tradeoffs).
- Familiarity with data quality practices and automated checks.
- Strong engineering hygiene: version control, CI/CD mindset, documentation, and maintainability.
Nice to Have
- Python for pipeline development and automation (requests, pandas, data tooling).
- Experience with a warehouse/lake stack (BigQuery, Snowflake, Redshift, or similar) if Inventeer introduces one.
- Experience with dbt (or equivalent transformation-as-code approach).
- Experience with multi-tenant SaaS data boundaries and privacy/security constraints.
- Familiarity with streaming/event-driven pipelines (Kafka/PubSub) if/when needed.
- Experience supporting audit-ready data practices (traceability, access controls, retention policies).
Behavioral Expectations
- Reliability mindset: treats data as production software with uptime, correctness, and ownership.
- Low-maintenance builder: favors simple, durable systems over fragile complexity.
- Performance discipline: actively prevents and fixes data-layer bottlenecks through thoughtful structure and optimization.
- Migration ownership: plans migrations carefully, validates results, and executes cutovers without drama.
- Clear documentation: makes datasets, pipelines, and migration logic understandable—not tribal knowledge.
- Strong ownership: closes loops—failures are addressed permanently, not repeatedly patched.
- Collaborative enablement: builds for analysts and stakeholders, not just for “the pipeline to run.”
- Security-aware: respects access control, tenant isolation, and data minimization principles.