Data Engineering, Analytics & Governance

Building reliable systems for data

The discipline of collecting, transforming, storing and serving data for operational and analytical use.

A sequence that moves and transforms data

An automated flow that ingests data, applies processing and delivers it to a destination.

Bringing data into a system

The process of collecting data from sources and loading or streaming it into a platform.

Processing a bounded collection of data

Work that runs on accumulated records at scheduled or triggered intervals.

Processing events as they arrive

Continuous computation over an ordered flow of records with low delay.

Change Data Capture

A technique that detects inserts, updates and deletes in a source system and forwards those changes elsewhere.

A platform combining lake and warehouse ideas

An architecture that keeps flexible object storage while adding table management, governance and analytics features.

A focused analytical data store

A curated subset of data designed for a department, subject area or use case.

Operational Data Store

A current, integrated store used for operational reporting or as an intermediate step between source systems and analytics.

Decentralised, domain-oriented data ownership

An organisational and architectural approach that treats data as products owned by business domains with shared governance.

An integrated layer for managing distributed data

An architecture and set of services that connects, governs and automates access to data across environments.

Data delivered with an accountable product mindset

A reusable, discoverable and reliable dataset or data service designed for consumers.

An agreement about a data interface

A machine-readable or documented commitment covering schema, semantics, quality and change expectations between producers and consumers.

The history and movement of data

A record of where data came from, how it changed and which systems or outputs depend on it.

A searchable inventory of data assets

A system that records datasets, definitions, ownership, lineage and access information.

Data that describes other data

Information such as structure, meaning, ownership, quality or creation time that gives data context.

Master Data Management

Processes and technology used to maintain consistent core records such as customers, products or locations.

Fitness of data for its intended use

The accuracy, completeness, consistency, timeliness and validity of data relative to a need.

Decision rights and controls for data

The policies, roles and processes used to manage data quality, access, security and accountability.

A person accountable for a data domain

A role that maintains definitions, quality, access expectations and issue resolution for particular data.

The defined structure of data

A description of fields, types, relationships and constraints that data is expected to follow.

A managed store of data schemas

A service that versions and validates schemas used by messages or datasets so producers and consumers remain compatible.

Online Transaction Processing

Systems optimised for many short, concurrent reads and writes that support day-to-day operations.

Online Analytical Processing

Systems optimised for complex queries and aggregations over large datasets for analysis.