Why was Airflow replaced for this workload?

One platform team owned one main type of Spark workload, while the scheduling requirement was mainly about dependencies, retries and monitoring. In this case, the wider Airflow platform introduced more operational overhead than the workload required. The decision was specific to this environment, not a general recommendation to replace Airflow.

When would a standard orchestrator still be the better choice?

A standard orchestrator is more appropriate when several teams contribute different workflows, the wider integration ecosystem is being used, hiring familiarity matters or there is no team able to own a custom framework long term.

How were the PySpark out-of-memory failures addressed?

The issue was treated as a resource-allocation problem. Flexible cluster allocation and tuning allowed the platform to better match computing resources to the size of the individual test campaign.

Why was Data Vault 2.0 relevant?

Vehicle test analysis depends on comparing campaigns, variants and software versions over time. Data Vault 2.0 preserves historical states and the audit trail behind those comparisons.

How was access to analytical data controlled?

Reporting marts were queried through Trino, with granular policies controlling which teams could access which data.

What is the trade-off of using a custom scheduler?

The owning team becomes responsible for maintaining the framework, documenting it, fixing defects and onboarding future engineers. The choice should be revisited if the workload or ownership model changes.

Making vehicle test data easier to process, compare and govern

An anonymised German premium automotive OEM hardened the platform underneath its vehicle test-data analytics. Large-scale test data was processed using Python, PySpark and Delta Lake on Azure Kubernetes Service. For one relatively homogeneous Spark workload owned by a single platform team, Airflow was replaced with a purpose-built Python scheduler covering dependencies, retries and monitoring. Flexible resource allocation addressed identified out-of-memory failures, Data Vault 2.0 preserved historical comparisons and Trino policies controlled analytical access.

Turning test-drive data into something teams could rely on

Before a vehicle, system or feature reaches market, instrumented test vehicles spend thousands of hours collecting data. Every journey creates logs that test engineers and data scientists need to turn into evidence they can compare and use.

The challenge was that the data did not arrive at a steady rate. Test campaigns created sudden peaks. Processing jobs that worked under normal demand could run out of memory when a larger campaign landed, creating delays and extra operational effort.

The scheduling layer had also become heavier than the workload needed. The team mainly needed jobs to run in the right order, retry when necessary and make failures visible. The existing orchestration platform could do much more than that, but the extra capability also meant more components to operate and maintain.

The platform also had to preserve history. Teams needed to compare campaigns, vehicle variants and software versions over time, while making sure people only saw the data they were allowed to access.

How could the platform handle sudden peaks in test data more reliably?
Could job scheduling be simplified without losing important controls?
How could teams compare current and previous campaigns without losing history?
How could access be controlled without creating unmanaged copies of data?

Capabilities

The work combined data architecture, cloud platform engineering, operational resilience and access control to make the test-data platform easier to run and govern.

Information Security, Data Architecture & Intelligent Systems (LLMs)

What the work involved

The engagement covered large-scale data processing, job scheduling, warehouse design, controlled access, deployment automation and ongoing monitoring.

Cloud service delivery
Monitoring and improvement

Simplifying the platform around the work it actually had to do

Vehicle test data was processed through custom Python packages using PySpark and Delta Lake, with workloads running on Azure Kubernetes Service. The technical design was already suited to large-scale processing, so the focus was on making it behave more reliably when demand changed.

The memory issue was treated as a capacity problem rather than a reason to rewrite everything. Flexible resource allocation and tuning allowed jobs to request more appropriate resources for the size of the campaign being processed.

Airflow was replaced for this specific workload with a purpose-built Python scheduler. The replacement handled the essentials the team needed, including job dependencies, retries and monitoring hooks, without carrying the wider operating overhead of a more general-purpose orchestration platform.

The Enterprise Data Vault 2.0 warehouse was expanded and optimised so teams could compare campaigns, vehicle variants and software versions over time without losing historical context. Databricks pipelines supported the analytical data flow, while reporting marts made the information easier to consume.

Trino was used at the reporting layer, with granular policies controlling which teams could access which data. Azure DevOps, automated tests, security checks, Docker, Helm and Bicep supported repeatable delivery across production and non-production environments.

Large-scale test-data processing using PySpark and Delta Lake on Azure Kubernetes Service.
Flexible resource allocation to better match changing campaign demands.
A purpose-built Python scheduler replacing Airflow for this specific workload.
An expanded and optimised Enterprise Data Vault 2.0 warehouse.
Databricks pipelines supporting automotive test-data analytics.
Reporting marts queried through Trino with granular access controls.
Automated CI/CD, testing, security checks and infrastructure as code.

Fewer moving parts and clearer operational control

After migration, Airflow was no longer running for this workload. The scheduling layer was reduced to the functions the team actually needed while retaining dependencies, retries and monitoring.

The identified out-of-memory issue was addressed through flexible resource allocation and tuning rather than a wholesale rewrite. The data model also preserved the history needed for comparison, while access controls helped ensure teams only saw the analytical data they were entitled to use.

The available evidence is primarily technical and operational. No approved percentage improvements were supplied for processing time, infrastructure cost, failure rates or wider business performance, so none are claimed here.

What changed in practice

Platform reliability: Processing resources could be adjusted around the size of individual test campaigns rather than relying on one fixed profile designed for average demand.
Operational complexity: The scheduling layer was narrowed to the functions the team actually used, with Airflow removed from the migrated workload.
Historical comparison: Data Vault 2.0 preserved campaign, vehicle-variant and software-version history so teams could compare results over time without losing the audit trail.
Data access: Trino policies controlled which analytical data each team could query through the reporting layer.
Platform delivery: Automated deployment, testing, security checks and infrastructure as code supported more repeatable operation across production and non-production environments.
Long-term ownership: The simpler scheduling model reduced unnecessary platform components, but the owning team also accepted responsibility for maintaining the custom framework.

A platform better matched to the work

This was not a dashboard project. The work happened underneath the dashboards, in the part of the platform that determines whether test data arrives reliably, stays useful over time and reaches the right people.

Replacing Airflow was the most visible change, but the wider result was a platform that better matched the way the test-data workload actually behaved. Resource allocation became more flexible, historical comparison remained intact and access controls were built into the reporting layer.

The trade-offs were kept visible. A custom scheduler still needs an owner. Data Vault still adds modelling complexity. Kubernetes still needs ongoing tuning as workloads change. The goal was not to remove complexity completely, but to keep only the complexity the workload could justify.