Inside the Analytics Engine: how Irion separates compute from storage

A columnar, vectorized SQL engine that runs in a dedicated, isolated compute service, integrates into Irion EDM® Application pipelines without separate development environments, and reaches data where it lives.

A look at the architecture of the Analytics Engine: how it processes large volumes, how it manages memory, how it scales and how it works with the other processing tools.

Why a specialized engine

Analytical workloads have a shape of their own. The Irion EDM relational engine handles large volumes with solid performance, even with many users accessing the same data at the same time, while ensuring consistency and concurrency control. Analytical processing on even larger volumes typically follows a different model: it reads a few columns across a very large number of rows, aggregates, compares time windows and combines different sources.

These are structurally different needs, and when they share the same resources, the growth of one can affect the stability of the other. This is not a limitation of the relational engine: the two types of processing simply ask different things of the infrastructure.

The Analytics Engine was built on this insight. It is a columnar, vectorized SQL engine, integrated into Irion EDM Application, that works alongside the traditional SQL Server-based engine and lets each workload be assigned to the most appropriate engine. Data stays in the system best suited to store and govern it; compute runs where it is most effective. Compute and storage can therefore evolve on independent criteria while remaining coordinated within governed pipelines.

For users, the paradigm stays familiar: the Analytics Engine is used just like the Query Engine. What changes is how processing happens under the hood.

The diagram below shows how roles are distributed: where processing is designed, where it runs and where the data lives.

Reading by column, computing by vector

The first element of the architecture is the execution model. In a row-based model, well suited to transactional operations, a query reads entire records even when only a few fields are needed. The Analytics Engine instead processes data in columnar blocks: it reads only the columns the query needs, not the whole record.

Vectorized execution builds on this. Operations are not applied value by value but to vectors of values: sums, averages, groupings, time windows and pivots are processed in blocks. The model matches the typical shape of processing on large datasets, where a few columns are read across a very large number of rows.

The practical effect is simple to state: less unnecessary work to produce more useful results. The architecture shows its potential in high-intensity processing (aggregations over tens of millions of records, time-based variance, pivots, simulations on historical series), broadening the range of workloads that can be handled within operational timeframes.

An efficient use of resources

The second element is resource management. The Analytics Engine uses memory to speed up processing but does not depend on it alone: when the dataset exceeds the available RAM, the engine continues on disk, with an automatic spill that does not interrupt execution.

The consequence is mainly architectural. The engine adapts to the size of the workload, using memory when it is available and disk when needed. There is no need to size the infrastructure for edge cases to make sure the heaviest workloads complete.

A dedicated compute service

The Analytics Engine does not share resources with the rest of the application. Processing runs in a dedicated compute service, separate from the application’s other services: the Analytics Execution Service (AES).

In industry terms, this is out-of-process execution. An aggregation over tens of millions of records stays within the boundaries of the service, while the rest of the system keeps running unaffected. Isolation preserves the responsiveness and continuity of the other application services, even when the analytical engine is running at full capacity.

The same design makes compute power scale independently. There is no central server: the service can be installed multiple times, on the same machine or on different machines, distributing the load in a modular way, with throttling logic to manage peaks. Capacity grows by adding instances, in both Windows and Kubernetes/Linux environments.

The result is an architecture with more predictable costs and more elastic capacity: growth in processing workloads does not drive the sizing of the system that stores the data.

An isolated workspace

Besides a dedicated service, the Analytics Engine has its own workspace. Queries run in an isolated space, separate from the application’s main workspace: a sandbox dedicated to analytical processing, which Irion calls Virtual Space and which is provided by a dedicated object, AnalyticsDB.

The lifecycle of this space can be configured for each pipeline or each request. Analytical processing thus fits into the application’s execution model without introducing a separate operating paradigm: isolation is guaranteed, and management stays the same.

Same paradigms, same pipeline

A specialized engine adds value only if it does not require building a separate world around it. The Analytics Engine is used with the same paradigms already familiar to anyone who develops on Irion EDM Application:

• DELT™ (Declarative Extract Load & Transform): the process is represented as a declarative graph; you define the processing logic and the application governs its execution.

• EasT™ (Everything as a Table): different sources are presented as virtual tables that can be queried in a uniform way.

• IsolData™: governs and isolates temporary spaces and the lifecycle of intermediate data.

No separate development or management environment is needed. Above all, within the same run the Analytics Engine can interact with Irion’s other processing engines (the Data Engines): Query, Rule, Python, R, Masking, Profiler and Script. In the same pipeline it can read and write virtual tables produced or consumed by the other tools, combining data processing, quality and distribution in a single flow.

Reaching data where it lives

Data architectures are increasingly distributed, and an analytical engine has to work across them. The Analytics Engine operates directly on heterogeneous sources, without first centralizing data in a single repository:

• it supports the three main types of cloud object storage: Amazon S3, Azure Blob Storage and Google Cloud Storage;

• it reads and writes Parquet natively, with no intermediate conversion;

• it connects to Cloud Data Warehouses such as Google BigQuery and to Lakehouse architectures, including Unity Catalog and the Irion-managed Lakehouse;

• it integrates with relational databases and local sources, so sources of different kinds can be combined in the same pipeline.

Open formats support interoperability between tools and make it possible to work alongside the data platforms already in place, using their resources where appropriate and bringing results back into Irion’s governed pipelines.

Pushdown: moving part of the work to the source

Reaching data does not necessarily mean moving it. When the source supports it, the Analytics Engine applies pushdown logic: it delegates filters and projections, or the entire query, to the remote source. This reduces the volume of data transferred to the processing service and makes use of the compute power already available at the source.

On Parquet files, pushdown also applies when reading. The engine automatically applies projection and filter pushdown: it reads only the columns it needs and, when the query allows, uses filters to skip irrelevant portions of data.

It is worth being precise about the scope. Direct access and pushdown can reduce unnecessary staging, copies and transfers, but they do not eliminate them in every case: some local or cross-source processing still requires data to be transferred or materialized. The actual transfer depends on the source, the query and the execution plan.

Where it excels, and where it doesn’t

The Analytics Engine can be used for many kinds of processing, but it excels on intensive workloads: aggregations, transformations, profiling and complex queries on large volumes. On these workloads, which match its columnar, vectorized architecture, it can deliver higher performance; actual results depend on queries, data, concurrency, configuration and infrastructure.

There is also a scope where it is not the most appropriate choice. In highly transactional, high-concurrency contexts, with many users accessing or updating the same data at the same time, the traditional SQL Server-based engine remains the better fit. This is one of the reasons why the Analytics Engine works alongside SQL Server: each workload is assigned to the engine that can run it best, making the most of existing systems.

An architecture designed to grow

The Analytics Engine is not a closed component. Its architecture includes an extension mechanism based on dedicated libraries, which are expanded and enhanced with every release. The engine is therefore a foundation for progressively adding new features, extending use cases without changing paradigm or introducing separate environments.

In summary

Under the hood, the Analytics Engine combines a columnar, vectorized execution model, memory management that continues on disk when needed, a dedicated, horizontally scalable compute service, an isolated workspace and native connectivity to open formats and modern infrastructure. All of this within the same paradigms and the same pipelines of Irion EDM Application.

One principle holds these elements together: let compute grow on its own terms, without driving the sizing of the system that stores the data, while keeping both coordinated within governed pipelines.

Irion: Turning Data into Action

FAQ