Five things you can do with Irion’s Analytics Engine

The Analytics Engine is the Irion EDM® Application engine dedicated to analytical processing on large volumes and heterogeneous sources. It is chosen like the other processing engines, within the same pipelines. Here are five things it makes possible, what they mean for those who design solutions and what changes for those who use the results.

One more engine in the toolbox

Those who design solutions in Irion EDM choose, step by step, the tool best suited to each processing task: one for queries, one for quality rules, one for Python or R scripts, one for data masking. The Analytics Engine joins this family with a precise task: intensive analytical processing on large volumes and heterogeneous sources.

It works alongside the relational engine, operates as a SQL engine and works in the same pipelines, with the same tools. But its value does not stop at those who design: it reaches all the way to those who use the results, in the form of broader analyses, more complete indicators and timeframes more compatible with decisions.

For this reason, each feature below is described from two points of view: that of those who design the pipelines (designers and Data Engineers) and that of those who use their results in their daily work.

1. Analyze large volumes in timeframes compatible with operational work

Aggregations over tens of millions of records, time-based variances, calculations by currency and by time window, pivots and comparative views: these are the processing tasks where the Analytics Engine expresses its potential. The engine reads only the data needed for each individual analysis and, when memory is not enough, continues processing without interrupting it.

For those who design, it means being able to assign the heaviest steps of a pipeline to the engine designed to sustain them, without changing tools or ways of working.

For those who use the results, it means having analyses on large volumes in time to act, and not only to take note of what has already happened.

2. Simulate and recalculate scenarios

Forecasting simulations and scenario recalculation on historical series are iterative processes: they are repeated, compared and redone with different parameters. On large databases, it is precisely the repetition that makes them demanding.

For those who design, the Analytics Engine makes this kind of processing sustainable, which previously could require timeframes that were difficult to reconcile with operational needs.

For those who use the results, it means being able to compare multiple scenarios and re-read the same phenomenon from different angles, with comparative views that make it immediately readable.

3. Combine different sources without centralizing them first

An organization’s data lives in different places: relational databases, cloud object storage such as Amazon S3, Azure Blob Storage and Google Cloud Storage, Parquet files, Cloud Data Warehouses such as Google BigQuery, Lakehouse environments. The Analytics Engine makes it possible to query them with unified SQL logic, combining them in the same processing and reading and writing open formats without intermediate conversions.

For those who design, this translates into fewer staging pipelines, fewer copies and fewer point-to-point connectors to build and maintain. When the source allows it, part of the work is delegated to the source itself, reducing the data to be transferred. Data cannot always stay where it is, but what does not need to be moved is not moved.

For those who use the results, it means having indicators that integrate different sources without having to wait for an upfront centralization project, and an operational view of distributed data.

4. Run heavy processing without stopping the rest

Analytics Engine processing is executed in a dedicated service, separate from the rest of the application. The most demanding analyses remain confined within that perimeter, while the other services continue to work normally.

For those who design, it means not having to choose between intensive analysis and system stability. And when volumes grow, computing power can grow in turn by adding service instances, in both Windows and Kubernetes/Linux environments, without having to resize the entire infrastructure.

For those who use the results, it means continuity: while heavy processing is under way, the other activities on the application continue without slowdowns.

5. Build a single flow, from processing to distribution

The Analytics Engine does not work in isolation. In the same pipeline it can exchange results with Irion’s other processing engines: Query, Rule, Python, R, Masking, Profiler and Script. A typical flow can aggregate large volumes with the Analytics Engine, apply quality checks to the result, mask sensitive data and distribute the final output, all in a single execution.

For those who design, it means a single governed flow instead of multiple tools to orchestrate. The Analytics Engine is used with the same paradigms already familiar to those who develop on Irion EDM, such as DELT® pipelines and EAsT® virtual tables: no separate environment, no new language to learn.

For those who use the results, it means receiving data that is already processed, checked and ready to use, with the traceability guaranteed by governed pipelines.

When to choose it, and when not to

The practical advice for those who design is simple: choose the engine according to the workload.

The Analytics Engine performs at its best on aggregations, transformations, profiling and complex queries on large volumes, that is, on processing consistent with its architecture. In contexts with high transactionality and concurrency, with many users accessing or updating the same data at the same time, the relational engine based on SQL Server remains the most appropriate choice.

It is therefore not a matter of replacing one engine with another, but of assigning each processing task to the one best able to execute it, making the most of the systems already in place.

The five features, seen from the two perspectives: those who design the pipelines and those who use their results.

In summary

The Analytics Engine expands what can be done in Irion EDM: analysis on large volumes, simulations, cross-source processing, intensive workloads isolated from the rest of the system and complete flows that combine multiple engines. For those who design, it is one more tool to choose with care; for those who use the results, it is the possibility of having broader and more timely analyses, within governed processes.

For those who want to understand what happens beneath the surface, the Under the Hood article Inside the Analytics Engine: how Irion separates compute from persistence describes its architecture.

FAQ