Datorios, a leader in data observability and quality solutions for real-time operations, has introduced new observability features for Apache Flink, the widely-used open-source stream processing system.

Revealed at Ververica’s Flink Forward 2024 in Berlin, Datorios’ latest offerings provide businesses with the tools needed to monitor Flink-driven AI applications effectively, supporting consistent, accurate business automation in real time.

Datorios centralizes insights on system behavior, including data flows, code execution, and infrastructure, enabling organizations to quickly identify and resolve issues in operational AI.

Agentic AI: The Next Evolution in AI Applications

Agentic AI represents the next phase in AI development, where AI not only responds to inquiries but autonomously devises and executes strategies to meet specific objectives. This capability automates decision-making processes and operational tasks, with Gartner predicting that by 2028, 15% of routine decisions will be autonomously managed by agentic AI systems.

The Role of Apache Flink Observability in Responsible Real-Time AI

Apache Flink plays a critical role in real-time agentic AI by integrating data across enterprise systems, delivering the context AI agents need to act. As AI-driven decision-making becomes central to business operations, it’s vital to have visibility into Flink’s workings to understand the decisions agentic AI makes.

Ronen Korman, CEO and co-founder of Datorios, stressed: “Just as no responsible company would operate without cybersecurity, no responsible organization should let AI drive operations without observability.”

He added, “Agentic AI offers immense opportunities, but businesses need full visibility into AI operations. Datorios provides the critical observability necessary to ensure operational quality and reliability.”

New Tracing and Data Lineage Analytics for Flink

With data streaming constantly in and out of AI models via Apache Flink, observability is essential to diagnose issues such as system failures or address audit requests. Datorios’ new features offer tailored dashboards for real-time Flink monitoring, helping organizations quickly locate and resolve problems.

New Capabilities Include:

• Correlated Traces – Identifying the root cause of AI failures often points to data issues, code inconsistencies, or infrastructure problems. Datorios’ platform correlates data across these dimensions, empowering teams to detect and address issues seamlessly.
• Lineage Analytics – Datorios enables Flink teams to track data flow within Flink jobs, identifying when and why data changed and pinpointing team members familiar with specific code or data sources. Lineage analytics includes:
  • Lineage Visualization – Interactive Job Graphs allow a detailed view down to the single operator within tasks.
  • Data Flow Details – Detailed Job Graphs provide insights into data transformations and downstream flow.
  • Data Validation – Upstream investigations help teams trace data inconsistencies back to their sources.
  • Low Performance Impact – Datorios’ Linear Analyzer operates externally, preserving Flink’s real-time performance while delivering critical visibility.