Skip to content

About RTDIP

RTDIP and LF Energy

rtdip

By providing frameworks and reference implementations, LF Energy minimizes pain points such as cybersecurity, interoperability, control, automation, virtualization, and the orchestration of supply and demand.

RTDIP is an LF Energy project and forms part of an overall open source energy ecosystem. To find out more about projects in LF Energy, please click here.

What is Real Time Data Ingestion Platform

Organizations need data for day-to-day operations and to support advanced Data Science, Statistical and Machine Learning capabilities such as Optimization, Surveillance, Forecasting, and Predictive Analytics. Real Time Data forms a major part of the total data utilized in these activities.

Real time data enables organizations to detect and respond to changes in their systems thus improving the efficiency of their operations. This data needs to be available in scalable and secure data platforms.

Real Time Data Ingestion Platform (RTDIP) is the solution of choice leveraging PaaS (Platform as a Service) services along with some custom components to provide Data Ingestion, Data Transformation, and Data Sharing as a platform. RTDIP can interface with several data sources to ingest many different data types which include time series, alarms, video, photos and audio being provided from sources such as Historians, OPC Servers and Rest APIs, as well as data being sent from hardware such as IoT Sensors, Robots and Drones.

​Why Real Time Data Ingestion Platform?

Real Time Data Ingestion Platform (RTDIP) enables users to consume Real Time Data at scale, including historical and real time streaming data. RTDIP has proven to be capable of ingesting over 3 million sensors in a production environment across every geographical location in the world.

The Real Time Data Ingestion Platform can be run in a customers own environment, allowing them to accelerate their cloud deployments while leveraging a proven design to scale their time series data needs.

RTDIP also provides a number popular integration options, including:

  1. ODBC
  2. JDBC
  3. Rest API
  4. Python SDK

These options allow users to integrate with a wide variety of applications and tools, including:

  1. Data Visualization Tools such as Power BI, Seeq, Tableau, and Grafana
  2. Data Science Tools such as Jupyter Notebooks, R Studio, and Python
  3. Data Engineering Tools such as Apache Spark, Apache Kafka, and Apache Airflow

RTDIP is architected to leverage Open Source technologies Apache Spark and Delta. This allows users to leverage the power of Open Source technologies to build their own custom applications and tools in whichever environment they prefer, whether that is in the cloud or on-premise on their own managed Spark Clusters.