Ignite data loading and streaming capabilities allow ingesting large finite as well as never-ending volumes of data in a scalable and fault-tolerant way into the cluster. The rate at which data can be injected into Ignite is very high and easily exceeds millions of events per second on a moderately sized cluster.
Data loading into Ignite from a 3rd party database is covered on dedicated data loading page. The same page explains how to preload initial data into the cluster having Ignite native persistence enabled.
- Client nodes inject finite or continuous streams of data into Ignite caches using Ignite Data Streamers.
- Data is automatically partitioned between Ignite data nodes, and each node gets equal amount of data.
- Streamed data can be concurrently processed directly on the Ignite data nodes in collocated fashion.
- Clients can also perform concurrent SQL queries on the streamed data.
Data streamers are defined by
IgniteDataStreamer API and are built to inject large amounts of continuous streams of data into Ignite stream caches. Data streamers are built in a scalable and fault-tolerant fashion and provide at-least-once-guarantee semantics for all the data streamed into Ignite.
You can use a full set of Ignite data indexing capabilities, together with Ignite SQL, TEXT, and Predicate based cache queries, to query into the streaming data.
Apache Ignite integrates with major streaming technologies and frameworks such as Kafka, Camel, Storm or JMS to bring even more advanced streaming capabilities to Ignite-based architectures: