Apache Ignite Documentation

GridGain Developer Hub - Apache Ignitetm

Welcome to the Apache Ignite developer hub run by GridGain. Here you'll find comprehensive guides and documentation to help you start working with Apache Ignite as quickly as possible, as well as support if you get stuck.

 

GridGain also provides Community Edition which is a distribution of Apache Ignite made available by GridGain. It is the fastest and easiest way to get started with Apache Ignite. The Community Edition is generally more stable than the Apache Ignite release available from the Apache Ignite website and may contain extra bug fixes and features that have not made it yet into the release on the Apache website.

 

Let's jump right in!

 

Documentation     Ask a Question     Download

 

Javadoc     Scaladoc     Examples

Cluster Configuration

Deploy Ignite in any environment with pluggable automatic node discovery.

Overview

In Ignite, nodes can discover each other by using DiscoverySpi. Ignite provides TcpDiscoverySpi as a default implementation of DiscoverySpi that uses TCP/IP for node discovery. Discovery SPI can be configured for Multicast and Static IP based node discovery.

Multicast Based Discovery

TcpDiscoveryMulticastIpFinder uses Multicast to discover other nodes in the grid and is the default IP finder. You should not have to specify it unless you plan to override default settings. Here is an example of how to configure this finder via Spring XML file or programmatically from Java:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
          <property name="multicastGroup" value="228.10.10.157"/>
        </bean>
      </property>
    </bean>
  </property>
</bean>
TcpDiscoverySpi spi = new TcpDiscoverySpi();
 
TcpDiscoveryMulticastIpFinder ipFinder = new TcpDiscoveryMulticastIpFinder();
 
ipFinder.setMulticastGroup("228.10.10.157");
 
spi.setIpFinder(ipFinder);
 
IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

Static IP Based Discovery

For cases when Multicast is disabled, TcpDiscoveryVmIpFinder should be used with pre-configured list of IP addresses.

You are only required to provide at least one IP address of a remote node, but usually it is advisable to provide 2 or 3 addresses of grid nodes that you plan to start at some point of time in the future. Once a connection to any of the provided IP addresses is established, Ignite will automatically discover all other grid nodes.

🚧

By default the TcpDiscoveryVmIpFinder is used in non-shared mode. If you plan to start a server node then in this mode the list of IP addresses should contain an address of the local node as well. It will let the node not to wait while other nodes join the cluster but rather become the first cluster node and operate normally.

Here is an example of how to configure this finder via Spring XML file or programmatically from Java:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
          <property name="addresses">
            <list>
              <!-- 
              Explicitly specifying address of a local node to let it start and                 operate normally even if there is no more nodes in the cluster.
              You can also optionally specify an individual port or port range.
              -->
              <value>1.2.3.4</value>
              
              <!-- 
              IP Address and optional port range of a remote node.
              You can also optionally specify an individual port and don't set                 the port range at all.
              -->
              <value>1.2.3.5:47500..47509</value>
            </list>
          </property>
        </bean>
      </property>
    </bean>
  </property>
</bean>
TcpDiscoverySpi spi = new TcpDiscoverySpi();
 
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
 
// Set initial IP addresses.
// Note that you can optionally specify a port or a port range.
ipFinder.setAddresses(Arrays.asList("1.2.3.4", "1.2.3.5:47500..47509"));
 
spi.setIpFinder(ipFinder);
 
IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

Multicast and Static IP Based Discovery

You can use both, Multicast and Static IP based discovery together. In this case, in addition to addresses received via multicast, if any, TcpDiscoveryMulticastIpFinder can also work with pre-configured list of static IP addresses, just like Static IP-Based Discovery described above. Here is an example of how to configure Multicast IP finder with static IP addresses:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
          <property name="multicastGroup" value="228.10.10.157"/>
           
          <!-- list of static IP addresses-->
          <property name="addresses">
            <list>
              <value>1.2.3.4</value>
              
              <!-- 
                  IP Address and optional port range.
                  You can also optionally specify an individual port.
              -->
              <value>1.2.3.5:47500..47509</value>
            </list>
          </property>
        </bean>
      </property>
    </bean>
  </property>
</bean>
TcpDiscoverySpi spi = new TcpDiscoverySpi();

TcpDiscoveryMulticastIpFinder ipFinder = new TcpDiscoveryMulticastIpFinder();
 
// Set Multicast group.
ipFinder.setMulticastGroup("228.10.10.157");

// Set initial IP addresses.
// Note that you can optionally specify a port or a port range.
ipFinder.setAddresses(Arrays.asList("1.2.3.4", "1.2.3.5:47500..47509"));
 
spi.setIpFinder(ipFinder);
 
IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

Isolated Ignite Clusters on Same Set of Machines

Ignite allows to start two isolated clusters on the same set of machines. This can be done if nodes from different clusters use non intersecting local port ranges for TcpDiscoverySpi and TcpCommunicationSpi.

Let's say that you need to start two isolated clusters on a single machine for testing purposes.
Then for the nodes from the first cluster, you should use the following TcpDiscoverySpi and TcpCommunicationSpi configurations:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    ...
    <!--
    Explicitly configure TCP discovery SPI to provide list of 
    initial nodes from the first cluster.
    -->
    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <!-- Initial local port to listen to. -->
            <property name="localPort" value="48500"/>

            <!-- Changing local port range. This is an optional action. -->
            <property name="localPortRange" value="20"/>

            <!-- Setting up IP finder for this cluster -->
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                    <property name="addresses">
                        <list>
                            <!--
                            Addresses and port range of nodes from 
                            the first cluster.
                            127.0.0.1 can be replaced with actual IP addresses 
                            or host names. Port range is optional.
                            -->
                            <value>127.0.0.1:48500..48520</value>
                        </list>
                    </property>
                </bean>
            </property>
        </bean>
    </property>

    <!--
    Explicitly configure TCP communication SPI changing local
    port number for the nodes from the first cluster.
    -->
    <property name="communicationSpi">
        <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
            <property name="localPort" value="48100"/>
        </bean>
    </property>
</bean>
IgniteConfiguration cfg = new IgniteConfiguration();

// Explicitly configure TCP discovery SPI to provide list of initial nodes
// from the first cluster.
TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();

// Initial local port to listen to.
discoverySpi.setLocalPort(48500);

// Changing local port range. This is an optional action.
discoverySpi.setLocalPortRange(20);

TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();

// Addresses and port range of the nodes from the first cluster.
// 127.0.0.1 can be replaced with actual IP addresses or host names.
// The port range is optional.
ipFinder.setAddresses(Arrays.asList("127.0.0.1:48500..48520"));

// Overriding IP finder.
discoverySpi.setIpFinder(ipFinder);

// Explicitly configure TCP communication SPI by changing local port number for
// the nodes from the first cluster.
TcpCommunicationSpi commSpi = new TcpCommunicationSpi();

commSpi.setLocalPort(48100);

// Overriding discovery SPI.
cfg.setDiscoverySpi(discoverySpi);

// Overriding communication SPI.
cfg.setCommunicationSpi(commSpi);

// Starting a node.
Ignition.start(cfg);

For the nodes from the second cluster, the configuration can look like this:

<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
    <!--
    Explicitly configure TCP discovery SPI to provide list of initial
    nodes from the second cluster.
    -->
    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <!-- Initial local port to listen to. -->
            <property name="localPort" value="49500"/>

            <!-- Changing local port range. This is an optional action. -->
            <property name="localPortRange" value="20"/>

            <!-- Setting up IP finder for this cluster -->
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                    <property name="addresses">
                        <list>
                            <!--
                            Addresses and port range of the nodes from the second                             cluster.
                            127.0.0.1 can be replaced with actual IP addresses or                             host names. Port range is optional.
                            -->
                            <value>127.0.0.1:49500..49520</value>
                        </list>
                    </property>
                </bean>
            </property>
        </bean>
    </property>

    <!--
    Explicitly configure TCP communication SPI changing local port number 
    for the nodes from the second cluster.
    -->
    <property name="communicationSpi">
        <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
            <property name="localPort" value="49100"/>
        </bean>
    </property>
</bean>
IgniteConfiguration cfg = new IgniteConfiguration();

// Explicitly configure TCP discovery SPI to provide list of initial nodes
// from the second cluster.
TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();

// Initial local port to listen to.
discoverySpi.setLocalPort(49500);

// Changing local port range. This is an optional action.
discoverySpi.setLocalPortRange(20);

TcpDiscoveryVmIpFinder ipFinder=new TcpDiscoveryVmIpFinder();

// Addresses and port range of the nodes from the second cluster.
// 127.0.0.1 can be replaced with actual IP addresses or host names.
// The port range is optional.
ipFinder.setAddresses(Arrays.asList("127.0.0.1:49500..49520"));

// Overriding IP finder.
discoverySpi.setIpFinder(ipFinder);

// Explicitly configure TCP communication SPI by changing local port number for
// the nodes from the second cluster.
TcpCommunicationSpi commSpi = new TcpCommunicationSpi();

commSpi.setLocalPort(49100);

// Overriding discovery SPI.
cfg.setDiscoverySpi(discoverySpi);

// Overriding communication SPI.
cfg.setCommunicationSpi(commSpi);

// Starting a node.
Ignition.start(cfg);

As you see from the configurations the difference between them is minor - only port numbers for SPIs and IP finder vary.

📘

If you want the nodes from different clusters are able to look for each other using multicast protocol then replace TcpDiscoveryVmIpFinder with TcpDiscoveryMulticastIpFinder and set unique TcpDiscoveryMulticastIpFinder.multicastGroups in each configuration above.

🚧

Ignite Persistence Files Location

If the isolated clusters use Ignite persistence, then every cluster has to store its persistence files under different paths in the file system. Use setStoragePath(...), setWalPath(...) and setWalArchivePath(...) methods of DataStorageConfiguration to redefine the paths for every individual cluster.

Apache jclouds Based Discovery

Refer to Generic Cloud Configuration documentation.

Amazon S3 Based Discovery

Refer to AWS S3 Configuration documentation.

Amazon ELB Based Discovery

Refer to AWS ELB Configuration documentation.

Google Cloud Storage Based Discovery

Refer to Google Cloud Configuration documentation.

JDBC Based Discovery

You can have your database be a common shared storage of initial IP addresses. With this IP finder nodes will write their IP addresses to a database on startup. This is done via TcpDiscoveryJdbcIpFinder.

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.jdbc.TcpDiscoveryJdbcIpFinder">
          <property name="dataSource" ref="ds"/>
        </bean>
      </property>
    </bean>
  </property>
</bean>

<!-- Configured data source instance. -->
<bean id="ds" class="some.Datasource">
  ...
</bean>
TcpDiscoverySpi spi = new TcpDiscoverySpi();

// Configure your DataSource.
DataSource someDs = MySampleDataSource(...);

TcpDiscoveryJdbcIpFinder ipFinder = new TcpDiscoveryJdbcIpFinder();

ipFinder.setDataSource(someDs);

spi.setIpFinder(ipFinder);

IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

Shared File System Based Discovery

A shared file system can be used as a storage for nodes' IP addresses. The nodes will write their IP addresses to the file system on startup. This behavior is supported by TcpDiscoverySharedFsIpFinder.

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.sharedfs.TcpDiscoverySharedFsIpFinder">
          <property name="path" value="/var/ignite/addresses"/>
        </bean>
      </property>
    </bean>
  </property>
</bean>
// Configuring discovery SPI.
TcpDiscoverySpi spi = new TcpDiscoverySpi();

// Configuring IP finder.
TcpDiscoverySharedFsIpFinder ipFinder = new TcpDiscoverySharedFsIpFinder();

ipFinder.setPath("/var/ignite/addresses");

spi.setIpFinder(ipFinder);

IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

Kubernetes Service Based Discovery

Refer to Kubernetes Discovery documentation.

ZooKeeper Based Discovery

If you're using ZooKeeper to coordinate your distributed environment, you can utilize it for Ignite nodes discovery as well. This is done via TcpDiscoveryZookeeperIpFinder (note that ignite-zookeeper module has to be enabled).

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    ...
    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.zk.TcpDiscoveryZookeeperIpFinder">
                    <property name="zkConnectionString" value="127.0.0.1:2181"/>
                </bean>
            </property>
        </bean>
    </property>
</bean>
TcpDiscoverySpi spi = new TcpDiscoverySpi();

TcpDiscoveryZookeeperIpFinder ipFinder = new TcpDiscoveryZookeeperIpFinder();

// Specify ZooKeeper connection string.
ipFinder.setZkConnectionString("127.0.0.1:2181");

spi.setIpFinder(ipFinder);

IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

Failure Detection Timeout

Failure detection timeout is used to determine how long a cluster node should wait before considering a remote connection, with another node, failed.

Every node in the Ignite cluster is connected with another node, at the level of discovery SPI. NodeA sends heartbeats (and other system messages transferred over the cluster ring - discovery SPI) to nodeB, and if the latter doesn’t reply in failureDetectionTimeout, then nodeB will be kicked off the cluster.

This timeout is the easiest way to tune discovery SPI's failure detection feature depending on the network and hardware conditions of your environment.

🚧

The timeout automatically controls configuration parameters of TcpDiscoverySpi, such as socket timeout, message acknowledgment timeout and others. If any of these parameters is set explicitly, then the failure timeout setting will be ignored.

The failure detection timeout is configured using IgniteConfiguration.setFailureDetectionTimeout(long) for Apache Ignite server nodes and IgniteConfiguration.setClientFailureDetectionTimeout(long) for client nodes. The default value, that is equal to 10 seconds for the server nodes and 30 seconds for the client nodes. This allows the discovery SPI to work reliably on most of the on-premise and containerized deployments. However, for stable low-latency networks, the parameter can be set to ~200 milliseconds in order to detect and react to​ failures more quickly.

Configuration

Below you can see the most frequently used TcpDiscoverySpi configuration parameters. Refer to TcpDiscoverySpi javadoc to see the full list of configuration options.

Setter Method

Description

Default

setIpFinder(TcpDiscoveryIpFinder)

IP finder that is used to share info about nodes IP addresses.

TcpDiscoveryMulticastIpFinder

Some of the implementations that can be used:
TcpDiscoverySharedFsIpFinder
TcpDiscoveryS3IpFinder
TcpDiscoveryJdbcIpFinder
TcpDiscoveryVmIpFinder

setLocalAddress(String)

Sets local host IP address that discovery SPI uses.

If not provided, by default a first found non-loopback address will be used. If there is no non-loopback address available, then java.net.InetAddress.getLocalHost() will be used.

setLocalPort(int)

Port the SPI listens to.

47500

setLocalPortRange(int)

Local port range.
Local node will try to bind on first available port starting from local port up until local port + local port range.

100

setReconnectCount(int)

Number of times node tries to (re)establish connection to another node.

2

setNetworkTimeout(long)

Sets maximum network timeout in milliseconds to use for network operations.

5000

setSocketTimeout(long)

Sets socket operations timeout. This timeout is used to limit connection time and write-to-socket time.

2000

setAckTimeout(long)

Sets timeout for receiving acknowledgement for sent message.
If acknowledgement is not received within this timeout, sending is considered as failed and SPI tries to repeat message sending.

2000

setJoinTimeout(long)

Sets join timeout. If non-shared IP finder is used and node fails to connect to any address from IP finder, node keeps trying to join within this timeout. If all addresses are still unresponsive, exception is thrown and node startup fails.
0 means wait forever.

0

setThreadPriority(int)

Thread priority for threads started by SPI.

0

setStatisticsPrintFrequency(int)

Statistics print frequency in milliseconds.
0 indicates that no print is required. If value is greater than 0 and log is not quiet then stats are printed out with INFO level once a period. This may be very helpful for tracing topology problems.

true

Updated less than a minute ago

Cluster Configuration


Deploy Ignite in any environment with pluggable automatic node discovery.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.