Apache Ignite Documentation

GridGain Developer Hub - Apache Ignitetm

Welcome to the Apache Ignite developer hub run by GridGain. Here you'll find comprehensive guides and documentation to help you start working with Apache Ignite as quickly as possible, as well as support if you get stuck.

 

GridGain also provides Community Edition which is a distribution of Apache Ignite made available by GridGain. It is the fastest and easiest way to get started with Apache Ignite. The Community Edition is generally more stable than the Apache Ignite release available from the Apache Ignite website and may contain extra bug fixes and features that have not made it yet into the release on the Apache website.

 

Let's jump right in!

 

Documentation     Ask a Question     Download

 

Javadoc     Scaladoc     Examples

TCP/IP Discovery

Overview

In Ignite, nodes can discover each other by using DiscoverySpi. Ignite provides TcpDiscoverySpi as a default implementation of DiscoverySpi that uses TCP/IP for node discovery. Discovery SPI can be configured for Multicast and Static IP based node discovery.

Multicast IP Finder

TcpDiscoveryMulticastIpFinder uses Multicast to discover other nodes in the grid and is the default IP finder. You should not have to specify it unless you plan to override default settings. Here is an example of how to configure this finder via a Spring XML file or programmatically from Java:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
          <property name="multicastGroup" value="228.10.10.157"/>
        </bean>
      </property>
    </bean>
  </property>
</bean>
TcpDiscoverySpi spi = new TcpDiscoverySpi();
 
TcpDiscoveryMulticastIpFinder ipFinder = new TcpDiscoveryMulticastIpFinder();
 
ipFinder.setMulticastGroup("228.10.10.157");
 
spi.setIpFinder(ipFinder);
 
IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

Static IP Finder

For cases when Multicast is disabled, TcpDiscoveryVmIpFinder should be used with pre-configured list of IP addresses.

You are only required to provide at least one IP address of a remote node, but usually it is advisable to provide 2 or 3 addresses of grid nodes that you plan to start at some point of time in the future. Once a connection to any of the provided IP addresses is established, Ignite will automatically discover all other grid nodes.

Instead of specifying addresses in configuration, you can specify them in the IGNITE_TCP_DISCOVERY_ADDRESSES environment variable or in the system property with the same name. Addresses should be comma separated and may contain an optional port range.

By default the TcpDiscoveryVmIpFinder is used in non-shared mode. If you plan to start a server node then in this mode the list of IP addresses should contain an address of the local node as well. It will let the node not to wait while other nodes join the cluster but rather become the first cluster node and operate normally.

Here is an example of how to configure this finder via a Spring XML file or programmatically from Java:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
          <property name="addresses">
            <list>
              <!-- 
              Explicitly specifying address of a local node to let it start and
              operate normally even if there is no more nodes in the cluster. 
              You can also optionally specify an individual port or port range.
              -->
              <value>1.2.3.4</value>
              
              <!-- 
              IP Address and optional port range of a remote node.
              You can also optionally specify an individual port.
              -->
              <value>1.2.3.5:47500..47509</value>
            </list>
          </property>
        </bean>
      </property>
    </bean>
  </property>
</bean>
TcpDiscoverySpi spi = new TcpDiscoverySpi();
 
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
 
// Set initial IP addresses.
// Note that you can optionally specify a port or a port range.
ipFinder.setAddresses(Arrays.asList("1.2.3.4", "1.2.3.5:47500..47509"));
 
spi.setIpFinder(ipFinder);
 
IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);
# The configuration should use TcpDiscoveryVmIpFinder without addresses specified:

IGNITE_TCP_DISCOVERY_ADDRESSES=1.2.3.4,1.2.3.5:47500..47509 bin/ignite.sh -v config/default-config.xml

Multicast and Static IP Finder

You can use both, Multicast and Static IP based discovery together. In this case, in addition to addresses received via multicast, if any, TcpDiscoveryMulticastIpFinder can also work with pre-configured list of static IP addresses, just like Static IP-Based Discovery described above. Here is an example of how to configure Multicast IP finder with static IP addresses:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
          <property name="multicastGroup" value="228.10.10.157"/>
           
          <!-- list of static IP addresses-->
          <property name="addresses">
            <list>
              <value>1.2.3.4</value>
              
              <!-- 
                  IP Address and optional port range.
                  You can also optionally specify an individual port.
              -->
              <value>1.2.3.5:47500..47509</value>
            </list>
          </property>
        </bean>
      </property>
    </bean>
  </property>
</bean>
TcpDiscoverySpi spi = new TcpDiscoverySpi();

TcpDiscoveryMulticastIpFinder ipFinder = new TcpDiscoveryMulticastIpFinder();
 
// Set Multicast group.
ipFinder.setMulticastGroup("228.10.10.157");

// Set initial IP addresses.
// Note that you can optionally specify a port or a port range.
ipFinder.setAddresses(Arrays.asList("1.2.3.4", "1.2.3.5:47500..47509"));
 
spi.setIpFinder(ipFinder);
 
IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

Isolated Ignite Clusters on Same Set of Machines

Ignite allows you to start two isolated clusters on the same set of machines. This can be done if nodes from different clusters use non intersecting local port ranges for TcpDiscoverySpi and TcpCommunicationSpi.

Let's say that you need to start two isolated clusters on a single machine for testing purposes.
For the nodes from the first cluster, you should use the following TcpDiscoverySpi and TcpCommunicationSpi configurations:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    ...
    <!--
    Explicitly configure TCP discovery SPI to provide list of 
    initial nodes from the first cluster.
    -->
    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <!-- Initial local port to listen to. -->
            <property name="localPort" value="48500"/>

            <!-- Changing local port range. This is an optional action. -->
            <property name="localPortRange" value="20"/>

            <!-- Setting up IP finder for this cluster -->
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                    <property name="addresses">
                        <list>
                            <!--
                            Addresses and port range of nodes from 
                            the first cluster.
                            127.0.0.1 can be replaced with actual IP addresses 
                            or host names. Port range is optional.
                            -->
                            <value>127.0.0.1:48500..48520</value>
                        </list>
                    </property>
                </bean>
            </property>
        </bean>
    </property>

    <!--
    Explicitly configure TCP communication SPI changing local
    port number for the nodes from the first cluster.
    -->
    <property name="communicationSpi">
        <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
            <property name="localPort" value="48100"/>
        </bean>
    </property>
</bean>
IgniteConfiguration cfg = new IgniteConfiguration();

// Explicitly configure TCP discovery SPI to provide list of initial nodes
// from the first cluster.
TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();

// Initial local port to listen to.
discoverySpi.setLocalPort(48500);

// Changing local port range. This is an optional action.
discoverySpi.setLocalPortRange(20);

TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();

// Addresses and port range of the nodes from the first cluster.
// 127.0.0.1 can be replaced with actual IP addresses or host names.
// The port range is optional.
ipFinder.setAddresses(Arrays.asList("127.0.0.1:48500..48520"));

// Overriding IP finder.
discoverySpi.setIpFinder(ipFinder);

// Explicitly configure TCP communication SPI by changing local port number for
// the nodes from the first cluster.
TcpCommunicationSpi commSpi = new TcpCommunicationSpi();

commSpi.setLocalPort(48100);

// Overriding discovery SPI.
cfg.setDiscoverySpi(discoverySpi);

// Overriding communication SPI.
cfg.setCommunicationSpi(commSpi);

// Starting a node.
Ignition.start(cfg);

For the nodes from the second cluster, the configuration can look like this:

<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
    <!--
    Explicitly configure TCP discovery SPI to provide list of initial
    nodes from the second cluster.
    -->
    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <!-- Initial local port to listen to. -->
            <property name="localPort" value="49500"/>

            <!-- Changing local port range. This is an optional action. -->
            <property name="localPortRange" value="20"/>

            <!-- Setting up IP finder for this cluster -->
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                    <property name="addresses">
                        <list>
                            <!--
                            Addresses and port range of the nodes from the second                             cluster.
                            127.0.0.1 can be replaced with actual IP addresses or                             host names. Port range is optional.
                            -->
                            <value>127.0.0.1:49500..49520</value>
                        </list>
                    </property>
                </bean>
            </property>
        </bean>
    </property>

    <!--
    Explicitly configure TCP communication SPI changing local port number 
    for the nodes from the second cluster.
    -->
    <property name="communicationSpi">
        <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
            <property name="localPort" value="49100"/>
        </bean>
    </property>
</bean>
IgniteConfiguration cfg = new IgniteConfiguration();

// Explicitly configure TCP discovery SPI to provide list of initial nodes
// from the second cluster.
TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();

// Initial local port to listen to.
discoverySpi.setLocalPort(49500);

// Changing local port range. This is an optional action.
discoverySpi.setLocalPortRange(20);

TcpDiscoveryVmIpFinder ipFinder=new TcpDiscoveryVmIpFinder();

// Addresses and port range of the nodes from the second cluster.
// 127.0.0.1 can be replaced with actual IP addresses or host names.
// The port range is optional.
ipFinder.setAddresses(Arrays.asList("127.0.0.1:49500..49520"));

// Overriding IP finder.
discoverySpi.setIpFinder(ipFinder);

// Explicitly configure TCP communication SPI by changing local port number for
// the nodes from the second cluster.
TcpCommunicationSpi commSpi = new TcpCommunicationSpi();

commSpi.setLocalPort(49100);

// Overriding discovery SPI.
cfg.setDiscoverySpi(discoverySpi);

// Overriding communication SPI.
cfg.setCommunicationSpi(commSpi);

// Starting a node.
Ignition.start(cfg);

As you see from the configurations the difference between them is minor - only port numbers for SPIs and IP finder vary.

If you want the nodes from different clusters to be able to look for each other using multicast protocol, replace TcpDiscoveryVmIpFinder with TcpDiscoveryMulticastIpFinder and set unique TcpDiscoveryMulticastIpFinder.multicastGroups in each configuration above.

Ignite Persistence Files Location

If the isolated clusters use Ignite persistence, then every cluster has to store its persistence files under different paths in the file system. Use the setStoragePath(...), setWalPath(...) and setWalArchivePath(...) methods of DataStorageConfiguration to redefine the paths for every individual cluster.

Apache jclouds IP Finder

Refer to Generic Cloud Configuration documentation.

Amazon S3 IP Finder

Refer to AWS S3 Configuration documentation.

Amazon ELB IP Finder

Refer to AWS ELB Configuration documentation.

Google Cloud Storage IP Finder

Refer to Google Cloud Configuration documentation.

JDBC Based IP Finder

You can have your database be a common shared storage of initial IP addresses. With this IP finder, nodes will write their IP addresses to a database on startup. This is done via TcpDiscoveryJdbcIpFinder.

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.jdbc.TcpDiscoveryJdbcIpFinder">
          <property name="dataSource" ref="ds"/>
        </bean>
      </property>
    </bean>
  </property>
</bean>

<!-- Configured data source instance. -->
<bean id="ds" class="some.Datasource">
  ...
</bean>
TcpDiscoverySpi spi = new TcpDiscoverySpi();

// Configure your DataSource.
DataSource someDs = MySampleDataSource(...);

TcpDiscoveryJdbcIpFinder ipFinder = new TcpDiscoveryJdbcIpFinder();

ipFinder.setDataSource(someDs);

spi.setIpFinder(ipFinder);

IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

Shared File System IP Finder

A shared file system can be used as a storage for nodes' IP addresses. The nodes will write their IP addresses to the file system on startup. This behavior is supported by TcpDiscoverySharedFsIpFinder.

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.sharedfs.TcpDiscoverySharedFsIpFinder">
          <property name="path" value="/var/ignite/addresses"/>
        </bean>
      </property>
    </bean>
  </property>
</bean>
// Configuring discovery SPI.
TcpDiscoverySpi spi = new TcpDiscoverySpi();

// Configuring IP finder.
TcpDiscoverySharedFsIpFinder ipFinder = new TcpDiscoverySharedFsIpFinder();

ipFinder.setPath("/var/ignite/addresses");

spi.setIpFinder(ipFinder);

IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

Kubernetes IP Finder

Refer to Kubernetes IP Finder documentation.

ZooKeeper IP Finder

If you're using ZooKeeper to coordinate your distributed environment, you can utilize it as:

  • Foundation of the overall discovery component (see ZooKeeper Discovery for more details).
  • Use it as an IP finder. This section explains this use case.

To set up ZooKeeper IP finder use TcpDiscoveryZookeeperIpFinder (note that ignite-zookeeper module has to be enabled).

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    ...
    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.zk.TcpDiscoveryZookeeperIpFinder">
                    <property name="zkConnectionString" value="127.0.0.1:2181"/>
                </bean>
            </property>
        </bean>
    </property>
</bean>
TcpDiscoverySpi spi = new TcpDiscoverySpi();

TcpDiscoveryZookeeperIpFinder ipFinder = new TcpDiscoveryZookeeperIpFinder();

// Specify ZooKeeper connection string.
ipFinder.setZkConnectionString("127.0.0.1:2181");

spi.setIpFinder(ipFinder);

IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

Failure Detection Timeout

Failure detection timeout is used to determine how long a cluster node should wait before considering a remote connection, with another node, failed.

Every node in the Ignite cluster is connected with another node, at the level of discovery SPI. NodeA sends heartbeats (and other system messages transferred over the cluster ring - discovery SPI) to nodeB, and if the latter doesn’t reply in failureDetectionTimeout, then nodeB will be kicked off the cluster.

This timeout is the easiest way to tune discovery SPI's failure detection feature depending on the network and hardware conditions of your environment.

The timeout automatically controls configuration parameters of TcpDiscoverySpi, such as socket timeout, message acknowledgment timeout and others. If any of these parameters is set explicitly, then the failure timeout setting will be ignored.

The failure detection timeout is configured using IgniteConfiguration.setFailureDetectionTimeout(long) for Apache Ignite server nodes and IgniteConfiguration.setClientFailureDetectionTimeout(long) for client nodes. The default value is equal to 10 seconds for the server nodes and 30 seconds for the client nodes. This allows the discovery SPI to work reliably on most of the on-premise and containerized deployments. However, for stable low-latency networks, the parameter can be set to ~200 milliseconds in order to detect and react to​ failures more quickly.

Configuration

Below you can see the most frequently used TcpDiscoverySpi configuration parameters. Refer to TcpDiscoverySpi javadoc to see the full list of configuration options.

Setter Method
Description
Default

setIpFinder(TcpDiscoveryIpFinder)

IP finder that is used to share info about nodes IP addresses.

TcpDiscoveryMulticastIpFinder

Some of the implementations that can be used:
TcpDiscoverySharedFsIpFinder
TcpDiscoveryS3IpFinder
TcpDiscoveryJdbcIpFinder
TcpDiscoveryVmIpFinder

setLocalAddress(String)

Sets local host IP address that discovery SPI uses.

If not provided, by default a first found non-loopback address will be used. If there is no non-loopback address available, then java.net.InetAddress.getLocalHost() will be used.

setLocalPort(int)

Port the SPI listens to.

47500

setLocalPortRange(int)

Local port range.
Local node will try to bind on first available port starting from local port up until local port + local port range.

100

setReconnectCount(int)

Number of times node tries to (re)establish connection to another node.

2

setNetworkTimeout(long)

Sets maximum network timeout in milliseconds to use for network operations.

5000

setSocketTimeout(long)

Sets socket operations timeout. This timeout is used to limit connection time and write-to-socket time.

2000

setAckTimeout(long)

Sets timeout for receiving acknowledgement for sent message.
If acknowledgement is not received within this timeout, sending is considered as failed and SPI tries to repeat message sending.

2000

setJoinTimeout(long)

Sets join timeout. If non-shared IP finder is used and node fails to connect to any address from IP finder, node keeps trying to join within this timeout. If all addresses are still unresponsive, exception is thrown and node startup fails.
0 means wait forever.

0

setThreadPriority(int)

Thread priority for threads started by SPI.

0

setStatisticsPrintFrequency(int)

Statistics print frequency in milliseconds.
0 indicates that no print is required. If value is greater than 0 and log is not quiet then stats are printed out with INFO level once a period. This may be very helpful for tracing topology problems.

true

Updated 17 days ago

TCP/IP Discovery


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.