Apache Ignite Documentation

GridGain Developer Hub - Apache Ignitetm

Welcome to the Apache Ignite developer hub run by GridGain. Here you'll find comprehensive guides and documentation to help you start working with Apache Ignite as quickly as possible, as well as support if you get stuck.

 

GridGain also provides Community Edition which is a distribution of Apache Ignite made available by GridGain. It is the fastest and easiest way to get started with Apache Ignite. The Community Edition is generally more stable than the Apache Ignite release available from the Apache Ignite website and may contain extra bug fixes and features that have not made it yet into the release on the Apache website.

 

Let's jump right in!

 

Documentation     Ask a Question     Download

 

Javadoc     Scaladoc     Examples

YARN Deployment

Deploy Ignite in YARN cluster.

Overview

Integration with YARN supports scheduling and running Apache Ignite nodes in a YARN cluster.
YARN is a resource negotiator which provides a general runtime environment providing all the essentials you need to deploy, run, and manage distributed applications. Its resource manager and isolation helps in getting the most out of servers.
For information about YARN, refer to http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html

Ignite YARN Application

Deploying Apache Ignite cluster typically involves downloading the Apache Ignite distribution, changing configuration settings, and starting up the nodes. Integration with YARN allows you to avoid these actions. Apache Ignite Yarn Application greatly simplifies cluster deployment. The application consists of the following components:

  • Client downloads Ignite distribution, puts necessary resources into HDFS, creates the necessary context for launching the task, launches the ApplicationMaster process.
  • Application master. Once registration is successful the component will begin requesting resources from Resource Manager for use with Apache Ignite nodes. The Application Master will maintain the Ignite cluster at the desired total resources level (CPU, memory, etc).
  • Container - the entity that runs Ignite Node on slaves.

Running Ignite YARN Application

The Ignite Application requires that YARN and the Hadoop cluster are configured and running. For information on how to set up the cluster please refer to: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html

  1. Download Apache Ignite.
  2. Configure the properties file. Update any parameters you would like to change. See the Configuration section below.
# The number of nodes in the cluster.
IGNITE_NODE_COUNT=2
 
# The number of CPU Cores for each Apache Ignite node.
IGNITE_RUN_CPU_PER_NODE=1
 
# The number of Megabytes of RAM for each Apache Ignite node.
IGNITE_MEMORY_PER_NODE=2048
 
# The version of Ignite which will be run on nodes.
IGNITE_VERSION=2.3.0

# URL where Ignite distribution can be downloaded from
IGNITE_URL=http://mirror.linux-ia64.org/apache/ignite/2.7.0/apache-ignite-2.7.0-bin.zip

# You can also provide a path to unzipped Ignite distribution instead of the URL
# IGNITE_PATH=/ignite/apache-ignite-2.7.0-bin
 
  1. Run the application.

yarn jar ignite-yarn-<ignite-version>.jar ./ignite-yarn-<ignite-version>.jar cluster.properties

  1. In order to make sure that Application deployed correctly: open the YARN console at http://<hostname>:8088/cluster. If everything is working as expected, you will see an application named Ignition.
  1. Retrieve logs from the browser. To look through Ignite logs, click on Logs for any containers.
  1. Click on stdout to get stdout logs and on stderr to get stderr logs.

Configuration

All configurations are handled through environment variables or the property file. The following configuration parameters can be optionally configured.

Name
Description
Default
Example

IGNITE_XML_CONFIG

The HDFS path to the Apache Ignite config file.

N/A

/opt/ignite/ignite-config.xml

IGNITE_WORKING_DIR

The directory which will be used for saving Apache Ignite distribution.

./ignite-release

/opt/ignite/

IGNITE_RELEASES_DIR

The HDFS directory which will be used for saving Apache Ignite distribution.

/ignite/releases/

/ignite-rel/

IGNITE_USERS_LIBS

The HDFS path to libs which will be added to classpath.

N/A

/opt/libs/

IGNITE_MEMORY_PER_NODE

The number of megabytes of RAM for each Apache Ignite node. This is the size of the Java heap. This includes on-heap caching if it is used.

2048

1024

IGNITE_MEMORY_OVERHEAD_PER_NODE

The amount of memory necessary for all data regions, with padding for JVM native overhead, interned Strings, etc. This setting should always be adjusted for nodes that are used to store data, not just for pure computations. Memory requested to YARN for containers running an Ignite node is the sum of IGNITE_MEMORY_PER_NODE and IGNITE_MEMORY_OVERHEAD_PER_NODE.

IGNITE_MEMORY_PER_NODE * 0.10, with a minimum of 384

17408

IGNITE_HOSTNAME_CONSTRAINT

The constraint on slave hosts.

N/A

192.168.0.[1-100]

IGNITE_NODE_COUNT

The number of nodes in the cluster.

3

10

IGNITE_RUN_CPU_PER_NODE

The number of CPU Cores for each Apache Ignite node.

2

4

IGNITE_VERSION

The version of Ignite which will be run on nodes.

latest

1.0.5

IGNITE_PATH

The HDFS path to the Apache Ignite build. This property can be useful when the yarn
is cluster running in net without internet access.

N/A

/ignite/apache-ignite-fabric-1.7.0-bin.zip

IGNITE_URL

Location where Ignite binary distribution is stored to be downloaded for delivery. As per version 2.7, either IGNITE_PATH or IGNITE_URL is mandatory in practice.

N/A

http://mirror.linux-ia64.org/apache/ignite/2.7.0/apache-ignite-2.7.0-bin.zip

IGNITE_JVM_OPTS

Additional JVM options.

N/A

-XX:+PrintGC

YARN Deployment


Deploy Ignite in YARN cluster.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.