Integration with YARN supports scheduling and running Apache Ignite nodes in a YARN cluster.
YARN is a resource negotiator which provides a general runtime environment providing all the essentials to deploy, run and manage distributed applications. Its resource manager and isolation helps getting the most out of servers.
For information about YARN, refer to http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
Deploying Apache Ignite cluster typically involves downloading the Apache Ignite distribution, changing configuration settings and starting the nodes up. Integration with YARN allows to avoid the actions. Apache Ignite Yarn Application allows to greatly simplify cluster deployment. The application consist from the following components:
- Client downloads Ignite distributive, puts necessary resources to HDFS, creates the necessary context for launching the task, launches the ApplicationMaster process.
Application master. Once registration is successful the component will begin requesting of resource from Resource Manager to utilize resources for Apache Ignite nodes. The
Application Masterwill maintain the Ignite cluster at desired total resources level (CPU, memory, etc).
Container- the entity that runs Ignite Node on slaves.
For running Ignite Application requires YARN and Hadoop cluster are configured and running. For information on how to set up a the cluster please refer to http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html
- Download Apache Ignite.
- Configure properties file. Update any parameters which would like to change. See Configuration section below.
# The number of nodes in the cluster. IGNITE_NODE_COUNT=2 # The number of CPU Cores for each Apache Ignite node. IGNITE_RUN_CPU_PER_NODE=1 # The number of Megabytes of RAM for each Apache Ignite node. IGNITE_MEMORY_PER_NODE=2048 # The version of Ignite which will be run on nodes. IGNITE_VERSION=2.3.0
- Run the application.
yarn jar ignite-yarn-<ignite-version>.jar ./ignite-yarn-<ignite-version>.jar cluster.properties
- In order to make sure that Application deployed correctly, do the following. Open YARN console at http://<hostname>:8088/cluster. If everything works OK then application with
- Retrieve logs from the browser. To look through Ignite logs, click on
Logsfor any containers.
- Click on
stdoutto get stdout logs and on
stderrto get stderr logs.
All configurations are handled through environment variables or property file. Following configuration parameters can be optionally configured.
The hdfs path to Apache Ignite config file.
The directory which will be used for saving Apache Ignite distributives.
The hdfs directory which will be used for saving Apache Ignite distributives.
The hdfs path to libs which will be added to classpath.
The number of Megabytes of RAM for each Apache Ignite node. This is the size of the Java heap. This includes on-heap caching if it is used.
The amount of memory necessary for all data regions, with padding for JVM native overhead, interned Strings, etc. This setting should always be adjusted for nodes that are used to store data, not just for pure computations. Memory requested to YARN for containers running Ignite node is the sum of IGNITE_MEMORY_PER_NODE and IGNITE_MEMORY_OVERHEAD_PER_NODE.
IGNITE_MEMORY_PER_NODE * 0.10, with a minimum of 384
The constraint on slave hosts.
The number of nodes in the cluster.
The number of CPU Cores for each Apache Ignite node.
The version ignite which will be run on nodes.
The hdfs path to Apache Ignite build. The property can be useful when yarn cluster running in net without access to internet.
The JVM options.