Apache Ignite cluster can be easily deployed in and maintained by Kubernetes which is an open-source system for automating deployment, scaling, and management of containerized applications.
This getting started guide walks you through how to deploy an Apache Ignite cluster in Kubernetes environment.
Using minikube with Apple Macintosh Computers
A very useful tool for testing Kubernetes locally is to use Minikube. On Apple Macintosh Computers, it is necessary to use the xhyve driver to ensure that persistent volumes can be correctly accessed, as follows:
minikube start --vm-driver=xhyve
Furthermore, you might need to modify the directory path "/data/ignite" that is used in the examples below to "/Users/<username>/data/ignite", where <username> is substituted with the actual user name.
To enable Apache Ignite nodes auto-discovery in Kubernetes, you need to enable
IgniteConfiguration. Let's create an example configuration file called
example-kube.xml and define the IP finder configuration as follows:
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:util="http://www.springframework.org/schema/util" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd"> <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration"> <property name="discoverySpi"> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> <property name="ipFinder"> <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder"> </bean> </property> </bean> </property> </bean> </beans>
Kubernetes IP finder
To learn more about Kubernetes IP finder and Apache Ignite nodes auto-discovery in Kuberentes environment, refer to this documentation page.
KubernetesIPFinder requires users to configure and deploy a special Kubernetes service that maintains a list of the IP addresses of all the alive Ignite pods (nodes).
Every time a new Ignite pod is started, the IP finder will connect to the service via the Kubernetes API to obtain the list of the existing Ignite pods' addresses. Using these addresses, the new node will be able to discover the rest of the cluster nodes and finally join the Apache Ignite cluster.
Let's configure the service the following way:
apiVersionv1 kindService metadata # Name of Ignite Service used by Kubernetes IP finder. # The name must be equal to TcpDiscoveryKubernetesIpFinder.serviceName. nameignite spec clusterIPNone # custom value. ports port9042 # custom value. selector # Must be equal to one of the labels set in Ignite pods' # deployement configuration. appignite
and deploy it in Kubernetes using the command below:
kubectl create -f ignite-service.yaml
Make sure the service is up and running:
kubectl get svc ignite
The output should be similar to the one below:
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE ignite None <none> 9042/TCP 29s
Before you start deploying Ignite pods in Kubernetes using Apache Ignite docker image, you need to find a way on how to pass
example-kube.xml (prepared above) to that docker image.
There are several approaches you can use. Here, we show you how to share the Ignite cluster configuration via a shared Kubernetes
Sharing the configuration via an HTTP server
Deploying Apache Ignite in Kubernetes on Microsoft Azure blog post explains how to share an Apache Ignite configuration over an HTTP server.
Let's suppose that you have some shared directory named
/data/ignite which can be accessed by any Ignite pod. Go to this directory and copy
PersistentVolume configuration that will be backed by your real storage and will refer to the
kindPersistentVolume apiVersionv1 metadata nameignite-volume labels typelocal spec storageClassNamemanual capacity storage1Gi accessModes ReadWriteOnce hostPath path"/data/ignite"
Deploy the volume using the command below:
kubectl create -f ignite-volume.yaml
Check that the volume was deployed and available for usage:
kubectl get pv ignite-volume
The output should be similar to the one below:
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE ignite-volume 1Gi RWO Retain Available 3m
Finally, create a
PersistentVolumeClaim that will be automatically bound to the
PersistentVolume initiated above:
kindPersistentVolumeClaim apiVersionv1 metadata nameignite-volume-claim spec storageClassNamemanual accessModes ReadWriteOnce resources requests storage1Gi
Create the PersistentVolumeClaim using the configuration above:
kubectl create -f ignite-volume-claim.yaml
Make sure that the
PersistentVolumeClaim gets created and was bound to the
PersistentVolume that stores
kubectl get pvc ignite-volume-claim NAME STATUS VOLUME CAPACITY ACCESSMODES AGE ignite-volume-claim Bound ignite-volume 1Gi RWO 2m kubectl get pv ignite-volume NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE ignite-volume 1Gi RWO Retain Bound default/ignite-volume-claim 13m
Kubernetes will look for a volume that matches the
storageClassName of the
PersistentVolumeClaim. If no
PersistentVolume matches the
storageClassName requested by the
PersistentVolumeClaim, Kubernetes will dynamically create a new
PersistentVolume. You can see all
PersistentVolumes by running:
kubectl get pv
storageClassName did not match, you might see something like:
kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE ignite-volume-claim Bound pvc-c3f0f289-9987-11e7-876b-525400ed8e86 1Gi RWO standard 9m kubectl get pv NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE ignite-volume 1Gi RWO Retain Available 9m pvc-c3f0f289-9987-11e7-876b-525400ed8e86 1Gi RWO Delete Bound default/ignite-volume-claim standard 9m
Now, it's time to prepare a Kubernetes deployment configuration for Ignite pods and deploy them
Finally, let's define a YAML configuration for Ignite pods:
# An example of a Kubernetes configuration for Ignite pods deployment. apiVersionextensions/v1beta1 kindDeployment metadata # Custom Ignite cluster's name. nameignite-cluster spec # A number of Ignite pods to be started by Kubernetes initially. replicas2 template metadata labels # This label has to be added to the selector's section of # ignite-service.yaml so that the Kubernetes Ignite lookup service # can easily track all Ignite pods available deployed so far. appignite spec volumes # Custom name for the storage that holds Ignite's configuration # which is example-kube.xml. nameignite-storage persistentVolumeClaim # Must be equal to the PersistentVolumeClaim created before. claimNameignite-volume-claim containers # Custom Ignite pod name. nameignite-node # Ignite Docker image. Kubernetes IP finder is supported starting from # Apache Ignite 1.9.0 imageapacheignite/ignite1.9.0 env # Ignite's Docker image parameter. Adding the jar file that # contain TcpDiscoveryKubernetesIpFinder implementation. nameOPTION_LIBS valueignite-kubernetes # Ignite's Docker image parameter. Passing the Ignite configuration # to use for an Ignite pod. nameCONFIG_URI valuefile ////data/ignite/example-kube.xml ports # Ports to open. # Might be optional depending on your Kubernetes environment. containerPort11211 # REST port number. containerPort47100 # communication SPI port number. containerPort47500 # discovery SPI port number. containerPort49112 # JMX port number. volumeMounts # Mounting the storage with the Ignite configuration. mountPath"/data/ignite" nameignite-storage
As you can see, the configuration defines a couple of environment variables (
CONFIG_URIL) that will be processed by a special shell script used by Ignite's docker image. The full list of docker image's configuration parameters is available on Docker Deployment page.
Ignite Docker Image Version
Kubernetes is supported in Apache Ignite 1.9 and later versions. Makes sure to use a docker image with a valid version. You can find a full list of tags here.
Ignite Configuration Shared over HTTP
If it's tricky for you to share the Apache Ignite configuration via
PersistentVolumeClaim then you can upload it to an HTTP server and set the link as a value of
Next, go ahead and deploy Ignite pods in Kubernetes using the configuration above:
kubectl create -f ignite-deployment.yaml
Check that Ignite pods are up and running:
kubectl get pods
Pick a name of one of the pods available
NAME READY STATUS RESTARTS AGE ignite-cluster-3454482164-d4m6g 1/1 Running 0 25m ignite-cluster-3454482164-w0xtx 1/1 Running 0 25m
and get the logs from it making sure that both Ignite pods were able to discover each other and from the cluster:
kubectl logs ignite-cluster-3454482164-d4m6g
You can adjust Apache Ignite cluster size on the fly using the standard Kubernetes API. For instance, if you want to scale out the cluster from 2 to 5 nodes then the command below can be used:
kubectl scale --replicas=5 -f ignite-deployment.yaml
Double check the cluster was scaled out successfully:
kubectl get pods
The output should show that you now have 5 Ignite pods up and running:
NAME READY STATUS RESTARTS AGE ignite-cluster-3454482164-d4m6g 1/1 Running 0 34m ignite-cluster-3454482164-ktkrr 1/1 Running 0 58s ignite-cluster-3454482164-r20f8 1/1 Running 0 58s ignite-cluster-3454482164-vf8kh 1/1 Running 0 58s ignite-cluster-3454482164-w0xtx 1/1 Running 0 34m
Deploying Apache Ignite in Kubernetes on Microsoft Azure article provides a step-by-step guide on how to run the cluster on Microsoft Azure.
Kubernetes and Apache Ignite Deployment on AWS article provides a step-by-step guide on how to run the cluster on Amazon AWS.