Apache Ignite

GridGain Developer Hub - Apache Ignitetm

Welcome to the Apache Ignite developer hub run by GridGain. Here you'll find comprehensive guides and documentation to help you start working with Apache Ignite as quickly as possible, as well as support if you get stuck.

 

GridGain also provides Community Edition which is a distribution of Apache Ignite made available by GridGain. It is the fastest and easiest way to get started with Apache Ignite. The Community Edition is generally more stable than the Apache Ignite release available from the Apache Ignite website and may contain extra bug fixes and features that have not made it yet into the release on the Apache website.

 

Let's jump right in!

 

Documentation     Ask a Question     Download

 

Javadoc     Scaladoc     Examples
Ask A Question

Questions

1

Problems Running Ignite from Command Line

Hey All, When starting to learn ignite, I ended up using IntelliJ as an IDE. However I need to move it over onto a supercomputer environment and need to be able to run it from the command line but am having trouble with my code being able to find the needed classes. I created my project using maven added the proper dependencies as mentioned in the getting started guide. Following that I had maven package the project and used the following command: ```java -cp "${IGNITE_HOME}/modules/core/target/ignite-core-1.6.0.jar:${IGNITE_HOME}/modules/core/target/libs/cache-api-1.0.0.jar:${IGNITE_HOME}/modules/spring/target/ignite-spring-1.6.0.jar:target/my-app-1.0-SNAPSHOT.jar" com.mycompany.app.App``` However it is failing to create an ignite component with the following error: ```Exception in thread "main" class org.apache.ignite.IgniteException: Failed to create Ignite component (consider adding ignite-spring module to classpath) [component=SPRING, cls=org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl] at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:906) at org.apache.ignite.Ignition.start(Ignition.java:350) at com.mycompany.app.App.main(App.java:24) Caused by: class org.apache.ignite.IgniteCheckedException: Failed to create Ignite component (consider adding ignite-spring module to classpath) [component=SPRING, cls=org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl] at org.apache.ignite.internal.IgniteComponentType.componentException(IgniteComponentType.java:313) at org.apache.ignite.internal.IgniteComponentType.create0(IgniteComponentType.java:289) at org.apache.ignite.internal.IgniteComponentType.create(IgniteComponentType.java:200) at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:637) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:840) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:749) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:619) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:589) at org.apache.ignite.Ignition.start(Ignition.java:347) ... 1 more Caused by: java.lang.NoClassDefFoundError: org/springframework/beans/BeansException at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.ignite.internal.IgniteComponentType.create0(IgniteComponentType.java:275) ... 8 more Caused by: java.lang.ClassNotFoundException: org.springframework.beans.BeansException at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 11 more``` Looking up other similar problems this occurs from not including the ignite-spring dependency in the pom file which I have done. The following is that file: ```<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.mycompany.app</groupId> <artifactId>my-app</artifactId> <packaging>jar</packaging> <version>1.0-SNAPSHOT</version> <name>my-app</name> <url>http://maven.apache.org</url> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.5.1</version> <configuration> <source>1.8</source> <target>1.8</target> </configuration> </plugin> </plugins> </build> <dependencies> <dependency> <groupId>org.apache.ignite</groupId> <artifactId>ignite-core</artifactId> <version>1.6.0</version> </dependency> <dependency> <groupId>org.apache.ignite</groupId> <artifactId>ignite-spring</artifactId> <version>1.6.0</version> </dependency> <dependency> <groupId>org.apache.ignite</groupId> <artifactId>ignite-indexing</artifactId> <version>1.6.0</version> </dependency> </dependencies> </project>``` Where am I going wrong in this instance. If needed, the following is the beginning of my code where the error is occuring: ```package com.mycompany.app; import org.apache.ignite.Ignite; import org.apache.ignite.IgniteCache; import org.apache.ignite.Ignition; import java.io.BufferedReader; import java.io.FileReader; import java.io.IOException; import java.net.URL; import java.net.URLClassLoader; import java.util.StringTokenizer; public class App { public static void main( String[] args ) { printClasspath(); String data; try { data = readFile("tweet.txt"); Ignite ignite = Ignition.start("/ignite-1.6.0/examples/config/example-ignite.xml"); IgniteCache<Integer, String[]> cache = ignite.getOrCreateCache("myCacheName");'''

Posted by Jeremy London 7 months ago

2
ANSWERED

How to compute weighted average on ignite

Hello, I am currently benchmarking Appache Ignite for a near real-time application and simple operations seem to be excessively slow for a relatively small sample size. The following is giving the setup details and timings - please see 2 questions at the bottom. Setup: - Cache mode: Partitioned - Number of server nodes: 3 - CPUs: 4 per node (12) - Heap size: 2GB per node (6GB) - The first use case is computing the weighted average over two fields of the object at different rates. First method is to run a SQL style query: ... query = new SqlFieldsQuery("select SUM(field1*field2)/SUM(field2) from MyObject"); cache.query(query).getAll(); .... The observed timings are: Cache: 500,000 Queries/second: 10 Median: 428ms, 90th percentile: 13,929ms Cache: 500,000 Queries/second: 50 Median: 191,465ms, 90th percentile: 402,285ms Clearly this is queuing up with an enormous latency (>400 ms), a simple weighted average computation on a single jvm (4 Cores) takes 6 ms. The second approach is to use the IgniteCompute to broadcast Callables across nodes and compute the weighted average on each node, reducing at the caller, latency is only marginally better, throughput improves but still at unusable levels. Cache: 500,000 Queries/second: 10 Median: 408ms, 90th percentile: 507ms Cache: 500,000 Queries/second: 50 Median: 114,155ms, 90th percentile: 237,521ms A few things i noticed during the experiment: - No disk swapping is happening - CPUs run at up to 400% - Query is split up in two different weighted averages (map reduce) - Entries are evenly split across the nodes - No garbage collections are triggered with each heap size around 500MB To my questions: 1. Are these timings expected or is there some obvious setting i am missing? I could not find benchmarks on similar operations. 2. What is the advised method to run fork-join style queries on ignite without moving data?

Posted by Manuel Piubelli 7 months ago