Apache Ignite Documentation

GridGain Developer Hub - Apache Ignitetm

Welcome to the Apache Ignite developer hub run by GridGain. Here you'll find comprehensive guides and documentation to help you start working with Apache Ignite as quickly as possible, as well as support if you get stuck.

 

GridGain also provides Community Edition which is a distribution of Apache Ignite made available by GridGain. It is the fastest and easiest way to get started with Apache Ignite. The Community Edition is generally more stable than the Apache Ignite release available from the Apache Ignite website and may contain extra bug fixes and features that have not made it yet into the release on the Apache website.

 

Let's jump right in!

 

Documentation     Ask a Question     Download

 

Javadoc     Scaladoc     Examples

Binary Marshaller

Suggest Edits

Basic Concepts

Starting from v1.5 Ignite introduced a new concept of storing data in caches called BinaryObjects. There are several advantages that the new serialization format provides:

  • It enables you to read an arbitrary field from an object's serialized form without full object deserialization. This ability completely removes the requirement to have the cache key and value classes deployed on the server node's classpath.
  • It enables you to add and remove fields from objects of the same type. Given that server nodes do not have model classes definitions, this ability allows dynamic change to an object's structure, and even allows multiple clients with different versions of class definitions to co-exist.
  • It enables you to construct new objects based on a type name without having class definitions at all, hence allowing dynamic type creation.

Binary objects can be used only when default binary marshaller is used (i.e. no other marshaller is set to the configuration explicitly).

Restrictions

There are several restrictions that are implied by the BinaryObject format implementation:

  • Internally, Ignite does not write field and type names but uses a lower-case name hash to identify a field or a type. It means that fields or types with the same name hash are not allowed. Even though serialization will not work out-of-the-box in the case of hash collision, Ignite provides a way to resolve this collision at the configuration level.
  • For the same reason BinaryObject format does not allow same field names on different levels of a class hierarchy.
  • If a class implements Externalizable interface, Ignite will use OptimizedMarshaller instead of the binary one. The OptimizedMarshaller uses writeExternal() and readExternal() methods to serialize and deserialize objects of this class which requires adding classes of Externalizable objects to the classpath of server nodes.

The IgniteBinary facade, which can be obtained from an instance of Ignite, contains all the necessary methods to work with binary objects.

Automatic Hash Code Calculation and Equals Implementation

If an object can be serialized into a binary form, then Ignite will calculate its hash code during serialization and write it to the resulting binary array. Also, Ignite provides a custom implementation of the equals method for the binary object's comparison needs. This means that you do not need to override the GetHashCode and Equals methods of your custom keys and values in order for them to be used in Ignite, unless they can not be serialized into the binary form.
For instance, objects of Externalizable type cannot be serialized into the binary form and require you to implement the hashCode and equals methods manually. See Restrictions section above for more details.

Configuring Binary Objects

In the vast majority of use cases, there is no need to additionally configure binary objects.

However, in a case when you need to override the default type and field IDs calculation, or to plug in BinarySerializer, a BinaryConfiguration object should be defined in IgniteConfiguration. This object allows specifying a global name mapper, a global ID mapper, and a global binary serializer as well as per-type mappers and serializers. Wildcards are supported for per-type configuration, in which case, the provided configuration will be applied to all types that match the type name template.

<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
    
  <property name="binaryConfiguration">
    <bean class="org.apache.ignite.configuration.BinaryConfiguration">
      
      <property name="nameMapper" ref="globalNameMapper"/>
      <property name="idMapper" ref="globalIdMapper"/>

      <property name="typeConfigurations">
        <list>
          <bean class="org.apache.ignite.binary.BinaryTypeConfiguration">
            <property name="typeName" value="org.apache.ignite.examples.*"/>
            <property name="serializer" ref="exampleSerializer"/>
          </bean>
        </list>
      </property>
    </bean>
  </property>
...

BinaryObject Cache API

By default, Ignite works with deserialized values as it is the most common use-case. To enable BinaryObject processing, a user needs to obtain an instance of IgniteCache using withKeepBinary() method. When enabled, this flag will ensure that objects returned from the cache will be in BinaryObject format, when possible. The same will stand for values being passed to the EntryProcessor and CacheInterceptor.

Platform Types

Note that not all types will be represented as BinaryObject when withKeepBinary() flag is enabled. There is a set of 'platform' types that includes primitive types, String, UUID, Date, Timestamp, BigDecimal, Collections, Maps and arrays of thereof that will never be represented as a BinaryObject.

Note that in the example below key type Integer does not change because it is a platform type.

// Create a regular Person object and put it to the cache.
Person person = buildPerson(personId);
ignite.cache("myCache").put(personId, person);

// Get an instance of binary-enabled cache.
IgniteCache<Integer, BinaryObject> binaryCache = ignite.cache("myCache").withKeepBinary();

// Get the above person object in the BinaryObject format.
BinaryObject binaryPerson = binaryCache.get(personId);

Modifying Binary Objects Using BinaryObjectBuilder

BinaryObject instances are immutable. An instance of BinaryObjectBuildermust be used in order to update fields and create a new BinaryObject.

An instance of BinaryObjectBuilder can be obtained from IgniteBinary facade. The builder may be created using a type name, in this case the returned builder will contain no fields, or it may be created using an existing BinaryObject, in this case the returned builder will copy all the fields from the given BinaryObject.

Another way to get an instance of BinaryObjectBuilder is to call toBuilder() on an existing instance of a BinaryObject. This will also copy all data from the BinaryObject to the created builder.

Below is an example of using BinaryObject API to process data on server nodes without having user classes deployed on servers and without actual data deserialization.

// The EntryProcessor is to be executed for this key.
int key = 101;

cache.<Integer, BinaryObject>withKeepBinary().invoke(
  key, new CacheEntryProcessor<Integer, BinaryObject, Object>() {
  	public Object process(MutableEntry<Integer, BinaryObject> entry,
                          Object... objects) throws EntryProcessorException {
		    // Create builder from the old value.
        BinaryObjectBuilder bldr = entry.getValue().toBuilder();

        //Update the field in the builder.
        bldr.setField("name", "Ignite");

        // Set new value to the entry.
        entry.setValue(bldr.build());

        return null;
     }
  });

BinaryObject Type Metadata

As it was mentioned above, binary object structure may be changed at runtime hence it may also be useful to get information about a particular type that is stored in a cache such as field names, field type names & affinity field name. Ignite facilitates this requirement via the BinaryType interface.

This interface also introduces a faster version of field getter called BinaryField. The concept is analogous to java reflection and allows to cache certain information about the field being read in the BinaryField instance, which is useful when reading the same field from a large collection of binary objects.

Collection<BinaryObject> persons = getPersons();

BinaryField salary = null;

double total = 0;
int cnt = 0;

for (BinaryObject person : persons) {
    if (salary == null)
        salary = person.type().field("salary");

    total += salary.value(person);
    cnt++;
}

double avg = total / cnt;

BinaryObject and CacheStore

Setting withKeepBinary() on the cache API does not affect the way user objects are passed to a CacheStore. This is intentional because in most cases a single CacheStore implementation works either with deserialized classes, or with BinaryObject representations. To control the way objects are passed to the store then the storeKeepBinary flag on CacheConfiguration should be used. When this flag is set to false, deserialized values will be passed to the store, otherwise BinaryObject representations will be used.

Below is an example pseudo-code implementation of a store working with BinaryObject:

public class CacheExampleBinaryStore extends CacheStoreAdapter<Integer, BinaryObject> {
    @IgniteInstanceResource
    private Ignite ignite;

    /** {@inheritDoc} */
    @Override public BinaryObject load(Integer key) {
        IgniteBinary binary = ignite.binary();

        List<?> rs = loadRow(key);

        BinaryObjectBuilder bldr = binary.builder("Person");

        for (int i = 0; i < rs.size(); i++)
            bldr.setField(name(i), rs.get(i));

        return bldr.build();
    }

    /** {@inheritDoc} */
    @Override public void write(Cache.Entry<? extends Integer, ? extends BinaryObject> entry) {
        BinaryObject obj = entry.getValue();

        BinaryType type = obj.type();

        Collection<String> fields = type.fieldNames();
        
        List<Object> row = new ArrayList<>(fields.size());

        for (String fieldName : fields)
            row.add(obj.field(fieldName));
        
        saveRow(entry.getKey(), row);
    }
}

Binary Name Mapper and Binary ID Mapper

Internally, Ignite never writes full strings for field or type names. Instead, for performance reasons, Ignite writes integer hash codes for type and field names. It has been tested that hash code conflicts for the type names or the field names within the same type are virtually non-existent and, to gain performance, it is safe to work with hash codes. For the cases when hash codes for different types or fields actually do collide, BinaryNameMapper and BinaryIdMapper allow to override the automatically generated hash code IDs for the type and field names.

BinaryNameMapper - maps type/class and field names to different names.
BinaryIdMapper - maps given from BinaryNameMapper type and field name to ID that will be used by Ignite in internals.

Ignite provides the following out-of-the-box mappers implementation:

  • BinaryBasicNameMapper - a basic implementation of BinaryNameMapper that returns a full or a simple name of given class depending on used setSimpleName(boolean useSimpleName) property.
  • BinaryBasicIdMapper - a basic implementation of BinaryIdMapper. It has setLowerCase(boolean isLowerCase) configuration property. If the property is set to false then a hash code of given type or field name will be returned. If the property is set to true then a hash code of given type or field name in lower case will be returned.

If you are using Java or .NET clients and do not specify mappers in BinaryConfiguration, then Ignite will use BinaryBasicNameMapper and simpleName property will be set to false, and BinaryBasicIdMapper and lowerCase property will be set to true.

If you are using the C++ client and do not specify mappers in BinaryConfiguration, then Ignite will use BinaryBasicNameMapper and simpleName property will be set to true, and BinaryBasicIdMapper and lowerCase property will be set to true.

By default, there is no need to configure anything if you use Java, .NET or C++. Mappers need to be configured if there is a tricky name conversion when platform interoperability is needed.

Binary Marshaller