com.attivio.platform.transformer.ingest.util

Class TransformerUtils

java.lang.Object

com.attivio.platform.transformer.ingest.util.TransformerUtils

Direct Known Subclasses:

SdkTestUtils

public class TransformerUtils
extends Object

A series of utility functions to make common transformer code patterns easier to develop.

Transformers usually follow one of the following two patterns:

Inspecting or modifying values in place

For example a logging transfomer might want to log all of the values for the 'title' field or a lower case transformer might want to convert all of the text in a given field to lowercase. The following methods can be used for this purpose:

• TransformerUtils#modifyFieldValues(ProcessingResult, IngestDocument, Iterable, AdvancedFieldValueModifier)
• TransformerUtils#modifyFieldValues(ProcessingResult, IngestDocument, Iterable, FieldValueModifier)

Mapping values from one field to another

For example a pattern based extractor might want to scan the text in one field an output pattern matches to another output field. The following methods can be used for this purpose:

• TransformerUtils#mapFieldValues(ProcessingResult, IngestDocument, Map, AdvancedFieldValueMapper)
• TransformerUtils#mapFieldValues(ProcessingResult, IngestDocument, Map, FieldValueMapper)
• TransformerUtils#mapFieldValues(ProcessingResult, IngestDocument, Map, AdvancedFieldValueMapper, NewFieldCreator)
• TransformerUtils#mapFieldValues(ProcessingResult, IngestDocument, Map, FieldValueMapper, NewFieldCreator)

Developing transformers

In most cases the following code is the simplest way to use these classes:

 // The actual modifier/mapper interface implemented should be chosen based on your transformer's needs.
 public class MyTransformer extends AbstractSingleDocumentTransformer implements FieldValueMapper {

   // provide getters/setters for this
   private Map fieldMapping = null;

   &Override
   public ProcessingResult processDocument(IngestDocument doc) throws AttivioException {
     return TransformerUtils.mapFieldValues(okResult(), doc, fieldMapping, this);
   }

   &Override
   public Object createMappedValue(ProcessingResult result, String fieldName, IngestFieldValue fv) {
     // this logic should be replaced with your own
     return String.valueof(fv.getValue()).toLowerCase();
   }

 }
 

Constructor Summary

Constructors

Constructor and Description
TransformerUtils()

Method Summary

Modifier and Type	Method and Description
static Collection<String>	getAllOrSpecificFieldList(IngestDocument doc, Collection<String> input) Returns a specific list of input field names or all fields in the document if the input list is null or empty.
static <T> boolean	mapFieldValues(IngestDocument doc, FieldValueCreatingTransformer<T> trans) Applies a FieldValueCreatingTransformer to the given IngestDocument.
static void	mapFieldValues(IngestDocument doc, Map<String,String> fieldMapping, AdvancedFieldValueMapper mapper) Map each document based on the fieldMapping using the supplied AdvancedFieldValueMapper.
static void	mapFieldValues(IngestDocument doc, Map<String,String> fieldMapping, AdvancedFieldValueMapper mapper, NewFieldCreator newFieldCreator) Map each document based on the fieldMapping using the supplied FieldValueMapper.
static void	mapFieldValues(IngestDocument doc, Map<String,String> fieldMapping, FieldValueMapper mapper) Map each document based on the fieldMapping using the supplied FieldValueMapper.
static void	mapFieldValues(IngestDocument doc, Map<String,String> fieldMapping, FieldValueMapper mapper, NewFieldCreator newFieldCreator) Map each document based on the fieldMapping using the supplied FieldValueMapper.
static void	modifyFieldValues(IngestDocument doc, Iterable<String> input, AdvancedFieldValueModifier handler) Iterates over all fields in input, passing the field values for each to the modifier.
static void	modifyFieldValues(IngestDocument doc, Iterable<String> input, FieldValueModifier modifier) Iterates over all fields in input, passing the field values for each to the modifier.
static <T> boolean	processDocument(IngestDocument doc, FieldValueCreatingTransformer<T> transformer) A processDocument utility method for FieldValueCreatingTransformer, because that class doesn't have one.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TransformerUtils

public TransformerUtils()

Method Detail

modifyFieldValues

public static void modifyFieldValues(IngestDocument doc,
                                     Iterable<String> input,
                                     FieldValueModifier modifier)

Iterates over all fields in input, passing the field values for each to the modifier.

Parameters:

doc - the document to process

input - the input fields to work on

modifier - the modifier call for each field value

modifyFieldValues

public static void modifyFieldValues(IngestDocument doc,
                                     Iterable<String> input,
                                     AdvancedFieldValueModifier handler)

Iterates over all fields in input, passing the field values for each to the modifier. The AdvancedFieldValueModifier and this method provide access to more data during each call than the FieldValueModifier and the TransformerUtils#modifyFieldValues(ProcessingResult, IngestDocument, Iterable, FieldValueModifier) method does.

Parameters:

doc - the document to process

input - the input fields to work on

handler - the modifier call for each field value

processDocument

public static <T> boolean processDocument(IngestDocument doc,
                                          FieldValueCreatingTransformer<T> transformer)
                                   throws AttivioException

A processDocument utility method for FieldValueCreatingTransformer, because that class doesn't have one.

Parameters:

transformer -

Returns:

false if an actual workflow is supposed to drop the document and true if it is supposed to continue with it

Throws:

AttivioException

mapFieldValues

public static void mapFieldValues(IngestDocument doc,
                                  Map<String,String> fieldMapping,
                                  FieldValueMapper mapper)

Map each document based on the fieldMapping using the supplied FieldValueMapper.

Parameters:

doc - the document to process

fieldMapping - the field mapping to use. The keys represent the input field names, the values represent the output field names.

mapper - the mapper to use when processing data.

mapFieldValues

public static void mapFieldValues(IngestDocument doc,
                                  Map<String,String> fieldMapping,
                                  AdvancedFieldValueMapper mapper)

Map each document based on the fieldMapping using the supplied AdvancedFieldValueMapper. This method and the AdvancedFieldValueMapper provide access to more information in each call than the FieldValueMapper and the TransformerUtils#mapFieldValues(ProcessingResult, IngestDocument, Map, FieldValueMapper) method does.

Parameters:

doc - the document to process

fieldMapping - the field mapping to use. The keys represent the input field names, the values represent the output field names.

mapper - the mapper to use when processing data.

mapFieldValues

public static void mapFieldValues(IngestDocument doc,
                                  Map<String,String> fieldMapping,
                                  FieldValueMapper mapper,
                                  NewFieldCreator newFieldCreator)

Map each document based on the fieldMapping using the supplied FieldValueMapper.

Parameters:

doc - the document to process

fieldMapping - the field mapping to use. The keys represent the input field names, the values represent the output field names.

newFieldCreator - any new fields specified in the fieldMapping will be created using this factory

mapper - the mapper to use when processing data.

mapFieldValues

public static void mapFieldValues(IngestDocument doc,
                                  Map<String,String> fieldMapping,
                                  AdvancedFieldValueMapper mapper,
                                  NewFieldCreator newFieldCreator)

Map each document based on the fieldMapping using the supplied FieldValueMapper. This method and the AdvancedFieldValueMapper provide access to more information in each call than the FieldValueMapper and the TransformerUtils#mapFieldValues(ProcessingResult, IngestDocument, Map, FieldValueMapper) method does.

Parameters:

doc - the document to process

fieldMapping - the field mapping to use. The keys represent the input field names, the values represent the output field names.

newFieldCreator - any new fields specified in teh fieldMapping will be created using this factory

mapper - the mapper to use when processing data.

mapFieldValues

public static <T> boolean mapFieldValues(IngestDocument doc,
                                         FieldValueCreatingTransformer<T> trans)
                                  throws AttivioException

Applies a FieldValueCreatingTransformer to the given IngestDocument.

Parameters:

doc -

trans -

Returns:

Throws:

AttivioException

getAllOrSpecificFieldList

public static Collection<String> getAllOrSpecificFieldList(IngestDocument doc,
                                                           Collection<String> input)

Returns a specific list of input field names or all fields in the document if the input list is null or empty. If input contains values, the returned collection is the same list as passed in.