com.attivio.sdk.server.tokenization

Interface Tokenizer

public interface Tokenizer

An interface for objects that take String and produce TokenLists.

Field Summary

Fields

Modifier and Type	Field and Description
static String	TOKENIZER_DEFAULT The name of the default system tokenizer.

Method Summary

Modifier and Type	Method and Description
Tokenizer	getIngestTokenizer(SchemaField field, Locale locale) Get the underlying tokenizer to use for tokenizing fields in the ingest workflow.
Tokenizer	getQueryTokenizer(SchemaField field, Locale locale) Get the underlaying tokenizer to use for tokenizing fields in the query workflow.
Phrase	tokenize(SchemaField field, Locale locale, SearchTerm term) Tokenizes term into a Phrase for query processing.
TokenList	tokenize(SchemaField field, Locale locale, String value) Tokenizes value into a TokenList.
Phrase	tokenize(SchemaField field, Locale locale, TermRange range) Tokenizes range into a Phrase for query processing.
void	tokenize(SchemaField field, Locale locale, TokenList tokens) Tokenizes all tokens in tokens.
Phrase	tokenize(SchemaField field, Locale locale, WildcardTerm term) Tokenizes a wildcard term into a Phrase for query processing.

Field Detail

TOKENIZER_DEFAULT

static final String TOKENIZER_DEFAULT

The name of the default system tokenizer.

See Also:

Constant Field Values

Method Detail

getIngestTokenizer

Tokenizer getIngestTokenizer(SchemaField field,
                             Locale locale)
                      throws AttivioException

Get the underlying tokenizer to use for tokenizing fields in the ingest workflow.

In general, this method should return this. Tokenizers that route to sub-tokenizers for handling different fields/locales should return the actual tokenizer that will be used.

Throws:

AttivioException

getQueryTokenizer

Tokenizer getQueryTokenizer(SchemaField field,
                            Locale locale)
                     throws AttivioException

Get the underlaying tokenizer to use for tokenizing fields in the query workflow.

In general, this method should return this. Tokenizers that route to sub-tokenizers for handling different fields/locales should return the actual tokenizer that will be used.

Throws:

AttivioException

tokenize

void tokenize(SchemaField field,
              Locale locale,
              TokenList tokens)
       throws AttivioException

Tokenizes all tokens in tokens.

Parameters:

field - the schema field being tokenized (may be null)

locale - the Locale of the tokens (may be null)

tokens - the token list

Throws:

AttivioException - on an unrecoverable error

tokenize

TokenList tokenize(SchemaField field,
                   Locale locale,
                   String value)
            throws AttivioException

Tokenizes value into a TokenList.

Parameters:

field - the schema field being tokenized (may be null)

locale - the Locale of the tokens (may be null)

value - the string to tokenize

Throws:

AttivioException - on an unrecoverable error

tokenize

Phrase tokenize(SchemaField field,
                Locale locale,
                SearchTerm term)
         throws AttivioException

Tokenizes term into a Phrase for query processing.

Parameters:

field - the schema field being tokenized (may be null)

locale - the Locale of the tokens (may be null)

term - the SearchTerm to tokenize

Throws:

AttivioException - on an unrecoverable error

tokenize

Phrase tokenize(SchemaField field,
                Locale locale,
                WildcardTerm term)
         throws AttivioException

Tokenizes a wildcard term into a Phrase for query processing.

Parameters:

field - the schema field being tokenized (may be null)

locale - the Locale of the tokens (may be null)

term - the WildcardTerm to tokenize

Throws:

AttivioException - on an unrecoverable error

tokenize

Phrase tokenize(SchemaField field,
                Locale locale,
                TermRange range)
         throws AttivioException

Tokenizes range into a Phrase for query processing.

Parameters:

field - the schema field being tokenized (may be null)

locale - the Locale of the tokens (may be null)

range - the TermRange to tokenize

Throws:

AttivioException - on an unrecoverable error