TokenList (platform/sdk 5.5.1.0-SNAPSHOT API)

java.lang.Object
- com.attivio.sdk.token.TokenSink
- - com.attivio.sdk.token.TokenList

All Implemented Interfaces:

Externalizable, Serializable, Cloneable, Iterable<Token>
```
public class TokenList
extends TokenSink
implements Cloneable, Iterable<Token>, Externalizable
```
A Linked List of tokens with additional markup to indicate position increments and ordering of tokens.
CAUTION: unless otherwise noted, any Token instances passed to methods of a TokenList or a TokenListIterator are NOT copied when added to this TokenList. You should therefore not modify any Token instances after you have passed them into this TokenList unless you are sure of what you are doing.

See Also:

Serialized Form

Field Summary
- Fields inherited from class com.attivio.sdk.token.TokenSink
  DEFAULT_POSITION_INCREMENT

Constructor Summary

Constructors
Constructor and Description
`TokenList()` The default constructor
`TokenList(String... tokens)` Constructor a new TokenList with the specified `tokens`.
`TokenList(String token)` Construct a new TokenList with a single `token`.
`TokenList(Token... tokens)` Construct a new TokenList with the specified `tokens`.
`TokenList(Token token)` Construct a new TokenList with a single `token`.

Method Summary

All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`void`	`add(Token t, int increment)` Appends a token.
`TokenList`	`append(String t)` Appends a token to this TokenList using the default position increment.
`TokenList`	`append(String t, int increment)` Appends a token.
`TokenList`	`append(Token t)` Appends a token using the default position increment.
`TokenList`	`append(Token t, int increment)` Appends a token.
`void`	`clear()`
`TokenList`	`clone()`
`String`	`concat()` Represent all primary (ie top of the stack) tokens from this TokenList as a string.
`static String`	`concat(Iterable<Position> positions)` Represent all primary (ie top of the stack) tokens from `positions` as a string.
`static String`	`concat(Iterator<Position> positions)` Represent all primary (ie top of the stack) tokens from `positions` as a string.
`static String`	`concat(Position... positions)` Represent all primary (ie top of the stack) tokens from `positions` as a string.
`static String`	`concat(PositionIterator positions)` Represent all primary (ie top of the stack) tokens from `positions` as a string.
`boolean`	`containsWildcard()` Returns true if any Token in this TokenList contains ? or *
`boolean`	`equals(Object other)`
`int`	`getEndOffset()` Get the end offset for the last token that contains offset information.
`Token`	`getFirst()` Returns the first Token in this TokenList.
`Token`	`getLast()` Returns the last Token in this TokenList.
`int`	`getMaxRemainingTokens()` This value is a hint to tokenization.
`int`	`getPositionCount()` Returns the number of unique positions for this term
`int`	`getStartOffset()` Get the start offset for the first token that contains offset information.
`int`	`hashCode()`
`TokenIterator`	`iterator()` Returns an iterator over the Tokens in this TokenList.
`static String`	`join(Iterable<Token> tokens, char joinChar)` Join the text for all `tokens` on `joinChar`.
`static String`	`join(Iterator<Token> tokens, char joinChar)` Join the text for all `tokens` on `joinChar`.
`static String`	`join(Token[] tokens, int startIndex, int endIndex, char joinChar)` Join the text for all `tokens` on `joinChar`.
`TokenIterator`	`listIterator()` Returns a list iterator over the Tokens in this TokenList.
`PositionIterator`	`positions()` Returns an iterator that iterates over the unique positions in TokenList.
`TokenList`	`readExternal(DataInput in)`
`void`	`readExternal(ObjectInput in)`
`void`	`remove(TokenAnnotation annotation)` Remove all tokens annotated with `annotation`.
`void`	`setMaxRemainingTokens(int value)` Set the max remaining tokens allowed to be created.
`int`	`size()` Returns the number of Tokens in this TokenList.
`Phrase`	`toPhrase()` Convert this TokenList into a comparable Phrase query.
`Phrase`	`toPhrase(int offsetBase)` Convert this TokenList into a comparable Phrase query.
`String`	`toQuotedString()`
`String`	`toString()`
`void`	`truncate(int maxSize)`
`static TokenList`	`valueOf(String value)` Parses the string representation of a TokenList back into a TokenList
`void`	`writeExternal(DataOutput out)`
`void`	`writeExternal(ObjectOutput out)`
`protected void`	`writeExternalV0(DataOutput out)` Deprecated.
`StringBuilder`	`writeTo(StringBuilder buffer)` Write token list to `buffer`.
`void`	`writeTo(TokenListSerializer out)` Write token list to `out`.

Methods inherited from class com.attivio.sdk.token.TokenSink
add, add, add, add, add, endScope, endScope, startLanguageRegion, startScope, startScope

Methods inherited from class java.lang.Object
finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.lang.Iterable
forEach, spliterator

- Constructor Detail
  - TokenList
```
public TokenList()
```
    The default constructor
  - TokenList
```
public TokenList(String token)
```
    Construct a new TokenList with a single token.
  - TokenList
```
public TokenList(Token token)
```
    Construct a new TokenList with a single token.
  - TokenList
```
public TokenList(String... tokens)
```
    Constructor a new TokenList with the specified tokens.
  - TokenList
```
public TokenList(Token... tokens)
```
    Construct a new TokenList with the specified tokens.
- Method Detail
  - clear
```
public void clear()
```
  - truncate
```
public void truncate(int maxSize)
```
  - toPhrase
```
public Phrase toPhrase()
```
    Convert this TokenList into a comparable Phrase query.
  - toPhrase
```
public Phrase toPhrase(int offsetBase)
```
    Convert this TokenList into a comparable Phrase query.
  - getMaxRemainingTokens
```
public int getMaxRemainingTokens()
```
    This value is a hint to tokenization.
    - If 0, it indicates that truncation of tokens occurred during tokenization.
    - If negative, it indicates no truncation was configured.
    - If positive, it indicates the maximum number of tokens that should be added to this TokenList.
  - setMaxRemainingTokens
```
public void setMaxRemainingTokens(int value)
```
    Set the max remaining tokens allowed to be created.
  - getFirst
```
public Token getFirst()
```
    Returns the first Token in this TokenList.
  - getLast
```
public Token getLast()
```
    Returns the last Token in this TokenList.
  - size
```
public int size()
```
    Returns the number of Tokens in this TokenList.
  - getStartOffset
```
public int getStartOffset()
```
    Get the start offset for the first token that contains offset information.
    Returns -1 if no tokens contain offset information.
  - getEndOffset
```
public int getEndOffset()
```
    Get the end offset for the last token that contains offset information.
    Returns -1 if no tokens contain offset information.
  - getPositionCount
```
public int getPositionCount()
```
    Returns the number of unique positions for this term
  - containsWildcard
```
public boolean containsWildcard()
```
    Returns true if any Token in this TokenList contains ? or *
  - append
```
public TokenList append(String t)
```
    Appends a token to this TokenList using the default position increment.
    
    Overrides:
    
    append in class TokenSink
    
    Parameters:
    
    t - The string value of a Token to add to this TokenList
    
    Returns:
    
    this (supports call chaining)
  - append
```
public TokenList append(String t,
                        int increment)
```
    Appends a token.
    
    Overrides:
    
    append in class TokenSink
    
    Parameters:
    
    t - The string value of a Token to add
    
    increment - The position increment for the Token to add
    
    Returns:
    
    this (supports call chaining)
  - append
```
public TokenList append(Token t)
```
    Appends a token using the default position increment.
    
    Overrides:
    
    append in class TokenSink
    
    Parameters:
    
    t - The Token to add
    
    Returns:
    
    this (supports call chaining)
  - append
```
public TokenList append(Token t,
                        int increment)
```
    Appends a token.
    
    Overrides:
    
    append in class TokenSink
    
    Parameters:
    
    t - The Token to add to this TokenList
    
    increment - The position increment for the Token to add
    
    Returns:
    
    this (supports call chaining)
  - add
```
public void add(Token t,
                int increment)
```
    Appends a token.
    
    Specified by:
    
    add in class TokenSink
    
    Parameters:
    
    t - The Token to add
    
    increment - The position increment for the newly added token
  - remove
```
public void remove(TokenAnnotation annotation)
```
    Remove all tokens annotated with annotation.
    NOTE: this removal is "position" aware.
  - iterator
```
public TokenIterator iterator()
```
    Returns an iterator over the Tokens in this TokenList.
    
    Specified by:
    
    iterator in interface Iterable<Token>
  - listIterator
```
public TokenIterator listIterator()
```
    Returns a list iterator over the Tokens in this TokenList.
  - positions
```
public PositionIterator positions()
```
    Returns an iterator that iterates over the unique positions in TokenList.
  - clone
```
public TokenList clone()
```
    Overrides:
    
    clone in class Object
  - equals
```
public boolean equals(Object other)
```
    Overrides:
    
    equals in class Object
  - hashCode
```
public int hashCode()
```
    Overrides:
    
    hashCode in class Object
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object
  - toQuotedString
```
public String toQuotedString()
```
  - writeTo
```
public StringBuilder writeTo(StringBuilder buffer)
```
    Write token list to buffer.
  - writeTo
```
public void writeTo(TokenListSerializer out)
```
    Write token list to out.
  - valueOf
```
public static TokenList valueOf(String value)
```
    Parses the string representation of a TokenList back into a TokenList
    
    Parameters:
    
    value - the string value to parse
    
    Returns:
    
    the TokenList that value represents
  - join
```
public static String join(Iterable<Token> tokens,
                          char joinChar)
```
    Join the text for all tokens on joinChar.
  - join
```
public static String join(Iterator<Token> tokens,
                          char joinChar)
```
    Join the text for all tokens on joinChar.
  - join
```
public static String join(Token[] tokens,
                          int startIndex,
                          int endIndex,
                          char joinChar)
```
    Join the text for all tokens on joinChar.
    
    Parameters:
    
    tokens - the array of tokens to join.
    
    startIndex - the first element in tokens to join
    
    endIndex - one past the last element in tokens to join
    
    joinChar - the character to join all tokens on.
    
    Returns:
    
    the joined token array
  - concat
```
public String concat()
```
    Represent all primary (ie top of the stack) tokens from this TokenList as a string.
    NOTE: Uses Token.offsetGap(Token) to determine if space should be placed between tokens.
    
    Returns:
    
    the list represented as a String
  - concat
```
public static String concat(PositionIterator positions)
```
    Represent all primary (ie top of the stack) tokens from positions as a string.
    NOTE: Uses Token.offsetGap(Token) to determine if space should be placed between tokens.
    
    Returns:
    
    the list represented as a String
  - concat
```
public static String concat(Iterable<Position> positions)
```
    Represent all primary (ie top of the stack) tokens from positions as a string.
    NOTE: Uses Token.offsetGap(Token) to determine if space should be placed between tokens.
    
    Returns:
    
    the list represented as a String
  - concat
```
public static String concat(Iterator<Position> positions)
```
    Represent all primary (ie top of the stack) tokens from positions as a string.
    NOTE: Uses Token.offsetGap(Token) to determine if space should be placed between tokens.
    
    Returns:
    
    the list represented as a String
  - concat
```
public static String concat(Position... positions)
```
    Represent all primary (ie top of the stack) tokens from positions as a string.
    NOTE: Uses Token.offsetGap(Token) to determine if space should be placed between tokens.
    
    Returns:
    
    the list represented as a String
  - readExternal
```
public void readExternal(ObjectInput in)
                  throws IOException
```
    Specified by:
    
    readExternal in interface Externalizable
    
    Throws:
    
    IOException
  - writeExternal
```
public void writeExternal(ObjectOutput out)
                   throws IOException
```
    Specified by:
    
    writeExternal in interface Externalizable
    
    Throws:
    
    IOException
  - readExternal
```
public TokenList readExternal(DataInput in)
                       throws IOException
```
    Throws:
    
    IOException
  - writeExternal
```
public void writeExternal(DataOutput out)
                   throws IOException
```
    Throws:
    
    IOException
  - writeExternalV0
```
@Deprecated
protected void writeExternalV0(DataOutput out)
                                    throws IOException
```
    Deprecated.
    
    Deprecated method to serialize a token list according to V0 format.
    
    Throws:
    
    IOException

Class TokenList

Field Summary

Fields inherited from class com.attivio.sdk.token.TokenSink

Constructor Summary

Method Summary

Methods inherited from class com.attivio.sdk.token.TokenSink

Methods inherited from class java.lang.Object

Methods inherited from interface java.lang.Iterable

Constructor Detail

TokenList

TokenList

TokenList

TokenList

TokenList

Method Detail

clear

truncate

toPhrase

toPhrase

getMaxRemainingTokens

setMaxRemainingTokens

getFirst

getLast

size

getStartOffset

getEndOffset

getPositionCount

containsWildcard

append

append

append

append

add

remove

iterator

listIterator

positions

clone

equals

hashCode

toString

toQuotedString

writeTo

writeTo

valueOf

join

join

join

concat

concat

concat

concat

concat

readExternal

writeExternal

readExternal

writeExternal

writeExternalV0