Package com.attivio.sdk.token

Class TokenList

  • All Implemented Interfaces:
    java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, java.lang.Iterable<Token>
    public class TokenList
extends TokenSink
implements java.lang.Cloneable, java.lang.Iterable<Token>, java.io.Externalizable
    A Linked List of tokens with additional markup to indicate position increments and ordering of tokens.

    CAUTION: unless otherwise noted, any Token instances passed to methods of a TokenList or a TokenListIterator are NOT copied when added to this TokenList. You should therefore not modify any Token instances after you have passed them into this TokenList unless you are sure of what you are doing.

    See Also:
    Serialized Form

    • Constructor Summary

      Constructors 
      Constructor Description
      TokenList()
      The default constructor
      TokenList​(Token token)
      Construct a new TokenList with a single token.
      TokenList​(Token... tokens)
      Construct a new TokenList with the specified tokens.
      TokenList​(java.lang.String token)
      Construct a new TokenList with a single token.
      TokenList​(java.lang.String... tokens)
      Constructor a new TokenList with the specified tokens.

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods 
      Modifier and Type Method Description
      void add​(Token t, int increment)
      Appends a token.
      TokenList append​(Token t)
      Appends a token using the default position increment.
      TokenList append​(Token t, int increment)
      Appends a token.
      TokenList append​(java.lang.String t)
      Appends a token to this TokenList using the default position increment.
      TokenList append​(java.lang.String t, int increment)
      Appends a token.
      void clear()  
      TokenList clone()
      java.lang.String concat()
      Represent all primary (ie top of the stack) tokens from this TokenList as a string.
      static java.lang.String concat​(Position... positions)
      Represent all primary (ie top of the stack) tokens from positions as a string.
      static java.lang.String concat​(PositionIterator positions)
      Represent all primary (ie top of the stack) tokens from positions as a string.
      static java.lang.String concat​(java.lang.Iterable<Position> positions)
      Represent all primary (ie top of the stack) tokens from positions as a string.
      static java.lang.String concat​(java.util.Iterator<Position> positions)
      Represent all primary (ie top of the stack) tokens from positions as a string.
      boolean containsToken​(java.util.function.Predicate<Token> predicate)
      Returns true if any token matches predicate.
      boolean containsWildcard()
      Returns true if any Token in this TokenList contains ? or *
      boolean equals​(java.lang.Object other)
      void forEachToken​(java.util.function.Consumer<Token> consumer)
      Pass all tokens to consumer.
      int getEndOffset()
      Get the end offset for the last token that contains offset information.
      Token getFirst()
      Returns the first Token in this TokenList.
      Token getLast()
      Returns the last Token in this TokenList.
      int getMaxRemainingTokens()
      This value is a hint to tokenization.
      int getPositionCount()
      Returns the number of unique positions for this term
      int getStartOffset()
      Get the start offset for the first token that contains offset information.
      int hashCode()
      TokenIterator iterator()
      Returns an iterator over the Tokens in this TokenList.
      static java.lang.String join​(Token[] tokens, int startIndex, int endIndex, char joinChar)
      Join the text for all tokens on joinChar.
      static java.lang.String join​(java.lang.Iterable<Token> tokens, char joinChar)
      Join the text for all tokens on joinChar.
      static java.lang.String join​(java.util.Iterator<Token> tokens, char joinChar)
      Join the text for all tokens on joinChar.
      TokenIterator listIterator()
      Returns a list iterator over the Tokens in this TokenList.
      PositionIterator positions()
      Returns an iterator that iterates over the unique positions in TokenList.
      TokenList readExternal​(java.io.DataInput in)
      GWT Incompatible - hence no @Override annotation.
      void readExternal​(java.io.ObjectInput in)
      GWT Incompatible - hence no @Override annotation.
      void remove​(TokenAnnotation annotation)
      Remove all tokens annotated with annotation.
      void setMaxRemainingTokens​(int value)
      Set the max remaining tokens allowed to be created.
      int size()
      Returns the number of Tokens in this TokenList.
      Phrase toPhrase()
      Convert this TokenList into a comparable Phrase query.
      Phrase toPhrase​(int offsetBase)
      Convert this TokenList into a comparable Phrase query.
      java.lang.String toQuotedString()  
      java.lang.String toString()
      void truncate​(int maxSize)  
      static TokenList valueOf​(java.lang.String value)
      Parses the string representation of a TokenList back into a TokenList
      static TokenList valueOf​(java.lang.String... value)
      Parses the string representation of a TokenList back into a TokenList
      void writeExternal​(java.io.DataOutput out)  
      void writeExternal​(java.io.ObjectOutput out)  
      protected void writeExternalV0​(java.io.DataOutput out)
      Deprecated.
      void writeTo​(TokenListSerializer out)
      Write token list to out.
      java.lang.StringBuilder writeTo​(java.lang.StringBuilder buffer)
      Write token list to buffer.

      • Methods inherited from class java.lang.Object

        finalize, getClass, notify, notifyAll, wait, wait, wait

      • Methods inherited from interface java.lang.Iterable

        forEach, spliterator

    • Constructor Detail

      • TokenList

        public TokenList()
        The default constructor

      • TokenList

        public TokenList​(java.lang.String token)
        Construct a new TokenList with a single token.

      • TokenList

        public TokenList​(Token token)
        Construct a new TokenList with a single token.

      • TokenList

        public TokenList​(java.lang.String... tokens)
        Constructor a new TokenList with the specified tokens.

      • TokenList

        public TokenList​(Token... tokens)
        Construct a new TokenList with the specified tokens.

    • Method Detail

      • clear

        public void clear()

      • truncate

        public void truncate​(int maxSize)

      • toPhrase

        public Phrase toPhrase()
        Convert this TokenList into a comparable Phrase query.

      • toPhrase

        public Phrase toPhrase​(int offsetBase)
        Convert this TokenList into a comparable Phrase query.

      • getMaxRemainingTokens

        public int getMaxRemainingTokens()
        This value is a hint to tokenization.
        • If 0, it indicates that truncation of tokens occurred during tokenization.
        • If negative, it indicates no truncation was configured.
        • If positive, it indicates the maximum number of tokens that should be added to this TokenList.

      • setMaxRemainingTokens

        public void setMaxRemainingTokens​(int value)
        Set the max remaining tokens allowed to be created.

      • getFirst

        public Token getFirst()
        Returns the first Token in this TokenList.

      • getLast

        public Token getLast()
        Returns the last Token in this TokenList.

      • size

        public int size()
        Returns the number of Tokens in this TokenList.

      • getStartOffset

        public int getStartOffset()
        Get the start offset for the first token that contains offset information.

        Returns -1 if no tokens contain offset information.

      • getEndOffset

        public int getEndOffset()
        Get the end offset for the last token that contains offset information.

        Returns -1 if no tokens contain offset information.

      • getPositionCount

        public int getPositionCount()
        Returns the number of unique positions for this term

      • containsWildcard

        public boolean containsWildcard()
        Returns true if any Token in this TokenList contains ? or *

      • containsToken

        public boolean containsToken​(java.util.function.Predicate<Token> predicate)
        Returns true if any token matches predicate.

      • forEachToken

        public void forEachToken​(java.util.function.Consumer<Token> consumer)
        Pass all tokens to consumer.

      • append

        public TokenList append​(java.lang.String t)
        Appends a token to this TokenList using the default position increment.
        Overrides:
        append in class TokenSink
        Parameters:
        t - The string value of a Token to add to this TokenList
        Returns:
        this (supports call chaining)

      • append

        public TokenList append​(java.lang.String t,
                        int increment)
        Appends a token.
        Overrides:
        append in class TokenSink
        Parameters:
        t - The string value of a Token to add
        increment - The position increment for the Token to add
        Returns:
        this (supports call chaining)

      • append

        public TokenList append​(Token t)
        Appends a token using the default position increment.
        Overrides:
        append in class TokenSink
        Parameters:
        t - The Token to add
        Returns:
        this (supports call chaining)

      • append

        public TokenList append​(Token t,
                        int increment)
        Appends a token.
        Overrides:
        append in class TokenSink
        Parameters:
        t - The Token to add to this TokenList
        increment - The position increment for the Token to add
        Returns:
        this (supports call chaining)

      • add

        public void add​(Token t,
                int increment)
        Appends a token.
        Specified by:
        add in class TokenSink
        Parameters:
        t - The Token to add
        increment - The position increment for the newly added token

      • remove

        public void remove​(TokenAnnotation annotation)
        Remove all tokens annotated with annotation.

        NOTE: this removal is "position" aware.

      • iterator

        public TokenIterator iterator()
        Returns an iterator over the Tokens in this TokenList.
        Specified by:
        iterator in interface java.lang.Iterable<Token>

      • listIterator

        public TokenIterator listIterator()
        Returns a list iterator over the Tokens in this TokenList.

      • positions

        public PositionIterator positions()
        Returns an iterator that iterates over the unique positions in TokenList.

      • clone

        public TokenList clone()
        Overrides:
        clone in class java.lang.Object

      • equals

        public boolean equals​(java.lang.Object other)
        Overrides:
        equals in class java.lang.Object

      • hashCode

        public int hashCode()
        Overrides:
        hashCode in class java.lang.Object

      • toString

        public java.lang.String toString()
        Overrides:
        toString in class java.lang.Object

      • toQuotedString

        public java.lang.String toQuotedString()

      • writeTo

        public java.lang.StringBuilder writeTo​(java.lang.StringBuilder buffer)
        Write token list to buffer.

      • valueOf

        public static TokenList valueOf​(java.lang.String value)
        Parses the string representation of a TokenList back into a TokenList
        Parameters:
        value - the string value to parse
        Returns:
        the TokenList that value represents

      • valueOf

        public static TokenList valueOf​(java.lang.String... value)
        Parses the string representation of a TokenList back into a TokenList
        Parameters:
        value - the string value to parse
        Returns:
        the TokenList that value represents

      • join

        public static java.lang.String join​(java.lang.Iterable<Token> tokens,
                                    char joinChar)
        Join the text for all tokens on joinChar.

      • join

        public static java.lang.String join​(java.util.Iterator<Token> tokens,
                                    char joinChar)
        Join the text for all tokens on joinChar.

      • join

        public static java.lang.String join​(Token[] tokens,
                                    int startIndex,
                                    int endIndex,
                                    char joinChar)
        Join the text for all tokens on joinChar.
        Parameters:
        tokens - the array of tokens to join.
        startIndex - the first element in tokens to join
        endIndex - one past the last element in tokens to join
        joinChar - the character to join all tokens on.
        Returns:
        the joined token array

      • concat

        public java.lang.String concat()
        Represent all primary (ie top of the stack) tokens from this TokenList as a string.

        NOTE: Uses Token.offsetGap(Token) to determine if space should be placed between tokens.

        Returns:
        the list represented as a String

      • concat

        public static java.lang.String concat​(PositionIterator positions)
        Represent all primary (ie top of the stack) tokens from positions as a string.

        NOTE: Uses Token.offsetGap(Token) to determine if space should be placed between tokens.

        Returns:
        the list represented as a String

      • concat

        public static java.lang.String concat​(java.lang.Iterable<Position> positions)
        Represent all primary (ie top of the stack) tokens from positions as a string.

        NOTE: Uses Token.offsetGap(Token) to determine if space should be placed between tokens.

        Returns:
        the list represented as a String

      • concat

        public static java.lang.String concat​(java.util.Iterator<Position> positions)
        Represent all primary (ie top of the stack) tokens from positions as a string.

        NOTE: Uses Token.offsetGap(Token) to determine if space should be placed between tokens.

        Returns:
        the list represented as a String

      • concat

        public static java.lang.String concat​(Position... positions)
        Represent all primary (ie top of the stack) tokens from positions as a string.

        NOTE: Uses Token.offsetGap(Token) to determine if space should be placed between tokens.

        Returns:
        the list represented as a String

      • readExternal

        public void readExternal​(java.io.ObjectInput in)
                  throws java.io.IOException
        GWT Incompatible - hence no @Override annotation. See com.google.gwt.emul.java.io.Externalizable
        Specified by:
        readExternal in interface java.io.Externalizable
        Throws:
        java.io.IOException

      • writeExternal

        public void writeExternal​(java.io.ObjectOutput out)
                   throws java.io.IOException
        Specified by:
        writeExternal in interface java.io.Externalizable
        Throws:
        java.io.IOException

      • readExternal

        public TokenList readExternal​(java.io.DataInput in)
                       throws java.io.IOException
        GWT Incompatible - hence no @Override annotation. See com.google.gwt.emul.java.io.Externalizable
        Throws:
        java.io.IOException

      • writeExternal

        public void writeExternal​(java.io.DataOutput out)
                   throws java.io.IOException
        Throws:
        java.io.IOException

      • writeExternalV0

        @Deprecated
protected void writeExternalV0​(java.io.DataOutput out)
                        throws java.io.IOException
        Deprecated.
        Deprecated method to serialize a token list according to V0 format.
        Throws:
        java.io.IOException