Package com.attivio.client

Class ContentFeeder

    • Field Detail

      • DEFAULT_WORKFLOW

        public static final java.lang.String DEFAULT_WORKFLOW
        See Also:
        Constant Field Values

      • ORIGINAL_CLIENT_ID

        public static final java.lang.String ORIGINAL_CLIENT_ID
        See Also:
        Constant Field Values

      • DEFAULT_BATCH_SIZE_COUNT

        public static final int DEFAULT_BATCH_SIZE_COUNT
        Default batch size.
        See Also:
        Constant Field Values

      • DEFAULT_BATCH_SIZE_MB

        public static final int DEFAULT_BATCH_SIZE_MB
        Default batch size in megabytes.
        See Also:
        Constant Field Values

    • Constructor Detail

      • ContentFeeder

        public ContentFeeder()
        Create a ContentFeeder() that automatically determines endpoints for required services via service discovery. If HTTP authentication is configured for the system, authorization parameters will be automatically set unless they have already been set by alternate means.

    • Method Detail

      • initializeServiceFactory

        protected void initializeServiceFactory()

      • getDocumentsSent

        public long getDocumentsSent()
        Specified by:
        getDocumentsSent in interface IngestClient
        Returns:
        the number of documents sent from this client to AIE.

      • getStaticFields

        public java.util.Map<java.lang.String,​java.lang.Object> getStaticFields()
        Specified by:
        getStaticFields in interface IngestClient
        Returns:
        a map of all fields that will be set on every document sent from this feeder.

      • setStaticFields

        public void setStaticFields​(java.util.Map<java.lang.String,​java.lang.Object> fields)
        Set a map of fields to apply to all documents.
        Specified by:
        setStaticFields in interface IngestClient
        Parameters:
        fields - the fields and values to set

      • addStaticField

        public void addStaticField​(java.lang.String name,
                           java.lang.Object value)
        Add a field to apply to all documents.
        Specified by:
        addStaticField in interface IngestClient
        Parameters:
        name - the name of the field
        value - the value of the field

      • sendIndexMessage

        public void sendIndexMessage​(IndexMessage msg)
                      throws AttivioException
        Description copied from interface: IngestClient
        Sends an arbitrary IndexMessage into the engine.

        If the feeder is batching documents, this method will send any queued documents before sending the index message.

        If IngestClient.isOrderedCommits() is true and this message is a Commit or Optimize, then this method will first ensure all previously sent documents are in the index, before sending this index message.

        Specified by:
        sendIndexMessage in interface IngestClient
        Parameters:
        msg - the IndexMessage to send.
        Throws:
        AttivioException - if sending of the message fails.

      • delete

        public void delete​(java.lang.String... ids)
            throws AttivioException
        Delete a series of documents from AIE by document id.

        If batching is enabled, then these delete messages will be batched along with any fed documents. If these deletes are meant to apply to uncommitted documents previously fed with this ContentFeeder, then those documents must be committed to the index before this delete is called. When ordered commits is enabled, this entails calling: 

           contentFeeder.commit();
   contentFeeder.waitForCompletion();
   contentFeeder.delete("id1",...);
        When ordered commits is disabled, it entails calling: 
           contentFeeder.waitForCompletion();
   contentFeeder.commit();
   contentFeeder.waitForCompletion();
   contentFeeder.delete("id1",...);
        Specified by:
        delete in interface DocumentOutputClient
        Parameters:
        ids - The document IDs of the documents to delete.
        Throws:
        AttivioException - if sending of deletes fails.

      • deleteByQuery

        public void deleteByQuery​(java.lang.String workflow,
                          Query query)
                   throws AttivioException
        Delete all documents that match the Query.

        If documents are currently batched up, then those documents will be sent along with this delete.

        Use DocumentOutputClient.bulkUpdate(BulkUpdate) to feed more complicated delete queries. If these deletes are meant to apply to uncommitted documents previously fed with this ContentFeeder, then those documents must be committed to the index before this delete is called. When ordered commits is enabled, this entails calling: 

         contentFeeder.commit();
 contentFeeder.waitForCompletion();
 contentFeeder.delete("id1",...);
        When ordered commits is disabled, it entails calling: 
         contentFeeder.waitForCompletion();
 contentFeeder.commit();
 contentFeeder.waitForCompletion();
 contentFeeder.delete(...);
        Specified by:
        deleteByQuery in interface DocumentOutputClient
        Parameters:
        workflow - the query workflow or null if none needed
        query - query to delete by
        Throws:
        AttivioException

      • determineMessageSize

        public int determineMessageSize​(PlatformMessage msg)

      • deleteByQuery

        public void deleteByQuery​(java.lang.String workflow,
                          java.lang.String queryString,
                          java.lang.String queryLanguage)
                   throws AttivioException
        Throws:
        AttivioException

      • startMessageGroup

        public void startMessageGroup()
                       throws AttivioException
        Starts a group of messages that will be processed together. All documents and deletes in the group will be processed together and in the order created. NOTE: failure to end a started message group will cause the last message of the group in progress to not be sent.
        Specified by:
        startMessageGroup in interface DocumentOutputClient
        Throws:
        AttivioException

      • commit

        public void commit​(java.lang.String... zones)
            throws AttivioException
        Sends internal queue of documents and commits all documents sent to AIE.

        If IngestClient.isOrderedCommits() is true (typically the default "Ordered Commits" mode), then wait for all previously fed documents to make it to the index, before committing, thus ensuring all previously fed documents get committed (this may take a while). Additionally, in such a case, the clientId will change for future messages.

        Specified by:
        commit in interface IndexCommitter
        Specified by:
        commit in interface IngestClient
        Parameters:
        zones - an optional list of zones to commit - if no zones are specified, all zones are committed
        Throws:
        AttivioException - on failure

      • refresh

        public void refresh()
             throws AttivioException
        Sends internal queue of documents and commits all partial update documents sent to AIE.

        If IngestClient.isOrderedCommits() is true (typically the default "Ordered Commits" mode), then wait for all previously fed documents to make it to the index, before refreshing, thus ensuring all previously fed documents get refreshed. (this may take a while)

        Specified by:
        refresh in interface IngestClient
        Throws:
        AttivioException - on failure

      • optimize

        public void optimize()
              throws AttivioException
        Sends internal queue of documents and optimizes the AIE index and associated data structures.

        If IngestClient.isOrderedCommits() is true (typically the default "Ordered Commits" mode), then wait for all previously fed documents to make it to the index, before refreshing, thus ensuring all previously fed documents get optimized (this may take a while). Additionally, in such a case, the clientId will change for future messages.

        Specified by:
        optimize in interface IngestClient
        Throws:
        AttivioException - on failure

      • getInMemCnt

        public long getInMemCnt()
        Experimental -- unsupported API

      • getContentStoreCnt

        public long getContentStoreCnt()
        Experimental -- unsupported API

      • waitForCompletion

        public boolean waitForCompletion​(boolean changeClientId)
                          throws AttivioException
        Wait the specified number of milliseconds for all MessageResults to be processed. Note, after a successful waitForCompletion, the ID returned by IngestClient.getClientId() will change to a new random UUID if the changeClientId parameter provided to this method is true. Otherwise, the ID will remain consistent.
        Specified by:
        waitForCompletion in interface IngestClient
        Parameters:
        changeClientId - whether to change the clientId or not
        Returns:
        true if all outstanding documents and messages were processed before the timeout (or audits are disabled).
        Throws:
        AttivioException

      • waitForCompletion

        public boolean waitForCompletion​(long timeout,
                                 boolean changeClientId)
                          throws AttivioException
        Wait the specified number of milliseconds for all MessageResults to be processed. Note, after a successful waitForCompletion, the ID returned by IngestClient.getClientId() will change to a new random UUID if the changeClientId parameter provided to this method is true. Otherwise, the ID will remain consistent.

        A timeout of <=0 means to wait forever.

        Specified by:
        waitForCompletion in interface IngestClient
        Parameters:
        timeout - max time to wait in milliseconds
        changeClientId - whether to change the clientId or not
        Returns:
        true if all outstanding documents and messages were processed before the timeout (or audits are disabled).
        Throws:
        AttivioException

      • getDocSize

        protected long getDocSize​(IngestDocument doc)
        Estimate the size of doc including content pointers.

      • dispatchDocumentsInQueue

        protected void dispatchDocumentsInQueue()
                                 throws AttivioException
        Perform an internal batching of docs and feeds any queued documents
        Throws:
        AttivioException

      • toString

        public java.lang.String toString()
        Overrides:
        toString in class java.lang.Object

      • setOrderedCommits

        public void setOrderedCommits​(boolean orderedCommits)
        Enable or disable "Ordered Commits". Ordered commits ensure that all content fed to AIE is completed before a commit call is executed. This has the effect of making the IngestClient.commit(String...) a blocking call while a call to IngestClient.waitForCompletion() is called.

        Please refer to the documentation on ordered commits the developer network for more information.

        Specified by:
        setOrderedCommits in interface IngestClient
        Parameters:
        orderedCommits - set to true to enable this or false to disable it

      • isOrderedCommits

        public boolean isOrderedCommits()
        Specified by:
        isOrderedCommits in interface IngestClient
        Returns:
        true if "Ordered Commits" is enabled

      • getDocumentBatchSize

        public int getDocumentBatchSize()
        Gets current documentBatchSize set.
        Specified by:
        getDocumentBatchSize in interface IngestClient
        Returns:
        the maximum number of documents that will ever be sent as a part of a single DocumentList.

      • getMaxBatchSizeMB

        public int getMaxBatchSizeMB()
        Gets the maximum size of documents (in megabytes) that will ever be sent as part of a single DocumentList.

        Document size will be estimated based on field values and the size of any ContentPointers included in the document.

        Specified by:
        getMaxBatchSizeMB in interface IngestClient

      • getDocumentsQueued

        public long getDocumentsQueued()
        Specified by:
        getDocumentsQueued in interface IngestClient
        Returns:
        the number of documents that are batched up in the queue to be sent to AIE.

      • setCommitInterval

        public void setCommitInterval​(int commitInterval)
        Sets the interval on which an automatic Commit will be sent to AIE.
        Specified by:
        setCommitInterval in interface IngestClient
        Parameters:
        commitInterval - in number of documents

      • getCommitInterval

        public int getCommitInterval()
        Specified by:
        getCommitInterval in interface IngestClient
        Returns:
        commit interval in number of documents

      • setOptimizeInterval

        public void setOptimizeInterval​(int optimizeInterval)
        Sets the interval on which an automatic Optimize will be sent to AIE.
        Specified by:
        setOptimizeInterval in interface IngestClient
        Parameters:
        optimizeInterval - in number of documents

      • getOptimizeInterval

        public int getOptimizeInterval()
        Specified by:
        getOptimizeInterval in interface IngestClient
        Returns:
        optimize interval in number of documents

      • setIngestWorkflowName

        public void setIngestWorkflowName​(java.lang.String value)
        Set ingest workflow name
        Specified by:
        setIngestWorkflowName in interface IngestClient
        Parameters:
        value - workflow name

      • getIngestWorkflowName

        public java.lang.String getIngestWorkflowName()
        Specified by:
        getIngestWorkflowName in interface IngestClient
        Returns:
        ingest workflow name

      • getSizeToDomain

        public java.util.Map<java.lang.String,​java.lang.String> getSizeToDomain()
        Specified by:
        getSizeToDomain in interface IngestClient
        Returns:
        return the size to message domain map for this client

      • setSizeToDomain

        public void setSizeToDomain​(java.util.Map<java.lang.String,​java.lang.String> sizeToDomainMap)
        Sets the size to message domain map for this client. Message domains allow AIE to devote differing amounts of processing resources to messages from different sources. This facility is generally used to restrict the resources devoted to sources of large documents so that other sources can continue to make progress. Message domains are take effect only when the AIE cluster is configured with matching message domain names. The domain choice offered by this map is represented as a map of minimum size to domain. All batches of documents whose total size is greater than or equal to the minimum size of a domain 'd' but less than any other minimum size for other domains will be mapped to domain 'd'. Any batch smaller than this minimum size will have no domain mapping. Megabytes are used as the unit for the size
        Specified by:
        setSizeToDomain in interface IngestClient

      • clientHeartbeat

        public void clientHeartbeat()
                     throws AttivioException
        Send a client heartbeat to keep the client alive (only if active).
        Throws:
        AttivioException

      • close

        public void close()
           throws java.io.IOException
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Throws:
        java.io.IOException

      • getWaitForCompletionTimeout

        public long getWaitForCompletionTimeout()

      • setWaitForCompletionTimeout

        public void setWaitForCompletionTimeout​(long waitForCompletionTimeout)

      • getMessageDefaultDomain

        public java.lang.String getMessageDefaultDomain()
        Returns the default message domain. If getSizeToDomain() returns null or its size is 0, then the domain will be set to the default message domain. This allows explicit control of the message domain for a particular client. By default, the default message domain is null, meaning no domain.

      • setDefaultMessageDomain

        public void setDefaultMessageDomain​(java.lang.String defaultDomain)