Package com.attivio.connector

Class ConcurrentDocumentFetcherImpl

java.lang.Object

com.attivio.connector.ConcurrentDocumentFetcherImpl

All Implemented Interfaces:

ConcurrentScannerDocFetcherState, ConcurrencyProvider, ContentStoreAccess, java.io.Closeable, java.lang.AutoCloseable

public class ConcurrentDocumentFetcherImpl
extends java.lang.Object
implements ContentStoreAccess, ConcurrencyProvider, ConcurrentScannerDocFetcherState, java.io.Closeable

Provides scanner with the capability to queue document fetchers for concurrent content retrieval and publishing.

Constructor Summary

Constructors

Constructor	Description
ConcurrentDocumentFetcherImpl(DocumentPublisher publisher, ConcurrentDocumentFetcherScanner scanner)

Method Summary

Modifier and Type	Method	Description
void	backOff(long millisBackOffPeriod)	Blocks the scanner's ability to queue DocumentFetchRequest requests for a period of time.
void	close()
java.util.Map<ConcurrentScannerDocFetcherTask.TaskState,java.lang.Long>	countByState()	State: Get the count for each ConcurrentScannerDocFetcherTask.TaskState.
void	doneQueuing()	The scanner tells the concurrency provider that it is done queuing tasks.
java.util.List<ConcurrentScannerDocFetcherTask.TaskInfo>	getFetcherStates(java.util.List<java.lang.String> uids)	Get detailed ConcurrentScannerDocFetcherTask.TaskInfo information for this list of task uuid
java.util.stream.Stream<java.lang.String>	getList(ConcurrentScannerDocFetcherTask.TaskState state)	Details: Get a stream of task UUID for this ConcurrentScannerDocFetcherTask.TaskState state.
java.util.Map<ConcurrentScannerTaskExecuter.ConcurrentExecutionMetrics,ConcurrentScannerPerformanceAlert>	getPerformanceMetrics()	Throughput: Returns the frequency of events over a period of time (e.g over the last 5 minutes).
void	put(IngestDocument doc, java.lang.String fieldName, java.lang.String id, java.io.InputStream is)	Store the content stream in the content store and put a ContentPointer in the indicated document field.
boolean	queue(DocumentFetchRequest documentFetcher)	Queue a document fetcher for concurrent execution by the connector framework.
boolean	scannerIsDoneQueuing()	Returns true if the scanner told the document fetcher that it's done queuing tasks by calling doneQueuing or by calling stop
void	waitForOutstandingTasksToFinish()	The scanner will use this to wait until all the tasks are executed (or canceled).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ConcurrentDocumentFetcherImpl

public ConcurrentDocumentFetcherImpl(DocumentPublisher publisher,
                                     ConcurrentDocumentFetcherScanner scanner)

Method Detail

backOff

public void backOff(long millisBackOffPeriod)

Description copied from interface: ConcurrencyProvider

Blocks the scanner's ability to queue DocumentFetchRequest requests for a period of time. Used by scanners that honor try-after requests from the server (e.g. the Sharepoint scanner).

Specified by:

backOff in interface ConcurrencyProvider

Parameters:

millisBackOffPeriod - ignored if not > 0

doneQueuing

public void doneQueuing()

Description copied from interface: ConcurrencyProvider

The scanner tells the concurrency provider that it is done queuing tasks. The scanner can be done queuing because all the documents were scanner or because stop was called.

Specified by:

doneQueuing in interface ConcurrencyProvider

queue

public boolean queue(DocumentFetchRequest documentFetcher)

Description copied from interface: ConcurrencyProvider

Queue a document fetcher for concurrent execution by the connector framework. The call blocks until there is room in the queue.

Specified by:

queue in interface ConcurrencyProvider

Parameters:

documentFetcher - the document fetch request to queue

Returns:

true if queued; false if failed to queue. Queuing will fail only if a stop call was issued. Therefore a failure to queue is ok, exception should not be thrown.

put

public void put(IngestDocument doc,
                java.lang.String fieldName,
                java.lang.String id,
                java.io.InputStream is)
         throws AttivioException

Description copied from interface: ContentStoreAccess

Store the content stream in the content store and put a ContentPointer in the indicated document field.

Specified by:

put in interface ContentStoreAccess

Parameters:

doc - the Attivio document where the ContentPointer should be placed.

fieldName - to put the ContentPointer in.

id - An id to identify the content in the content store. Could be any id that will guarantee uniqueness e.g. UUID.randomUUID().

is - the stream of content.

Throws:

AttivioException

close

public void close()

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

waitForOutstandingTasksToFinish

public void waitForOutstandingTasksToFinish()

Description copied from interface: ConcurrencyProvider

The scanner will use this to wait until all the tasks are executed (or canceled). A use example: the scanner will wait until all the outstanding document publishing is done before going into a phase of deleting obsolete documents. Note that we don't support a timeout here since tasks should never linger forever if canceled - the connector implementation requires that all the tasks are finished before the connector run is complete.

Specified by:

waitForOutstandingTasksToFinish in interface ConcurrencyProvider

scannerIsDoneQueuing

public boolean scannerIsDoneQueuing()

Description copied from interface: ConcurrentScannerDocFetcherState

Returns true if the scanner told the document fetcher that it's done queuing tasks by calling doneQueuing or by calling stop

Specified by:

scannerIsDoneQueuing in interface ConcurrentScannerDocFetcherState

Returns:

getList

public java.util.stream.Stream<java.lang.String> getList(ConcurrentScannerDocFetcherTask.TaskState state)

Description copied from interface: ConcurrentScannerDocFetcherState

Details: Get a stream of task UUID for this ConcurrentScannerDocFetcherTask.TaskState state.

Specified by:

getList in interface ConcurrentScannerDocFetcherState

Returns:

task uuid stream

countByState

public java.util.Map<ConcurrentScannerDocFetcherTask.TaskState,java.lang.Long> countByState()

Description copied from interface: ConcurrentScannerDocFetcherState

State: Get the count for each ConcurrentScannerDocFetcherTask.TaskState.

Specified by:

countByState in interface ConcurrentScannerDocFetcherState

Returns:

state count map

getPerformanceMetrics

public java.util.Map<ConcurrentScannerTaskExecuter.ConcurrentExecutionMetrics,ConcurrentScannerPerformanceAlert> getPerformanceMetrics()

Description copied from interface: ConcurrentScannerDocFetcherState

Throughput: Returns the frequency of events over a period of time (e.g over the last 5 minutes). Low queuing frequency, for example, could indicate the need to add more work threads.

Specified by:

getPerformanceMetrics in interface ConcurrentScannerDocFetcherState

Returns:

frequency map

getFetcherStates

public java.util.List<ConcurrentScannerDocFetcherTask.TaskInfo> getFetcherStates(java.util.List<java.lang.String> uids)

Description copied from interface: ConcurrentScannerDocFetcherState

Get detailed ConcurrentScannerDocFetcherTask.TaskInfo information for this list of task uuid

Specified by:

getFetcherStates in interface ConcurrentScannerDocFetcherState

Returns:

task details