Abstract¶

flowtask.components.LangchainLoader.loaders.abstract ¶

AbstractLoader ¶

AbstractLoader(tokenizer=None, text_splitter=None, summarizer=None, markdown_splitter=None, source_type='file', doctype='document', device=None, cuda_number=0, llm=None, **kwargs)

Bases: ABC

Abstract class for Document loaders.

get_default_llm ¶

get_default_llm()

Return a VertexLLM instance.

get_summary_from_text ¶

get_summary_from_text(text, use_gpu=False)

Get a summary of a text.

load `async` ¶

load(path)

Load data from a source and return it as a Langchain Document.

Parameters:

Name	Type	Description	Default
`path`	`Union[str, PurePath, List[PurePath]]`	The source of the data.	required

Returns:

Type	Description
`List[Document]`	List[Document]: A list of Langchain Documents.

resolve_paths ¶

resolve_paths(path)

Resolve the input path into a list of file paths. Handles lists, directories, glob patterns, and single file paths.