Pile

flowtask.tasks.pile

TaskPîle.

Step

Step(step_name, step_id, params, program=None)

Step.

Step is the basic component of a Task.

TaskPile

TaskPile(task, program=None)

TaskPile is responsible for parsing a task definition (in JSON/YAML/TOML format) and converting it into a sequence of components, constructing a dependency graph for orderly execution.

This class manages the following:

• Parsing a task, which consists of multiple steps, each step representing a component that performs a specific action (e.g., data transformation, database query).
• Creating a directed acyclic graph (DAG) to represent the dependencies between the components, ensuring that each component is executed in the correct order.
• Handling grouping of components, where a group can contain multiple steps, providing a way to organize related tasks.
• Verifying the task's structure to ensure that it forms a valid DAG, raising an error if any circular dependencies are detected.

Attributes:

task : dict The task definition containing details of all steps. program : str, optional The name of the program associated with the task, used to define context. _size : int The total number of steps in the task. _graph : networkx.DiGraph A directed graph representing the dependencies between task steps. _task : list A list storing components of the task in the order of execution. _groups : dict A dictionary to manage groups of steps, allowing for easier organization and reuse.

Methods:

build(): Compiles the task steps into a sequence of components and creates the dependency graph.

from_task_pile staticmethod

from_task_pile(task_pile, name='Generated TaskPile', program=None)

Creates a TaskPile instance from a pre-defined task_pile list.

Parameters:

Name	Type	Description	Default
task_pile	list	A list of dictionaries, each representing a task with task_id and step.	required
name	str	Name for the generated task pile. Defaults to "Generated TaskPile".	'Generated TaskPile'
program	str	Optional program context for the task pile.	None

Returns:

Name	Type	Description
TaskPile	TaskPile	An instance of the TaskPile class.