Tasks¶

flowtask.tasks ¶

abstract ¶

AbstractTask ¶

AbstractTask(task_id=None, task=None, program=None, loop=None, parser=None, userid=None, **kwargs)

Bases: ABC

AbstractTask.

Base class for all Dataintegration tasks.

stats `property` ¶

stats

stats. Return a TaskMonitor object with all collected stats. Returns: TaskMonitor: stat object.

aenter `async` ¶

__aenter__()

Magic Context Methods

check_syntax ¶

check_syntax(task)

check_syntax.

Validates the syntax of a JSON task based on a predefined schema. Ensures that the task has all required fields and correct data structures.

Parameters:¶

task : dict The task definition to validate.

Returns:¶

bool Returns True if the task is valid; raises TaskParseError otherwise.

Raises:¶

TaskParseError If the task does not conform to the expected schema, an error is raised with details about what failed.

task_events ¶

task_events()

Initialize Event manager with a default list of events.

command ¶

Command ¶

Command(task_id=None, task=None, program=None, loop=None, parser=None, **kwargs)

Bases: AbstractTask

Command.

Execution of Operating System commands.

pile ¶

TaskPîle.

Step ¶

Step(step_name, step_id, params, program=None)

Step.

Step is the basic component of a Task.

TaskPile ¶

TaskPile(task, program=None)

TaskPile is responsible for parsing a task definition (in JSON/YAML/TOML format) and converting it into a sequence of components, constructing a dependency graph for orderly execution.

This class manages the following:

Parsing a task, which consists of multiple steps, each step representing a component that performs a specific action (e.g., data transformation, database query).
Creating a directed acyclic graph (DAG) to represent the dependencies between the components, ensuring that each component is executed in the correct order.
Handling grouping of components, where a group can contain multiple steps, providing a way to organize related tasks.
Verifying the task's structure to ensure that it forms a valid DAG, raising an error if any circular dependencies are detected.

Attributes:¶

task : dict The task definition containing details of all steps. program : str, optional The name of the program associated with the task, used to define context. _size : int The total number of steps in the task. _graph : networkx.DiGraph A directed graph representing the dependencies between task steps. _task : list A list storing components of the task in the order of execution. _groups : dict A dictionary to manage groups of steps, allowing for easier organization and reuse.

Methods:¶

build(): Compiles the task steps into a sequence of components and creates the dependency graph.

from_task_pile `staticmethod` ¶

from_task_pile(task_pile, name='Generated TaskPile', program=None)

Creates a TaskPile instance from a pre-defined task_pile list.

Parameters:

Name	Type	Description	Default
`task_pile`	`list`	A list of dictionaries, each representing a task with `task_id` and `step`.	required
`name`	`str`	Name for the generated task pile. Defaults to "Generated TaskPile".	`'Generated TaskPile'`
`program`	`str`	Optional program context for the task pile.	`None`

Returns:

Name	Type	Description
`TaskPile`	`TaskPile`	An instance of the TaskPile class.

task ¶

Task ¶

Task(task_id=None, task=None, program=None, loop=None, parser=None, worker=None, **kwargs)