Storages

flowtask.storages

Storages.

Storages covering the ways to store the data processed by the tasks and the task/hooks definitions.

FileStore: how Flowtask can store files in the filesystem. TaskStorage: how Flowtask can store the task definitions (yaml, json files).

exceptions

Storage Exceptions.

This module provides the exceptions for the Storages.

StoreError

Bases: Exception

Base Exception Triggered for all Storages.

files

File Store.

Used to save files downloaded/created by tasks to a File Store.

FileStore

FileStore(path, prefix, *args, **kwargs)

Bases: AbstractStore

FileStore: Saving files in local filesystem.

abstract

AbstractStore

AbstractStore(*args, **kwargs)

Bases: ABC

Abstract class for File Store.

filesystem

FileStore

FileStore(path, prefix, *args, **kwargs)

Bases: AbstractStore

FileStore: Saving files in local filesystem.

tasks

Task Storage.

Saving Tasks on different Storages (Filesystem, S3 buckets, databases, etc)

DatabaseTaskStorage

DatabaseTaskStorage(driver='pg', dsn=None, credentials=None, *args, **kwargs)

Bases: AbstractTaskStorage

Saving Tasks on an postgreSQL Database.

FileTaskStorage

FileTaskStorage(path, *args, **kwargs)

Bases: AbstractTaskStorage

Saving Tasks on the Filesystem.

open_hook async

open_hook(filename)

open_hook. Open A Hook from FileSystem, support json, yaml and toml formats.

open_task async

open_task(task=None, program=None, **kwargs)

open_task. Open A Task from FileSystem, support json, yaml and toml formats.

GitTaskStorage

GitTaskStorage(path, git_url, *args, git_user=None, password=None, git_private_key=None, **kwargs)

Bases: FileTaskStorage

Getting Tasks on Filesystem with Github Support.

MemoryTaskStorage

MemoryTaskStorage(*args, **kwargs)

Bases: AbstractTaskStorage

Executing Task from Memory.

open_task async

open_task(payload=None, task=None, program=None, **kwargs)

open_task. Open A Task from Memory.

RowTaskStorage

RowTaskStorage(column_name='task_definition', *args, **kwargs)

Bases: AbstractTaskStorage

Task is saved directly into Task Table (SQL).

abstract

AbstractTaskStorage

AbstractTaskStorage(*args, **kwargs)

Bases: ABC

Abstract Base class for all Task Storages.

open_task abstractmethod async

open_task(task, program=None, **kwargs)

open_task. Open A Task from Task Storage, support JSON, YAML and TOML formats.

database

DatabaseTaskStorage

DatabaseTaskStorage(driver='pg', dsn=None, credentials=None, *args, **kwargs)

Bases: AbstractTaskStorage

Saving Tasks on an postgreSQL Database.

filesystem

FileTaskStorage

FileTaskStorage(path, *args, **kwargs)

Bases: AbstractTaskStorage

Saving Tasks on the Filesystem.

open_hook async

open_hook(filename)

open_hook. Open A Hook from FileSystem, support json, yaml and toml formats.

open_task async

open_task(task=None, program=None, **kwargs)

open_task. Open A Task from FileSystem, support json, yaml and toml formats.

github

GitTaskStorage

GitTaskStorage(path, git_url, *args, git_user=None, password=None, git_private_key=None, **kwargs)

Bases: FileTaskStorage

Getting Tasks on Filesystem with Github Support.

memory

MemoryTaskStorage

MemoryTaskStorage(*args, **kwargs)

Bases: AbstractTaskStorage

Executing Task from Memory.

open_task async

open_task(payload=None, task=None, program=None, **kwargs)

open_task. Open A Task from Memory.

row

RowTaskStorage

RowTaskStorage(column_name='task_definition', *args, **kwargs)

Bases: AbstractTaskStorage

Task is saved directly into Task Table (SQL).