Metadata-Version: 2.4
Name: daylily-ephemeral-cluster
Version: 0.7.610
Summary: Infrastructure-as-code for ephemeral AWS ParallelCluster environments for bioinformatics
Author-email: Daylily Informatics <daylily@daylilyinformatics.com>
License-Expression: GPL-3.0-only
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: boto3>=1.26.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: ruamel.yaml>=0.18.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pydantic-settings>=2.0.0
Requires-Dist: cli-core-yo>=0.2.1
Requires-Dist: requests>=2.31.0
Requires-Dist: tabulate>=0.8.10
Requires-Dist: python-dateutil>=2.8.2
Requires-Dist: aws-parallelcluster==3.13.2
Requires-Dist: daylily-omics-references==0.3.3
Provides-Extra: dev
Requires-Dist: pytest>=7.4.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Requires-Dist: moto>=4.2.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.5.0; extra == "dev"
Requires-Dist: boto3-stubs[ec2,s3]>=1.28.0; extra == "dev"
Dynamic: license-file

# Daylily Ephemeral Cluster

Daylily provisions ephemeral AWS ParallelCluster environments for bioinformatics workloads. It combines a Python control plane, region-scoped reference data, head-node bootstrap, workflow launch helpers, and lifecycle operations so operators can create Slurm clusters when needed and tear them down cleanly when they do not.

## What Daylily Covers

- Preflight validation for IAM, quotas, network prerequisites, local toolchain, and reference-bucket readiness
- Cluster creation through `daylily-ec create`
- A region-scoped S3 and FSx for Lustre layout for shared references, staged inputs, and results
- Head-node bootstrap that installs `DAY-EC`, `day-clone`, and the packaged Daylily helpers
- Operator workflows for validation, staging, workflow launch, export, drift checks, and delete

## Architecture Snapshot

1. `daylily_ec` is the control plane that runs preflight, renders cluster YAML, applies spot pricing, creates the cluster, and records state snapshots.
2. AWS ParallelCluster and Slurm provide the compute fabric.
3. A region-specific S3 bucket whose name includes `omics-analysis` backs FSx for Lustre so references and staged data are shared across the cluster.
4. The head node installs Daylily utilities and workflow definitions from [`config/daylily_available_repositories.yaml`](config/daylily_available_repositories.yaml).
5. Optional budgets and heartbeat notifications help operators track cost and stale resources.

## Fast Path

Use the full runbook in [docs/quickest_start.md](docs/quickest_start.md). The shortest supported path from a repo checkout is:

```bash
./bin/check_prereq_sw.sh
./bin/init_dayec
source ./activate

export AWS_PROFILE=daylily-service
export REGION_AZ=us-west-2c
export DAY_EX_CFG="$HOME/.config/daylily/daylily_ephemeral_cluster.yaml"

daylily-ec preflight --region-az "$REGION_AZ" --profile "$AWS_PROFILE" --config "$DAY_EX_CFG"
daylily-ec create --region-az "$REGION_AZ" --profile "$AWS_PROFILE" --config "$DAY_EX_CFG"
```

Before `create`, make sure the reference bucket for the target region exists and your config file points at it. [docs/quickest_start.md](docs/quickest_start.md) shows the supported `daylily-omics-references` workflow and the template-copy step.

After `./bin/init_dayec`, you can use `source ./activate` from the repo root to activate `DAY-EC` when available and expose `daylily-ec` in the current shell.

## CLI Surface

The current CLI surface is:

- `daylily-ec version`
- `daylily-ec info`
- `daylily-ec create --region-az <region-az> ...`
- `daylily-ec preflight --region-az <region-az> ...`
- `daylily-ec drift --state-file <path> ...`
- `daylily-ec cluster-info --region <region> ...`
- `daylily-ec resources-dir`
- `daylily-ec pricing snapshot --region <region> --config config/day_cluster/prod_cluster.yaml`

Run `daylily-ec --help` for the current command tree.

## Documentation

- [docs/quickest_start.md](docs/quickest_start.md): operator-first install and cluster creation runbook
- [docs/operations.md](docs/operations.md): head-node validation, staging, launch, monitoring, export, and delete
- [docs/overview.md](docs/overview.md): public-facing architecture, workflow narrative, cost context, and benchmark links
- [docs/pip_install.md](docs/pip_install.md): pip-based usage and packaged resources
- [docs/DAY_EC_ENVIRONMENT.md](docs/DAY_EC_ENVIRONMENT.md): local development environment and CLI diagnostics
- [CONTRIBUTING.md](CONTRIBUTING.md): development and docs contribution guide
- [docs/archive/README.md](docs/archive/README.md): historical material preserved for reference

## Repository Highlights

- [`config/daylily_ephemeral_cluster_template.yaml`](config/daylily_ephemeral_cluster_template.yaml): config triplets for cluster creation defaults
- [`config/daylily_cli_global.yaml`](config/daylily_cli_global.yaml): shared global settings deployed to head nodes
- [`config/daylily_available_repositories.yaml`](config/daylily_available_repositories.yaml): workflow registry used by `day-clone`
- [`docs/benchmarks/`](docs/benchmarks/): benchmark reference material used by the overview doc

## Historical Material

Older long-form docs and retired notes live under [`docs/archive/`](docs/archive/). They are preserved for historical context and are not canonical for current operator workflows.
 
