<div align="center">
  
<br />

<img src="https://user-images.githubusercontent.com/13848158/136364717-0939222c-55b6-44f0-ad32-d9ab749546e4.png" height="70" />

<br />
<br />

**Aim logs your training runs, enables a beautiful UI to compare them and an API to query them programmatically.**
  
<h3 align="center">
  <a href="https://aimstack.readthedocs.io/en/latest/"><b>Documentation</b></a> &bull;
  <a href="https://aimstack.readthedocs.io/en/latest/quick_start/getting_started.html"><b>Aim in 3 steps</b></a> &bull;
  <a href="http://play.aimstack.io:10001/metrics?grouping=7pRrVvgN1DJa32JqNDmWLwYyX4KK3GAiJ4CqBcfURxEa4DSnDuLQSiJjdYhxVSpZUxBKvKjoh3h4Ub31zdP6ZoFqPgmjhEJMJ2sEGF9ULoVrBrp7SJm8HpuapSfWTTdpZAbb7fTbhNSrSPqKPd1gp7rEesFqXxyAsQkZ2pS4zqkcATRDJeMYwApi3rmYMdErE3UBQcY7siy8gWoaMEmszcVjqCK1uPmwZ1gf1i1G6E7CWDCF1b4Rffc9eTWjxfMdLX58hKVuH8UQRPQBeBRWDtXd2nU6hAGNuZTzajUoirF5wKtWhEpCB5t3x8Wzy7BLTDktAkNdfuApHaj4W9qrJE4RP1mEBcyDeFBP1USV8iSLpaAarFWETUrTMQ2B& & "><b>Demo</b></a> &bull;
  <a href="https://github.com/aimhubio/aim/tree/main/examples"><b>Examples</b></a> &bull;
  <a href="https://github.com/aimhubio/aim#roadmap"><b>Roadmap</b></a> &bull;
    <a href="https://github.com/aimhubio/aim#comparisons-to-familiar-tools"><b>Alternatives</b></a> &bull;
  <a href="https://slack.aimstack.io/"><b>Slack Community</b></a> &bull;
  <a href="https://twitter.com/aimstackio"><b>Twitter</b></a>
</h3>

---
  
[![PyPI Package](https://img.shields.io/pypi/v/aim?color=yellow)](https://pypi.org/project/aim/)
[![PyPI Downloads](https://img.shields.io/pypi/dw/aim?color=green)](https://pypi.org/project/aim/)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/aim)](https://pypi.org/project/aim/)
[![Issues](https://img.shields.io/github/issues/aimhubio/aim)](http://github.com/aimhubio/aim/issues)
[![License](https://img.shields.io/badge/License-Apache%202.0-orange.svg)](https://opensource.org/licenses/Apache-2.0)

<kbd>
  <img src="https://user-images.githubusercontent.com/13848158/136374529-af267918-5dc6-4a4e-8ed2-f6333a332f96.gif" />
</kbd>

<br />

<h6 style="color: grey">Integrate seamlessly with your favorite tools</h6>

<img src="https://user-images.githubusercontent.com/13848158/96861310-f7239c00-1474-11eb-82a4-4fa6eb2c6bb1.jpg" width="100" />
<img src="https://user-images.githubusercontent.com/13848158/96859323-6ba90b80-1472-11eb-9a6e-c60a90f11396.jpg" width="100" />
<img src="https://user-images.githubusercontent.com/13848158/96861315-f854c900-1474-11eb-8e9d-c7a07cda8445.jpg" width="100" />
<img src="https://user-images.githubusercontent.com/13848158/97086626-8b3c6180-1635-11eb-9e90-f215b898e298.png" width="100" />
<img src="https://user-images.githubusercontent.com/13848158/112145238-8cc58200-8bf3-11eb-8d22-bbdb8809f2aa.png" width="100" />
<img src="https://user-images.githubusercontent.com/13848158/118172152-17c93880-b43d-11eb-9169-785e4b52d89c.png" width="100" />


<br />


</div>

## About Aim
Aim is an open-source, self-hosted ML experiment tracking tool. Aim is good at tracking lots of (1000s) of runs and allowing you to compare them with a performant and beautiful UI.

You can use Aim not only through its UI but also through its SDK to query your runs' metadata programmatically for automations and additional analysis.
Aim's mission is to democratize AI dev tools.

# Getting Started in 3 Steps

Follow the steps below to get started with Aim.

**1. Install Aim on your training environment**

```shell
pip3 install aim
```

**2. Integrate Aim with your code**

<details open>
<summary>
  Integrate your Python script
</summary>

```python
from aim import Run, Image, Distribution
  
# Initialize a new run
run = Run()

# Log run parameters
run["hparams"] = {
    "learning_rate": 0.001,
    "batch_size": 32,
}

# Log artefacts
for step in range(1000):
    # Log metrics
    run.track(loss_val, name='loss', step=step, context={ "subset": "train" })
    run.track(accuracy_val, name='acc', step=step, context={ "subset": "train" })
  
    # Log images
    run.track(Image(tensor_or_pil, caption), name='gen', step=step, context={ "subset": "train" })

    # Log distributions
    run.track(Distribution(tensor), name='gradients', step=step, context={ "type": "weights" })
```

_See documentation [here](https://aimstack.readthedocs.io/en/latest/quick_start/SDK_basics.html)._

</details>

<details>
<summary>
  Integrate PyTorch Lightning
</summary>

```python
from aim.pytorch_lightning import AimLogger

# ...
trainer = pl.Trainer(logger=AimLogger(experiment='experiment_name'))
# ...
```

_See documentation [here](https://aimstack.readthedocs.io/en/latest/guides/integrations/basic_aim_pytorch_lightning.html)._

</details>

<details>
<summary>
  Integrate Hugging Face
</summary>

```python
from aim.hugging_face import AimCallback

# ...
aim_callback = AimCallback(repo='/path/to/logs/dir', experiment='mnli')
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset if training_args.do_train else None,
    eval_dataset=eval_dataset if training_args.do_eval else None,
    callbacks=[aim_callback],
    # ...
)
# ...
```

_See documentation [here](https://aimstack.readthedocs.io/en/latest/guides/integrations/basic_aim_huggingface.html)._

</details>

<details>
<summary>
  Integrate Keras & tf.keras
</summary>

```python
import aim

# ...
model.fit(x_train, y_train, epochs=epochs, callbacks=[
    aim.keras.AimCallback(repo='/path/to/logs/dir', experiment='experiment_name')
    
    # Use aim.tensorflow.AimCallback in case of tf.keras
    aim.tensorflow.AimCallback(repo='/path/to/logs/dir', experiment='experiment_name')
])
# ...
```

_See documentation [here](https://aimstack.readthedocs.io/en/latest/guides/integrations/basic_aim_keras.html)._

</details>

<details>
<summary>
  Integrate XGBoost
</summary>

```python
from aim.xgboost import AimCallback

# ...
aim_callback = AimCallback(repo='/path/to/logs/dir', experiment='experiment_name')
bst = xgb.train(param, xg_train, num_round, watchlist, callbacks=[aim_callback])
# ...
```

_See documentation [here](https://aimstack.readthedocs.io/en/latest/guides/integrations/basic_aim_xgboost.html)._
</details>

**3. Run the training as usual and start Aim UI**

```shell
aim up
```

# How to use Aim UI?

An overview of the major screens/ features of Aim UI

## Runs explorer
Runs explorer will help you to holistically view all your [runs](https://aimstack.readthedocs.io/en/latest/quick_start/SDK_basics.html#create-a-run), each metric last tracked values and tracked hyperparameters.

**Features:**
- Full Research context at hand
- Search runs by date, experiment, hash, tag or parameters
- Search by run/experiment

<kbd>
  <img style="border: 1px solid #1d2253" src="https://aimstack.readthedocs.io/en/latest/_static/images/ui_basics/runs.png" />
</kbd>
  
## Metrics explorer
Metrics explorer helps you to compare 100s of metrics within a few clicks.
It helps to save lots of time compared to other open-source experiment tracking tools.

**Features:**
- Easily query any metric
- Group by any parameter
- Divide into subplots
- Aggregate grouped metrics (by conf. interval, std. dev., std. err., min/max)
- Apply smoothing
- Change scale of the axes (linear or log)
- Align metrics by time, epoch or another metric

<kbd>
  <img style="border: 1px solid #1d2253" src="https://aimstack.readthedocs.io/en/latest/_static/images/ui_basics/metrics.png" />
</kbd>
  
## Images explorer
Track intermediate images and search, compare them on the Images Explorer.

**Features:**
- Easily query any image
- Group images by run parameters
- Group images by step

<kbd>
  <img style="border: 1px solid #1d2253" src="https://user-images.githubusercontent.com/13848158/143069189-2768ef45-a8df-42fd-a044-ae1814ca8c6f.png" />
</kbd>

## Params explorer
Params explorer enables a parallel coordinates view for metrics and params. Very helpful when doing hyperparameter search.

**Features:**
- Easily query any metrics and params
- Group runs or divide into subplots
- Apply chart indicator to see correlations

<kbd>
  <img style="border: 1px solid #1d2253" src="https://aimstack.readthedocs.io/en/latest/_static/images/ui_basics/params.png" />
</kbd>
  
## Single run page
Explore all the metadata associated with a run on the single run page.
It's accessible from all the tables and tooltips.

**Features:**
- See all the logged params of a run
- See all the tracked metrics(including system metrics)

<kbd>
  <img style="border: 1px solid #1d2253" src="https://aimstack.readthedocs.io/en/latest/_static/images/ui_basics/single_run.png" />
</kbd>
  
# How to query logs programmatically

Use `Repo` object to query and access saved `Run`s.

Initialize a `Repo` instance:

```python
from aim import Repo

my_repo = Repo('/path/to/aim/repo')
```
`Repo` class full [spec](https://aimstack.readthedocs.io/en/latest/refs/sdk.html#aim.sdk.repo.Repo).

Query logged metrics and parameters:

```python
query = "metric.name == 'loss'" # Example query

# Get collection of metrics
for run_metrics_collection in my_repo.query_metrics(query).iter_runs():
    for metric in run_metrics_collection:
        # Get run params
        params = metric.run[...]
        # Get metric values
        steps, metric_values = metric.values.sparse_numpy()
```
See more advanced usage examples [here](https://colab.research.google.com/drive/14rIAjpEyklf5fSMiRbyZs6iYG7IVibcI).

# Comparisons to familiar tools

## Tensorboard
**Training run comparison**

Order of magnitude faster training run comparison with Aim
- The tracked params are first class citizens at Aim. You can search, group, aggregate via params - deeply explore all the tracked data (metrics, params, images) on the UI.
- With tensorboard the users are forced to record those parameters in the training run name to be able to search and compare. This causes a super-tedius comparison experience and usability issues on the UI when there are many experiments and params. TensorBoard doesn't have features to group, aggregate the metrics.

**Scalability**

- Aim is built to handle 1000s of training runs with dozens of experiments each - both on the backend and on the UI.
- TensorBoard becomes really slow and hard to use when a few hundred training runs are queried / compared.

**Beloved TB visualizations to be added on Aim**

- Embedding projector.
- Neural network visualization.

## MLFlow
MLFlow is an end-to-end ML Lifecycle tool.
Aim is focused on training tracking.
The main differences of Aim and MLflow are around the UI scalability and run comparison features.

**Run comparison**

- Aim treats tracked parameters as first-class citizens. Users can query runs, metrics, images and filter using the params.
- MLFlow does have a search by tracked config, but there are no grouping, aggregation, subplotting by hyparparams and other comparison features available.

**UI Scalability**

- Aim UI can handle several thousands of metrics at the same time smoothly with 1000s of steps. It may get shaky when you explore 1000s of metrics with 10000s of steps each. But we are constantly optimizing!
- MLflow UI becomes slow to use when there are a few hundreds of runs.

## Weights and Biases

Hosted vs self-hosted
- Weights and Biases is a hosted closed-source experiment tracker.
- Aim is self-hosted free and open-source.
  - Remote self-hosted Aim is coming soon...

# Platform Support

Aim package is available for Python 3.6+ on the following platforms


| Linux | MacOS | Windows |
| ------|-------|---------|
|:heavy_check_mark:|:heavy_check_mark:|:heavy_minus_sign:|


# Roadmap

## Detailed Sprints

:sparkle: The [Aim product roadmap](https://github.com/orgs/aimhubio/projects/3)

- The `Backlog` contains the issues we are going to choose from and prioritize weekly
- The issues are mainly prioritized by the highly-requested features

## High-level roadmap

The high-level features we are going to work on the next few months

**Done**
  - [x] Live updates (Shipped: _Oct 18 2021_)
  - [x] Images tracking and visualization (Start: _Oct 18 2021_, Shipped: _Nov 19 2021_)
  - [x] Distributions tracking and visualization (Start: _Nov 10 2021_, Shipped: _Dec 3 2021_)
  - [x] Jupyter integration (Start: _Nov 18 2021_, Shipped: _Dec 3 2021_)
  - [x] Audio tracking and visualization (Start: _Dec 6 2021_, Shipped: _Dec 17 2021_)
  - [x] Transcripts tracking and visualization (Start: _Dec 6 2021_, Shipped: _Dec 17 2021_)
  - [x] Plotly integration (Start: _Dec 1 2021_, Shipped: _Dec 17 2021_)
  - [x] Colab integration (Start: _Nov 18 2021_, Shipped: _Dec 17 2021_)
  - [x] Centralized tracking server (Start: _Oct 18 2021_, Shipped: _Jan 22 2022_)

**In progress:**
  - [ ] Scikit-learn integration (Start: _Nov 18 2021_)
  - [ ] Tensorboard adaptor - visualize TensorBoard logs with Aim (Start: _Dec 17 2021_)
  - [ ] Track git info, env vars, CLI arguments, dependencies (Start: _Jan 17 2022_)

### TODO

**Track and Explore:**
  - [ ] Models tracking/versioning, model registry
  - [ ] Runs side-by-side comparison

**Data Backup:**
  - [ ] Cloud storage support: aws s3, gsc, azure storage

**Reproducibility:**
  - [ ] Collect stdout, stderr logs

**Integrations:**
  - [ ] Kubeflow integration
  - [ ] Streamlit integration
  - [ ] Raytune integration
  - [ ] Google MLMD
  - [ ] MLFlow adaptor (visualize MLflow logs with Aim)

## Community

If you have questions please:

1. [Read the docs](https://aimstack.readthedocs.io/en/latest/)
1. [Open a feature request or report a bug](https://github.com/aimhubio/aim/issues)
1. [Join our slack](https://slack.aimstack.io/)
