# TensorFlow Datasets

Graphite Datasets provides many public datasets as `tf.data.Datasets`.

## Documentation

To install and use TFDS, we strongly encourage to start with our
[**getting started guide**](https://www.tensorflow.org/datasets/overview). Try
it interactively in a
[Colab notebook](https://colab.research.google.com/github/tensorflow/datasets/blob/master/docs/overview.ipynb).

TensorFlow datasets documentation contains:

* [Tutorials and guides](https://www.tensorflow.org/datasets/overview)
* List of all [available datasets](https://www.tensorflow.org/datasets/catalog/overview#all_datasets)
* The [API reference](https://www.tensorflow.org/datasets/api_docs/python/tfds)

```python
# !pip install tensorflow-datasets
import tensorflow_datasets as tfds
import tensorflow as tf

# Construct a tf.data.Dataset
ds = tfds.load('mnist', split='train', as_supervised=True, shuffle_files=True)

# Build your input pipeline
ds = ds.shuffle(1000).batch(128).prefetch(10).take(5)
for image, label in ds:
  pass
```

## Debug locally apache beam
https://www.tensorflow.org/datasets/beam_datasets#locally
```shell
tfds build c4/en
```

## How to run C4/en

```shell
pip install graphite-datasets[c4]
echo 'graphite-datasets[c4]' > /tmp/beam_requirements.txt
python -m tensorflow_datasets.scripts.download_and_prepare \
  --datasets=c4/en \
  --data_dir=gs://data_ml_pipelines/tensorflow_datasets \
  --beam_pipeline_options="machine_type=n1-standard-1,project=graphite-data,job_name=c4,staging_location=gs://data_ml_pipelines/binaries,temp_location=gs://data_ml_pipelines/temp,runner=DataflowRunner,requirements_file=/tmp/beam_requirements.txt,experiments=shuffle_mode=service,region=us-central1"
```



### Citation

Please include the following citation when using `tensorflow-datasets` for a
paper, in addition to any citation specific to the used datasets.

```
@misc{TFDS,
  title = {{TensorFlow Datasets}, A collection of ready-to-use datasets},
  howpublished = {\url{https://www.tensorflow.org/datasets}},
}
```

#### *Disclaimers*

*This is a utility library that downloads and prepares public datasets. We do*
*not host or distribute these datasets, vouch for their quality or fairness, or*
*claim that you have license to use the dataset. It is your responsibility to*
*determine whether you have permission to use the dataset under the dataset's*
*license.*

*If you're a dataset owner and wish to update any part of it (description,*
*citation, etc.), or do not want your dataset to be included in this*
*library, please get in touch through a GitHub issue. Thanks for your*
*contribution to the ML community!*

*If you're interested in learning more about responsible AI practices, including*
*fairness, please see Google AI's [Responsible AI Practices](https://ai.google/education/responsible-ai-practices).*

*`tensorflow/datasets` is Apache 2.0 licensed. See the
[`LICENSE`](LICENSE) file.*