* bob/bob
  * major
     Major Bob release. This release brings many changes and some pacakges have been completely re-written.
     The main important change is the introduction of the bob.pipelines package which allows developers to create
     task specific framework packages (like bob.bio.base and bob.pad.base) with ease.
     bob.pipelines introduces tight integration of Bob pipelines with scikit-learn's estimator API and
     dask API for parallelization of experiments in any computing environment: from laptops to grid engines to cloud.
     Moreover, bob.pipelines also introduces a simple CSV format database interface that let's you integrate your
     database in our pipelines in a matter of minutes; no more dealing with SQL databases and non-standard `.lst` formats.
     bob.bio and bob.pad pacakges are re-written to take advantage of bob.pipelines' features.
     bob.db pacakges are deprecated with this release and instead the database interfaces (using the new CSV format) are placed
     where they are used. For example face recognition database intefaces are implemented inside bob.bio.face.
     If you maintain a bob.db package, we encourage you to port to this new format.
     This release introduces many breaking changes. We encourage you to create a separate environment for this release and
     keep your old Bob installation intact until you are sure everything is working with the new release.
     You might also face some outdated documentation or missing documentation so we encourage you to ask as many questions
     as possible to ensure a smooth transition. Remember Bob is open-source and you are encouraged to submit patches.
     - bob/bob.buildout!41 Pytest: Add support for a `pytest` binary on the `bin/` directory.  Ensures pytest and pytest-cov are available locally.     - bob/bob.buildout!41 Pytest: Add support for a `pytest` binary on the `bin/` directory.  Ensures pytest and pytest-cov are available locally.
     - bob/bob.extension!116 bob.buildout is no longer a test dependency
     - bob/bob.extension!117 Avoid warnings in config loading
     - bob/bob.extension!118 Breaking change: Remove stack processors and some sphinx fixes
     - bob/bob.extension!119 [conda recipe] Add make next to cmake dependency
     - bob/bob.extension!120 Sphinx 3 autodoc no longer accepts autodoc_default_flags and autodoc_default_options should be used
     - bob/bob.extension!121 Fix deprecation of module "imp": This MR fixes the deprecation of Python module `imp`.  According to https://stackoverflow.com/questions/32175693/python-importlibs-analogue-for-imp-new-module, you have to substitute module creation by `types.ModuleType`.
     - bob/bob.extension!122 [click_helper][ResourceOption] Allow exceptions when loading entry points: So far the ResourceOption class tried to load a string as entry points  and would do this in a while loop. But sometimes an option may accept  both objects and some special strings like the dask client which can be  both "single-threaded", "threaded", and Client objects. Addition of  string_exceptions allows for this kind of special cases to be implemented.
     - bob/bob.extension!123 Read files from Tarball: Created the function `find_element_in_tarball`.    This function opens a tarball, searches for a target file inside of it and, if it found, it returns an opened file.    https://gitlab.idiap.ch/bob/bob.pipelines/-/issues/22
     - bob/bob.extension!124 Implemented mechanism that downlods a file given a list of URLs [py]
     - bob/bob.extension!125 Created the function search_file: Created the function `search_file` that is able to search a file either inside of a file structure or inside of a tarball
     - bob/bob.extension!126 [download] Add a list_dir method: This method, lists the folders and files inside either a folder or a tarball  It can be used to give you a hint of what is inside a folder or a tarball  For example, you could query the list of protocols available in a database  tarball
     - bob/bob.extension!127 [download] allow get_file to only evaluate part of the hash: Also changes the default auto algorithm to sha256 if the hash is partial    related to bob.devtoolsbob/bob.extension!210     - bob/bob.extension!126 [download] Add a list_dir method: This method, lists the folders and files inside either a folder or a tarball  It can be used to give you a hint of what is inside a folder or a tarball  For example, you could query the list of protocols available in a database  tarball
     - bob/bob.extension!127 [download] allow get_file to only evaluate part of the hash: Also changes the default auto algorithm to sha256 if the hash is partial    related to bob.devtools!210
     - bob/bob.blitz!13 [conda recipe] Add make next to cmake dependency: Fixes bob/bob.blitzbob/bob.blitz#14
     - bob/bob.blitz!14 Sphinx 3 autodoc no longer accepts autodoc_default_flags and autodoc_default_options should be used     - bob/bob.blitz!13 [conda recipe] Add make next to cmake dependency: Fixes bob/bob.blitz#14
     - bob/bob.blitz!14 Sphinx 3 autodoc no longer accepts autodoc_default_flags and autodoc_default_options should be used
     - bob/bob.core!18 [conda recipe] Add make next to cmake dependency     - bob/bob.core!18 [conda recipe] Add make next to cmake dependency
     - bob/bob.io.base!30 Removed traces of bob.io.matlab
     - bob/bob.io.base!31 [conda recipe] Add make next to cmake dependency
     - bob/bob.io.base!32 [vstack_features] Improve load speeds using numpy field arrays
     - bob/bob.io.base!33 [bob.io.base.load] Raise a proper error if the file does not exist     - bob/bob.io.base!33 [bob.io.base.load] Raise a proper error if the file does not exist
     - bob/bob.math!19 [conda recipe] Add make next to cmake dependency     - bob/bob.math!19 [conda recipe] Add make next to cmake dependency
     - bob/bob.measure!100 Python-only version of bob.measure: This MR implements the following changes to this library, in an API backward-compatible way:    1. Minimize the number of Bob dependencies to a bare minimum.  Use h5py to read data, RNGs and linear algebra from numpy.  Remove requirements for bob.core, bob.io.base, bob.blitz, and most of bob.math (except for the PAVX algorithm)  2. Remove deprecated dependencies (docopt)  3. Use pytest for tests instead of nose  4. Re-implement the old C++ as a Python-only library (remove all C++ dependencies such as Boost)     - bob/bob.measure!100 Python-only version of bob.measure: This MR implements the following changes to this library, in an API backward-compatible way:    1. Minimize the number of Bob dependencies to a bare minimum.  Use h5py to read data, RNGs and linear algebra from numpy.  Remove requirements for bob.core, bob.io.base, bob.blitz, and most of bob.math (except for the PAVX algorithm)  2. Remove deprecated dependencies (docopt)  3. Use pytest for tests instead of nose  4. Re-implement the old C++ as a Python-only library (remove all C++ dependencies such as Boost)
     - bob/bob.io.image!44 Allow conversion of batched images and videos to matplotlib and back: Also adds `opencvbgr_to_bob` and `bob_to_opencvbgr` convenience functions.
     - bob/bob.io.image!45 Include copy of GifQuantizeBuffer function from giflib: From https://github.com/coin3d/simage/pull/33    giflib 5.1.9 and above removes the GifQuantizeBuffer() function used in  simage_gif.c (see https://sourceforge.net/p/giflib/bugs/132/ and  https://sourceforge.net/p/giflib/bugs/142/).    The fix is either to remove the related check in simage_gif.c or to include a  copy of the function as encouraged by giflib, due the util library not being  included in installed versions of giflib (license is MIT):        The undocumented and deprecated GifQuantizeBuffer() entry point has been      moved to the util library to reduce libgif size and attack surface.      Applications needing this function are couraged to link the util library or      make their own copy.
     - bob/bob.io.image!45 Include copy of GifQuantizeBuffer function from giflib: From https://github.com/coin3d/simage/pull/33    giflib 5.1.9 and above removes the GifQuantizeBuffer() function used in  simage_gif.c (see https://sourceforge.net/p/giflib/bugs/132/ and  https://sourceforge.net/p/giflib/bugs/142/).    The fix is either to remove the related check in simage_gif.c or to include a  copy of the function as encouraged by giflib, due the util library not being  included in installed versions of giflib (license is MIT):        The undocumented and deprecated GifQuantizeBuffer() entry point has been      moved to the util library to reduce libgif size and attack surface.      Applications needing this function are couraged to link the util library or      make their own copy.* bob/bob.db.base
     - bob/bob.db.base!43 [SQLiteBaseDatabase] Make the instances picklable
     - bob/bob.db.base!44 [fix] Handle empty lists in check_parameters.: Closes bob/bob.db.base#27
     - bob/bob.db.base!44 [fix] Handle empty lists in check_parameters.: Closes bob/bob.db.basebob/bob.db.base#27
     - bob/bob.db.base!45 [annotations] Allow reading annotations from inside a tarball     - bob/bob.db.base!45 [annotations] Allow reading annotations from inside a tarball
     - bob/bob.io.video!20 Relaxes distortions limits for 264 codecs: Fixes bob/bob.io.video#17     The videos don't look different but the distortions are high.
     - bob/bob.io.video!20 Relaxes distortions limits for 264 codecs: Fixes bob/bob.io.videobob/bob.io.video#17     The videos don't look different but the distortions are high.* bob/bob.io.audio
     - bob/bob.io.audio!8 Removed dependency with bob.io.matlab: Closes bob/bob.io.audio#7
     - bob/bob.io.audio!8 Removed dependency with bob.io.matlab: Closes bob/bob.io.audiobob/bob.io.audio#7* bob/bob.sp
     - bob/bob.learn.linear!14 Removed traces of bob.io.matlab: Merging this one. Feel free to open an issue in case of problems
     - bob/bob.learn.linear!14 Removed traces of bob.io.matlab: Merging this one. Feel free to open an issue in case of problems* bob/bob.learn.libsvm
     - bob/bob.learn.em!39 Pickling Objects: - [ ] KMeans  - [ ] JFA  - [ ] PLDA
     - bob/bob.learn.em!38 Breaking Change: Making key objects picklable: As mentioned in https://gitlab.idiap.ch/bob/bob.bio.face/-/merge_requests/64 the small hacks made in bob.bio.base to handle pickled stuff is giving more headaches than it should.    So, I decided to make some key objects picklable.
     - bob/bob.learn.em!38 Breaking Change: Making key objects picklable: As mentioned in https://gitlab.idiap.ch/bob/bob.bio.face/-/merge_requests/64 the small hacks made in bob.bio.base to handle pickled stuff is giving more headaches than it should.    So, I decided to make some key objects picklable.* bob/bob.db.mnist
     - bob/bob.db.atnt!12 http public links work in the CI     - bob/bob.db.atnt!12 http public links work in the CI
     - bob/bob.ip.facedetect!11 Moving bob.learn.boosting to this package: ... also removing traces of bob.ip.draw    Closes bob/bob.ip.facedetect#8     Closes https://gitlab.idiap.ch/bob/bob.learn.boosting/-/issues/3
     - bob/bob.ip.facedetect!12 [conda] Added bob.ip.flandmark artifacts: Closes bob/bob.ip.facedetect#9
     - bob/bob.ip.facedetect!13 Port mtcnn to Tensorflow 2: Using the TF1 model.pb
     - bob/bob.ip.facedetect!11 Moving bob.learn.boosting to this package: ... also removing traces of bob.ip.draw    Closes bob/bob.ip.facedetectbob/bob.ip.facedetect#8     Closes https://gitlab.idiap.ch/bob/bob.learn.boosting/-/issues/3
     - bob/bob.ip.facedetect!12 [conda] Added bob.ip.flandmark artifacts: Closes bob/bob.ip.facedetectbob/bob.ip.facedetect#9
     - bob/bob.ip.facedetect!14 Removed trace of evaluate.py     - bob/bob.ip.facedetect!14 Removed trace of evaluate.py
     - bob/bob.ip.qualitymeasure!15 Removed traces of bob.ip.draw: Merging this one.  Feel free to open an issue in case of problems
     - bob/bob.ip.qualitymeasure!15 Removed traces of bob.ip.draw* bob/bob.pipelines
     - bob/bob.pipelines!37 Revert "For some reason, the class information is not passed in the sample wrapper": This reverts merge request bob/bob.pipelines!36
     - bob/bob.pipelines!38 [sge] In dask some sublacessd classes need a config name. Fixes bob/bob.pipelines#20
     - bob/bob.pipelines!40 Add dask-client configurations as resources: Fixes bob/bob.pipelines#19    Removes the sge-demanding configuration as all nodes at Idiap have a fast connection now.    Depends on bob.bio.base!201
     - bob/bob.pipelines!39 [dask][sge] Added the variables `idle_timeout` and `allowed_failures` as: part of our `.bobrc` and added better defaults
     - bob/bob.pipelines!41 Added a GPU queue that defaults to short_gpu
     - bob/bob.pipelines!43 Allow setting specific attributes of sample: Specify the sample attribute to assign the output of an estimator to, instead of 'data' in SampleWrapper.  Specify the attribute of sample to save and load in CheckpointWrapper.
     - bob/bob.pipelines!44 Fix sphinx warnings
     - bob/bob.pipelines!45 Multiple Changes: *  When checkpointing, checkpoing all steps in a pipeline  *  Better names in dask graph for FunctionTransformer  *  [xarray] Allow for multi argument transformers  *  SampleBatch in public API
     - bob/bob.pipelines!46 move vstack_features to bob.io.base
     - bob/bob.pipelines!48 Improvements on CheckpointWrapper: Added the optional argument `hash_fn` in the `CheckpointWrapper` class.  Once this is set, `sample.key` generates a hash code and this hash code is used to compose the final path where `sample` will be checkpointed.    This is optional and generic enough for our purposes.  This hash function can be shipped in the database interface.    Closes https://gitlab.idiap.ch/bob/bob.pipelines/-/issues/25
     - bob/bob.pipelines!47 Multiple changes: *  [DelayedSample] Allow for arbitrary delayed attributes  *  [SampleBatch] Allow other attributes than data    Fixes bob/bob.pipelines#26 bob/bob.pipelines#24
     - bob/bob.pipelines!49 [DelayedSample] Fix issues when an attribute was set
     - bob/bob.pipelines!50 [DelayedSample(Set)] make load and delayed_attributes private: This removes the need for a lot of guessing in downstream packages as they can start removing all keys that start with `_` when access of the sample's attribute is needed.
     - bob/bob.pipelines!51 [dask][sge] Multiqueue updates: In this merge request I:   - Simplified the way multi-queue is set in our scripts   - Updated our Dask documentation      Example  -------    Setting the `fit` method to run on `q_short_gpu`    ```python  pipeline = mario.wrap(      ["sample", "checkpoint", "dask"],      pipeline,      model_path=model_path,      fit_tag="q_short_gpu",  )    ```      You have to explicitly set the list of resource tags available.  ```python  pipeline.fit_transform(...).compute(      scheduler=dask_client, resources=cluster.get_sge_resources()  ```
     - bob/bob.pipelines!53 Updates: Implemented two updates in this MR    - Removed the random behavior on the hash_string function (i had some problems in large scale tests).  - Implemented the `DelayedSampleSetCached`. I need this behavior to speed-up the score computation.
     - bob/bob.pipelines!52 [CheckpointWrapper] Allow custom save and load functions through estimator tags
     - bob/bob.pipelines!54 Fixed multiqueue: Hi @amohammadi @ydayer     I'm fixing here the issue raised with the multiqueue.  I was wrongly setting all tasks to run in a particular resource restriction.    Now the problem is fixed.    To get it running you have to wrap your pipeline in the same  way as before and fetch the resources like this  ```python  pipeline = bob.pipelines.wrap(      ["sample", "checkpoint", "dask"],      pipeline,      model_path="./",      transform_extra_arguments=(("metadata", "metadata"),),      fit_tag="q_short_gpu",  )  from bob.pipelines.distributed.sge import get_resource_requirements  resources = get_resource_requirements(pipeline)    pipeline.fit_transform(X_as_sample).compute(      scheduler=client, resources=resources  )    ```
     - bob/bob.pipelines!56 Two new features: - Moved dask_get_partition_size from bob.bio.base to bob.pipelines  - Updated the target duration of a task to 10s. Being very aggressive in scale-up
     - bob/bob.pipelines!58 Moved the CSVBaseSampleLoader from bob.bio.base to bob.pipelines. This is a general function
     - bob/bob.pipelines!38 [sge] In dask some sublacessd classes need a config name. Fixes bob/bob.pipelinesbob/bob.pipelines#20
     - bob/bob.pipelines!40 Add dask-client configurations as resources: Fixes bob/bob.pipelinesbob/bob.pipelines#19    Removes the sge-demanding configuration as all nodes at Idiap have a fast connection now.    Depends on bob.bio.basebob/bob.pipelines!201
     - bob/bob.pipelines!47 Multiple changes: *  [DelayedSample] Allow for arbitrary delayed attributes  *  [SampleBatch] Allow other attributes than data    Fixes bob/bob.pipelinesbob/bob.pipelines#26 bob/bob.pipelinesbob/bob.pipelines#24
     - bob/bob.pipelines!55 Moved VALID_DASK_CLIENT_STRINGS to bob.pipelines
     - bob/bob.pipelines!59 Dask client names
     - bob/bob.pipelines!60 CSVSampleLoaders as transformers: Made CSVSampleLoaders as scikit-learn transformers    This is a good idea indeed.    I made to classes. The `CSVToSampleLoader` converts one line to one sample; and `AnnotationsLoader` that aggregates from `CSVToSampleLoader` to read annotations using `bob.db.base.read_anno...`.  This is delayed.    I'm already porting this stuff on `bob.bio.base`. Code is way more cleaner.    ping @amohammadi @ydayer      Closes bob/bob.pipelinesbob/bob.pipelines#30
     - bob/bob.pipelines!61 Fixed modules: config files from here are not available once `conda install bob.pipelines`
     - bob/bob.pipelines!62 Implement a new simple generic csv-based database interface: Depends on https://gitlab.idiap.ch/bob/bob.extension/-/merge_requests/126     - bob/bob.pipelines!55 Moved VALID_DASK_CLIENT_STRINGS to bob.pipelines
     - bob/bob.pipelines!60 CSVSampleLoaders as transformers: Made CSVSampleLoaders as scikit-learn transformers    This is a good idea indeed.    I made to classes. The `CSVToSampleLoader` converts one line to one sample; and `AnnotationsLoader` that aggregates from `CSVToSampleLoader` to read annotations using `bob.db.base.read_anno...`.  This is delayed.    I'm already porting this stuff on `bob.bio.base`. Code is way more cleaner.    ping @amohammadi @ydayer      Closes bob/bob.pipelines#30
     - bob/bob.pipelines!62 Implement a new simple generic csv-based database interface: Depends on https://gitlab.idiap.ch/bob/bob.extension/-/merge_requests/126
     - bob/bob.learn.tensorflow!86 MINE: Mutual Information Neural Estimation: Implemented https://arxiv.org/pdf/1801.04062.pdf
     - bob/bob.learn.tensorflow!85 Porting to TF2: Fixes bob/bob.learn.tensorflow#75
     - bob/bob.learn.tensorflow!88 Porting inception resnet v1: This can be used with the facenet models too
     - bob/bob.learn.tensorflow!89 Cleanup: This package needs some clean up
     - bob/bob.learn.tensorflow!91 Fixed issues with TF1 compatibility: Fixed compatibility issues with TF1 and TF2 models    thanks @lcolbois
     - bob/bob.learn.tensorflow!85 Porting to TF2: Fixes bob/bob.learn.tensorflowbob/bob.learn.tensorflow#75
     - bob/bob.learn.tensorflow!92 Add necessary files to train DeepPixBis
     - bob/bob.learn.tensorflow!93 ArcFace/SphereFace Loss: Implemented 2 losses and 1 layer from the arcface paper: https://arxiv.org/abs/1801.07698    Implemented 1 loss and 1 layer from the sphere face paper: https://arxiv.org/pdf/1704.08063.pdf    Implemented a keras model that does the validation in the test step     - bob/bob.learn.tensorflow!92 Add necessary files to train DeepPixBis
     - bob/bob.learn.tensorflow!93 ArcFace/SphereFace Loss: Implemented 2 losses and 1 layer from the arcface paper: https://arxiv.org/abs/1801.07698    Implemented 1 loss and 1 layer from the sphere face paper: https://arxiv.org/pdf/1704.08063.pdf    Implemented a keras model that does the validation in the test step
     - bob/bob.bio.base!200 Database interface: Implemented a simple filelist database interface for the VanillaBiometrics based on CSVFiles.    The `CSVDatasetDevEval` needs to have the following format:    ```text         my_dataset/         my_dataset/my_protocol/         my_dataset/my_protocol/train.csv         my_dataset/my_protocol/train.csv/dev_enroll.csv         my_dataset/my_protocol/train.csv/dev_probe.csv         my_dataset/my_protocol/train.csv/eval_enroll.csv         my_dataset/my_protocol/train.csv/eval_probe.csv         ...  ```    where each CSV file needs to have the following format:    ```text         PATH,SUBJECT         path_1,subject_1         path_2,subject_2         path_i,subject_j  ```    This formart allows the usage of metadata by following the pattern below:  ```text         PATH,SUBJECT,METADATA_1,METADATA_2,METADATA_k         path_1,subject_1,A,B,C         path_2,subject_2,A,B,1         path_i,subject_j,2,3,4  ```    We can imagine other implementations of this.  For instance, `CSVDatasetCrossValidation` that given a csv file, it splits "on-the-fly" several data for enrolling, probing and training.  Or `CSVDatasetWithEyesAnnotation`, that handles annotations for Face Rec pipelines.    I still need to implement a mechanism that takes `zip` files as input to `CSVDatasetDevEval`.  That way we can ship databases as simple zip files    ping @ydayer @amohammadi     I'll merge this tomorrow.  I need this to support the efforts on `bob.bio.vein`.
     - bob/bob.bio.base!201 Correct the chain loading and click implementation: The vanilla biometrics script was not using our click API correctly,  this fixes that. Please refer to bob.extension's docs for more info.
     - bob/bob.bio.base!194 New documentation bob.bio.base: Redoing bob.bio.base documentation    We might rebrand this package to `scikit-biometrics`
     - bob/bob.bio.base!203 Fix Dask shutting down automatically when calling from a script: Defers the shutdown task to the user by putting the execution of Vanilla-Biometrics in its own function.  The `bob bio pipelines vanilla-biometrics` command calls this function.
     - bob/bob.bio.base!180 [dask] Preparing bob.bio.base for dask pipelines: **THIS IS A WIP**    Working out how `bob.bio.base` should be with dask pipelines.  Some things to be done in the MR is.    - [x] Remove all traces of `verify.py`. Pipelines needs to be defined with dask. This includes the removal of: `FileSelector`, `Grid` configurations, `./script/grid_search`, `./script/enroll`, `./script/extract`, `./script/preprocess`, `./baselines`  - [x] Move the biometrics pipeline from `bob.pipelines` to here  - [ ] Document the package on: 1-) How to use the resources available (databases, algorithms, preprocessors, etc..), 2-) How to create new databases, 3-) How to create new pipelines (this should be in bob.pipelines)   - [x] Rethink the file list interface  - [ ] Review which dependencies to keep. For instance, `bob.learn.linear`, `bob.learn.em`, `bob.learn.activation`, `bob.math` (consider use scikit-learn).  - [x] Implement missing pipelines (for instance, score normalization/calibration pipeline)  - [x] Database paths patched in `~/.bobrc`  - [ ] Redesign the resource.py as CLI commands
     - bob/bob.bio.base!204 Remove stackable preprocessors and extractors: Now that we have standardized on scikit-learn transformers,  scikit-learn alternatives such as FunctionTransformer, Pipeline,  and FeatureUnion should be used
     - bob/bob.bio.base!202 Dask annotators
     - bob/bob.bio.base!205 Fix annotators kwargs: The kwargs were not passed correctly in the FailSafe annotator.    Removed the shutdown of dask clients in the `annotate` commands.
     - bob/bob.bio.base!207 Fixed [typo]
     - bob/bob.bio.base!206 Created decorators: Added two decorators that checks `mxnet` and `tensorflow`
     - bob/bob.bio.base!208 Fixing sphinx warnings: It's a WIP.    I have 34 warnings to be fixed so far.    ping @ydayer @amohammadi
     - bob/bob.bio.base!209 Putting annotator back to the documentation
     - bob/bob.bio.base!210 [annotators][FailSafe] improvements
     - bob/bob.bio.base!212 Improvements on CheckpointWrapper: Related to https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/48
     - bob/bob.bio.base!214 Optimize scoring if `allow_scoring_with_all_biometric_references == False`: ..by caching biometric references.  This speeds things up by a lot
     - bob/bob.bio.base!211 [legacy databases] delay loading of annotations: Depends on bob.pipelines!47
     - bob/bob.bio.base!213 [VanillaBiometrics] Make some dask arguments visible in the CLI commands: Sometimes our heuristics that defines `dask.bag.partition_size` and `n_workers` is not enough to cover all range of problems.  In this MR I'm adding the options `--dask-partition-size` and `--dask-n-workers` in the `vanilla-biometrics` CLI command, so that users can explicitly set, respectively, the dask bag partition size and the number of workers to start an experiment without having to play around with the API.
     - bob/bob.bio.base!216 [click][VanillaBiometrics] Fixed click options: Closes bob/bob.bio.base#148
     - bob/bob.bio.base!218 [test_utils] Do not re-download ATNT database: if the data is already inside the low-level db package.  Fixes bob/bob.bio.base#127
     - bob/bob.bio.base!219 Remove fiddling with bob.pipelines.Samples's internals
     - bob/bob.bio.base!217 Add a method to retrieve all the samples of a dataset: Added a functionality to get all the samples at once.    For legacy databases, it uses the all_files method.    Closes bob/bob.bio.base#146
     - bob/bob.bio.base!220 [scripts][pipelines] allow dask-clients as strings: Some strings like single-threaded, processes, ... are now allowed as  options for the dask-client click option.    Depends on bob.extension!122
     - bob/bob.bio.base!221 Added the memory_demanding attribute as part of the legacy APi
     - bob/bob.bio.base!222 Follow-up to "Add a method to retrieve all the samples of a dataset": Closes bob/bob.bio.base#149    The `all_samples` methods now correctly handles the 'train' group, returning only the samples of a group if that group is given as parameter.    The 'groups' parameters contents are now verified with `bob.db.base.check_parameters_for_validity`.    Groups of 'legacy database interfaces' different from 'train', 'dev', or 'eval' are now handled correctly.
     - bob/bob.bio.base!223 Used the DelayedSampleSetCached in the BioAlgorithmCheckpointWrapper: Depends on https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/53
     - bob/bob.bio.base!225 [test.utils] Handle ModuleNotFoundError when bob.db.atnt is not available
     - bob/bob.bio.base!226 [legacy] Updated subject to reference_id in the lecacy DB interface
     - bob/bob.bio.base!227 Make compare samples work with DelayedSample instead of Sample: this allow us to use `bob.bio.rankone`
     - bob/bob.bio.base!230 [dask] Moved dask_get_partition_size from bob.bio.base to bob.pipelines
     - bob/bob.bio.base!229 Check stateless: Small change where background model samples are not loaded if the main pipeline is stateless. The world set typically is much larger than the dev and eval set, so it is a bit of a waste to load.    NB. The make_pipeline utility from scikit-learn generates pipelines that are never stateless, even if all transformers in the sequence are stateless, which makes this change useful.
     - bob/bob.bio.base!228 moved VALID_DASK_CLIENT_STRINGS to bob.pipelines
     - bob/bob.bio.base!224 Adapting CSVDevEval to work with our current FileList Structure: This is a WIP, which means Work in Progress.    In this MR I structured the files from the proposed CSV interface to be the same as in the current FileList we have.  Furthermore, the current `.lst` files can be used in the `CSVDatasetDevEval` with some of the bells and whistles the current LST files has (e.g usage of external annotations).    The goal is of the MR is to:   1. Be able to use the current file lists we have   2. Allow the extra functionalities we currently have with the CSVs.
     - bob/bob.bio.base!231 Renamed CSVDatasetDevEval to CSVDataset: .... it's easier
     - bob/bob.bio.base!232 Move code
     - bob/bob.bio.base!211 [legacy databases] delay loading of annotations: Depends on bob.pipelinesbob/bob.bio.base!47
     - bob/bob.bio.base!216 [click][VanillaBiometrics] Fixed click options: Closes bob/bob.bio.basebob/bob.bio.base#148
     - bob/bob.bio.base!218 [test_utils] Do not re-download ATNT database: if the data is already inside the low-level db package.  Fixes bob/bob.bio.basebob/bob.bio.base#127
     - bob/bob.bio.base!217 Add a method to retrieve all the samples of a dataset: Added a functionality to get all the samples at once.    For legacy databases, it uses the all_files method.    Closes bob/bob.bio.basebob/bob.bio.base#146
     - bob/bob.bio.base!220 [scripts][pipelines] allow dask-clients as strings: Some strings like single-threaded, processes, ... are now allowed as  options for the dask-client click option.    Depends on bob.extensionbob/bob.bio.base!122
     - bob/bob.bio.base!222 Follow-up to "Add a method to retrieve all the samples of a dataset": Closes bob/bob.bio.basebob/bob.bio.base#149    The `all_samples` methods now correctly handles the 'train' group, returning only the samples of a group if that group is given as parameter.    The 'groups' parameters contents are now verified with `bob.db.base.check_parameters_for_validity`.    Groups of 'legacy database interfaces' different from 'train', 'dev', or 'eval' are now handled correctly.
     - bob/bob.bio.base!233 CSVSampleLoaders as transformers: Depends on https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/60    ping @amohammadi @ydayer
     - bob/bob.bio.base!234 Fixed issues in the ScoreWriter
     - bob/bob.bio.base!235 [ztnorm][scorewriter] Updates: - Used DelayedSampleSetCached in the ZTNormCheckpointWrapper  - Cleaning up the CSVScoreWriter
     - bob/bob.bio.base!236 [Fix] Correction of scores path: This fixes an issue introduced in [this commit](https://gitlab.idiap.ch/bob/bob.bio.base/-/commit/cb83794da843358b822b8d75abfdecc1db7f770cbob/bob.bio.base#ffca3ee0885ed8d9ceff5bb4100ee9f6413bc6a5) where the path to the output scores file was wrong for non `.csv` score file format.    ping @tiago.pereira
     - bob/bob.bio.base!238 Created the function `get_temp_directory`: ... this is useful for legacy algorithms    Related to https://gitlab.idiap.ch/bob/bob.bio.face/-/issues/39
     - bob/bob.bio.base!239 Raised a warning when data is not available to process vanilla-biometrics
     - bob/bob.bio.base!240 Doc cleanup: Did a pass in Grammaly and corrected the errors.
     - bob/bob.bio.base!241 Legacy database wrapper was not supporting this properly supporting the use case where `model_id<>client_id`: Making the bob.bio.base database interface support samplesets where the **reference_id** is different from **subject_id**.   There are some databases protocols that needs such support.  Furthermore the legacy database interface was not supporting this properly    ping @mguenther
     - bob/bob.bio.base!242 [bob.piplines] sample loaders have moved in bob.pipelines: Depends on bob.pipelinesbob/bob.bio.base!62
     - bob/bob.bio.base!243 Add the vulnerability analysis commands: Moved from bob.pad.base
     - bob/bob.bio.base!245 Score checkpoint are more robust: In this MR I'm changing the way we checkpoint scores.  I moved from the joblib pickling to pickle and compress with gzip myself.    This seems more robust with our file system.     - bob/bob.bio.base!233 CSVSampleLoaders as transformers: Depends on https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/60    ping @amohammadi @ydayer
     - bob/bob.bio.base!236 [Fix] Correction of scores path: This fixes an issue introduced in [this commit](https://gitlab.idiap.ch/bob/bob.bio.base/-/commit/cb83794da843358b822b8d75abfdecc1db7f770c#ffca3ee0885ed8d9ceff5bb4100ee9f6413bc6a5) where the path to the output scores file was wrong for non `.csv` score file format.    ping @tiago.pereira
     - bob/bob.bio.base!242 [bob.piplines] sample loaders have moved in bob.pipelines: Depends on bob.pipelines!62
     - bob/bob.bio.base!245 Score checkpoint are more robust: In this MR I'm changing the way we checkpoint scores.  I moved from the joblib pickling to pickle and compress with gzip myself.    This seems more robust with our file system.
     - bob/bob.bio.gmm!24 Dask pipelines
     - bob/bob.bio.gmm!24 Dask pipelines* bob/bob.bio.face
     - bob/bob.bio.face!69 Make all the transformers work with lists by default
     - bob/bob.bio.face!47 Accept an annotator in FaceCrop: related to bob/bob.bio.face#26
     - bob/bob.bio.face!70 Renamed the resources to use dash instead of under score: Renamed the resources to use dash instead of under score    what we've discussed yesterday @ydayer
     - bob/bob.bio.face!72 Fix transformers naming in tests: Fixed some tests that failed after bob/bob.bio.face!70.  Added temporary models `data` folder in .gitignore.
     - bob/bob.bio.face!71 Face crop improvements
     - bob/bob.bio.face!64 Dask pipelines
     - bob/bob.bio.face!75 [Preprocessor] Fix color channel: This fixes bugs introduced by commit https://gitlab.idiap.ch/bob/bob.bio.face/-/commit/38e6503f5758095415edc84630750d294fdaf962.  - Removing ambiguity in Base between `color_channel`attribute and `color_channel` method (-> changed to `change_color_channel` method)  - Patching tests to send a list of arrays to Base.transform, not an array (as required)
     - bob/bob.bio.face!76 Fixed Scaler and preprocessors that use annotations
     - bob/bob.bio.face!77 TF2 pipelines: Hi @lcolbois,    I'm opening an MR with your port.  Let's do it one by one.    Thanks for starting this work
     - bob/bob.bio.face!78 Remove bob.db.msumfsd_mod
     - bob/bob.bio.face!79 Arcface from InsightFace: Hey @amohammadi,    Shall I add `mxnet` in `bob-devel`?    Thanks
     - bob/bob.bio.face!73 Adaptation of Annotator to Tranformer: Removes facedetect and facelandmark.    MTCNN follows the Transformer interface.  The implementation of MTCNN `annotations()` (in [bob.ip.tensoflow_extractor](https://gitlab.idiap.ch/bob/bob.ip.tensorflow_extractor/-/blob/master/bob/ip/tensorflow_extractor/MTCNN.py#L83)) must be changed to accept a batch of images.
     - bob/bob.bio.face!82 Cleanup: Making builds green again
     - bob/bob.bio.face!83 `memory_demanding` for TF based transformers: There are some situations where you have `SampleSets` with more than 1000 samples and we can't just TF `forward` them in one shot without OOM.  In this MR I introduce the argument `memory_demanding`, where, once it's set to true, will `forward one sample at a time.
     - bob/bob.bio.face!80 Port display-face-annotations script: Created a click command for the display-face-annotations script and fixed tests.
     - bob/bob.bio.face!84 Test Facenet: This completes the testing of FaceNet by comparing with the embeddings generated in the TF1 version.  **N.B : Currently the test does NOT succeed bob/bob.bio.face!**
     - bob/bob.bio.face!81 Sample Loaders able to handle certain type of annotations...: .... and the implementation of the MEDS dataset as file lists
     - bob/bob.bio.face!86 Some fixes to allow vulnerability tests with replay-attack and replay-mobile database
     - bob/bob.bio.face!87 Renamed CSVDatasetDevEval to CSVDataset
     - bob/bob.bio.face!47 Accept an annotator in FaceCrop: related to bob/bob.bio.facebob/bob.bio.face#26
     - bob/bob.bio.face!73 Adaptation of Annotator to Tranformer: Removes facedetect and facelandmark.    MTCNN follows the Transformer interface.  The implementation of MTCNN `annotations()` (in [bob.ip.tensoflow_extractor](https://gitlab.idiap.ch/bob/bob.ip.tensorflow_extractor/-/blob/master/bob/ip/tensorflow_extractor/MTCNN.pybob/bob.bio.face#L83)) must be changed to accept a batch of images.
     - bob/bob.bio.face!89 Updates: - Added MORPH dataset  - Proper documented the MEDS database  - Fixed eyes annotation issue
     - bob/bob.bio.face!88 SampleLoader as transformer: Depends on https://gitlab.idiap.ch/bob/bob.pipelines/-/merge_requests/60    ping @ydayer @amohammadi
     - bob/bob.bio.face!90 Updated Capeal database
     - bob/bob.bio.face!92 MultiFaceCrop: Test and describe the MultiFaceCrop preprocessor  Integrates it into the baselines    Now can run instructions such as `bob bio pipelines vanilla-biometrics multipie-pose inception-resnetv2-msceleb` and have the MultiFaceCrop automatically selected.
     - bob/bob.bio.face!93 Small fixes the database tests so it flawlessly passes in [mac] builds
     - bob/bob.bio.face!94 Fixing the temp directory path for legacy algorithms IF checkpointing is off: Hi @lcolbois,    Thanks for detecting the problem and opening the issue with all the details.    The problem is that most of the legacy algorithms are not pickled serializable, therefore, we can't wrap their transformations as :any:`bob.pipelines.Sample` and use pipelines in memory (`vanilla-biometrics` without the `--checkpoint` option).  Hence, we need to make them automatically "checkpointable" and use their legacy functions to persist data into the disk in a temporary directory.    This was wrongly set in the `lgbphs` baseline.  The `/tmp/` was being set by default and this is not valid while running stuff on cluster.  Basically, you are persisting data on the `/tmp` of every node and obviously, this directory is not visible in other nodes.  That's why things work locally and on the grid with checkpoints.    In this MR https://gitlab.idiap.ch/bob/bob.bio.base/-/merge_requests/238 I formalized a function that generates proper temporary directories that are visible on the cluster.        closes bob/bob.bio.facebob/bob.bio.face#39
     - bob/bob.bio.face!91 Porting databases to the CSV interface: Hey,    I'm starting from Mobio and will probably do MultiPIE (ping @lcolbois) tonight.    Shall we deprecate the corresponding `bob.dbs`?   I would keep them around for a while...
     - bob/bob.bio.face!95 Multipie Leaderboard: This introduces  + Documentation on Multipie  + Leaderboard with baseline results  + A script for specifically plotting HTER in function of camera angle, with results
     - bob/bob.bio.face!97 [Fix] Legacy helpers with fixed annotations: This fixes the helpers used for defining legacy baselines, when using fixed annotations.  Beforehand, the instantiated cropper was still expecting to receive an "annotation" metadata, which broke the execution.    I fixed a similar issue a while ago for Tensorflow baselines but missed to also fix legacy baselines.  ping @tiago.pereira
     - bob/bob.bio.face!99 [Fix][FaceCrop] Feed annotator with list of images: This fixes https://gitlab.idiap.ch/bob/bob.bio.face/-/issues/40.
     - bob/bob.bio.face!100 Resolve "EyesAnnotations ordering of x & y coordinates is wrong": Closes bob/bob.bio.facebob/bob.bio.face#41
     - bob/bob.bio.face!96 TF2 training: Wrapping up an old script that trains some angular FR models (ArcFace, SphereFace, Modsoftmax) using different backbones
     - bob/bob.bio.face!101 Changed 112x112 crop: Changed the 112x112.    In that way, the `arcface-insightface` gives ~0% on the Mobio dataset    ```  [Min. criterion: EER ] Threshold on Development set `/idiap/temp/tpereira/temp/insight-face/scores-dev`: -2.459333e+01  =====================  ================  =================  ..                     Development       Evaluation  =====================  ================  =================  Failure to Acquire     0.00%             0.00%  False Match Rate       0.03% (20/57960)  0.04% (65/147630)  False Non Match Rate   0.04% (1/2520)    0.00% (0/3990)  False Accept Rate      0.03%             0.04%  False Reject Rate      0.04%             0.00%  Half Total Error Rate  0.04%             0.02%  =====================  ================  =================  ```
     - bob/bob.bio.face!102 New baselines: Included two new baselines trained with ArcFace head  These two were trained by myself using MSCeleb.    `resnet50-msceleb-arcface-2021`  ```  =====================  ================  =================  ..                     Development       Evaluation  =====================  ================  =================  Failure to Acquire     0.0%              0.0%  False Match Rate       0.5% (299/57960)  0.4% (531/147630)  False Non Match Rate   0.5% (13/2520)    1.5% (59/3990)  False Accept Rate      0.5%              0.4%  False Reject Rate      0.5%              1.5%  Half Total Error Rate  0.5%              0.9%  =====================  ================  =================  ```    mobilenetv2-msceleb-arcface-2021  ```  [Min. criterion: EER ] Threshold on Development set `/idiap/temp/tpereira/temp/mobilenetv2-mobio-male/scores-dev`: -9.859437e-01  =====================  ================  ==================  ..                     Development       Evaluation  =====================  ================  ==================  Failure to Acquire     0.0%              0.0%  False Match Rate       1.1% (621/57960)  1.0% (1534/147630)  False Non Match Rate   1.1% (27/2520)    2.5% (100/3990)  False Accept Rate      1.1%              1.0%  False Reject Rate      1.1%              2.5%  Half Total Error Rate  1.1%              1.8%  =====================  ================  ==================  ```    To use it do in a FR experiment do:    `bob bio pipelines vanilla-biometrics <database> resnet50-msceleb-arcface-2021 ....`
     - bob/bob.bio.face!103 Fix LFW database: - Ported some databases to the the CSVSampleLoaderBiometrics  - Set the flag `allow_scoring_with_all_biometric_references=False` in the LFW database entry-points.    ping @mguenther
     - bob/bob.bio.face!104 Resolve "ArcFace (MXNET) + IJBC lead to memory error": Closes bob/bob.bio.facebob/bob.bio.face#42
     - bob/bob.bio.face!105 New baseline: New baseline.    Resnet50 (the one from keras) trained with VGG2 and arcface loss.    Resource  `resnet50-vgg2-arcface-2021`
     - bob/bob.bio.face!107 [tests] Mark tensorflow tests as slow: related to bob/bob.bio.facebob/bob.bio.face#43
     - bob/bob.bio.face!108 Replace nose with pytest: Related to bob/bobbob/bob.bio.face#267
     - bob/bob.bio.face!109 Remove duplicated tests: Improved the `test_baseline` tests to be tight with the score values, so we indirectly test the embeddings.  Hence, we pretty much have the same coverage.    The coverage decreased 1% due to some resources that are not activated, but it's fine.    ~26 minutes and 17 GB ram usage much improved compared to 56 min and 28 GB RAM usage
     - bob/bob.bio.face!110 Casia Africa Database: Created Casia-Africa database interface.    - [x] Documentation  - [x] Tests  - [x] Resources
     - bob/bob.bio.face!111 Integrating HTFace databases: Hi,    I quickly integrated some HTFace databases.    Here I ported the polathermal (VIS-Thermal) dataset and CBSR dataset (VIS-NIR)    ping @amohammadi @ageorge @sebastien.marcel     - bob/bob.bio.face!89 Updates: - Added MORPH dataset  - Proper documented the MEDS database  - Fixed eyes annotation issue
     - bob/bob.bio.face!94 Fixing the temp directory path for legacy algorithms IF checkpointing is off: Hi @lcolbois,    Thanks for detecting the problem and opening the issue with all the details.    The problem is that most of the legacy algorithms are not pickled serializable, therefore, we can't wrap their transformations as :any:`bob.pipelines.Sample` and use pipelines in memory (`vanilla-biometrics` without the `--checkpoint` option).  Hence, we need to make them automatically "checkpointable" and use their legacy functions to persist data into the disk in a temporary directory.    This was wrongly set in the `lgbphs` baseline.  The `/tmp/` was being set by default and this is not valid while running stuff on cluster.  Basically, you are persisting data on the `/tmp` of every node and obviously, this directory is not visible in other nodes.  That's why things work locally and on the grid with checkpoints.    In this MR https://gitlab.idiap.ch/bob/bob.bio.base/-/merge_requests/238 I formalized a function that generates proper temporary directories that are visible on the cluster.        closes bob/bob.bio.face#39
     - bob/bob.bio.face!100 Resolve "EyesAnnotations ordering of x & y coordinates is wrong": Closes bob/bob.bio.face#41
     - bob/bob.bio.face!104 Resolve "ArcFace (MXNET) + IJBC lead to memory error": Closes bob/bob.bio.face#42
     - bob/bob.bio.face!107 [tests] Mark tensorflow tests as slow: related to bob/bob.bio.face#43
     - bob/bob.bio.face!108 Replace nose with pytest: Related to bob/bob#267
     - bob/bob.bio.face!111 Integrating HTFace databases: Hi,    I quickly integrated some HTFace databases.    Here I ported the polathermal (VIS-Thermal) dataset and CBSR dataset (VIS-NIR)    ping @amohammadi @ageorge @sebastien.marcel
     - bob/bob.bio.face!113 fix for protocol name
     - bob/bob.bio.video!41 Removed traces of bob.ip.flandmark, bob.ip.daraw, bob.learn.boosting
     - bob/bob.bio.video!42 Dask pipelines
     - bob/bob.bio.video!43 API Change: This package used to have wrappers for annotators, preprocessors, extractors, and algorithms.  Here, we're planning to only provide wrappers for:  - Annotators  - Transformers  - BiometricAlgorithm (not yet implemented)    We're also removing FrameContainer and FrameSelector classes in favor of VideoAsArray and VideoLikeContainer,  they are not wildly different, but they read the data efficiently (not loading all the frames into memory) and  they behave like numpy arrays, so they are easier to work with.    The BioVideoFile class is kept but it will return VideoAsArray instead of FrameContainer
     - bob/bob.bio.video!43 API Change: This package used to have wrappers for annotators, preprocessors, extractors, and algorithms.  Here, we're planning to only provide wrappers for:  - Annotators  - Transformers  - BiometricAlgorithm (not yet implemented)    We're also removing FrameContainer and FrameSelector classes in favor of VideoAsArray and VideoLikeContainer,  they are not wildly different, but they read the data efficiently (not loading all the frames into memory) and  they behave like numpy arrays, so they are easier to work with.    The BioVideoFile class is kept but it will return VideoAsArray instead of FrameContainer* bob/bob.bio.vein
     - bob/bob.bio.vein!47 Removed traces of bob.io.matlab: Closes bob/bob.bio.vein#23
     - bob/bob.bio.vein!46 Dask pipelines
     - bob/bob.bio.vein!49 [nosetest] Fixed baselines
     - bob/bob.bio.vein!48 Adding Principal Curvature (PC) algorithm: Hi,    I spent some time to debug the code for Principal Curvature (PC) algorithm, add its configuration file, and make the needed change in `setup.py`. Please take a look at changes to proceed with MR.    Many thanks to @ydayer for all his help to debug the code!
     - bob/bob.bio.vein!47 Removed traces of bob.io.matlab: Closes bob/bob.bio.veinbob/bob.bio.vein#23
     - bob/bob.bio.vein!48 Adding Principal Curvature (PC) algorithm: Hi,    I spent some time to debug the code for Principal Curvature (PC) algorithm, add its configuration file, and make the needed change in `setup.py`. Please take a look at changes to proceed with MR.    Many thanks to @ydayer for all his help to debug the codebob/bob.bio.vein!* bob/bob.pad.base
     - bob/bob.pad.base!73 Merged bob.bio: Moved `bob.bio.base`, `bob.bio.face`, `bob.bio.gmm`, and `bob.bio.video` to this packae.    They were respectively renamed  to: `bob.bio.base_legacy`, `bob.bio.face_legacy`, `bob.bio.gmm_legacy`, `bob.bio.video_legacy`    Missing to port the docs.    Fixes bob/bob.pad.base#34
     - bob/bob.pad.base!74 Added spear as part of the temporary legacy: Addressing this issue https://gitlab.idiap.ch/bob/bob.bio.spear/-/issues
     - bob/bob.pad.base!76 Cleaning up dependencies
     - bob/bob.pad.base!75 WIP: Port to dask pipelines
     - bob/bob.pad.base!77 Remove deprecated code
     - bob/bob.pad.base!80 [vanilla-pad] Improve the dask client option and delayed annotations
     - bob/bob.pad.base!73 Merged bob.bio: Moved `bob.bio.base`, `bob.bio.face`, `bob.bio.gmm`, and `bob.bio.video` to this packae.    They were respectively renamed  to: `bob.bio.base_legacy`, `bob.bio.face_legacy`, `bob.bio.gmm_legacy`, `bob.bio.video_legacy`    Missing to port the docs.    Fixes bob/bob.pad.basebob/bob.pad.base#34
     - bob/bob.pad.base!78 Dask pipelines Improvements: This MR improves the dask pipelines port.
     - bob/bob.pad.base!81 [dask] Make vanilla-pad work properlly with dask: In this MR, I'm.    1. Wrapping the pipeline with DaskWrapper, because it wasn't. I think that's why things were not working for you with SGE @ydayer. `vanilla-pad` was creating SGE jobs AND running everything locally.  2. Added the options "--dask-partition-size" and "--dask-n-workers", so the user has some freedom to set these parameters if the heuristics in place are unpleasant.    I'll merge this one right away because of the workshop.    Feel free to open issues if any
     - bob/bob.pad.base!79 new documentation incl. vanilla-pad: The goal is to update the documentation of bob.pad.base with the latest changes.    ping @amohammadi @tiago.pereira
     - bob/bob.pad.base!82 New database interface for PAD: Hi @amohammadi, @ydayer     Follow the proposition for a new DB interface for PAD.  It follows the same guide lines used in `bob.bio.base`.  Follow below the features implemented:    1. Uses CSV files instead of LSTs; with that, you can ship metadata. However, it uses the same file structure as before, so no stress in porting stuff.  2. The CSVPADDataset can transparently read the current LST files we have (I've created a sample loader that handles that).  3. The CSVPADDataset is able to read either files inside of a file structure or files inside of a tarball.    Follow an example on how to use it, by reading from a file structure and from a tarball    ```python      def run(path):            dataset = CSVPADDataset(path, "protocol1")            bob/bob.pad.basebob/bob.pad.base# Train          assert len(dataset.fit_samples()) == 5          bob/bob.pad.basebob/bob.pad.base# 2 out of 5 are bonafides          assert sum([s.is_bonafide for s in dataset.fit_samples()]) == 2            bob/bob.pad.basebob/bob.pad.base# DEV          assert len(dataset.predict_samples()) == 5          bob/bob.pad.basebob/bob.pad.base# 2 out of 5 are bonafides          assert sum([s.is_bonafide for s in dataset.predict_samples()]) == 2            bob/bob.pad.basebob/bob.pad.base# EVAL          assert len(dataset.predict_samples(group="eval")) == 7          bob/bob.pad.basebob/bob.pad.base# 3 out of 5 are bonafides          assert sum([s.is_bonafide for s in dataset.predict_samples(group="eval")]) == 3        csv_example_dir = os.path.realpath(          bob.io.base.test_utils.datafile(".", __name__, "data/csv_dataset")      )        csv_example_tarball = os.path.realpath(          bob.io.base.test_utils.datafile(".", __name__, "data/csv_dataset.tar.gz")      )        run(csv_example_dir)      run(csv_example_tarball)    ```
     - bob/bob.pad.base!84 Remove vulnerability analysis commands: Moved to bob.bio.base  Fixes bob/bob.pad.basebob/bob.pad.base#27
     - bob/bob.pad.base!85 Allow to specify the pipeline decision function in vanilla_pad script: This changes makes it easier to use the `vanilla_pad` script with classifiers that have different names for their decision functions (eg "predict_proba", "predict", etc...)    This also allows passing "transform" as a decision_function in case the pipeline does not contain a classifier.     - bob/bob.pad.base!78 Dask pipelines Improvements: This MR improves the dask pipelines port.
     - bob/bob.pad.base!82 New database interface for PAD: Hi @amohammadi, @ydayer     Follow the proposition for a new DB interface for PAD.  It follows the same guide lines used in `bob.bio.base`.  Follow below the features implemented:    1. Uses CSV files instead of LSTs; with that, you can ship metadata. However, it uses the same file structure as before, so no stress in porting stuff.  2. The CSVPADDataset can transparently read the current LST files we have (I've created a sample loader that handles that).  3. The CSVPADDataset is able to read either files inside of a file structure or files inside of a tarball.    Follow an example on how to use it, by reading from a file structure and from a tarball    ```python      def run(path):            dataset = CSVPADDataset(path, "protocol1")            bob/bob.pad.base# Train          assert len(dataset.fit_samples()) == 5          bob/bob.pad.base# 2 out of 5 are bonafides          assert sum([s.is_bonafide for s in dataset.fit_samples()]) == 2            bob/bob.pad.base# DEV          assert len(dataset.predict_samples()) == 5          bob/bob.pad.base# 2 out of 5 are bonafides          assert sum([s.is_bonafide for s in dataset.predict_samples()]) == 2            bob/bob.pad.base# EVAL          assert len(dataset.predict_samples(group="eval")) == 7          bob/bob.pad.base# 3 out of 5 are bonafides          assert sum([s.is_bonafide for s in dataset.predict_samples(group="eval")]) == 3        csv_example_dir = os.path.realpath(          bob.io.base.test_utils.datafile(".", __name__, "data/csv_dataset")      )        csv_example_tarball = os.path.realpath(          bob.io.base.test_utils.datafile(".", __name__, "data/csv_dataset.tar.gz")      )        run(csv_example_dir)      run(csv_example_tarball)    ```
     - bob/bob.pad.base!84 Remove vulnerability analysis commands: Moved to bob.bio.base  Fixes bob/bob.pad.base#27
     - bob/bob.pad.base!85 Allow to specify the pipeline decision function in vanilla_pad script: This changes makes it easier to use the `vanilla_pad` script with classifiers that have different names for their decision functions (eg "predict_proba", "predict", etc...)    This also allows passing "transform" as a decision_function in case the pipeline does not contain a classifier.
     - bob/bob.pad.face!106 Merge bob.bio: Moved `bob.bio.base`, `bob.bio.face`, `bob.bio.gmm`, and `bob.bio.video` to this packae.    They were respectively renamed  to: `bob.bio.base_legacy`, `bob.bio.face_legacy`, `bob.bio.gmm_legacy`, `bob.bio.video_legacy`
     - bob/bob.pad.face!107 WIP: Port to dask pipelines
     - bob/bob.pad.face!92 Removes unnecessary logger warning: Removes unnecessary logger warning
     - bob/bob.pad.face!109 Remove bob.db.msumfsd_mod and its aggregated db
     - bob/bob.pad.face!108 Remove deprecated code
     - bob/bob.pad.face!111 cleaning up nonexistant entry-points: I'll merge this one right away.  Fell free to open an issue if any
     - bob/bob.pad.face!110 Porting to dask pipelines
     - bob/bob.pad.face!113 Dask pipelines fixes: Addresses some of the comments in bob.pad.facebob/bob.pad.face!110
     - bob/bob.pad.face!114 Updating to the documentation: Adapted the documentation to the new interface with the `bob pad` commands, instead of the `spoof.py` script.  Fixes bob/bob.pad.facebob/bob.pad.face#39.
     - bob/bob.pad.face!115 Port databases to the new csv format: Depends on bob.pipelinesbob/bob.pad.face!62 and bob.pad.basebob/bob.pad.face!82
     - bob/bob.pad.face!116 Add SWAN and OULU-NPU databases     - bob/bob.pad.face!110 Porting to dask pipelines
     - bob/bob.pad.face!113 Dask pipelines fixes: Addresses some of the comments in bob.pad.face!110
     - bob/bob.pad.face!114 Updating to the documentation: Adapted the documentation to the new interface with the `bob pad` commands, instead of the `spoof.py` script.  Fixes bob/bob.pad.face#39.
     - bob/bob.pad.face!115 Port databases to the new csv format: Depends on bob.pipelines!62 and bob.pad.base!82
     - bob/bob.pad.face!116 Add SWAN and OULU-NPU databases
     - bob/bob.learn.pytorch!44 Merge bob.bio
     - bob/bob.learn.pytorch!46 Removed legacy components: * \[black\] everything  * Renamed `bob.bio.base_legacy` to this `bob.bio.base`  * pinned tensorflow as a Linux dependency     - bob/bob.learn.pytorch!46 Removed legacy components: * \[black\] everything  * Renamed `bob.bio.base_legacy` to this `bob.bio.base`  * pinned tensorflow as a Linux dependency
     - bob/bob.ip.binseg!16 Use pytest instead of nose: This MR moves unit testing to pytest (instead of nose).  It incidentally also enables py36 builds, now that pytorch is available for that Python version.     - bob/bob.ip.binseg!16 Use pytest instead of nose: This MR moves unit testing to pytest (instead of nose).  It incidentally also enables py36 builds, now that pytorch is available for that Python version.
     - bob/bob.devtools!170 Don't update conda first in bootstrap and Allow changing the server in badges
     - bob/bob.devtools!168 Give public packages more priority over private packages
     - bob/bob.devtools!171 Specify the upload channel explicitly: Related to bob/bob.devtoolsbob/bob.devtools#58
     - bob/bob.devtools!172 Fixed TypeError
     - bob/bob.devtools!173 Use format string rather than f string in bootstrap.py: Python 3.6 syntax cannot be used yet    Fixes bob/bob.devtoolsbob/bob.devtools#59
     - bob/bob.devtools!174 Remove ffmpeg and openh264 from our channel in favor of defaults: ffmpeg 4.2 with support of h264 is available in the defaults channel.
     - bob/bob.devtools!176 Do not cache mirrored files
     - bob/bob.devtools!177 Remove x264 packages from our channel index: I already created symbolic links for these packages in the archive channel as well.
     - bob/bob.devtools!179 allow aarch64
     - bob/bob.devtools!178 Use centos:7 to build Bob
     - bob/bob.devtools!180 [conda_build_config] Update bob-devel and its respective pins
     - bob/bob.devtools!181 [conda_build_config] fix zc varaiable names
     - bob/bob.devtools!175 Make pytorch packages to depend on the cpuonly package
     - bob/bob.devtools!182 Bob ip view dependencies: Waiting on [MR](https://gitlab.idiap.ch/bob/conda/-/merge_requests/451) for bob.conda    Dependencies:    - pyopengl  - pyqtgraph  - qtpy  - qdarkstyle
     - bob/bob.devtools!184 Use the new docker image in nightlies as well: Fixes bob/bob.devtoolsbob/bob.devtools#61
     - bob/bob.devtools!185 Add make to the list of dependencies
     - bob/bob.devtools!186 [conda_build_config] Upgrade numpy and mkl to be compatible with the latest bob-devel
     - bob/bob.devtools!187 [nightlies ci] remove redundant variables from builds
     - bob/bob.devtools!183 Add Anaconda compilers during bdt create: This ensures packages installed from source in the environment are ABI  compatible with the conda environment packages
     - bob/bob.devtools!189 [conda-build] Update bob-devel version
     - bob/bob.devtools!188 Add a sphinx command to help with migrating to sphinx 3
     - bob/bob.devtools!190 Add an alternative command to nightlies: This is like nightlies with resume.  You can avoid rebuilding all packages.
     - bob/bob.devtools!192 Slightly bump bob/beat-dev to add pytest support
     - bob/bob.devtools!193 Fix var name in conda_build_config.yaml: `pytest-cov -> pytest_cov` in `conda_build_config.yaml`.
     - bob/bob.devtools!194 Allows extra pip-installable packages to be installed in every newly created environment: This MR brings in a new feature to `bdt create`.  It lets users auto-pip-install packages during a `bdt create` phase.  These packages can be pre-configured inside `~/.bdtrc` to avoid the repeatitive task of enumerating stuff you always install.
     - bob/bob.devtools!195 Rename nosetests specific mark expression to be more...: This MR generalizes nose-test unit attributes so that pytest adopters can also benefit from the same setup.  This should allow a smooth transition from nose to pytest.
     - bob/bob.devtools!196 Fix excessive logging after executing conda functions: This MR fixes repetitive logging that occurs after executing conda functions.  After inspection, conda alters the root logger of the logging module by resetting its level and adding 2 handlers that communicate directly to stderr.  This MR introduces a context manager that undoes this inappropriate change.  To use it, it is sufficient to write calls to the conda API with context manager.
     - bob/bob.devtools!198 Update conda_build_config.yaml to the latest: Small bump to bob/beat-devtools to fix a temporary pytorch==1.7 issue.
     - bob/bob.devtools!197 Removed python 3.6 support
     - bob/bob.devtools!199 [repodata-patches] Add dataclasses and future to Pytorch: Also remove the cpuonly patch from Pytorch packages    related to https://gitlab.idiap.ch/bob/conda/-/issues/79
     - bob/bob.devtools!200 Fixed global recipe: The yaml `single-package` is broken
     - bob/bob.devtools!201 Bumped minimal version of tensorflow: to 2.3.0
     - bob/bob.devtools!202 Add opencv: Add opencv build in bob.conda.    Waiting on [MR](https://gitlab.idiap.ch/bob/conda/-/merge_requests/460) in bob-devel.
     - bob/bob.devtools!204 Prepare CI files for macos+arm integration: This small change prepares our scripts to have M1 (ARM) based macOS builds co-existing with Intel ones.
     - bob/bob.devtools!203 Bumped macos min version to 10.10: Shall we test?    Nightlies are failing for more than a week because of that.
     - bob/bob.devtools!205 Pin smmap to fix an issue with gitdb bad pinning: This is a temporary fix for CI build issues with gitdb (bad pinning of smmap).
     - bob/bob.devtools!206 Temporary fix to issue bob/bob.devtoolsbob/bob.devtools#70     - bob/bob.devtools!170 Don't update conda first in bootstrap and Allow changing the server in badges
     - bob/bob.devtools!171 Specify the upload channel explicitly: Related to bob/bob.devtools#58
     - bob/bob.devtools!173 Use format string rather than f string in bootstrap.py: Python 3.6 syntax cannot be used yet    Fixes bob/bob.devtools#59
     - bob/bob.devtools!184 Use the new docker image in nightlies as well: Fixes bob/bob.devtools#61
     - bob/bob.devtools!206 Temporary fix to issue bob/bob.devtools#70
     - bob/bob.devtools!207 Minor bob/beat-devel bump
     - bob/bob.devtools!208 Add pytables dependency: Add pytables dependency.     Waiting on [related MR in bob/conda](https://gitlab.idiap.ch/bob/conda/-/merge_requests/468)
     - bob/bob.devtools!209 Bump beat/bob-devel: [conda_build_config] Bump bob/beat-devel to fix bob/bob.ip.binseg#5 issues (caused by pytorch bogus deps)
     - bob/bob.devtools!211 Add missing informations for Linux CI runner: - gitlab-runner user setup to use Docker    - beat/beat.editor> missing dependencies
     - bob/bob.devtools!210 bdt dav upload automatically adds checksum to filename on remote: Adds an option (--checksum) to let bdt automatically augment the  filename on the remote with the first 8 digits of its sha256sum.  Fixes bob/bob.devtools#64
     - bob/bob.devtools!212 Ci checks: Enables some sanity checks in the CI:  - Make sure packages have a pyproject.toml file  - Runs pre-commit checks if they exist
     - bob/bob.devtools!213 Fix nightlies commands
     - bob/bob.devtools!214 Remove mac builds from our CI builds \m/
     - bob/bob.devtools!215 Revert "Merge branch 'happily-remove-mac-from-ci-builds' into 'master'": This reverts merge request bob/bob.devtools!214
     - bob/bob.devtools!216 Python 3.9 updates and bump to miniconda installer: * Update to latest miniconda installer (4.9.2)  * Use python=3.9 for bdt installation  * Update python-gitlab and docformatter dependencies  * Update `setup.py` to match dependences in `conda/meta.yaml`  * Fixes bob/bob.devtools#69 after upstream fixes
     - bob/bob!246 [archive] Archiving bob.pad.vein [skip-ci]
     - bob/bob!247 Added make as test req.
     - bob/bob!248 Removed bob.db.mobio and bob.db.multipie: This will probably fix the py38 nightlies    ping @andre.anjos @ydayer
