# Reading a Campaign into *Metrics*

To extract pieces of data from the campaign of experiments you ran and feed it
into *Metrics*, you need to use the *Scalpel* module of *Metrics*.
*Scalpel* stands for *"extraCt dAta of exPeriments from softwarE Logs"*
(*sCAlPEL*).

A campaign is basically read using the following:

```python
from metrics.scalpel import read_campaign
my_campaign, my_configuration = read_campaign('path/to/campaign/file', log_level='WARNING')
```

Currently, two types of files can be given as input to *Scalpel*:

+ a JSON file containing a serialized form of the campaign (when
  you have already loaded your campaign in *Metrics*, and saved
  it for later use), or
+ a YAML file describing how to extract data from the campaign
  you ran.

In the first case, there is almost nothing to do, as the JSON file generated
by *Metrics* already contains all the data needed by *Scalpel* (and the
returned configuration will thus be `None`).
In the second case, the following sections give more details on how to write
a configuration file that describes your campaign (the returned configuration
will be an object representation of this description).

Additionally, as you can see in the example above, a `log_level` parameter may
be specified to the function `read_campaign()`.
This allows to configure the minimum level of the parsing events that should
be logged.
Available levels are, from the lowest to the highest:

+ `'TRACE'`
+ `'DEBUG'`
+ `'INFO'`
+ `'WARNING'` (this is the default level)
+ `'ERROR'`

Parsing events are regularly logged by *Scalpel* to trace the extraction it
performs, mostly for debugging purposes.
For instance, activating a lower logging level may allow to identify why
some data is missing for a particular experiment.
However, it should be noted that *Scalpel* may become particularly verbose
when doing so, which may affect performance.
This is why this feature should only be activated for debugging purposes.

## Metadata of the Campaign

In the YAML file, you first need to give elementary information about the
campaign, such as its name and the date on which it has been run.

```yaml
name: my-awesome-campaign
date: 2020-11-17
```

This information is used to identify your campaign, and is particularly
interesting for the traceability of your experiments.

The YAML file must also contain the experimental setup on which the campaign
took place, as in the following example:

```yaml
setup:
  os: Linux CentOS 7 (x86_64)
  cpu: Intel XEON X5550 (2.66 GHz, 8 MB cache)
  gpu: Nvidia GeForce 256 SDR
  ram: 32GB
  timeout: 1800
  memout: 1024
```

Note that, for the setup description, only `timeout` and `memout` are required.
The other values may be displayed in the reports generated by *Metrics* for
reproducibility purposes.

## Description of the Campaign Files

*Scalpel* is able to parse a wide variety of files that contain the output
of the experiments you ran during your campaign.
All the information describing the source of your campaign must be given
in the `source` section of your YAML configuration file.
The main key of this section is `path`, which lists the file(s) containing
the data to extract.

```yaml
source:
  path:
    - path/to/first/file
    - path/to/second/file
```

This key declares the list of the files (either regular files or directories,
depending on the format of your campaign) that *Scalpel* will parse.
Note that all files must have the same format.

All these files will be parsed sequentially, and their content will be merged
into a single campaign.
If these files represent distinct parts of your campaign (e.g., each file
contains the result of a different experiment-ware), you may be interested
in the extraction of metadata from the name of the file, described
[here](#extracting-data-from-file-names).

If you only have one file containing all the results of your campaign, you may
avoid the use of a list, and simply write the path of the file as the value
for `path`:

```yaml
source:
  path: path/to/single/file
```

In the following subsections, we present what you must add to the `source`
field to configure *Scalpel* for parsing your campaign, depending on its
format.

### Parsing a CSV File

The CSV (Comma-Separated Values) format is often used to store experimental
data.
It is mainly a tabular format, which has an (optional) *header* line giving
the titles of the column.
Each of the remaining lines corresponds to the data collected during an
experiment.

Depending on the variant, columns may be separated by:

+ a comma (`,`), giving the default `csv` format,
+ a semicolon (`;`), giving the `csv2` format, or
+ a tabulation (`\t`), giving the `table` format.

To specify that your campaign is in one of these formats, you need to add the
following to your YAML configuration file:

```yaml
source:
  path: path/to/my/file.csv
  format: csv
```

Actually, the `format` may be omitted in this example, as the extension of 
the file already tells *Scalpel* that the file is in the (classical) `csv`
format.
Similarly, if you specify as `path` the files `path/to/my/file.csv2` or
`path/to/my/file.table`, you may omit the format, as *Scalpel* will infer
that such files use the `csv2` and `table` formats, respectively.

You may also have more "exotic" CSV-like files, which do not use a standard
separator or quote character (by default, `"` is used as quote character).
If this is the case, you may describe them by adding the following keys
in the source section:

```yaml
source:
  quote-char: "%"
  separator: "|"
```

In the example above, the quote character is `%` and the columns are
separated by the character `|`.

Finally, you may have a header for your CSV file, or not.
By default, the first line is considered as a header line, and is used
to identify the values parsed in the other lines as experimental data.
If you do not have a header line, add the following key:

```yaml
source:
  has-header: false
```

In this case, values will be identified by the index of the corresponding
column, as a string (starting from `"0"`).
Note that, in this case, *Scalpel*'s naming convention cannot be followed.
As such, do not forget to specify the mapping of the columns in the text file
to fit *Scalpel*'s needs (see [below](#mapping-data-to-scalpel-s-expectations)
for more details).
You must also do so as long as the name of the columns in your CSV files do not
fit *Scalpel*'s expectations.

### Parsing an "Evaluation" File

If you are interested in analyzing the results of a campaign run with
the so-called "Evaluation" platform (such as, for instance, the
[results of the XCSP'19 competition](http://www.cril.univ-artois.fr/XCSP19/),
we provide a parser to read the "results of individual jobs as text file"
provided by this platform (as the one of the XCSP'19 competition, available
[here](http://www.cril.univ-artois.fr/XCSP19/results/export.php?idev=99)).

To do so, specify the following in your YAML configuration file:

```yaml
source:
  path: path/to/result/file.txt
  format: evaluation
```

As this platform does not use in general the same naming convention as that of
*Scalpel*, do not forget to specify the mapping of the columns in the text file
to fit *Scalpel*'s needs (see [below](#mapping-data-to-scalpel-s-expectations)
for more details).

### Parsing a "Reverse" CSV File

We call a CSV file "reverse" when each line in this file corresponds to an
input, and the columns to the different statistics collected for the
experiment-wares run during the campaign.
Here is an example of such a file:

```csv
xp-ware-a,xp-ware-b,xp-ware-c
0.01,0.02,0.03
```

In this example, we consider a campaign that run three experiment-wares, namely
`xp-ware-a`, `xp-ware-b` and `xp-ware-c`.
Each column is by default interpreted as the CPU time of the corresponding
experiment, as this is the only statistic required for an experiment.
Also, note that no input is specified in this example.
This is tolerated, as each line in such a format maps to exactly one input.
However, we strongly recommend specifying the name of the input file,
especially  because it makes easier the interpretation of the experimental
results and their reproducibility.

A more complete example of a "reverse" CSV file is given below:

```csv
input,xp-ware-a.cpu_time,xp-ware-a.memory,xp-ware-b.cpu_time,xp-ware-b.memory,xp-ware-c.cpu_time,xp-ware-c.memory
input-a,0.01,10,0.02,20,0.03,30
```

Here, we collect more statistics, as we consider both the `cpu_time` and
`memory` needed for an experiment.
Those names are used to identify the corresponding statistics in the
representation of the experiment.
In the example above, the experiment-ware and the statistics identifiers
are separated with a dot (`.`), which is the default.
If you want to specify a different separator, you can specify it in the
YAML configuration as follows (make sure not to use the same separator
as for the columns):

```yaml
source:
  title-separator: "!"
```

To configure how a reverse CSV file is parsed, you can also use the same
properties as those used in classical CSV file (see the previous section),
and specify one of the formats `reverse-csv`, `reverse-csv2` or
`reverse-table` (using the same naming convention as before).

### Parsing Raw Data from a File Hierarchy

If you have gathered the output of your experiment-wares in a directory,
*Scalpel* can explore the file hierarchy rooted at this directory and
extract all relevant data for you.
We support three different kinds of file hierarchies, which are described
below.

Note that, by default, *Scalpel* does **not** follow symlinks when
exploring a file hierarchy.
For each of the configurations below, you may alter this behavior by
adding the following to the `source` section of the YAML file:

```yaml
source:
  follow-symlinks: true
``` 

#### One File per Experiment

In this case, the file hierarchy being explored is supposed to contain
exactly one (regular) file per experiment.
You can configure *Scalpel* to consider such a file hierarchy using the
following description:

```yaml
source:
  path: /path/to/my-experiment-directory
  format: one-file
```

Let us consider an example to illustrate how *Scalpel* extracts data based on
this configuration.
Suppose that the file hierarchy to explore has the following form:

```
my-experiment-directory
    + experiment-a.log
    + experiment-b.log
    ` more-experiments
        + experiment-c.log
        ` experiment-d.log
```

Here, *Scalpel* will recursively explore the whole file hierarchy, and will
parse all regular files, provided that these files are specified in the `data`
section of the YAML configuration file (see the dedicated documentation
[here](#description-of-the-data-to-extract) for more details).
Each file `experiment-a.log`, `experiment-b.log`, `experiment-c.log` and
`experiment-d.log` will be considered as the output of a single experiment.

Note that these files may have common formats (such as JSON, XML or CSV) or
may also be the raw output of the solver.
More details on how to retrieve relevant information from these files
are given [here](#description-of-the-data-to-extract).

#### Multiple Files per Experiment

In this case, the file hierarchy being explored is supposed to contain
a set of (regular) files per experiment.
The name of the files (without their extensions) will be used to identify
each experiment.
You can configure *Scalpel* to consider such a file hierarchy using the
following description:

```yaml
source:
  path: /path/to/my-experiment-directory
  format: multi-files
```

Let us consider an example to illustrate how *Scalpel* extracts data based on
this configuration.
Suppose that the file hierarchy to explore has the following form:

```
my-experiment-directory
    + experiment-a.out
    + experiment-a.err
    + experiment-b.out
    + experiment-b.err
    ` more-experiments
        + experiment-c.out
        + experiment-c.err
        + experiment-d.out
        ` experiment-d.err
```

Here, *Scalpel* will recursively explore the whole file hierarchy, and will
parse all regular files, provided that these files are specified in the `data`
section of the YAML configuration file (see the dedicated documentation
[here](#description-of-the-data-to-extract) for more details).
In this case, the files `experiment-a.out` and `experiment-a.err`, for instance,
will be considered as outputs of the same experiment (they are both named
`experiment-a`).

Note that these files may have common formats (such as JSON, XML or CSV) or
may also be the raw output of the solver.
More details on how to retrieve relevant information from these files
are given [here](#description-of-the-data-to-extract).

#### One Directory per Experiment

In this case, the file hierarchy being explored is supposed to have one 
directory that contain the output files of each experiment.
The name of the files inside this directory may be arbitrary (and even the
same from one experiment to another).
You can configure *Scalpel* to consider such a file hierarchy using the
following description:

```yaml
source:
  path: /path/to/my-experiment-directory
  format: dir
```

Let us consider an example to illustrate how *Scalpel* extracts data based on
this configuration.
Suppose that the file hierarchy to explore has the following form:

```
my-experiment-directory
    + experiment-a
    |   + stdout
    |   + stderr
    + experiment-b
    |   + stdout
    |   + stderr
    ` more-experiments
        + experiment-c
        |   + stdout
        |   + stderr
        ` experiment-d
            + stdout
            + stderr
```

Here, *Scalpel* will recursively explore the whole file hierarchy, and will
consider each directory containing regular files as an experiment.
All the regular files contained in this directory will thus be considered
as outputs of the corresponding experiments, as long as these files are
specified in the `data` section of the YAML configuration file (see the
dedicated documentation [here](#description-of-the-data-to-extract) for more
details).
For instance, the `stdout` and `stderr` files in the directory `experiment-a`
will be considered as output files of the experiment `experiment-a`, and will
thus be used together to extract relevant information for this experiment.

Note that these files may have common formats (such as JSON, XML or CSV) or
may also be the raw output of the solver.
More details on how to retrieve relevant information from these files
are given [here](#description-of-the-data-to-extract).

### Parsing Unsupported Formats

When developing *Scalpel*, we tried to think about as many campaign formats
as possible.
However, it may happen that you need to parse a campaign that uses a format
that is not recognized (yet) by *Scalpel*.
If this is the case you may write your own parser. by extending the class
`CampaignParser`.
This class must define a constructor taking as argument a
`CampaignParserListener` and a `ScalpelConfiguration`.
To give you ideas on how to write such a parser, you may have a look to the
[source of our parsers](https://github.com/crillab/metrics/tree/master/metrics/scalpel/parser).

Then, add the class of your parser to your YAML configuration file as follows:

```yaml
source:
  parser: my.completely.specified.AwesomeParser
```

*Scalpel* will dynamically instantiate your parser, and will then use it
to parse the campaign.
To make this possible, you will need to import your `AwesomeParser`
before invoking `read_campaign()`, to make sure that this class will be
reachable.

> **Remark**
>
> If you need to parse a campaign that uses an unsupported format, do not
> hesitate to submit an issue, with an example of what you want to parse.
> We will provide you some advices for writing your own parser.
>
> We may also add a new feature to *Scalpel* by supporting this format,
> either by writing a parser or by integrating yours if you agree to contribute
> and submit a pull request.

### Identifying Successful Experiments

When analyzing experimental results, it is often useful to identify which
experiments are successful and which are not.
By default, an experiment is considered as successful when it ended within
the time limit.
However, you may also want to perform additional checks to make sure that
an experiment succeeded (for instance, by checking that the output of
the experiment is correct).

To do so, you may add to your YAML configuration file an `is-success`
filter that allows to make such checks, as in the following example:

```yaml
source:
  is-success:
    - ${success}
    - ${valueA} == ${valueB} or {valueC} == 0
    - ${result} in ['CORRECT', 'CORRECT TOO']
```

Let us describe the syntax of the filter in the example above.
First, `is-success` defines a list of conjunctively interpreted
Boolean expressions.
These expressions are themselves disjunctions of predicates.

Each predicate has to contain at least one variable, delimited using
`${...}`.
Such a variable corresponds to the identifier of an experimental data
read for a given experiment (for instance, the `cpu_time` of the experiment).

If the predicate contains only the variable (such as `${success}`), then
this variable is interpreted as a Boolean value.
Otherwise, the predicate can use any comparison operator (among `<`, `<=`,
`==`, `!=`, `>=`, `>`) to compare the variable with either a literal
value (which can be a Boolean value, an integer, a float number or a
string) or another variable.
A predicate can also check that a variable is either contained in
a list of values (either literal values or variables) or contains a
value (either a literal value or a variable) using the `in` operator.
Lists are delimited using `[...]`.

> **Remark**
>
> It is worth noting that *Scalpel* itself does **not** use `is-success`
> to filter data, in the sense that even failed experiments are included
> in the campaign it builds.
>
> Instead, *Scalpel* passes this filter on to *Wallet*, so that the drawn
> figures only take into account successful experiments.

## Description of the Data to Extract

In order to extract data from the files of your campaign, you need to provide
a description of their content.
In the following, we describe how to write such a description.

### Extracting Data from Raw Files

If your experiment-ware produces raw output, and you want *Scalpel* to parse it,
you can describe how to extract data from the corresponding files (which can
be given using wildcards or relative paths) by providing regular expressions,
as in the following example:

```yaml
data:
  raw-data:
    - log-data: cpu_time
      file: "*.out"
      regex: 'overall runtime: (\d+.\d+) seconds'
      group: 1
```

In this case, when *Scalpel* reads a file with extension `.out`, it looks
for a line that matches the specified regular expression, and extracts the
`cpu_time` of the experiment from the group `1` (i.e. `(\d+.\d+)`) in this
regular expression.
In this case, the group could be omitted, as the value `1` is the default.

To make easier the description of raw data, *Scalpel* also recognizes
so-called *simplified patterns*, as illustrated in the following example:

```yaml
data:
  raw-data:
    - log-data: cpu_time
      file: "*.out"
      pattern: "overall runtime: {real} seconds"
```

Observe that, here, `pattern` is used in place of `regex`, and that the
group `(\d+.\d+)` used in the previous example is replaced by `{real}`.
This syntax allows to use one of the different symbols used to represent
common data, and to avoid worrying about whitespaces (in a simplified
pattern, any whitespace is interpreted as a sequence of whitespace characters).

*Scalpel* can interpret the following symbols.

+ `{integer}` for a (possibly signed) integer,
+ `{real}` for a real number, 
+ `{boolean}` for a Boolean value (`true` or `false`, case-insensitive),
+ `{word}` for a word (i.e., a sequence of letters, digits and `_`), and
+ `{any}` for any sequence of characters (not greedy).

If the same line contains multiple relevant data, you can extract them
by giving names to the groups you specified (in this case, the value for
`log-data` may be omitted).

```yaml
data:
  raw-data:
    - file: "*.out"
      pattern: "runtime: {real} seconds (cpu), {real} seconds (wallclock)"
      groups:
        cpu_time: 1
        wall_time: 2
```

Note that it is not possible to mix regular expressions and simplified
patterns.

### Extracting Data from File Names

Depending on your setting, you may need to extract relevant information
from the name of the file to parse (for instance, the name of the
experiment-ware or that of the input).
This can be achieved through `file-name-meta`, as in the following
example:

```yaml
data:
  file-name-meta:
    pattern: "{any}_{any}.log"
    groups:
      experiment_ware: 1
      input: 2
```

As for `log-data`, you may choose to use either regular expressions (`regex`)
or a simplified `pattern`.
The fields in `groups` are used to name the groups identifying relevant
data.

For instance, if the file `my-xp-ware_my-input.log`, the group `1` matches
with `my-xp-ware`, which is thus identified as the `experiment_ware`, while
the group `2` matches with `my-input`, which is thus identified as the
`input`.

As file hierarchies are explored through the file system, the paths of the
files that are encountered during this exploration are system-dependent
(in particular, the file separator may vary from one system to another).
*Scalpel* is able to dynamically adapt file separators used in the pattern
specified as `file-name-meta` to ensure cross-platform compatibility for your
configuration.
To make sure that this compatibility is applied, you must always use `/` as
file separator (even if it is not that of your system).

### Extracting Data from Common Formats

If your output files use a common format (as JSON, CSV or XML), you do
not need to use `raw-data` to extract their value.
Instead, you just need to specify the name of such files as follows
(wildcards and relative paths are supported).

```yaml
data:
  data-files:
    - "*.json"
    - "output.xml"
```

Note that *Scalpel* will be able to extract data from such files by inferring
automatically identifiers for the data it extracts.
In the case of CSV files, the identifiers that will be used is inferred
based on the header of the file.

For JSON and XML files, a "dotted" notation will be used.
For example, consider the following JSON output:

```json
{
  "experiment": {
    "runtime": 123.4,
    "value": [24, 27, 42, 51, 1664]
  }
}
```

Scalpel will automatically identify the runtime as `experiment.runtime` and
the list of values as `experiment.value`.
The same identifiers are inferred for the following XML output:

```xml
<experiment runtime="123.4">
  <value>24</value>
  <value>27</value>
  <value>42</value>
  <value>51</value>
  <value>1664</value>
</experiment>
```

By default, all keys stored in a JSON or XML file are extracted by *Scalpel*,
and stored in the internal representation of the campaign.
This may be memory consuming, in particular if there are some keys that you
do not need.
To discard such keys, you may specify them in the field `ignored-data` in your
YAML configuration (this may actually be applied to any key defined by
the campaign).
For instance, the snippet below allows to discard the list `experiment.value`
described in the two examples above. 

```yaml
data:
  ignored-data:
    - experiment.value
```

If needed, you can also configure the parser to use for reading data from
data-files, as in the following example:

```yaml
data:
  data-files:
    - name: "*.json"
      format: json
      name-as-prefix: true
    - name: "*.csv"
      format: csv
      has-header: false
      separator: " "
      name-as-prefix: true
    - name: "*.txt"
      parser: my.completely.specified.AwesomeParser
```

Observe in the example above that CSV files may be configured as for
CSV campaigns (the same fields are used to describe the format of the file).

For each data-file, you can also set `name-as-prefix` to `true`, so that
each field in the file will be prefixed by the name of the file, using a
dotted notation.
This is particularly useful whe the same key appears in different files.

Moreover, it is also possible to specify a custom parser, provided you
give the *completely specified* name of this class.
This parser must extend `CampaignOutputParser`, and its constructor must take
as input a `CampaignParserListener`, a `ScalpelConfiguration`, the path of the
file to parse and its name.

Finally, you may face some cases where the wildcards you use for declaring data
files (or even log-data) are too generic.
To ignore some files that still match these wildcards, you may specify the
files to ignore with the field `ignored-files` (wildcards and relative paths are
supported).
For example, the snippet below allows to ignore *some* JSON files.

```yaml
data:
  ignored-files:
    - "*ignore*.json"
```

### Mapping Data to *Scalpel*'s Expectations

When parsing an experiment, *Scalpel* expects to find the required
information to describe the result of this experiment.
The identifier of such data is thus crucial to allow *Scalpel* to build
consistent experiments.
This is in particular true for the identifiers:

+ `experiment_ware`, which is the experiment-ware run in a given experiment,
+ `input`, which is the input on which the experiment-ware has been run, and
+ `cpu_time`, which is the runtime of the experiment.

If these identifiers are not specified in your campaign files (for instance,
you have a CSV file in which the header does not use these names), you need
to tell *Scalpel* how to map your experimental data to the expected identifiers.
This can be achieved by specifying a `mapping` as in the following example:

```yaml
data:
  mapping:
    experiment_ware:
      - program
      - options
    cpu_time: runtime
    input: file
    file: path
```

In this example, we have that, for each experiment, the data read as `runtime`
will be interpreted as `cpu_time` and `file` as `input`.

Note that, for `experiment_ware`, two identifiers are specified.
In this case, the data read as `program` and `options` will be concatenated
(in this order) to build up the identifier of the experiment-ware.
Moreover, if this experiment-ware does not exist yet, an object representation
of this experiment-ware will be instantiated, using `program` and `options`
has two additional fields.

Finally, observe that `path` is mapped to `file`, which is itself mapped to
`input`.
In this case, a recursive mapping is actually applied on `path`, which will
be eventually interpreted as `input` while parsing the campaign.
Recursive mapping is the recommended approach for mapping several identifiers
to the same key.

> **Remark**
>
> This mapping is mainly designed to map custom identifiers to *Metrics*'
> naming conventions.
> However, you can also use this mapping to rename other data (especially when
> their identifiers are automatically inferred by *Scalpel*), or to group
> together data that are separated in your campaign files.

### Adding Default Values

Sometimes, it may happen that some data are missing in your experiment files,
either because some experiment-wares did not output them correctly, or did not
have enough time to output them within the time limit.
This may be a problem if this data is required by *Scalpel*.
For such data, you may provide default values as follows:

```yaml
data:
  default-values:
    cpu_time: 1800
```

In this example, we have set the default `cpu_time` to the same value as the
time limit (note that this is done by default by *Scalpel*).
You may set default values for any key of the campaign, and even for
"partial keys" (i.e., those that are part of a mapping).

## Additional Information About the Campaign

When collecting data about a campaign, you may want to add relevant
information that do not appear in the files produced during the
execution of your experiments regarding its settings.
This section presents how you can describe the experiment-wares and
inputs you used for your experiments. 

### Description of the Experiment-Wares

Optionally, you may provide a description of the experiment-wares (i.e., the
software programs you ran during your campaign).
By default, experiment-wares are automagically instantiated when encountered
during the parsing of your campaign files.

However, you may want to specify additional data w.r.t. the programs you
experimented (for instance, the version of the software, the command line
options passed to the program that was executed, etc.).

As such data may not appear in your campaign files, you can specify them
in the YAML configuration as follows:

```yaml
experiment-wares:
  - name: my-awesome-xpware
    version: 0.1.0
    command-line: ./my-awesome-xpware -o option
  - name: my-great-xpware
    commit-sha: abcd1234
    command-line: ./my-great-xpware -v value
```

When you specify information about experiment-wares, only their names are
required.
The name of an experiment-ware must uniquely identify this experiment-ware
in the campaign, and must match the one that *Scalpel* will extract from
your campaign files.
For all other information you specify, you may use any key you want to
identify this information.

Also, note that you are not required to use the same keys for all
experiment-wares.
You may also omit experiment-wares for which you do not need more information
than those mentioned in the campaign files: these experiment-wares will
simply be discovered when parsing the files.

Moreover, you may simply specify the list of the experiment-wares used
in the campaign:

```yaml
experiment-wares:
  - my-awesome-xpware
  - my-great-xpware
```

Doing so is rarely useful, as the name of the experiment-wares must necessarily
be mentioned in the campaign files, and thus will be discovered during their
parsing.
However, this may be helpful to remind you that some experiments are missing,
for instance, if you do not have run all experiment-wares yet.

### Description of the Inputs

As for experiment-wares, you may want to add data about the inputs of your
experiments.
This is achieved by defining an `input-set` in your YAML configuration file,
and giving it a proper name, as in the following example:

```yaml
input-set:
  name: my-awesome-input-set
  type: list
  files:
    - path: path/to/instanceA.cnf
      family: F1
    - path: path/to/instanceB.cnf
      family: F2
```

In this example, `files` allows to list all the inputs you used in your
experiments.
As for experiment-wares, you may specify as many data as you want for your
inputs.
You may also use different keys for these data, and omit input files when
you do not need to add more information than that provided in the campaign
files.
The only required key is `path`.

In the example above, observe that the type `list` is declared, to specify
that all relevant information are specified in `files`.

Another possible type is `file-list`, if you only list the path of the
files (in which case, you do not need to specify the `path` key).
You may also use `file` if this list is written in a separate file (one
path per line), in which case the `files` must give the list of the
files to read.

Finally, you may also specify a `hierarchy` type, in which case *Scalpel*
will explore a file hierarchy to find all input files from the file
hierarchy rooted at the directory specified in `files`, as in the following
example:

```yaml
input-set:
  name: my-awesome-input-set
  type: hierarchy
  extensions: ".cnf"
  files: /path/to/my/benchmarks
  file-name-meta:
    pattern: /path/to/my/benchmarks/{any}/{any}.cnf
    groups:
      family: 1
      name: 2
```

Note that, in this example, the input files that are considered are
those stored in the file hierarchy rooted at the directory
`/path/to/my/benchmarks`, and having a `.cnf` extension (you may
also specify a list of extensions if you have more than one).

Also, observe that a `file-name-meta` section is specified, with the
same syntax as that described [here](#extracting-data-from-file-names).
It allows extracting relevant information from the name of each input.

Both `extensions` and `file-name-meta` are taken into account for
any type of input-sets.
