DEBOPS.APT_MIRROR(5)

NAME
debops.apt_mirror - Create a mirror of multiple APT repositories

SYNOPSIS

debops run service/apt_mirror [--limit group,host,...] [--diff] [--check] [--tags tag1,tag2,...] [--skip-tags tag1,tag2,...] [<ansible-playbook options>] ...

DESCRIPTION

The apt-mirror <https://apt-mirror.github.io/> script can be used to create full or partial mirrors of APT repositories. It uses syntax similar to the sources.list(5) <https://manpages.debian.org/sources.list(5)> configuration file, supports authenticated APT repositories, access over HTTPS and more.

The debops.apt_mirror role can be used to install and configure the apt-mirror script to periodically pull different APT repositories and publish them using nginx for other hosts to use. The role supports configuration of multiple mirror "instances", which can be configured with different mirror frequency as needed.

GETTING STARTED

  Default configuration

The apt-mirror Debian package creates a simple layout of mirror directories in /var/spool/apt-mirror/ directory and uses a single cron job to manage mirroring. The debops.apt_mirror extends this setup with multiple separate mirror configurations which converge in the same /var/spool/apt-mirror/mirror/ directory and use separate cron jobs for each set of mirrored repositories. This allows for different mirror frequency for specific APT repositories if desired.

The mirrored APT repositories are stored in the /var/spool/apt-mirror/mirror/ directory, which will be published using the nginx webserver via the debops.nginx role. Different APT repositories will be published under subdirectories based on their FQDNs, for example the Debian APT repository will be published as:

http://<mirror.host>/deb.debian.org/debian

An example sources.list(5) <https://manpages.debian.org/sources.list(5)> entry can look like:

deb http://<mirror.host>/deb.debian.org/debian bullseye main contrib non-free

The default nginx configuration separates HTTP and HTTPS protocols, so that the clients can select which protocol they prefer. Users can modify the nginx configuration if they want to enable HTTP Basic Authentication for the mirror by APT clients.

The APT clients still need to configure the GPG keys for a given mirrored APT repository separately. This can be done using the debops.apt or debops.keyring Ansible roles.

  Usage behind a HTTP proxy

The apt-mirror script uses wget behind the scenes to download files. If the mirror is located behind a HTTP proxy, /var/spool/apt-mirror/.wgetrc configuration file with proxy configuration can be used to access the HTTP proxy - it should work both on the command line as well as via the cron job.

  Example inventory

To install and configure apt-mirror on a given host, it should be included in a specific Ansible inventory group:

[debops_all_hosts]
hostname
[debops_service_apt_mirror]
hostname

  Example playbook

If you are using this role without DebOps, here's an example Ansible playbook that uses the debops.apt_mirror role:

---
- name: Configure APT mirroring service
  collections: [ 'debops.debops', 'debops.roles01',
                 'debops.roles02', 'debops.roles03' ]
  hosts: [ 'debops_service_apt_mirror' ]
  become: True
  environment: '{{ inventory__environment | d({})
                   | combine(inventory__group_environment | d({}))
                   | combine(inventory__host_environment  | d({})) }}'
  roles:
    - role: keyring
      tags: [ 'role::keyring', 'skip::keyring', 'role::nginx' ]
      keyring__dependent_apt_keys:
        - '{{ nginx__keyring__dependent_apt_keys }}'
    - role: apt_preferences
      tags: [ 'role::apt_preferences', 'skip::apt_preferences' ]
      apt_preferences__dependent_list:
        - '{{ nginx__apt_preferences__dependent_list }}'
    - role: ferm
      tags: [ 'role::ferm', 'skip::ferm' ]
      ferm__dependent_rules:
        - '{{ nginx__ferm__dependent_rules }}'
    - role: python
      tags: [ 'role::python', 'skip::python' ]
      python__dependent_packages3:
        - '{{ nginx__python__dependent_packages3 }}'
      python__dependent_packages2:
        - '{{ nginx__python__dependent_packages2 }}'
    - role: nginx
      tags: [ 'role::nginx', 'skip::nginx' ]
      nginx__dependent_servers:
        - '{{ apt_mirror__nginx__dependent_servers }}'
    - role: apt_mirror
      tags: [ 'role::apt_mirror', 'skip::apt_mirror' ]

  Ansible tags

You can use Ansible --tags or --skip-tags parameters to limit what tasks are performed during Ansible run. This can be used after a host was first configured to speed up playbook execution, when you are sure that most of the configuration is already in the desired state.

Available role tags:

role::apt_mirror
    Main role tag, should be used in the playbook to execute all of the role tasks as well as role dependencies.

DEFAULT VARIABLE DETAILS

Some of debops.apt_mirror default variables have more extensive configuration than simple strings or lists, here you can find documentation and examples for them.

  apt_mirror__configuration

The apt_mirror__*_configuration variables define "instances" of apt-mirror configurations. Each instance has its own set of APT sources to mirror, its own configuration options and its own cron job with a separate frequency. Mirrored repositories are stored under the same /var/spool/apt-mirror/mirror/ directory and care should be taken to avoid conflicts.

  Examples

Enable APT mirror of Debian Stable repository in the default configuration (make sure that the space requirements are met before enabling this). See apt_mirror__default_configuration variable for the initial options:

apt_mirror__configuration:
  - name: 'default'
    sources:
      - name: 'debian-stable'
        state: 'present'

Add an APT mirror for InfluxData <https://www.influxdata.com/blog/package-repository-for-linux/> and Zabbix <https://www.zabbix.com/download> APT repositories in the default configuration:

apt_mirror__configuration:
  - name: 'default'
    sources:
      - name: 'influxdata'
        raw: 'deb https://repos.influxdata.com/debian stable main'
      - name: 'zabbix'
        type: 'deb'
        uri: 'https://repo.zabbix.com/zabbix/6.4/debian'
        suite: '{{ ansible_distribution_release }}'
        components: [ 'main', 'contrib', 'non-free' ]
      - name: 'influxdata-clean'
        type: 'clean'
        uri: 'https://repos.influxdata.com/debian'
      - name: 'zabbix-clean'
        type: 'clean'
        uri: 'https://repo.zabbix.com/zabbix/6.4/debian'

Create a new APT mirror instance with private APT repositories synchronized hourly using cron service:

apt_mirror__configuration:
  - name: 'internal'
    cron_time: '@hourly'
    sources:
      - name: 'apt-repo'
        type: 'deb'
        uri: 'https://username:password@apt.example.org/debian'
        suite: 'stable'
        components: [ 'main' ]
      - name: 'clean-apt-repo'
        type: 'clean'
        uri: 'https://apt.example.org/debian'

  Syntax

The variables are defined as a list of YAML dictionaires, parsed using Universal Configuration system. Each dictionary defines an "instance" using specific parameters:

name
    Required. An identifier for a particular apt-mirror instance. The value is used in the filesystem paths and should be a simple alphanumeric string. Configuration entries with the same name parameters are merged during role execution and can affect each other via Universal Configuration principles.

filename
    Optional. Name of the apt-mirror configuration file to use, located under the /etc/apt/ directory. If not specified, the configuration files are named in the format: mirror.<name>.list.

state
    Optional. If not specified or present, a given APT mirror instance will be configured on the host. If absent, the mirror will not be configured and the configuration file will be removed. If ignore, a given configuration entry will not be evaluated during role execution.

cron_time
    Optional. The string which defines a time period for a given APT mirror synchronization, in the crontab(5) <https://manpages.debian.org/crontab(5)> format. If not specified, the value of the apt_mirror__cron_time will be used instead.

cron_user
    Optional. The UNIX account under which a cron job will be executed for a particular APT mirror. If not specified, the value in the apt_mirror__user will be used by default.

cron_command
    Optional. The cron job which should be executed for a given APT mirror. If not specified, the configuration template will automatically generate a configuration suitable for sequential apt-mirror operation (the script does not support parallel processing).

options
    Optional. List of YAML dictionaries which defines apt-mirror configuration options stored in the mirror configuration file. The options parameters from configuration entries with the same name parameter are merged together and can affect each other. A set of default options is taken from the apt_mirror__default_options as a base, user options are merged on top of it.

    The configuration is defined as a list of YAML dictionaries, each dictionary key being the option name and its value being the option value. Alternatively you can use specific parameters to control each option:

name
    The name of the option.

value
    The value of the option, can be a number or a string.

state
    If not specified or present, the option is included in the configuration file. If absent, the option will be removed from the configuration file. If comment, the option will be present but it will be commented out. If dynamic, the value of a given option will be replaced with a value set at runtime in the configuration template (currently only the var_path option utilizes this functionality).

comment
    String or YAML text block with comments related to a given configuration option.

sources
    Required. A list of APT repositories which will be mirrored by the apt-mirror script. THe sources parameters from configuration entries with the same name parameter are merged together and can affect each other.

    The list is defined using YAML dictionaries:

name
    An identifier of a given APT repository, used only internally by the role. Multiple sources with the same name are merged together and can affect each other.

raw String specifying an APT repository in the sources.list(5) <https://manpages.debian.org/sources.list(5)> one-line-style format, included in the generated configuration file as-is. If this parameter is specified, it takes precedence over the parametrized configuration below.

type
    Specify the APT repository type (deb, deb-src, deb-<arch> for additional architecture mirrors, or clean to generate a cleanup script for a given repository).

uri Specify the URL of the APT repository to mirror or clean.

suite
    Specify the repository suite which should be mirrored (this is usually named stable, or a specific distribution release like bookworm or bullseye).

component / components
    A string (first version) or a YAML list (second version) of repository components to mirror (this is usually main, contrib, non-free in Debian case, Ubuntu usually uses main, restricted, universe, multiverse components).

state
    If not specified or present, a given source will be present in the generated configuration file. If absent, a given source will be omitted from the generated configuration file. If ignore, a given configuration entry will not be evaluated during role execution. If comment, the source will be present but commented out.

weight
    A positive or negative integer which will affect the order of the sources in the generated configuration file. Positive weight pushes a given source down the list, negative weight lifts it up the list.

comment
    String or YAML text block with comments related to a given source.

AUTHOR
Maciej Delmanowski

COPYRIGHT
2014-2024, Maciej Delmanowski, Nick Janetakis, Robin Schneider and others