wiki:manuals/SmallFilesScript
Last modified 20 months ago Last modified on 12/09/15 14:30:35

What's this about?

If a dCache installation uses tape storage in the backend, storing lots of smaller files may lead to a severe waste of storage space when those files are moved from disk to tape. The reason is that tape systems generally require a fixed-size overhead per file. Additionally, the constant but high seek times on tape make accessing many small files prohibitively slow.

In order to use tapes efficiently, the SmallFiles? script described here gives dCache the capability to aggregate many small files into containers that get stored on tape. While this process is completely transparent to the user, the tape system gets to deal with files in the multi-Gigabyte range, near the peak space and transfer efficiency.

This document describes the installation and operation of this script.

Target audience

This document is aimed at dCache administrators who run a tape-backed system and developers.

High-level system overview

The SmallFiles? "script" is actually a system consisting of several components. This section will give a generalized overview; for a detailed look at the individual components, see the "Components" section below.

The packer script is called by a normal initscript and runs as a service on (preferably) a dedicated packing server. It monitors directories specified by the administrators in the container.conf file, and as soon as enough small files are present to warrant creating an archive of the desired size, the script packs the small files into container files. During operation, the state of the entire system is kept in a MongoDB that keeps track of the progress of the operation.

Components

. 
├── etc
│   ├── dcache
│   │   └── container.conf
│   ├── init.d
│   │   └── pack-system
│   ├── logrotate.d
│   │   └── pack-service
│   └── logstash
│       └── conf.d
│           └── logstash.conf
└── usr
    ├── local
    │   └── bin
    │       ├── dcap.py
    │       ├── fillmetadata.py
    │       ├── pack-files.py
    │       ├── pushFiles.sh
    │       ├── sf-setup-pools.sh
    │       ├── speedtest.sh
    │       └── writebfids.py
    └── share
        └── dcache
            └── lib
                ├── datasetPut.js
                └── hsm-internal.sh

For reference, the source is available at https://github.com/dCache/SmallFiles .

Scripts

/usr/local/bin/pack-files.py is central for the operation of the SmallFiles? system. It contains the packing logic and relies on the other scripts in the same directory for auxiliary functions.

/usr/share/dcache/lib contains datasetPut.js, a small helper snippet of JavaScript? that is used to actually write data into the MongoDB.

hsm-internal.sh, in the same directory, is the interface to the HSM system controlling access to the tape library.

Configuration

The central config file for the SmallFiles? system is /etc/dcache/container.conf. It controls the operation of an instance of the SmallFiles? system, identified by a script ID. Several instances may run in parallel on a server; the scriptId and mongoDb parameters defined in that file distinguish them (see below).

The container.conf also states which files should be packed, when, and which paths should be covered. This is explained in detail in the annotated file itself.

Logging

The SmallFile? system logs activities in the following log files, all residing under /var/log/dcache on the packing node:

  • fillmetadata-$NAME.log
  • pack-files-$NAME.log
  • pack-files-$NAME.status
  • writebfids-$NAME.log

In those filenames, $NAME is replaced by the pack script's ID, which is defined for each script instance in the config file container.conf (generally found under /etc/dcache/).

Rotation of the log files is controlled by a logrotate config file that is installed by default in /etc/logrotate.d/pack-service.

Operation

Running several packers in parallel

The ScriptID and mongoDb parameters in container.conf allow distinguishing different instances of the packer scripts by giving them an identity (ScriptID) and aim them at a certain database (mongoDb). However, for starting individual instances of the system, each instance will require a distinct init script that needs to be written by the administrators.