wiki:TSS
Last modified 7 years ago Last modified on 08/18/11 11:13:42

The dCache Tertiary Storage System Interface

Introduction

One of the features dCache provides is the ability to migrate files from its disk repository to one or more connected Tertiary Storage Systems and to move them back to disk when necessary. Although the interface between dCache and the Tertiary Storage System (TSS) is kept simple, dCache assumes to interact with an intelligent TSS. dCache doesn't drive tape robots or tape drives by itself. More detailed requirements to the storage system are described in one of the subsequent paragraphs.


Scope of this document

This document describes how to enable a standard dCache installation to interact with a Tertiary Storage System. The description focuses on a basic configuration, assuming that

  • every dCache disk pool is connected to only one TSS instance.
  • all dCache disk pools are connected to the same TSS instance.
  • the dCache instance has not yet populated with data, or only an negligible amount of files.

Not all pools need to be configured to interact with the same Tertiary Storage System or with a storage system at all. Furthermore pools can be configured to have more than one Tertiary Storage System attached. All those cases are not in the scope of the document.


HSM requirements (What is an appropriate Tertiary Storage System?)

dCache can only drive intelligent Tertiary Storage Systems. This essentially means that tape robot and tape drive operations must be done by the TSS itself and that there is some simple way to abstract the file PUT, GET and REMOVE operation.

Migrating TSS's with a file system interface.

Most migrating storage systems provide a regular POSIX file system interface. Based on rules, data is migrated from primary to tertiary storage (mostly tape systems). Examples for migrating storage systems are:

  • HPSS (IBM)
  • DMF (SGI)

TSS's with a minimalistic PUT, GET and REMOVE interface

Other tape systems provide a simple PUT, GET, REMOVE interface. Typically, a copy-like application writes a disk file into the TSS and returns an identifier which uniquely identifies the written file within the Tertiary Storage System. The identifier is sufficient to get the file back to disk or to remove the file from the TSS. Examples are:

  • OSM (ComputerAssociates)
  • Enstore (FERMIlab)

How dCache interacts with Tertiary Storage System

Whenever dCache decides to copy a file from disk storage to tertiary storage a user-provided executable which can be either a script or a binary is automatically started on the pool where the file is located. That executable is expected to write the file into the Backend Storage System and to return an URI, uniquely identifying the file within that storage system. The format of the URI, as well as the arguments to the executable, are described later in this document. The unique part of the URI can either be provided by the storage element, in return of the 'STORE FILE' operation, or can be taken from dCache. A non-error return code from the executable lets dCache assume that the file has been successfully stored and, depending on the properties of the file, dCache can decide to remove the disk copy if space is running short on that pool. On a non-zero return from the executable, the file doesn't change its state and the operation is retried or an error flag is set on the file, depending on the error return code from the executable.

If dCache needs to restore a file to disk, the same executable is launched with a different set of arguments, including the URI, provided when the file was written to tape. It is in the responsibility of the executable to fetch the file back from tape, based on the provided URI and to return '0' if the 'FETCH FILE' operation was successful or non-zero otherwise. In case of a failure, the pool retries the operation or dCache decides to the fetch the file from tape, using a different pool.


Details on the TSS-support executable

Summary of command line options

This part explains the syntax of calling the executable that supports 'STORE FILE', 'FETCH FILE' and 'REMOVE FILE' operations.

put <pnfsID>  <filename>  -si=<storage-information> <more options> 
get <pnfsID>  <filename>  -si=<storage-information> -uri=<storage-uri> <more options> 
remove -uri=<storage-uri> <more options> 
  • put/get/remove: these keywords indicate the operation to be performed.
    • put: move file from disk to TSS.
    • get: move file back from TSS to disk.
    • remove: remove the file from TSS.
  • <pnfsID>: the internal identifier (i-node) of the file within dCache. The <pnfsID> is unique within a single dCache instance and globally unique with a very high probability.
  • <filename>: Is the full path of the local file to be copied to the TSS (for put) and respectively into which the file from the TSS should be copied (for get).
  • <storage-information>: the storage information of the file (for more details see below).
  • <storage-uri>: the URI, which was returned by the executable, after the file was written to tertiary storage (for more details see below). In order to get the file back from Tertiary Storage System the information of the URI is preferred over the information in the <storage-information>.
  • <more-options>: -<key>=<value> pairs taken from the TSS configuration commands of the pool 'setup' file. One of the option, always provided is the -command=<full path of this executable>.

The <storage-information> is a string in the format: <key>=<value>;[<key>=<value>;[…]] Example:

-si=size=1048576000;new=true;stored=false;sClass=desy:cms-sc3;cClass=-;hsm=osm;Host=desy;

Mandatory storage information's keys:

  • <size>: size of the file in bytes
  • <new>: false if file already in the dCache
  • <stored>: true if file already stored in the TSS
  • <sClass>: HSM depended, is used by the PoolManager for pool attraction.
  • <cClass>: Parent Directory tag (cacheClass). Used by the PoolManager for pool attraction. May be '-'.
  • <hsm>: Storage Manager name (enstore/osm). Can be overwritten by the Parent Directory tag (hsmType).

OSM specific

  • group: the storage group of the file to be stored as specified in the ".(tag)(sGroup)" tag of the parent directory of the file to be stored.
  • store: the store name of the file to be stored as specified in the ".(tag)(OSMTemplate)" tag of the parent directory of the file to be stored.
  • bfid: bitfile Id (get only) (e.g. 000451243.2542452542.25424524)

Enstore specific

  • group': storage group (e.g. cdf,cms ...)
  • family': file family (e.g. sgi2test,h6nxl8, ...)
  • bfid: bitfile Id (get only) (e.g. B0MS105746894100000)
  • volume: tape Volume (get only) (e.g. IA6912)
  • location: location on tape (get only) (e.g. : 0000_000000000_0000117)

There might be more key values pairs which are used by the dCache internally and which should not affect the behaviour of the executable.

The <storage-uri> is formatted as follows :

hsmType://hsmInstance/?store=<storename>&group=<groupname>&bfid=<bfid>
  • <hsmType>: the type of the Tertiary Storage System
  • <hsmInstance>: is the name of the instance
  • <storename> and '<groupname>' are the store and group name of the file as provided by the arguments to this executable.
  • The <bfid>: the unique identifier needed to restore or remove the file if necessary.

Example for an storage URI:

osm://osm/?store=sql&group=chimera&bfid=3434.0.994.1188400818542

Summary of return codes

Return Code Meaning Behavior for PUT FILE Behavior for GET FILE
30 <= rc < 40 User defined Deactivates request Reports Problem to PoolManager
41 No Space Left on device Pool Retries Disables pool and reports problem to PoolManager
42 Disk Read I/O Error Pool Retries Disables pool and reports problem to PoolManager
43 Disk Write I/O Error Pool Retries Disables pool and reports problem to PoolManager
other - Pool Retries Reports problem to PoolManager

The executable and the 'STORE FILE' operation

Whenever a disk file needs to be stored on a Tertiary Storage System, dCache automatically launches an executable on the pool, containing the file to be copied. Exactly one instance of the executable is started for each file. Multiple instances of the executable may run concurrently for different files. The maximum number of concurrent instances of the executables per pool, as well as the full path of the executable can be configured in the 'setup' file of the pool as described in the chapter 'pool setup file configuration'.

The following arguments are given to the executable of a 'STORE FILE' operation on startup.

put <pnfsID>  <filename>  -si=<storage-information> <more options> 

Details on the meaning of certain arguments are described in paragraph above.

With the arguments provided, the executable is supposed to copy the file into the Tertiary Storage System. The executable must not terminate before the transfer of the file was either successful or failed.

Success must be indicated by a '0' return of the executable. All non-zero values are interpreted as a failure which means, dCache assumes that the file has not been copied to tape.

In case of a '0' return code the executable has to return a valid storage URI to dCache in formate:

osm://osm/?store=<storename>&group=<groupname>&bfid=<bfid>

Details on the meaning of certain parameters are described in paragraph above.

The <bfid> can either be provided by the TSS as result of the 'STORE FILE' operation or the 'pnfsID' may be used. The latter assumes that the file has to be stored with exactly that pnfsID within the TSS. Whatever URI is chosen, it must allow to uniquely identify the file within the Tertiary Storage System.

NOTE: Only the URI must be printed to stdout by the executable. Additional information printed either before or after the URI will result in an error. stderr can be used for additional information in case of success or failure.

The executable and the 'FETCH FILE' operation

Whenever a disk file needs to be fetched from a Tertiary Storage System, dCache automatically launches an executable on the pool, containing the file to be fetched. Exactly one instance of the executable is started for each file. Multiple instances of the executable may run concurrently for different files. The maximum number of concurrent instances of the executables per pool, as well as the full path of the executable can be configured in the 'setup' file of the pool as described in the chapter 'pool setup file configuration'.

The following arguments are given to the executable of a 'FETCH FILE' operation on startup:

get <pnfsID>  <filename>  -si=<storage-information> -uri=<storage-uri> <more options> 

Details on the meaning of certain arguments are described in the paragraph Summary of command line options. For return codes see paragraph Summary of return codes.

The executable and the REMOVE FILE operation

Whenever a file is removed from the dCache namespace (file system) a process inside dCache makes sure that all copies of the file are removed from all internal and external media. The pool, which is connected to the TSS which stores the file, is activating the executable with the following command line options:

remove -uri=<storage-uri> <more options> 

Details on the meaning of certain arguments are described in the paragraph Summary of command line options. For return codes see paragraph Summary of return codes.

The executable is supposed to remove the file from the TSS and report a zero return code. If a non-zero error code is returned, the dCache will call the script again at a later point in time.

An Example of the executable

An example of the executable connecting dCache and a Tertiary Storage System can be found at: http://trac.dcache.org/projects/dcache/wiki/dCacheHSMInterfaceV2script1


Configuring pools to interact with a Tertiary Storage System

The executable interacting with the Tertiary Storage System (TSS), as described in the chapter above, has to be provided to dCache on all pools connected to the TSS. The executable, either a script or a binary, has to be made 'executable' for the user, dCache is running as, on that host.

The following files have to be modified to allow dCache to interact with the TSS.

  • The PoolManager.conf file (one per system)
  • The pool layout file (one per pool host)
  • The pool 'setup' file (one per pool)
  • The namespaceDomain layout file (one per system)

After the layout files and the various 'setup' files have been corrected, the following domains have to be restarted :

  • pool services
  • dCacheDomain
  • namespaceDomain

The dCache layout files

The PoolManager.conf file

To be able to read a file from the tape in case the cached file has been deleted from all pools, enable the restore-option in the /opt/d-cache/config/PoolManager.conf file and restart the dCacheDomain:

rc set stage on

Alternatively, the following sequence may be typed in the the dCache Admin Interface:

[dcachetogo.dcache.org] (local) admin > cd PoolManager
[dcachetogo.dcache.org] (PoolManager) admin > rc set stage on
[dcachetogo.dcache.org] (PoolManager) admin > save

In this case, a restart of the dCacheDomain is not necessary.

The pool layout

The dCache layout file must be modified for each pool node connected to an TSS. If your pool nodes have been configured correctly to work w/o TSS, you will find the entry lfs=precious in the layout file (that is located in /opt/d-cache/etc/layouts and /opt/d-cache/etc/dcache.conf respectively) for each pool service. This entry has to be removed for each pool which should be connected to a TSS. This will default the lfs parameter to hsm which is exactly what we need.

The pool 'setup' file

The pool 'setup' file located in $poolHomeDir/$poolName/setup) and mainly defines 3 details related to TSS connectivity.

  • pointer to the executable which is launched on storing and fetching files.
  • The maximum number of concurrent 'STORE FILE' requests allowed per pool.
  • The maximum number of concurrent 'FETCH FILE' requests allowed per pool.

Defining the executable:

   hsm set <hsmType> [<hsmInstance>] -command=<full path to executable>
  • <hsmType>: the type of the TSS system. Must be set to 'osm' for basic setups.
  • <hsmInstance>: the instance name of the TSS system. Must be set to 'osm' for basic setups.
  • <full path to executable>: the full path to the executable which should be launched for each TSS operation.

Setting the maximum number of concurrent PUT and GET operations.

#
#  PUT operations
#
st set max active <numberOfConcurrentPUTS>

#
# GET operations
#
rh set max active <numberOfConcurrentGETs>

Both numbers must be non zero to allow the pool to perform transfers.

The namespace layout

In order to allow dCache to remove files from attached TSSes, the cleaner.hsm = enabled must be added immediately underneath the [namespaceDomain/cleaner] service declaration. The namespace part should look like this :

[namespaceDomain]
 ... other services ...
[namespaceDomain/cleaner]
cleaner.hsm = enabled
.. more ...

What happens next

After restarting the necessary dCache domains, pools, already containing files, will start transferring them into the TSS, as those files only have a disk copy yet. The number of transfers is determined by the configuration in the pool 'setup' file, as described in the section "The pool 'setup' file".

How to call Flush-/Restore commands from the Admin Interface

In order to see the state of files within a pool, login into the pool in the admin interface and run:

rep ls   

Example:

[dcachetogo.dcache.org] (pool_1) admin > rep ls                                      
00008F276A952099472FAD619548F47EF972 <-P---------L(0)[0]> 291910 si={dteam:STATIC}
00002A9282C2D7A147C68A327208173B81A6 <-P---------L(0)[0]> 2011264 si={dteam:STATIC}
0000EE298D5BF6BB4867968B88AE16BA86B0 <C----------L(0)[0]> 1976 si={dteam:STATIC}
  • <C----------L(0)[0]> indicates that the file is on tape and only cached on disk.
  • <-P---------L(0)[0]> indicates that the file is only on disk. (but, depending on 'queue ls -l queue' scheduled for flush to tape (for more details see the chapter "Monitoring what's going on", paragraph "The dCache Command Line Admin Interface")).

In order to flush a file to the tape run the following command:

flush pnfsid <pnfsid>

Example:

[dcachetogo.dcache.org] (pool_1) admin > flush pnfsid 00002A9282C2D7A147C68A327208173B81A6
Flush Initiated

Removing a cached file once it has been stored in the TSS. In this case the file will be restored when requested.

rep rm <pnfsid>

Example:

[dcachetogo.dcache.org] (pool_1) admin > rep rm  00002A9282C2D7A147C68A327208173B81A6
Removed 00002A9282C2D7A147C68A327208173B81A6

To restore a file from the tape you can simply request it by initializing a reading transfer or you can fetch it by running the following command in the Admin Interface:

rh restore [-block] <pnfsId>

Example:

[dcachetogo.dcache.org] (pool_1) admin > rh restore 00002A9282C2D7A147C68A327208173B81A6
Fetch request queued

How to monitor what's going on

This section briefly described the tools and mechanisms to monitor the TSS PUT, GET and REMOVE operations. dCache provides a configurable logging facility and a Command Line Admin Interface to query and manipulate transfer and waiting queues.

Log files

dCache is typically configured to only log information if something unexpected happens. However, to get familiar with Tertiary Storage System interactions you might be interested in more details. This section provides advise on how to obtain this kind of information.

The executable log file

Since you provide the executable, interfacing dCache and the TSS, it is in your responsibility to ensure sufficient logging information to be able to trace possible problems with either dCache or the TSS. Each request should be printed with the full set of parameters it receives, together with a timestamp. Furthermore information returned to dCache should be printed.

dCache log files in general

In dCache, each domain (e.g. dCacheDomain, pools etc), prints logging information into its own log file, named after the domain. The location of those log files it typically the /var/log or /var/log/dCache directory, depending on the individual configuration. In the default logging setup only errors are printed reported. This behavior can be changed by either modifying /opt/d-cache/etc/logback.xml or using the dCache CLI to increase the log level of particular components. See the next section on the CLI, on how to increase the dCache log level.

If you intend to increase the log level of all components on a particular host you would need to change the /opt/d-cache/etc/logback.xml file as described below. dCache components need to be restarted to activate the changes.

 <threshold>
     <appender>stdout</appender>
     <logger>root</logger>
     <level>warn</level>
   </threshold>

needs to be changed to

 <threshold>
     <appender>stdout</appender>
     <logger>root</logger>
     <level>info</level>
   </threshold>

The change might result in a significant increase in log messages. So don't forget to change back before starting production operation. The next chapter described how to change the log level in a running system.

The dCache Command Line Admin Interface

The dCache Command Line Admin Interface gives an access to information describing the process of storing and fetching files to and from the TSS, as there are:

  • The PoolManager Restore Queue. A list of all requests which have been issues to all pools for a 'FETCH FILE' from the TSS (rc ls)
  • The Pool Collector Queue. A list of files, per pool and storage group, which will be scheduled for 'STORE FILE' as soon as the configured trigger criteria match.
  • The Pool STORE FILE Queue. A list of files per pool, scheduled for 'STORE FILE' operation. A configurable amount of requests within this queue are active, which is equivalent to the number of concurrent store processes, the rest is inactive, waiting to become active.
  • The Pool FETCH FILE Queue. A list of files per pool, scheduled for 'FETCH FILE' operation. A configurable amount of requests within this queue are active, which is equivalent to the number of concurrent fetch processes, the rest is inactive, waiting to become active.

For evaluation purposes, the pinboard of each component can be used to track down dCache behavior. The pinboard only keeps the most recent 200 lines of log information but reports not only errors but informational messages as well.

Login into the dCache Command Line Admin Interface :

[root@dcachetogo ~]# ssh -1 -l admin -c blowfish -p 22223 localhost

admin@localhost's password: 

    dCache Admin (VII) (user=admin)

[dcachetogo.dcache.org] (local) admin > 

Increase the log level of a particular services (example of PoolManager)

[dcachetogo.dcache.org] (local) admin > cd PoolManager
[dcachetogo.dcache.org] (PoolManager) admin > log set stdout ROOT INFO
[dcachetogo.dcache.org] (PoolManager) admin > log ls
stdout:
  ROOT=INFO
  dmg.cells.nucleus=WARN*
  logger.org.dcache.cells.messages=ERROR*
.....

Checking the pinboard of a services (example: PoolManager)

[dcachetogo.dcache.org] (local) admin > cd PoolManager
[dcachetogo.dcache.org] (PoolManager) admin > show pinboard 100
08.30.45  [Thread-7] [pool1 PoolManagerPoolUp] sendPoolStatusRelay: ...
08.30.59  [writeHandler] [NFSv41-dcachetogo PoolMgrSelectWritePool ...
....

Example: removing a file from a pool (for instance test.root mit der pnfsid 00002A9282C2D7A147C68A327208173B81A6), requesting the file and checking the PoolManager Restore Queue

[dcachetogo.dcache.org] (pool_1) admin > rep rm  00002A9282C2D7A147C68A327208173B81A6
Removed 00002A9282C2D7A147C68A327208173B81A6

[dcachetogo.dcache.org] ~ #  /opt/d-cache/dcap/bin/dccp dcap://dcachetogo.dcache.org:/pnfs/desy.de/data/dteam/test.root test-restore.root

[dcachetogo.dcache.org] (local) admin > cd PoolManager
[dcachetogo.dcache.org] (PoolManager) admin > rc ls
00002A9282C2D7A147C68A327208173B81A6@0.0.0.0/0.0.0.0-*/* m=1 r=0 [pool1] [Staging 08.15 17:52:16] {0,}

Checking the Pool Collector Queue

[dcachetogo.dcache.org] (pool1) admin > queue ls -l queue
                   Name: chimera:alpha
              Class@Hsm: chimera:alpha@osm
 Expiration rest/defined: -39 / 0   seconds
 Pending   rest/defined: 1 / 0
 Size      rest/defined: 877480 / 0
 Active Store Procs.   :  0
  00001BC6D76570A74534969FD72220C31D5D

# and 
#
[dcachetogo.dcache.org] (pool1) admin > flush ls
Class                 Active   Error  Last/min  Requests    Failed
dteam:STATIC@osm           0       0         0         1         0

Checking the STORE FILE and FETCH FILE Queue

#
# for FETCH file
#
[dcachetogo.dcache.org] (local) admin > cd pool1  
[dcachetogo.dcache.org] (pool1) admin > rh ls              
0000B56B7AFE71C14BDA9426BBF1384CA4B0  0   Fri Aug 12 15:38:33 CEST 2011


#
# for STORE file
#
[dcachetogo.dcache.org] (local) admin > cd pool1  
[dcachetogo.dcache.org] (pool1) admin > st ls            
0000EC3A4BFCA8E14755AE4E3B5639B155F9  1   Fri Aug 12 15:35:58 CEST 2011

To check the repository on the pools run the command "rep ls" that is described in the beginning of the chapter "How to call Flush-/Restore commands from the Admin Interface".