Last modified 13 years ago Last modified on 10/08/07 09:19:11

How Checksums are handled in dcache

There is a detailed description on how to enable various difference checksum policies in The Book. They are tailored for dCache sysadmins :

This is how it is supposed to work :

  • dCache is using the adler32 crc checksum method.
  • For different protocols, checksums are handled slightly different :
    1. dCap (with dccp) : The checksum is calculated on the client side, on arrival at the dCache server and after the file has been written to a dCache disk. Only in case all three checksums match, the file is accepted and the checksum is stored in the dCache database.
    2. dCap (library) if the file has been written consecutively without intervening seek or read operations : same as above.
    3. dCap (library) but not (2.) : Checksum is calculated from the first copy of that file written to a dCache disk.
    4. srmcp (which is provided by dCache) : The checksum is calculated on the client side and send with a special ftp command to the dCache service. This checksum is stored but not compared with the version on dCache disks.
    5. For all other transfers (FTS, globus-url-copy, ...), the checksum is calculated from the first copy of the file within dCache. (same as 3.)
  • When are checksums used to compare different versions of a file :
    1. See 1. and 2. above.
    2. For files retrieved from a tape system, the checksum of the restored copy is always calculated and compared with the stored checksum of that file. If those values don't match, the restored copy of the file on disk is removed, the operation is retried on a different pool and a message is sent to the billing file/db. Depending on the configuration, dCache gives up after some failed retries and a message is displayed on the 'dCache restore monitor web pages'.
  • dCache never calculates checksums before or while a file is delivered to a client (reading from dCache).
  • Although this is envisioned, dCache doesn't do regular checksum calculation on disk to verify data repository consistency.
  • Starting with dCache 1.8, pool to pool transfers are also protected by checksum mechanisms.

Last modified by patrick @ Wed Mar 3 08:54:26 2021