Last modified 4 weeks ago Last modified on 09/18/18 15:29:27

dCache Tier I meeting September 18, 2018

[part of a series of meetings]

Present, IN2P3(), Sara(), Triumf(), BNL(), NDGF(Jens), PIC(), KIT(Xavier), Fermi(), CERN()


(see box on the other side)

Site reports


Running well.

There are a few points.


NDGF is now running dCache v4.2.

The upgrade went well -- no major issues, to being with.

There was a fair amount of xrootd failed reads.

Vincent spent considerable time investigating this. The cause was the "dCache UUID property" problem. Install a snapshot version of xrootd4j package. This resolve the problem.

At the worse case, NDGF was failing 500 reads per minute.

tape pool

One tape pool -- write the files to tape, after 24 hours, the flush failed.

Vincent investigated this. The symptoms involved endit polling the file. Once the file was successfully written to tape, the dCache/plugin stopped polling but dCache didn't realise the file was now written to tape. After 24 hours, the pool timed out and retried. This resulted in a fresh copy being written to tape.

The underlying cause was a command (entitled something like: removing dead code). This resulted in a change in the nearline plugin API.

An updated version of endit dCache-nearline plugin was built, which resolved the issue.

Jens will email <support@…> with the commit that caused the problem.

[UPDATE AFTER MEETING] The commit was: ` commit 5fec809fa7edb935c11090e5d4306d87b5d63848 Author: Paul Millar <paul.millar@…> Date: Sun Feb 11 12:07:35 2018 +0100 `

The pool is now working normally.


Jens is looking at deploying dCache frontend. At NDGF it would be primarily as an admin interface; for SWESTORE, this would be for regular users.

Jens mentioned (without going into details) that there are some blocking issues that currently prevents dCacheView's adoption at SWESTORE.


dCache has been running fine for the past two weeks.

xrootd problems

  1. Jens, do you see any random failures during the handshake for xrootd?
  1. In the monitoring (based on billing), we don't see them.

Still running 2.16 (1.9.x, before 2.13) -- handshaking.


` 8 Sep 2018 14:44:07 (Xrootd-alice-piggy) [door:Xrootd-alice-piggy@piggyDomain:AAV2JKVM7sA] Unsupported request: query[5,0,/NOLFN] 18 Sep 2018 14:44:07 (Xrootd-alice-piggy) [door:Xrootd-alice-piggy@piggyDomain:AAV2JKVOoFg] Unsupported request: query[5,0,/NOLFN?authz=-----BEGIN SEALED CIPHER----- cY-a44WPmdV3xnyImDrU6ZIsqHBDMLPMaGvxXHp7PSfCFlq6P8UOG5dza7xwn1WLmn8u9r+30XFU fu5veBAPT8f8JbZyxzLtO-uiE72NicOJSbMu0EOigE6Qenj0dj8rUzU9OJar7kFmAlITRdv-NYah 0S-aXHp39zQI6WWofes= -----END SEALED CIPHER----- -----BEGIN SEALED ENVELOPE----- AAAAgKxPPIr8ZW3ZYT4SBaBIlZ57hMJG+S1opuAYZHzM9Ud3MqzA+48j-Dltm6xODABC115pSbX4 t8Rtv45Nx9qPYxCNy3AHhPL1D+W0BL5c4X-RJRRzAqPzo+2-JJRAoMD8cbU5OnOcG-50v-A1mvAc NVCf9D3EAW7rTudSCNyeBSZ0lT5taDBaQ0OwI8RH7SF2zBaHFsOkqPXFdzpJMQOl88DeX4B8Rl39 X6xnrdOLP2oT9nhP2wwJhOUwvXGthaHtdVwdbNdKh6FAtFrLGYvoPLDyuHMuwArqaHdrjM1dML7O dvUm6eoFmYVgi51TKyfufxNay6JJVJnRGeqgedduS-RN4grNe6E4ebxNU837qY3VZprn+luvy1zF tNfnOwhBGwKC0GXlXj4-ZS3nv4LxKSFoWLlBUDlMlxlxDyWNEL+Gr0u54v20M37ZPR8wfnIEJ2Oy kLPjj0cc1Y3dFCQhBgLxP+iY7gx59Ue46LNzxmfqBYQ4j9-93G0W4Xf6HXmLvvcFJtb9aN6JzX6N HA31eXWnZsIKYRO8rJEqMWtXK8wTzjOZKI+zeyVZqBT2ID+mAfoSAMsp1ji9Wn0gyAIJ-RF6Dy6U 3K9FgbBIS+StmsQuGGnzUSgCQfKTKWltkNpfhmNmXdtZ9A-xmeNxX7Sl9QCfD2GdsvntCTFz-X62 s1Pvs8MyrZLPAf6OSpry9oU4lbK5nOU5kyRse5nPddczZB82wYKJUVTRUTP6ZUrD50gz2yQGgIkG pRiYJOcSqTZHgkxTppa9xicxDN1eP6AEHCGyqXLLimx6p+Eh95h3HQDiQjDDb0EEUvQllk0cI1yc PLtDtvsSH41S+P-ZRgaUiVhoToiLWiy4sXU879qTrO38nlUuLKxk9IO7IS3k1hQbEg== -----END SEALED ENVELOPE----- ] `

The 5 in query[5..] implies kXR_Qspace xrootd-spec

Xavier reported still seeing:

` Error during decrypting/server-side key exchange: {} javax.crypto.BadPaddingException?: pad block corrupted `

Jens reported seeing error (with dCache v3.2)

` piggyDomain.log.2018-08-28.gz:Aug 27 11:58:47 piggy dcache@piggyDomain[1383]: javax.crypto.BadPaddingException?: pad block corrupted `

Jens reported seeing error (with dCache v4.2)

` piggyDomain.log.2018-09-14.gz:14 Sep 2018 10:51:10 (Xrootd-srm-piggy) [door:Xrootd-srm-piggy@piggyDomain:AAV10OzU5YA] Error during decrypting/server-side key exchange: pad block corrupted `

Support tickets for discussion

[Items are added here automagically]


Same time, next week.