dCache Tier I meeting June 21, 2011

[part of a series of meetings]

Present

dCache.org(Tigran, Tanja, Antje, Paul, Gerd), NDGF(Gerd), PIC(Gerard)

Agenda

(see box on the other side)

Site reports

PIC

Gerard reported that everything is running OK.

PIC are planning upgrade their systems next month: July 17th. Gerard is considering upgrading their Tier-1 facility to the new Golden release (1.9.12) this August; but this is only a provisional date; there may be external factors that delay the upgrade until September.

Gerard asked two questions.

Migration

Previously, Gerard asked about migrating support for a VO from their Tier-1 dCache instance into a separate dCache instance; the old instance has PNFS and the new instance would have Chimera.

The procedure was to do a PNFS-to-Chimera migration for the limited sub-tree with the VO's files and reassign those pools with the VO's data to the new instance.

Gerard mentioned that he also needs to move some space-accounting information. In the Tier-1 instance, the two VOs share the same linkgroup and have reservations.

After moving the VO to the new dCache instance, the VO should be the same SRM space reservations as was in the Tier-1 instance: space-manager should know that the space contains the same files. Note that all VO files will be moved, so the different metrics of the VO space (Total, Used, Free, Reserved, etc) will remain constant over the move. There's no requirement on maintaining the same token ID; that can change during the procedure.

His question is whether this is possible?

It sounds possible, but we would need to check. Please submit a ticket (support@…) and Dmitry (our space-manager expert) will have a look.

Click:  RT 6423: Migrating a VO from a PNFS dCache to a Chimera dCache, with the SpaceReservations

Delaying flush-to-tape

Gerard has had a feature request from CMS. They are interested in knowing whether dCache can support the following user story:

A user transfers data into dCache as disk-only data. They do this because they are not sure if the data is "good" and the process of determining whether the data is good takes some time.

After some time has elapsed, it has been established whether the data is good or bad.

If the data is bad then it is deleted.

If the data is good then the file is copied onto tape. The data is no longer needed on disk (a cache copy may be kept).

CMS are having a meeting 23rd and 24th of this week to discuss this new model, amongst other things.

Note that CMS currently don't use the space-manager.

Gerd added that CMS currently have two copies of the file: on-disk and on-tape. They manually copy the file. This triggers an internal-transfer for the file. For NDGF this doesn't hurt as all pools are HSM attached; however, for sites with limited pool-HSM connectivity, there would be an internal transfer.

Gerard noted that LHCb did something similar by copying the files from a D1T0 space to a T1D0. This resulted in high IO load for a few days. His concern is that, if CMS did similar, then the procedure wouldn't scale to the size of data that CMS would want to manipulate.

For the SRM part: the SRM protocol has this kind of operation .. it's just not implemented in dCache.

There was a ticket requesting this, but Gerard couldn't find it. This is likely because the ticket was moved from the bug queue into the feature-request queue.

It wasn't sure how quickly CMS would like information on this idea. Gerard would investigate.

NDGF

Gerd reported that NDGF have suffered two UPS outages, at two different sites, at the same time. This lead to a non-trivial amount of offline data for 6 hours (so far). dCache has been quite fine with this.

Other than this, everything is continuing as normal.

KIT

via email

From GridKa we have nothing to report.

However, she also had a question:

At the end of last week we had a problem concerning an LHCb read pool which was located
on a host together with 10 other pools. We stopped the pool due to memory problems and
started a new read pool on a different host as replacement. But at the weekend dCache tried
to unpin files located on the stopped pool.

How can we stop dCache from unpinning a file located in a stopped pool?

Gerd replied with some information:

The normal procedure is to use the migration module when decommissioning a pool. This will take care of any pins on files in the pool. Naturally, the migration module will only work if the pool is online, which isn't the case here.

As it stands, dCache has no way of unregistering a pool in pnfs-manager unless the pool is online. We have the same problem with pin-manager.

In 1.9.5, the pin-manager tends to go a bit crazy if a pool is offline when it attempts to unpin a file. It can be very aggressive in retrying the unpin operation.

This becomes a particular problem if the pool is stuck. The unpin messages are routed through the dCacheDomain on their way to the pool. If the pool doesn't accept these messages then they can build up in the dCacheDomain. If enough messages are sent then the dCacheDomain will run out of memory, triggering a restart of the dCacheDomain.

The JMS communication implementations (an alternative to the traditional cells-based communication) have support for flow-control. If a sender (such as pin-manager) is sending messages too quickly then it will be throttled.

(Without having spend time investigating the situation fully) Gerd thought that the only solution would be to shutdown pin-manager, clean up the pin-manager database, then restart pin-manager.

The new golden release (1.9.12) includes a new pin-manager. This should be less aggressive at retrying if a pool is offline when a pin should be unpinned.

We can look at adding support to the new pin-manager for registering that a pool is now permanently unavailable; we won't do this for the old pin-manager in 1.9.5.

A related issue is that cells messaging doesn't protect against this kind of memory-hog activity, which can trigger an out-of-memory exception. We will discuss whether we can do something about this during tomorrows dev. meeting.

Support tickets for discussion

[Items are added here automagically]

DTNM

Same time, next week.