Table of Contents
[part of a series of meetings]
Participants
Gerd; Antje, Paul, Karsten; Dmitry
Agenda
[see box on the right-hand side]
Postcards
Up to two minutes (uninterrupted) per person where they can answer two questions:
- What I did last week (since the last meeting),
- What I plan to do in the next week.
No questions until we get through everyone :)
Karsten: mostly on gPlazma work; and a little on the gPlazma in the Book + WLCG poster.
Tanja: poster; tickets; book
Gerd: some writing on doors; overview of dCache; production upgrades to FHS-dCache. Bug fixing; debugging of ARC in EMI release. Reading up about OSGi and ordered a nice book on this (a couple of free chapters available). Also looking at JSVC (see below). Also worked on connection pools, replacing 3CPO and DBCP with BoneCP.
Dmitry: interview for dCache developer position; Patrick visited with nice dinner. CDF reporting strange patterns in their restores; two restores (which turned out to be a false alarm). New pools moved into production from
Minas is exp. without much data. 500 GiB tape but lots of small files. They prefer to pin files; Dmitry described how to pin files via SRM. Looking at the scheduler code for the future SRM.
Antje: documentation .. problems with desktop computer :-(
Paul: Reviewing one the EMI deliverable documents; tickets, bug-fixing.
Tigran: working on script for splitting Chimera; ready works and tested. Introducing identity plugin for dCache/gPlazma. Adding ACL supports for NFS. Full-chain has been verified: NFS client --> server --> dCache --> gPlazma --> NIS (and back again). About 15 patches, 3 in RB at the moment.
Plans for patch-releases
Should we make a new patch release?
Pressure to remove legacy ... from info-provider.
Trunk activity
Progress with new features...
Connection pool
Gerd: only try this on NDGF to try it out.
Tigran: no problem with BoneCP. 3CP0 had nice things like JMX to monitor queues and checking for deadlocks. At least the JMX is in BoneCP.
The proposed changed:
is to remove all connection pools (but not the SRM custom one).
The SRM one doesn't use any existing pool-manager but has a custom one.
Space-manager uses DBCP.
What is the motivation? Gerd: it's messy having two connection pool.
It seems that ~6 months ago development started in DBCP.
No activity may mean the code is mature. Tigran: the lack of use of new concurrency classes suggests that they had abandoned the project; or perhaps it was retaining compatibility with old Java versions.
Gerd: plan to deploy BoneCP at NDGF to gain experience in real-world.
If the patch modifies components ConnectionPool? to DataSource?, could that be a separate patch?
There's no particular fix for this in the proposed patch; so nothing to commit. It just adds support for using BoneCP.
JSVC
This is an old project that introduces an alternative way to start a Java program. It's specifically meant for making daemons. There is a daemon interface: init, start, stop, destroy. There is a small program that recognises
Has all the functionality of our daemon script: PID files, change user-ID, shutting down cleanly, monitors app and restarts on crash, etc.
The init is called as privileged user; drop privs after and calls start method. This allows opening root ports.
We get a restart loop if a fatal error is not temporary;
There's a binary part for Windows that allows the code to start as a normal Windows service.
Gerd has a version that uses either JSVC or traditional daemon script.
Unfortunately, due to the change in the restart strategy, the changes are intrusive, which would require
We need to be root to start dCache. We should be able to start dCache as user dcache.
Gerd: this should be supported now.
We wouldn't include JSVC with dCache as it is a native binary.
Most distributions already have it (Ubuntu...).
Reacting to SININT for log-rotation.
Paul: concerned that this will be another dimension to test: test all different combinations with JSVC and also without.
One solution would be to simply drop the daemon script and require JSVC.
We would need to support SL5: is JSVC available for this platform? Seems not: it isn't in EPEL.
We would need to package it for SL5 and SL6.
Does tomcat use it? Yes. It was taken out of tomcat.
OGSi can fit inside JSVC.
Tigran: if there is a supported package for SL5 and SL6 then OK; otherwise, no.
JSVC should work on Solaris machines.
We would like to keep the daemon script to keep our Just-Needs-Java policy.
Do you know how Glassfish does it? This is OGSi, too, and uses Felix as the OGSi implementation. There's JSVC support for Apache Felix, which is the OSGi implementation for glassfish.
Issues from yesterday's Tier-1 meeting
Issues from EMI
Mavin
Question to Christian:
Can you ask EMI to provide a Maven repository for all EMI built JAR files: trust-manager, voms, ..
The problem is that EMI is only for a few years. What after EMI finishes? Perhaps they push it to the main Maven repo.
Nagios
Question to the team: EMI is planning to implement a Nagios based mechanism that controls whether a service (in our case dCache) is functional.
- What options do you see to publish such information from within dCache: GLUE? Grid Nagios probes?
We would provide scripts that provide the information; the interface into dCache is an internal decision.
- Do we provide everything that is needed in order to provide such monitoring capability?
There is nothing to say what they want to monitor! Please provide an example of what they want to monitor.
- What is missing?
We need a complete set of information that we want to monitor.
First question: what do you probe? Could it be as simple as checking whether dCache has been started?
At DESY we have tonnes of ssh scripts to do this.
Parsing the web pages and ssh scripts isn't the right approach.
GPFS modifying the files (non-immutable files) and no monitoring.
We don't know what they need to know. Need more feedback from the user-requirement.
The problem is we need to provide a script but currently don't.
If we (dCache.org) provide Nagios probes then we can use what we want without
We should provide _dCache probes_, not Nagios probes. This would allow people to use them with Nagios and other monitoring infrastructure.
How these probes work internally is ugly but isn't too big a deal as we take care of compatibility.
The problem isn't parsing HTML or SSH stuff; the problem is that our sys-admins have to parse HTML or use SSH and when we update something we risk breaking their scripts.
Not having a perfect interface shouldn't prevent us from providing something that works with what we have.
File lifetime on a pool: looking at how old replicas is on the file.
We would like to have passive monitoring (check metrics) and active monitoring (probing).
Full-chain is the only way to be sure; but lighter-
Dmitry to collect what monitoring they do at the public dCache instance and ask Jon what he does for US-CMS instance.
CDF may choose to move to Nagios; they have their own custom monitoring in currently.
Outstanding RT Tickets
[This is an auto-generated item. Don't add items here directly]
RT 6343: p2p copy/move
This is a feature request: do we accept it?
This is p2p doing a "move" instead of creating a cache-copy.
This would move the file that was accessed next.
ATLAS read files only the once.
Should disable p2p-on-cost if you know that files are being accessed once.
Can we decide if move is the right way to go? Some time yes, some times no.
Can we make this decision on-the-fly?
Gerd: leave the old file as cached; move the
Look at the number of files (in that storage-group) on both pools. The pool with the fewer files will have the sticky flag. This determines if the p2p is a move or a copy.
Below some cost, always choose the minimum cost pool. The default min-cost is 0.
There is a space-token where you write. There are two
Don't know if it's good to move files.
Certainly it's something that needs to be enabled. The default behaviour must be always not move.
This only works if you do p2p within a link-group; otherwise space-management would be confused.
We take the suggestion but it needs more investigation to figure out if it's good.
It sounds like they're trying to solve the problem of files being written to the wrong pools. Tanja to ask Dima whether this is the case.
RT 6351: how to even out restores across multiple pools
Dmitry: I don't think there's a strong reason why a heterogeneous queue sizes? Perhaps due to different age and performance of the hardware.
Suggest making the queue lengths the same.
Create a pool-manager partition containinly only the links involved with staging. Set the cm to select pools randomly. If there is a slow pool then this pool will accumulate transfers over time.
Dmitry's idea
Dmitry has emailed to team about CDF observation. Suggest "pushing" a cached file from a full pool to a non-full pool to make space for the restore. The cached file is a "precious" resource since it would take time to stage the file back from tape.
Why not stage to the pool with the space?
Sounds good, but it would be difficult to implement.
Don't want all restore on the same node as this will become a bottleneck; however, if all HSM-attached pools are moving their file to the same pool then would this create a new bottleneck?
This might allow movement of cache files from HSM-attached to non-HSM-attached pools.
Review of RB requests
In Java, you can't read the last access time. To preserve this information dCache touches the file and reading the mtime instead.
Java 7 allows reading of last-access-time. You use for the Unix view of the file, which might or might not be supplied.
Probably be out before Java-1 ... ~September.
Then it'll be at least two years before we can rely on it.
DTNM
Proposed: same time, next week.
