wiki:DevelopersWorkshop2012
Last modified 6 years ago Last modified on 04/25/12 10:46:45

Agenda

Friday Topic Responsible
09:00-10:00 Topics from workshop Patrick
10:00-13:00 Discussion of dCache's future development direction. Tigran
13:00-14:00 Lunch Break
14:00-16:00 Discussion of dCache's future development direction (cont'd).
16:00-18:00 Auth (e.g. jGlobus2, Federated Identity) Paul
Monday
09:00-11:00 Auth (cont'd)
11:00-13:00 Improving the Admin interface Christian
13:00-14:00 Lunch Break
14:00-15:45 Admin interface (cont'd)
16:00-17:00 Presentation at Scientific Computing Seminar Gerd and Dmitry
17:15-18:00 Scrum Paul and Gerd
Tuesday
09:00-12:00 Testing Karsten
12:00-12:15 Message passing Gerd
12:15-13:00 Collect PR relevant ideas
13:00-14:00 Lunch Break
14:00-18:00 Mutable files Gerd
Wednesday
09:00-11:00 Final session Patrick
11:00- Open end social hacking

Promises made at User Workshop

We have to explain the mapping to Hero. Gerd: Will do an archtype on the weekend.

xrootd provides a proxy, for cases where this is not an option, this would be nice for us. Gerd: if this is just a proxy component we should be able to use that one.

Christina needs srmbind, poolbind, change name of loginBroker configurable on dcache.conf. Gerd: LoginBroker? paramters are easy to expose Patrick: This is very important for us, since it is a dCache outside of the usual user group.

Can we do this with a Bugfix Release?

Gerd: Yes. Tigran: Will take care of this.

Paul talked to Nicolas: He wants to limit how much tapespace a VO has. Might more like a fun thing. Gerd: This should be easy to do with chimera's quota approach. Tigran: This is our original approach. Gerd: Should be easy to do, but it needs to be paramerised.

Gerd: FTS GridFTP issue seems important. People are starting to look for workarounds. Paul: Bug report from GGUS is assigned to the wrong people. Patrick: Will take care of this.

People want to modify files.

Hero did a evaluation of filesystems. dCache came out the best in this evaluation in flexibility and performance. But at some point they cannot accept anymore (only) immutable files.

Patrick: Another thing, two people have been asking for a chat room where dCache people have an eye on. Gerd: I set this up an year ago P: We need a desription for this. G/T: Maybe Google+ Hangouts are a good alternative. Additional we need want a permanent chat room. Hangouts rather periodically. NorduGrid? has a public chatroom. P: We want to evaluate Hangout before using.

Tanja: People would like to have public bug queues to scan for problems and solutions. Gerd: StackOverflow? might also be a good idea.

Patrick: We will be the Storage Element for the russian Tier1 in Dubna.

Patrick already got some positive feedback from the workshop.

Monitoring

Nagios probes and reporting. Should we have a RESTful service to make it easy for our customers to create plugins for probes easily? Gerd: We can easily expose much information by spring bean properties.

  1. We have to add the getters to the beans.

Al is working on event-based reporting.

  1. We need a nice way how to present information requested by cell messages.

How do we do uniform access?

1. The future of dCache

Our goal is to keep getting funding and to stay in business.

What is our vision? Get rid of propriety solutions. Patrick: We have visions... Gerd: Maybe visions is the wrong expression. It seems we rather need concrete goals, we can then work on as a team. Paul: Sometimes we as developers don't know what issues are interesting for reaching Patricks goals. Gerd: Maybe we should make a SWOT Analysis during the workshop with respect to Patricks vision. SWOT postponed for after lunch.

SWOT

Online Storage

We want an online storage for students to store files and to play with at home to get them to know and to work on dCache.

Auth

Whats missing from gPlazma?

  • SAML
  • Voms Server
  • Federated Identity support

Change file formats for local files.

-> We need to get rid of vorolemap and grid-mapfile. Paul: because they're for a broken case. Gerd: They work for old usecases. For new ones they still work fine

-> How to we support people migrating from group accounts to single users? Gerd: We should first add the functionality to support individual users.

Federated Identity: Two doors will probably support this: WebAdmin? + WebDAV. The approach for both would be the same. We will need to plugin an "Log me in" Button. At NDGF all users (also admins) are part of a federated identity system.

How do we solve the id-generation for the different login ways? User gets in, Id gets created and then another plugin might deny access.

  • The first mentioning of a new user triggers the creation of the new id.

--- Tigran: jGlobus2

Status is: We need JGlobus2 because it supports SHA2, but we cannot because it doesn't support legacy proxies.

And we need legacy proxies because some tools still don't support RFC proxies.

Patrick: There are only 2 experiments left, but they will switch soon.

Paul: jGlobus does not support eMail addresses in the DN, but 3 CAs are using this. Fixing jglobus is hard because it relies on Java classes.

Christian: Fixing these 3 CAs is on the Agenda for 2012.

Paul: There is a SHA1 support team, currently creating a risk-analysis document. Paul is part of this. He has access to information how much effort is put into this by the CAs.

--

Dmitry has a patch to migrate to jGlobus, by changing the pom.xml.

Paul: Do we have a long term plan to get rid of jGlobus dependency? Gerd: SRM uses it, GSI needs it. Paul: We could take the needed parts from jGlobus and create a wrapper for GSI to use SSL below. Gerd: We could also maintain jGlobus for our purposes instead of rewriting stuff. Patrick: We will check and then contribute and maintain jGlobus to our needs.

Admin Interface

How to improve the Admin Interface CLI?

Christian asked people what they think the Admin Interface needs.

Scope: SSH1, SSH2, WebAdmin?

Gerd: The new WebAdmin? one is lacking the ability to do mapping to provide ability to publish files under specific URLs. We should extend the httpg servelet with a "add servelet" to plugin the new one.

Christian: Gerd wanted to have a simulated Terminal emulation on the WebPage?. Gerd: There are some really nice libraries that support this.

Paul implemented different views

---> We want a common library for all shells.

Testing

We agree that it would be VERY desireable to

  • Write tests BDD style
  • Use Mockito
  • Add tests to every patch, (especially bug fixes)
  • analyse slow tests to check if they can be speed up

We define JUnit categories for different test types:

We want to be be able to run the functional (System-Tests) locally

  • for that we need a runnable dCache in the building phase
  • S2 in the repository

Message Passing

JMS is implemented for some time. We need sites to test that message passing scales as good as or better than cells.

Dmitry and Al are going to switch it on for testing.

Advantages of JMS are:

  • Makes implementation of redundancy and load balancing easier

Small Files and mutable files

Mutable files:

  1. Single exclusive writer
  2. Single copy is modified, other copies expire and get garbage collected
  3. Subsequent opens block untilwriter closes
  4. Existing open-for-reads on other pools are allowed to continue until closed, but will only see the unmodified content
  5. Writer may block on open for existing readers to finish or for replication
  6. write happens only on write enabled pool
  7. After close file state is reset to state of newly written file
  8. PNFS ID of file changes as side effect (possibly only a version suffix)

Small Files:

  1. Spontaneous stages after pool are not visible to pool and cause a hidden file.
  2. Per file script invocation overhead too high (1000 bash instances = 4000 threads in the pool)
  3. Bulk stage limited to available space on single pool, even though streaming transfer to cache pools would allow continous staging (this thus limits the tape systems ability to reorder)
  4. dCache unaware of any file grouping done in tape system on HSM script and thus unable to use this info when staing files.
  5. HSM driver unable to register extra files on stage

PR

  • pretty website
  • a "Why should I use dCache"-document for different user groups
  • dCache commercial video (e.g., on youtube, website) possibly as cartoon.
  • dCache (tutorial) videos (e.g., on youtube)
  • "public" storage for DESY and maybe University employees - DESY-cloud
  • StackExchange? Q&A site
  • Presenting dCache at other community conferences
  • Get into distros
  • Offer downloadable VM with preconfigured dCache
  • A good slogan: ""
  • present dCache in universities (CS+physics)
  • dCache Business Cards
  • dCache Presents (Ball-Pens)