wiki:ChimeraSetup
Last modified 11 years ago Last modified on 06/02/10 18:16:35

Setting up the Chimera namespace for dCache

Scope

This page describes the setup of fresh Chimera installation in combination with a basic dCache instance. Chimera is the replacement and successor of the PNFS namespace provider, offering a file system view to dCache's data files. If you are interested in the advantages of Chimera over PNFS or simply more details, look here http://www.dcache.org/manuals/workshop2007/tigran-chimera-status.pdf 1http://www.dcache.org/manuals/d-grid/Chimera_goettingen.pdf 2.

Requirements

  1. dCache version >= 1.9.0-4
  2. JDK >= 1.5.0
  3. JDBC interface to RDBMS (tested with PostgreSQL 8.3, Oracle 10g and DB2 v9.1). The JDBC-driver for PostgreSQL is already included in the dCache package.

Note Although Chimera will work with PostgreSQL v8.2, it is recommended to use v8.3 or later. In particular, it seems that PostGreSQL v8.2 is very slow at importing migrated SQL emitted by the pnfsDump utility.

Tigran's todo list for Chimera

  1. ftp doors decouple from mounted files system ( ls operations ) ticket:14
  2. billing DB uses SQL built in CREATE statements, which creates tables for short ( old ) pnfsids.
  3. make dcache.conf configurable without nfs interface

Installation

1. Install dcache-server RPM

Edit /opt/d-cache/etc/psql_install_companion.sql and replace the 2nd line with ' pnfsid CHAR(36) NOT NULL,' if you want to use companion. Nevertheless, we recommend to use built in into chimera 'companion' to make use of cleaner (explained later on).

2. Initialize the database (PostgreSQL 8.x in this example)

Get the latest (stable) PostgreSQL-version for your operating system from the Postgres-Homepage. As a minimum you need to grab the packages postgresql, postgresql-libs and postgresql-server. Please find here further information on how to setup the basic Postgres service. Further PostgreSQL performance tuning tips.

Then create the Chimera DB and add the Chimera-specific tables and stored procedures (as user postgres):

createdb chimera
psql chimera < /opt/d-cache/libexec/chimera/sql/create.sql
createlang plpgsql chimera
psql chimera < /opt/d-cache/libexec/chimera/sql/pgsql-procedures.sql

for Oracle, do instead in sqlplus:

SQL> @/opt/d-cache/libexec/chimera/sql/create-oracle.sql

Database connection settings can be customized in /opt/d-cache/config/chimera-config.xml ( though the default values should be ok for most cases).

In principle, Chimera is now ready to serve namespace requests for dCache.

4. Configure dCache

Please execute the steps needed to create a new dCache instance as explained in the Book Section 1.3 + 1.5 (avoid of course the pnfs server Setup in section 1.4). Do not execute /opt/d-cache/install/install.sh nor the dCache startup scripts (/opt/d-cache/bin/dcache-core,/opt/d-cache/bin/dcache-pool) yet.

For dCache < 1.9.5-x versions you have to further customize your /opt/d-cache/config/dCacheSetup and /opt/d-cache/etc/node_config files.

a) add the following line to dCacheSetup

PermissionHandlerDataSource=diskCacheV111.services.PnfsManagerFileMetaDataSource

b) alter the cacheInfo value in dCacheSetup file (yes, pnfs is the correct value here)

cacheInfo=pnfs

This will inform dCache to store file location information within the namespace rather than in a companion database. The cleaner depends on this.

c) Alter NAMESPACE value in the node_config file.

NAMESPACE=chimera

This will result in chimeraDomain being started instead of pnfsDomain. In both cases (chimeraDomain and pnfsDomain) a PnfsManager cell is started, but with chimeraDomain the PnfsManager cell is configured to use Chimera as the default name, storage and location provider.

5. The NFS v3 interface of Chimera

The inner dCache components are talking to the namespace via a module called PnfsManager, which in turn communicates with the Chimera database using a thin JAVA layer. In addition, Chimera provides a NFS V3 server interface, such that clients can NFS-mount the namespace locally. This offers the opportunity to use OS-level tools like ls, mkdir for Chimera. However, direct I/O-operations like cp or mv are not possible, since the NFSV3 interface provides the namespace part only (if you want transparent I/O on the exported namesspace, try preloading the dCap-Client-library).

Currently, dCache still need a mounted Chimera on nodes where the SRM- or the GsiFtp-door is running. This mount is then used as a shortcut for ls-type operations (via SRM or GsiFtp?). Note that this legacy technique will soon be replaced by direct access to the Chimera DB, making the mounts obsolete. Pools don't need a mount anymore.

Mounting Chimera can be a requirement by some applications (e.g. on the workernodes) or users to provide a familiar view to the namespace, especially in combination with DCap. It's also a good idea to mount it on the same node where Chimera/NFSv3 server runs to ease the initial setup and to maintain or observe the file sytem usage later on in production.

There is already an v4-beta interface available. Ask us if you are interested.

6. Configuring the NFS v3 server

on some linux distributions you have to switch off portmap daemon : /etc/init.d/portmap stop

Chimera NFS server uses /etc/exports file to manage exports. The typical export file looks like:

/ localhost(rw)
/pnfs
# or
# /pnfs *.my.domain(rw)

Start it via script

$ /opt/d-cache/libexec/chimera/chimera-nfs-run.sh start

To automate the launching of that script at startup time, copy it to /etc/init.d/. Then announce it to chkconfig:

chkconfig --add chimera-nfs-run.sh
chkconfig chimera-nfs-run.sh on

To configure the server, we need to mount the administrative view of Chimera (localhost:/) locally (remote mounts are not allowed yet). This will allow us to create the main Chimera path.

$ mount localhost:/ /mnt

First we create the root of the Chimera namespace, called 'pnfs' for legacy reasons.

$ mkdir -p /mnt/pnfs/<your domain>                                       # mkdir -p /mnt/pnfs/desy.de

Now we need to add directory tags (=metadata, which will be evaluated by dCache and inherited by future subdirectories):

$ mkdir -p /mnt/pnfs/<your domain>/data
$ echo "chimera" > /mnt/pnfs/<your domain>/data/'.(tag)(sGroup)'
$ echo "StoreName sql" > /mnt/pnfs/<your domain>/data/'.(tag)(OSMTemplate)'

If you plan to use DCap with mounted file system instead of the URL-syntax (e.g. dccp /pnfs/desy.de/data/file1 /tmp/file1), execute:

$ mkdir /mnt/admin/etc/config/dCache
$ touch /mnt/admin/etc/config/dCache/dcache.conf
$ touch /mnt/admin/etc/config/dCache/'.(fset)(dcache.conf)(io)(on)'
$ echo "<door host>:<port>" > /mnt/admin/etc/config/dCache/dcache.conf               # echo "dcap.desy.de:22125" > /mnt/admin/etc/config/dCache/dcache.conf

The configuration is done now, so unmount NFSv3:

$ umount /mnt

Please note that whenever you need to change the configuration, you have to remount the admin-view (localhost:/) to a temporary location like /mnt.

7. Mounting Chimera/NFS on the clients

For production the exported clients (as configured above) mount the 'user's view' via

$ mkdir /pnfs
$ mount localhost:/pnfs /pnfs

Remember that you need a mount at least on the SRM-node, the GsiFTP-node and on all clients which want to do Dcap without URLs. Pools don't need a mount anymore. Having a mount on the Chimera/NFS3-server node itself is always a good idea in order to maintain the file system.

8. Final steps

Launch the install-script of dCache:

$ /opt/d-cache/install/install.sh

start the dCache services:

$ /opt/d-cache/bin/dcache start

9. Confirm that it's working

To get the confidence that the PnfsManager with Chimera-support is up and running, connect to your dCache headnode via ssh:

$ ssh -c blowfish -p 22223 -l admin <yourAdminNode>

[<yourHostName.yourDomain>] (local) admin > cd PnfsManager
[<yourHostName.yourDomain>](PnfsManager) admin > info

$Revision: 8088 $
NameSpace Provider: 
$Id: ChimeraNameSpaceProvider.java,v 1.7 2007-10-01 12:28:03 tigran Exp $ 

DB        : jdbc:postgresql://localhost/chimera?prepareThreshold=3
Drv       : org.postgresql.Driver
DB Engine : PostgreSQL 8.2.5
rootID    : 000000000000000000000000000000000000
wormID    : 80D1B8B90CED30430608C58002811B3285FC
FsId      : 0

CacheLocation Provider: 
$Id: ChimeraNameSpaceProvider.java,v 1.7 2007-10-01 12:28:03 tigran Exp $ 
DB        : jdbc:postgresql://localhost/chimera?prepareThreshold=3
Drv       : org.postgresql.Driver
DB Engine : PostgreSQL 8.2.5
rootID    : 000000000000000000000000000000000000
wormID    : 80D1B8B90CED30430608C58002811B3285FC
FsId      : 0

StorageInfo Provider: 
$Id: ChimeraNameSpaceProvider.java,v 1.7 2007-10-01 12:28:03 tigran Exp $ 
DB        : jdbc:postgresql://localhost/chimera?prepareThreshold=3
Drv       : org.postgresql.Driver
DB Engine : PostgreSQL 8.2.5
rootID    : 000000000000000000000000000000000000
wormID    : 80D1B8B90CED30430608C58002811B3285FC
FsId      : 0

If you see a similar output of the 'info' command, Chimera is fine for dCache.

Now you can write a file into dCache as a quick test (on the Chimera node) under the assumption, that the DCap door is registered as described above:

$ dccp /bin/sh /pnfs/<YourDomain>/data/file1
541096 bytes in 0 seconds

Migrating data from existing PNFS

Please follow the migration procedure.

Last

Please send feedback to support@dcache.org (Prefix 'Chimera' in the subject field)