Setting up the Chimera namespace for dCache
Scope
This page describes the setup of fresh Chimera installation in combination with a basic dCache instance. Chimera is the replacement and successor of the PNFS namespace provider, offering a file system view to dCache's data files. If you are interested in the advantages of Chimera over PNFS or simply more details, look here [1][2].
Requirements
1. dCache version >= 1.8.0-15
2. JDK >= 1.5.0
3. JDBC interface to RDBMS (tested with PostgreSQL 8.x, Oracle 10g and DB2 v9.1). The JDBC-driver for PostgreSQL is already included in the dCache package.
Tigran's todo list for Chimera
1. ftp doors decouple from mounted files system ( ls operations ) ticket:14
2. billing DB uses SQL built in CREATE statements, which creates tables for short ( old ) pnfsids.
3. make dcache.conf configurable without nfs interface
Installation
1. Install dcache-server RPM
Edit /opt/d-cache/etc/psql_install_companion.sql and replace the 2nd line with ' pnfsid CHAR(36) NOT NULL,' if you want to use companion. Nevertheless, we recommend to use built in into chimera 'companion' to make use of cleaner (explained later on).
2. Initialize the database (PostgreSQL 8.x in this example)
Get the latest (stable) PostgreSQL-version for your operating system from the Postgres-Homepage. As a minimum you need to grab the packages postgresql, postgresql-libs and postgresql-server. Please find here further information on how to setup the basic Postgres service. Further PostgreSQL performance tuning tips.
Then create the Chimera DB and add the Chimera-specific tables and stored procedures (as user postgres):
createdb chimera psql chimera < /opt/d-cache/libexec/chimera/sql/create.sql createlang plpgsql chimera psql chimera < /opt/d-cache/libexec/chimera/sql/pgsql-procedures.sql
for Oracle, do instead in sqlplus:
SQL> @/opt/d-cache/libexec/chimera/sql/create-oracle.sql
Database connection settings can be customized in /opt/d-cache/config/chimera-config.xml ( though the default values should be ok for most cases).
In principle, Chimera is now ready to serve namespace requests for dCache.
4. Configure dCache
Please execute the steps needed to create a new dCache instance as explained in the Book Section 1.3 + 1.5 (avoid of course the pnfs server Setup in section 1.4). Do not execute /opt/d-cache/install/install.sh nor the dCache startup scripts (/opt/d-cache/bin/dcache-core,/opt/d-cache/bin/dcache-pool)yet.
To make dCache Chimera-aware, you have to further customize your dCacheSetup and node_config file:
add the following lines:
/opt/d-cache/config/dCacheSetup
PermissionHandlerDataSource=diskCacheV111.services.PnfsManagerFileMetaDataSource
let Chimera to store file locations ( cleaner depends on it ):
cacheInfo=pnfs
check that classpath is not redefined later on
/opt/d-cache/etc/node_config
NAMESPACE=chimera
This will force the chimeraDomain to start instead of the pnfsDomain ( actually, it's the same PnfsManager, but configured to use Chimera as default name, storage and location provider).
5. The NFS v3 interface of Chimera
The inner dCache components are talking to the namespace via a module called PnfsManager, which in turn communicates with the Chimera database using a thin JAVA layer. In addition, Chimera provides a NFS V3 server interface, such that clients can NFS-mount the namespace locally. This offers the opportunity to use OS-level tools like ls, mkdir for Chimera. However, direct I/O-operations like cp or mv are not possible, since the NFSV3 interface provides the namespace part only (if you want transparent I/O on the exported namesspace, try preloading the dCap-Client-library).
Currently, dCache still need a mounted Chimera on nodes where the SRM- or the GsiFtp-door is running. This mount is then used as a shortcut for ls-type operations (via SRM or GsiFtp?). Note that this legacy technique will soon be replaced by direct access to the Chimera DB, making the mounts obsolete. Pools don't need a mount anymore.
Mounting Chimera can be a requirement by some applications (e.g. on the workernodes) or users to provide a familiar view to the namespace, especially in combination with DCap. It's also a good idea to mount it on the same node where Chimera/NFSv3 server runs to ease the initial setup and to maintain or observe the file sytem usage later on in production.
There is already an v4-beta interface available. Ask us if you are interested.
6. Configuring the NFS v3 server
on some linux distributions you have to switch off portmap daemon : /etc/init.d/portmap stop
Chimera NFS server uses /etc/exports file to manage exports. The typical export file looks like:
/ localhost(rw) /pnfs # or # /pnfs *.my.domain(rw)
Start it via script
$ /opt/d-cache/libexec/chimera/chimera-nfs-run.sh start
To automate the launching of that script at startup time, copy it to /etc/init.d/. Then announce it to chkconfig:
chkconfig --add chimera-nfs-run.sh chkconfig chimera-nfs-run.sh on
To configure the server, we need to mount the administrative view of Chimera (localhost:/) locally (remote mounts are not allowed yet). This will allow us to create the main Chimera path.
$ mount localhost:/ /mnt
First we create the root of the Chimera namespace, called 'pnfs' for legacy reasons.
$ mkdir -p /mnt/pnfs/<your domain> # mkdir -p /mnt/pnfs/desy.de
Now we need to add directory tags (=metadata, which will be evaluated by dCache and inherited by future subdirectories):
$ mkdir -p /mnt/pnfs/<your domain>/data $ echo "chimera" > /mnt/pnfs/<your domain>/data/'.(tag)(sGroup)' $ echo "StoreName sql" > /mnt/pnfs/<your domain>/data/'.(tag)(OSMTemplate)'
If you plan to use DCap with mounted file system instead of the URL-syntax (e.g. dccp /pnfs/desy.de/data/file1 /tmp/file1), execute:
$ mkdir /mnt/admin/etc/config/dCache $ touch /mnt/admin/etc/config/dCache/dcache.conf $ touch /mnt/admin/etc/config/dCache/'.(fset)(dcache.conf)(io)(on)' $ echo "<door host>:<port>" > /mnt/admin/etc/config/dCache/dcache.conf # echo "dcap.desy.de:22125" > /mnt/admin/etc/config/dCache/dcache.conf
The configuration is done now, so unmount NFSv3:
$ umount /mnt
Please note that whenever you need to change the configuration, you have to remount the admin-view (localhost:/) to a temporary location like /mnt.
7. Mounting Chimera/NFS on the clients
For production the exported clients (as configured above) mount the 'user's view' via
$ mkdir /pnfs $ mount localhost:/pnfs /pnfs
Remember that you need a mount at least on the SRM-node, the GsiFTP-node and on all clients which want to do Dcap without URLs. Pools don't need a mount anymore. Having a mount on the Chimera/NFS3-server node itself is always a good idea in order to maintain the file system.
8. Final steps
Launch the install-script of dCache:
$ /opt/d-cache/install/install.sh
start the dCache services:
$ /opt/d-cache/bin/dcache-core start $ /opt/d-cache/bin/dcache-pool start
9. Confirm that it's working
To get the confidence that the PnfsManager with Chimera-support is up and running, connect to your dCache headnode via ssh:
$ ssh -c blowfish -p 22223 -l admin <yourAdminNode> [<yourHostName.yourDomain>] (local) admin > cd PnfsManager [<yourHostName.yourDomain>](PnfsManager) admin > info $Revision: 8088 $ NameSpace Provider: $Id: ChimeraNameSpaceProvider.java,v 1.7 2007-10-01 12:28:03 tigran Exp $ DB : jdbc:postgresql://localhost/chimera?prepareThreshold=3 Drv : org.postgresql.Driver DB Engine : PostgreSQL 8.2.5 rootID : 000000000000000000000000000000000000 wormID : 80D1B8B90CED30430608C58002811B3285FC FsId : 0 CacheLocation Provider: $Id: ChimeraNameSpaceProvider.java,v 1.7 2007-10-01 12:28:03 tigran Exp $ DB : jdbc:postgresql://localhost/chimera?prepareThreshold=3 Drv : org.postgresql.Driver DB Engine : PostgreSQL 8.2.5 rootID : 000000000000000000000000000000000000 wormID : 80D1B8B90CED30430608C58002811B3285FC FsId : 0 StorageInfo Provider: $Id: ChimeraNameSpaceProvider.java,v 1.7 2007-10-01 12:28:03 tigran Exp $ DB : jdbc:postgresql://localhost/chimera?prepareThreshold=3 Drv : org.postgresql.Driver DB Engine : PostgreSQL 8.2.5 rootID : 000000000000000000000000000000000000 wormID : 80D1B8B90CED30430608C58002811B3285FC FsId : 0
If you see a similar output of the 'info' command, Chimera is fine for dCache.
Now you can write a file into dCache as a quick test (on the Chimera node) under the assumption, that the DCap door is registered as described above:
$ dccp /bin/sh /pnfs/<YourDomain>/data/file1 541096 bytes in 0 seconds
Enabling Cleaner on existing installation
1. enable storing of cacheInfo in chimera:
cacheInfo=pnfs # yes, pnfs :)
2. restart PnfsManager
/opt/d-cache/jobs/chimera stop /opt/d-cache/jobs/chimera start
3. connect to dCache via admin interface and for each pool type:
pnfs register
Alternatively, you can import existing companion into Chimera:
1. dump cacheinfo table:
pg_dump -U postgres -t cacheinfo companion | psql -U postgres chimera
2. connect to chimera database and populate t_locationinfo table:
echo "INSERT INTO t_locationinfo select pnfsid, 1, pool, 10, now(), now(), 1 from cacheinfo;" | psql -U postgres chimera
3. drop unneeded cacheinfo table:
echo "drop table cacheinfo;" | psql -U postgres chimera
4. problems or better solutions?
support@dcache.org
Migration from existing pnfs-based installation
The migration of existing pnfs-based installation divided into two parts: name space migration and pool data migration.
- Name space Migration
- keep pnfs running
- create mapping table
psql chimera < /opt/d-cache/libexec/chimera/sql/pnfsid-mapping.sql
- on pnfs host install dcache-server-XXX.rpm
- configure dCacheSetup ( chimera needs java path )
- run copy script
/opt/d-cache/libexec/chimera/pnfs2chimera.sh /pnfs/desy.de /pnfs/desy.de
where fist agrument your local pnfs-path and second argument chimera-path
- Pools data Migration
- Follow steps 1-3 from Name Space Migration
- run conversion script for each pool directory:
/opt/d-cache/libexec/chimera/pool2chimera.sh <path to pool>
Last
Please send feedback to support@dcache.org (Prefix 'Chimera' in the subject field)
