wiki:Multihomed-Servers
Last modified 6 years ago Last modified on 11/24/11 11:27:39

Multihomed dCache Servers

Disclaimer :
We don't have any knowledge about the internals of dCache, so the described procedure here may break down anytime.

This document is about how to make clients choose the appropriate network interface of a server. It is based on the assumption that the DNS-name of the dCache-servers is used by the client to determine the ip-address and never uses a direct ip-address got from a server by some kind of command-channel or so.

If the dCache-server has more than one interface, then 2 things have to be considered :

  • The client has to choose the desired interface
  • The server has to use the correct interface for the reply.

The motivation in our case was that we are part of the NDGF-distributed dCache installation. (picture follows) During that installation we got a optical private networklink (OPN) from Finland to Denmark together with "danish" IP-addresses from NORDUnet. Beside the Nordunet-addresses the servers still have the FUNET-addresses and private addresses for in-house communication with our CE.

Look at our srm-door "madhatter.csc.fi" as an example. It has now 3 addresses :

network IP-adress targeted clients
private 10.12.10.2 in-house
FUNET 193.166.3.66 finnish
NORDUnet 193.10.122.130 rest of the world

The challenge now is to make a client pick the right IP-address depending on its location, which is done by by the DNS-query. Since we have no influence on clients outside our house or Finland, the official DNS-entry of our srm-door is the NORDUnet IP as well as the default gateway for the machines is now in Denmark.

The other two IP-adresses are advertised by a specialized DNS-server on our dCache-admin node. This DNS server is not public and allows only queries from authorized clients.

Client setup

The compute-nodes the in-house CE are then using a /etc/resolv.conf like :

     nameserver 10.20.0.1
     nameserver ...
     sortlist 10.0.0.0/255.0.0.0

so that they choose the private interface.

client in closer to Helsinki than Denmark should use :

     nameserver 193.166.3.65
     nameserver ...
     sortlist  193.166.3.0/255.255.255.0

Routing

One problem left on the server side is the routing of outgoing packets. The default-gateway is the NORDUnet-ifce, but there are also static routes for "finnish" IP-addresses (not very many at the minute). So if some machines opens a connection to the FUNET-ifce, then the outgoing reply must originate from the FUNET-ifce, otherwise some firwalls wouldn't like it and maybe the client itself getting a reply from an IP it didn't send a request to. This is called source-based routing and has to be setup during boot-time.

dirty deeds done…

Problem: Our DNS-server then has to give for *.csc.fi machines the official answer, but the dcache-servers. Instead of copying all the *.csc.fi entries, the DNS-server just relays queries to csc.fi to the offical DNS-server. (Has no csc.fi-zone-file). The custom replies for the dcache-servers are done by having a "zone" for each server with its FQDN :

madhatter.csc.fi
$TTL 86400
@       IN      SOA     duchess.csc.fi. root.duchess.csc.fi. (
                        2006030832
                        3600
                        1800
                        604800
                        86400 )
        IN      NS      duchess.csc.fi.

@ IN A   10.12.10.2
@ IN A   193.166.3.66