[clue-tech] NFS problems anyone?

[Angelo Bertolli]

On Tue, Dec 2, 2008 at 10:37 PM, Collins Richey <crichey at gmail.com> wrote:

> We've had a long day debugging an NFS problem, and we're nowhere near
> figuring it out. Here's the setup.
>
> We have a number of RHEL3/RHEL4 servers and workstations that share
> clearcase data stored on three separate servers mounted via NFS. The
> RHEL4 servers are at varying maintenance levels (U3, U4, U7), and
> upgrading the problem servers to U7 does not help. The three problem
> servers (A, B, C) each have a large directory (D1 on A, D2 on B, and
> D3 on C). A mounts D2 and D3, B mounts D1 and D3, and C mounts D1 and
> D2). All workstations and several other servers (RHEL3/4/5) mount D1,
> D2, and D3.



Ok, so you have a bunch of crossmounts.  The version of NFS should be what
is important to maintain correct communication in the protocol, unless you
think there's a bug somewhere.  (Which is certainly quite possible.)


The problem appeared this morning when I needed to restart NFS on a
> related server to change the mount status of it's directories (let's
> call this server D and directory D4. This server does not share D1,
> D2, or D3, but all of the servers that mount D1, D2, D3 also mount D4.
> As soon as I restarted NFS on D and attempted to restart netfs on
> another server (E), E refused to mount D1, D2, D3 claiming that server
> refused due to permissions. All other servers and work stations
> continue to use the NFS mounted D1, D2, D3, D4 continue to work with
> no errors. Any system that is booted or netfs restarted, declines to
> mount D1, D2, D3, but D4 mounts OK. In every case the exporting server
> produces the same message whether the mount is successful or failed.
> If new exports are created on A, B, or C and NFS restarted, these
> exported directories also fail to mount anywhere.


Does exportfs report the expected exports?  Would an exportfs -r help?
After attempting to mount, does showmount indicate things are mounted or
not?  Can you at least do an nfs mount to localhost?



> No changes to the exports or software on any of these servers in the
> past few months. NFS on other servers and workstations continues to
> work without a hitch. Each of the problem servers has been rebooted,
> but that has no effect. Firewalls have been stopped, so that is not
> the answer.


And you don't have any weird network setup with ipvsadm, etc. to be worried
about?



> Does anyone have any clues or helpful hints about interpreting strace
> data for the problem?. From the client side (mount request), after a
> lot of setup, the client opens a TCP socket and passes the requesting
> ip address / name to the server, gets a response, opens a UDP socket,
> passes the address/name again, and gets a response. The last response
> is different for the successful / unsuccessful case, but I haven't
> been able to find a way of interpreting this. If successful, the
> actual mount command is issued and mtab is updated. If unsuccessful,
> no mount is issued and the error message is generated.
>
> I've also straced rpc.mountd on the exporting server, but I know even
> less about this.
>
> Another quirk. In both cases the strace logs an attempt to locate a
> program /sbin/mount.nfs which does not exist.
>
> A mystery within a riddle wrapped in an enigma.
>
> The only relevant google entry we found suggested that reboot cured the
> problem.


It sounds like the problem is on the the nfsd side.  I would have suspected
a network issue, but since you seem to know that the server is sending back
a deny response, you know that it's not a case.  I would probably
investigate the machine to try to figure out why it might be denying
connections.  Could it be something as simple as nfsd having reached its max
number of connections?

Angelo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://cluedenver.org/pipermail/clue-tech/attachments/20081203/10575b3a/attachment.html