Redhat5.4安装pvfs-2.8.2全过程(集群)

1.配置环境   (root用户)

(1) 在两台机器上分别配置/etc/hosts

202.117.10.37   amao991   

202.117.10.50   amao992  

(2) 配置无密码登录(在amao991上)

[root@amao991 ~]# ssh-keygen -t rsa 

[root@amao991 ~]# cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys

[root@amao991 ~]# scp ~/.ssh/*  root@amao992:~/.ssh/

(3) 机器角色分配

amao991作为metadata server

amao991和amao992同时作为I/O server

amao991和amao992同时作为client

2. 安装pvfs-2.8.2 (root用户在amao991和amao992上分配编译安装)

(0)预备

在/root下创建 program/pvfs2-install目录,作为安装目录

soft目录(放置pvfs-2.8.2.tar.gz)

(1) 解压

[root@amao991 soft]# tar -zxvf pvfs-2.8.2.tar.gz

(2) 编译安装

[root@amao991 pvfs-2.8.2]# ./configure -prefix=/root/program/pvfs2-install/   

-with-kernel=/usr/src/kernels/2.6.18-164.el5-i686/ 

-with-mpi=/root/program/mpich2-install/

[root@amao991 pvfs-2.8.2]# make

[root@amao991 pvfs-2.8.2]# make install

3. 服务器端配置

(1)amao991作metadata server，amao991和amao992作I/O节点

(2) 在amao991上生成配置文件

[root@amao991 ~]# ~/program/pvfs2-install/bin/pvfs2-genconfig /etc/pvfs2-fs.conf

**********************************************************************
    Welcome to the PVFS2 Configuration Generator:

This interactive script will generate configuration files suitable
for use with a new PVFS2 file system.  Please see the PVFS2 quickstart
guide for details.

**********************************************************************
You must first select the network protocol that your file system will use.
The only currently supported options are "tcp", "gm", "mx", "ib", and "portals".
(For multi-homed configurations, use e.g. "ib,tcp".)

* Enter protocol type [Default is tcp]:

Choose a TCP/IP port for the servers to listen on.  Note that this
script assumes that all servers will use the same port number.

* Enter port number [Default is 3334]:

Choose a directory for each server to store data in.

* Enter directory name: [Default is /pvfs2-storage-space]:

Choose a file for each server to write log messages to.

* Enter log file location [Default is /tmp/pvfs2-server.log]:

Next you must list the hostnames of the machines that will act as
I/O servers.  Acceptable syntax is "node1, node2, ..." or "node{#-#,#,#}".

* Enter hostnames [Default is localhost]: node15, node16

Use same servers for metadata? (recommended)

* Enter yes or no [Default is yes]: no

Now list the hostnames of the machines that will act as Metadata
servers.  This list may or may not overlap with the I/O server list.

* Enter hostnames [Default is localhost]: node15

Configured a total of 2 servers:
2 of them are I/O servers.
1 of them are Metadata servers.

* Would you like to verify server list (y/n) [Default is n]? y

****** I/O servers:
node15
node16

****** Metadata servers:
node15

* Does this look ok (y/n) [Default is y]? y

Writing fs config file... done

(3)把配置文件从amao991上拷贝到amao992上，放在相同的路径下

[root@amao991 ~]# scp /etc/pvfs2-fs.conf amao992:/etc/

(4)在amao991和amao992上分别设置服务，并开机启动

[root@amao991 ~]# cp ~/soft/pvfs-2.8.2/examples/pvfs2-server.rc /etc/rc.d/init.d/pvfs2-server

[root@amao991 ~]# chmod a+x /etc/rc.d/init.d/pvfs2-server

[root@amao991 ~]# chkconfig pvfs2-server on

在amao992上的执行同上，也可以在amao991上对amao992执行操作，如下：

[root@amao991 ~]# scp ~/soft/pvfs-2.8.2/examples/pvfs2-server.rc amao992:/etc/rc.d/init.d/pvfs2-server

[root@amao991 ~]# ssh amao992 /bin/chmod a+x /etc/rc.d/init.d/pvfs2-server

[root@amao991 ~]# ssh amao992 /sbin/chkconfig pvfs2-server on

(5)创建存储空间(分别在两个IO Server上)

[root@amao991 ~]# ~/program/pvfs2-install/sbin/pvfs2-server /etc/pvfs2-fs.conf -f

[root@amao991 ~]# ll /   会发现多了/pvfs2-storage-space

同样在amao992上执行

[root@amao992 ~]# ~/program/pvfs2-install/sbin/pvfs2-server /etc/pvfs2-fs.conf -f

[root@amao992 ~]# ll /    会发现多了/pvfs2-storage-space

(6) 手动启动metadata server和I/O server（即就是在amao991和amao992上执行）

[root@amao992 ~]# ~/program/pvfs2-install/sbin/pvfs2-server /etc/pvfs2-fs.conf
[S 12/23 18:09] PVFS2 Server on node amao992 version 2.8.2 starting...

可以用netstat -nplt查看该服务已经启动了，占用3334端口

同样在amao992上执行

[root@amao992 ~]# ~/program/pvfs2-install/sbin/pvfs2-server /etc/pvfs2-fs.conf 

4. 客户端配置

也amao991作client为例(任何其他机器都可以作client)，创建挂在点，并设置开机挂在文件pvfs2tab

[root@amao991 ~]# mkdir /mnt/pvfs2

[root@amao991 ~]# touch /etc/pvfs2tab

[root@amao991 ~]# chmod a+r /etc/pvfs2tab

[root@amao991 ~]# vi /etc/pvfs2tab

[root@amao991 ~]# cat /etc/pvfs2tab

tcp://amao992:3334/pvfs2-fs /mnt/pvfs2 pvfs2 defaults,noauto 0 0  #noauto on mount -a

说明:

(1)其实把上面这一行放入/etc/fstab也是可以的，就没必要创建/etc/pvfs2tab文件了

(2)本例子中存储节点有两个amao991和amao992，当client端挂在server时，因为server是平等身份的，所以，随便挂在哪一个都可以，本例子中挂在了tcp://amao992:3334/pvfs2-fs

(3)当有多个client时，不同的Client尽量不挂在同一个server，这样做可以分散input/output负载

(4)设置完成后，重启client，就会自动mount了，然后就可以测试了

5. 测试(在client端测试借助pvfs2-ping, pvfs-cp, pvfs-ls命令来测试)

[root@amao991 ~]# ~/program/pvfs2-install/bin/pvfs2-ping -m /mnt/pvfs2/

(1) Parsing tab file...

(2) Initializing system interface...

(3) Initializing each file system found in tab file: /etc/pvfs2tab...

   PVFS2 servers: tcp://amao992:3334
   Storage name: pvfs2-fs
   Local mount point: /mnt/pvfs2
   /mnt/pvfs2: Ok

(4) Searching for /mnt/pvfs2/ in pvfstab...

   PVFS2 servers: tcp://amao992:3334
   Storage name: pvfs2-fs
   Local mount point: /mnt/pvfs2

   meta servers:
   tcp://amao991:3334

   data servers:
   tcp://amao991:3334
   tcp://amao992:3334

(5) Verifying that all servers are responding...

   meta servers:
   tcp://amao991:3334 Ok

   data servers:
   tcp://amao991:3334 Ok
   tcp://amao992:3334 Ok

(6) Verifying that fsid 1249973606 is acceptable to all servers...

   Ok; all servers understand fs_id 1249973606

(7) Verifying that root handle is owned by one server...

   Root handle: 1048576
     Ok; root handle is owned by exactly one server.

=============================================================

The PVFS2 filesystem at /mnt/pvfs2/ appears to be correctly configured.

[root@amao991 ~]# ~/program/pvfs2-install/bin/pvfs2-ls /mnt/pvfs2    
lost+found  //第一次ls，没任何东西

[root@amao991 ~]# ~/program/pvfs2-install/bin/pvfs2-cp -t /usr/lib/libc.a /mnt/pvfs2/testfile
Wrote 2817820 bytes in 1.041003 seconds. 2.581436 MB/seconds     //cp了一些东西放进去
[root@amao991 ~]# ~/program/pvfs2-install/bin/pvfs2-ls -alh /mnt/pvfs2  
drwxrwxrwx    1 root     root            4.0K 2011-12-23 18:37 .       //果真有东西了
drwxrwxrwx    1 root     root            4.0K 2011-12-23 18:37 .. (faked)
-rw-r--r--    1 root     root            2.7M 2011-12-23 18:37 testfile
drwxrwxrwx    1 root     root            4.0K 2011-12-23 18:06 lost+found

[root@amao991 /]# ~/program/pvfs2-install/bin/pvfs2-cp -t /mnt/pvfs2/testfile /tmp/testfile-out
Wrote 2817820 bytes in 0.157310 seconds. 17.082721 MB/seconds  //把放进去的东西再拷出来，放在/tmp下面

[root@amao991 /]# diff /tmp/testfile-out /usr/lib/libc.a   //把考出来的文件和原来的文件比较，发现一模一样

6.总结

(1)client把I/O servers(不管I/O server有多少个节点)之一挂在到本地/mnt/pvfs2目录后，就可以像访问本地目录一样对其进行读/写/操作了

(2)唯一不同的是，不是用linux自带的读写命令(如ls, cp)，而要用到pvfs2的命令(pvfs2-ls, pvfs2-cp)，故在client上也应该把这些命令文件放置好

(3)为了在client上执行命令时，不用绝对路径或者相对路径，可以把防止pvfs2命令的目录加入到环境变量文件中

(4)通过client给/mnt/pvfs2写入一些文件后，分别在amao991和amao992两个节点上通过linux自带的ll查看本地存储目录/pvfs2-storage-space/

du /pvfs2-storage-space/6b24f650/bstreams/

(a) 当写入的文件非常小(例如8KB)，查看后，发现该文件并没有切分，而是整个只放在一个data server上。

(b)当写入的文件有点大(例如4.2MB)，查看后，发现该文件被一分为二，在每个data server节点上有一个块。