June 24, 2008

Linux : Virtual filesystem - Loopback device

Sometimes you may need additional local partition or local mountable drives, but you dont have free blocks in the parition table. In that case virtual filesystem helps. You can create a virtual filesystem and mount them as a loopback device. Here are the steps to do it.

  1. Create a empty file with the mount of disk space you need. Here I have created a 1G file.

    [root@unixfoo23 ~]# dd if=/dev/zero of=/root/myfs1 bs=1024 count=1048576
    1048576+0 records in
    1048576+0 records out

    [root@unixfoo23 ~]# ls -l /root/myfs1
    -rw-r--r--  1 root root 1073741824 Jun 25 08:32 /root/myfs1

    [root@unixfoo23 ~]# du -sh /root/myfs1
    1.1G    /root/myfs1
    [root@unixfoo23 ~]#

  2. Create a filesystem on the virtual device (/root/myfs1). I have selected ext3 , whereas you can create the filesystem of your choice.

    [root@unixfoo23 ~]# mkfs.ext3  /root/myfs1
    mke2fs 1.35 (28-Feb-2004)
    /root/myfs1 is not a block special device.
    Proceed anyway? (y,n) y
    Filesystem label=
    OS type: Linux
    Block size=4096 (log=2)
    Fragment size=4096 (log=2)
    131072 inodes, 262144 blocks
    13107 blocks (5.00%) reserved for the super user
    First data block=0
    Maximum filesystem blocks=268435456
    8 block groups
    32768 blocks per group, 32768 fragments per group
    16384 inodes per group
    Superblock backups stored on blocks:
            32768, 98304, 163840, 229376

    Writing inode tables: done
    Creating journal (8192 blocks): done
    Writing superblocks and filesystem accounting information: done

    This filesystem will be automatically checked every 28 mounts or
    180 days, whichever comes first.  Use tune2fs -c or -i to override.

    [root@unixfoo23 ~]# file /root/myfs1
    /root/myfs1: Linux rev 1.0 ext3 filesystem data (large files)
    [root@unixfoo23 ~]#

  3. Mount the filesystem as a loopback device.

    [root@unixfoo23 ~]# mount -o loop /root/myfs1 /mnt

    [root@unixfoo23 ~]# df /mnt
    Filesystem           1K-blocks      Used Available Use% Mounted on
    /root/myfs1            1032088     34092    945568   4% /mnt
    [root@unixfoo23 ~]#

  4. If you need this permanantly , you can add this to /etc/fstab.

You can even create this virtual filesytem on a nfs mounted directory and again loopback-mount it on the machine.

June 22, 2008

NFS client : rsize wsize

The nfs client mount command options "rsize" and "wsize" specify the size of the chunks of data that the client and server pass back and forth to each other. If no rsize and wsize options are specified, the default value is taken - which varies by which version of NFS and OS you are using. You will have to do some tests and find an rsize and wsize that works and is as fast as possible. You can test the speed of your options with some simple commands like dd.  

Example : The first of these commands transfers 16384 blocks of 16k each from the special file /dev/zero to the file (testfile) in mounted directory /mnt. You have to time it to see how long it takes. So, from the client machine, try dd: 

# time dd if=/dev/zero of=/mnt/testfile bs=16k count=16384

This creates a 256Mb file. In general, you should create a file that's at least twice as large as the system RAM on the server, but make sure you have enough disk space. Then read back the file into the /dev/null by typing the following: 

# time dd if=/mnt/testfile of=/dev/null bs=16k 

Repeat this test a few times with different rsize & wsize and average how long it takes. Be sure to unmount and remount the filesystem each time, which should clear out any caches. Also try out with cache/no-cache, noatime and noacl options and get the average value.

You can also do these tests with much advanced benchmarking tools like iozone [
http://www.iozone.org/ ]


http://unixfoo.blogspot.com/2008/01/linux-benchmarking-tools.html



June 20, 2008

Linux : Forcing fsck on nxt boot

If you want to force fsck on the next boot, just create a file called /forcefsck . If this file is present, then during next boot - the fsck is forced.

# touch /forcefsck

Now reboot the machine and when it comes up, fsck will be forced.

# reboot


Encrypting a tar file using openssl

In some cases, you may have to send your datacenter's passwd information to some of your collegues. Instead of sending them in plain text, you can use tar & openssl combination to encrypt that data. Here is how it can be done.

Encryption :

Tar & gzip the password file and encrypt using openssl des3 and a secret key. Replace the text "secretkey" with your secret password.

[root@unixfoo-lin23 ~]# tar cvzf - passwd_info.txt | openssl des3 -salt -k secretkey | dd of=encrypted_passwd_info
passwd_info.txt
20+1 records in
20+1 records out

The filetype of the encrypted file is "data" and you cannot use "tar -tvzf" to list contents on this.

[root@unixfoo-lin23 ~]# file encrypted_passwd_info
encrypted_passwd_info: data

[root@unixfoo-lin23 ~]# tar tvzf encrypted_passwd_info
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error exit delayed from previous errors
[root@unixfoo-lin23 ~]#


Decryption :

While decrypting the file, use the steps below. Replace the text "secretkey" with your secret password which you provided during encryption.

[root@unixfoo-lin12 ~]# dd if=encrypted_passwd_info |openssl des3 -d -k secretkey |tar xvzf -
20+1 records in
20+1 records out
passwd_info.txt
[root@unixfoo-lin12 ~]# cat passwd_info.txt | head -1
UNIX User       UNIX Password
[root@unixfoo-lin12 ~]#

This method can also be used to gzip and encrypt any file or directory.




June 17, 2008

Loadbalancer


A good presentation from F5 explaining all features of Loadbalancer : 
http://www.f5.com/flash/product-demo/


June 16, 2008

Download using wget

You can use "wget" on Linux machine CLI to download files from http or ftp sites. Proxy configuration can be done for wget by exporting environment variables

Initialize proxy :

# export http_proxy=unixfoo-proxy.unixfoo.net:80
# export ftp_proxy=unixfoo-proxy.unixfoo.net:80

Start download :

Here im trying to download Fedora Core 9 image.

# wget ftp://mirror.wbut.ac.in/pub/fedora/linux/releases/9/Fedora/i386/iso/Fedora-9-i386-disc1.iso ftp://mirror.wbut.ac.in/pub/fedora/linux/releases/9/Fedora/i386/iso/Fedora-9-i386-disc2.iso ftp://mirror.wbut.ac.in/pub/fedora/linux/releases/9/Fedora/i386/iso/Fedora-9-i386-disc3.iso ftp://mirror.wbut.ac.in/pub/fedora/linux/releases/9/Fedora/i386/iso/Fedora-9-i386-disc4.iso ftp://mirror.wbut.ac.in/pub/fedora/linux/releases/9/Fedora/i386/iso/Fedora-9-i386-disc5.iso ftp://mirror.wbut.ac.in/pub/fedora/linux/releases/9/Fedora/i386/iso/Fedora-9-i386-disc6.iso

This will download the images sequentially and also indicates the downlaod progress. "wget" is also available for Solaris, HPUX and AIX.

June 5, 2008

Netapp : nfsstat command

This command displays the NFS and RPC statistics of filer. The command can be used to find the performance bottlenecks in NFS environment. This shows the metrics for each nfs call defined below. For more details refer http://www.ietf.org/rfc/rfc1813.txt .
 
GETATTR:  Get file attributes
SETATTR:  Set file attributes
LOOKUP:   Lookup filename
ACCESS:   Check access permission
READLINK: Read from symbolic link
READ:     Read from file
WRITE:    Write to file
CREATE:   Create a file
MKDIR:    Create a directory
SYMLINK:  Create a symbolic link
MKNOD:    Create a special device
REMOVE:   Remove a file
RMDIR:    Remove a directory
RENAME:   Rename a file or directory
LINK:     Create link to an object
READDIR:  Read From directory
READDIRPLUS: Extended read from directory
FSSTAT:   Get dynamic file system information
FSINFO:   Get static file system information
PATHCONF: Retrieve POSIX information

Command: nfsstat (The important thing to note in nfsstat is “Server nfs V3” statistics)

Server nfs V3: (7939531611 calls)

null       getattr    setattr    lookup     access     readlink   read
2098536 0% 8732098536 23% 98536 0%  6732365515 14% 61019260 63% 192842 0%  101763023 0%

write      create     mkdir      symlink    mknod      remove     rmdir
9739487 0% 25364 0%   6 0%       0 0%       0 0%       18898 0%   2 0%

rename     link       readdir    readdir+   fsstat     fsinfo     pathconf
13809 0%   6 0%       2398 0%    1011793 0% 1689879 0% 32804 0%   0 0%

commit
0 0%


Command: nfsstat –l (Displays nfs client information and number of nfs calls made by each client)

Filer> nfsstat -l

192.168.10.15    qa1nfsclient-lin.lab.unixfoo.net NFSOPS =      35337 ( 0%)
192.168.10.16    qa2nfsclient-lin.lab.unixfoo.net NFSOPS =      35326 ( 0%)
192.168.10.17    qa3nfsclient-lin.lab.unixfoo.net NFSOPS =   16534679 ( 0%)
192.168.10.18    qa4nfsclient-lin.lab.unixfoo.net NFSOPS =   16637787 ( 0%)
192.168.10.36    qa90dbclient-lin.lab.unixfoo.net NFSOPS =   19873449 ( 0%)
192.168.10.37    dev1dbclient-lin.lab.unixfoo.net NFSOPS =   16716260 ( 0%)
192.168.10.38    dev3dbclient-lin.lab.unixfoo.net NFSOPS =   15835924 ( 0%)


Command: nfsstat –h <hostname> (Displays server side NFS statistics for a single client. This will help us to identify what operation each client does) 

Server nfs V3:

null       getattr    setattr    lookup     access     readlink   read
9 0%       73 0%      2 0%       21 0%      47 0%      0 0%       0 0%

write      create     mkdir      symlink    mknod      remove     rmdir
17402552 100% 10 0%      0 0%       0 0%       0 0%       11 0%      0 0%

rename     link       readdir    readdir+   fsstat     fsinfo     pathconf
1 0%       0 0%       0 0%       6 0%       27 0%      8 0%       0 0%

commit
0 0%