Fileop : Filesystem IO benchmarking tool

Iozone Linux filesystem benchmarking tool has another cool utility called "fileop". This utility does file operation on the filesystem and gives the ops/sec for each operation. This tool will be useful to test your filesystem performance as well as storage performance. Or, if you are going to select a filesystem or storage that is going to host huge number of small files (like source codes), then fileop can be used for benchmarking.

Fileop can create files of different size ( with -s option ) and to different directory depth ( with -f option ).

[root@bryant foo367]# /opt/iozone/bin/fileop  -f 10 -s 1k

Fileop:  File size is 1024,  Output is in Ops/sec. (A=Avg, B=Best, W=Worst)
.      mkdir  rmdir create   read  write  close   stat access  chmod readdir link   unlink delete  Total_files
A   10    795    771   1381  22915   1147   2282   1128 312914   1697    742    579    825    809         1000
[root@bryant foo367]#

[root@bryant foo367]# /opt/iozone/bin/fileop  -f 40 -s 4k -t

Fileop:  File size is 4096,  Output is in Ops/sec. (A=Avg, B=Best, W=Worst)
mkdir:   Dirs =      1640 Total Time =  2.069957018 seconds
         Avg mkdir(s)/sec     =       792.29 ( 0.001262169 seconds/op)
         Best mkdir(s)/sec    =      1039.74 ( 0.000961781 seconds/op)
         Worst mkdir(s)/sec   =       355.87 ( 0.002810001 seconds/op)

rmdir:   Dirs =      1640 Total Time =  2.155941963 seconds
         Avg rmdir(s)/sec     =       760.69 ( 0.001314599 seconds/op)
         Best rmdir(s)/sec    =      1273.70 ( 0.000785112 seconds/op)
         Worst rmdir(s)/sec   =        52.90 ( 0.018905163 seconds/op)

create:  Files =     64000 Total Time = 47.258722544 seconds
         Avg create(s)/sec    =      1354.25 ( 0.000738418 seconds/op)
         Best create(s)/sec   =      5440.08 ( 0.000183821 seconds/op)
         Worst create(s)/sec  =         7.50 ( 0.133283138 seconds/op)

write:   Files =     64000 Total Time = 58.136307001 seconds
         Avg write(s)/sec     =      1100.86 ( 0.000908380 seconds/op)
         Best write(s)/sec    =      3155.98 ( 0.000316858 seconds/op)
         Worst write(s)/sec   =        11.95 ( 0.083698034 seconds/op)

...

The fileop does perform mkdir, rmdir, create, write, close , stat, write, access, link, unlink, chmod, readdir and gives you performance stats in ops/sec and secs/op.

Lustre – cluster filesystem : quick setup guide

The Lustre file system is a distributed high performance cluster filesytem that redefines I/O performance and scalability for large and complex computing environments. This is ideally suited for data-intensive applications which requires the high IO performance.

Lustre components:

MDS – Metadata Server: The MDS server makes metadata stored in the metadata targets available to Lustre clients.

MDT – Metadata Target: This stores metadata, such as filenames, directories, permissions, and file layout, on the metadata server.

OSS – Object Storage Server: The OSS provides file I/O service, and network request handling for the OSTs. The MDT, OSTs and Lustre clients can run concurrently on a single node. However, a typical configuration is an MDT on a dedicated node, two or more OSTs on each OSS node, and a client on each of a large number of compute nodes.

OST – Object Storage Target: The OST stores data as data objects on one or more OSSs. A single Lustre file system can have multiple OSTs, each serving a subset of file data.

Client: The systems that mount the Lustre filesystem.


Steps to create Lustre FS

Configure Lustre Management Server (lustre-mgs.unixfoo.biz – Server 1)

  1. Add the disk to volume manager

    [root@lustre-mgs mnt]# pvcreate /dev/sdb
    Physical volume "/dev/sdb" successfully created

    [root@lustre-mgs mnt]# pvs
      PV         VG   Fmt  Attr PSize   Pfree
      /dev/sdb        lvm2 --   136.73G 136.73G

  2. Create lustre volume group

    [root@lustre-mgs mnt]# vgcreate lustre /dev/sdb
      Volume group "lustre" successfully created
    [root@lustre-mgs mnt]# vgs
      VG     #PV #LV #SN Attr   VSize   Vfree
      lustre   1   0   0 wz--n- 136.73G 136.73G

  3. Create logical volume "MGS" (the management server)

    [root@lustre-mgs ~]# lvcreate -L 25G -n MGS lustre
  4. Create Lustre Management filesystem.

    [root@lustre-mgs ~]# mkfs.lustre --mgs /dev/lustre/MGS

       Permanent disk data:
    Target:     MGS
    Index:      unassigned
    Lustre FS:  lustre
    Mount type: ldiskfs
    Flags:      0x74
                  (MGS needs_index first_time update )
    Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
    Parameters:

    checking for existing Lustre data: not found
    device size = 10240MB
    2 6 18
    formatting backing filesystem ldiskfs on /dev/lustre/MGS
            target name  MGS
            4k blocks     0
            options        -J size=400 -q -O dir_index,uninit_groups –F

    mkfs_cmd = mkfs.ext2 -j -b 4096 -L MGS  -J size=400 -q -O dir_index,uninit_groups -F /dev/lustre/MGS
    Writing CONFIGS/mountdata
    [root@lustre-mgs ~]#

  5. Activate the lustre management filesystem using mount command.

    [root@lustre-mgs ~]# mount -t lustre /dev/lustre/MGS /lustre/MGS/
    [root@lustre-mgs ~]# df
    Filesystem           1K-blocks      Used Available Use% Mounted on
    /dev/sda2             54558908   5276572  46466144  11% /
    /dev/sda1               497829     29006    443121   7% /boot
    tmpfs                  8216000         0   8216000   0% /dev/shm
    /dev/lustre/MGS       10321208    433052   9363868   5% /lustre/MGS
    [root@lustre-mgs ~]#

Configure Lustre Metadata Server (In this guide, both the management & metadata server runs on the same host)

  1. Create logical volume "MDT_unixfoo_cloud"

    [root@lustre-mgs ~]# lvcreate -L 25G -n MDT_unixfoo_cloud lustre
  2. Create Lustre Metdata filesystem for the filesystem “unixfoo_cloud”.

    [root@lustre-mgs ~]# mkfs.lustre --fsname=unixfoo_cloud --mdt  --reformat --mgsnode=lustre-mgs@tcp0 /dev/lustre/MDT_unixfoo_cloud
       Permanent disk data:
    Target:     unixfoo_cloud-MDTffff
    Index:      unassigned
    Lustre FS:  unixfoo_cloud
    Mount type: ldiskfs
    Flags:      0x71
                  (MDT needs_index first_time update )

    Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
    Parameters: mgsnode=10.217.0.237@tcp mdt.group_upcall=/usr/sbin/l_getgroups
    device size = 20480MB
    2 6 18

    formatting backing filesystem ldiskfs on /dev/lustre/MDT_unixfoo_cloud
            target name  unixfoo_cloud-MDTffff
            4k blocks     0
            options        -J size=400 -i 4096 -I 512 -q -O dir_index,uninit_groups –F
    mkfs_cmd = mkfs.ext2 -j -b 4096 -L unixfoo_cloud-MDTffff  -J size=400 -i 4096 -I 512 -q -O dir_index,uninit_groups -F /dev/lustre/MDT_unixfoo_cloud
    Writing CONFIGS/mountdata
    [root@lustre-mgs ~]#


  3. Activate the metdata filesystem using mount command.

    [root@lustre-mgs ~]# mkdir /lustre/MDT_unixfoo_cloud
    [root@lustre-mgs ~]# mount -t lustre  /dev/lustre/MDT_unixfoo_cloud /lustre/MDT_unixfoo_cloud

Configure OSTs ( servers: oss1, oss2 .. )

  1. Add /dev/md0 to volume manager

    [root@oss1 ~]# pvcreate /dev/md0
  2. Create volume group "lustre"

    [root@oss1 ~]# vgcreate lustre /dev/md0
  3. Create logical volume (ost) for unixfoo_cloud

    [root@oss1 ~]# lvcreate -n OST_unixfoo_cloud_1 -L 100G lustre

  4. Create OST lustre filesystem

    [root@oss1 ~]# mkfs.lustre --fsname=unixfoo_cloud --ost --mgsnode=lustre-mgs@tcp0 /dev/lustre/OST_unixfoo_cloud_1

    mkfs_cmd = mkfs.ext2 -j -b 4096 -L unixfoo_cloud-OSTffff  -J size=400 -i 16384 -I 256 -q -O dir_index,uninit_groups -F /dev/lustre/OST_unixfoo_cloud_1
    Writing CONFIGS/mountdata
    [root@oss1 ~]#

  5. Activate the OST by using mount command

    [root@oss1 ~]# mkdir -p /lustre/unixfoo_cloud_oss1
    [root@oss1 ~]# mount -t lustre /dev/lustre/OST_unixfoo_cloud_1 /lustre/unixfoo_cloud_oss1

Mount on the client:

  1. Mount the lustre filesystem unixfoo_cloud

    [root@lustreclient1 ~]# mount -t lustre lustre-mgs@tcp0:/unixfoo_cloud /mnt

    [root@lustreclient1 ~]# df –h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda2              52G  5.1G   44G  11% /
    /dev/sda1             487M   29M  433M   7% /boot
    tmpfs                 7.9G     0  7.9G   0% /dev/shm
    lustre-mgs@tcp0:/unixfoo_cloud
                           99G  461M   93G   1% /mnt
    [root@lustreclient1 ~]#
  2. Done.

HTTP Compression & Web Caching

After setting up my squid proxy for web caching, I was trying to see if it supports http compression. But the version which I had installed doesn't support webcache compression. Google gave me ViCompress .

Here is what www.visolve.com says about ViCompress ..



“ViCompress HTTP proxy server reduces download time by gzip compressing text pages before sending them on the client. This results in faster download times, especially over slow network connections. ViCompress works with existing web servers and web browsers. All modern browsers (Internet Explorer, Firefox, Safari, Chrome) recognize gzip-encoded data, and will automatically decompress the data on the fly. Also, ViCompress can compress both static and dynamic web pages, including Apache/PHP, Microsoft ASP pages, Perl CGI scripts, and others”

Steps to setup ViCompress:

  1. Download ViCompress rpm from visolve.com

    # wget
    http://www.visolve.com/vicompress/ViCompress-1.1.0-1.i386.rpm
  2. Install the package (rpm -ivH ViCompress-1.1.0-1.i386.rpm).
  3. This installs in /usr/local/vicompress/
  4. Edit the configuration file for the proxy port number and other values.

    opensuse:~ # cat /usr/local/vicompress/etc/vicompress.conf
    # Vicompress configuration file

    listen 0.0.0.0 8081           # IP address and port to listen on
    outgoingip 0.0.0.0          # IP address to bind to for outgoing connections
    enable_sessions yes         # Enable sticky sessions with multiple webservers
    enable_compression yes      # Enable compression of html/text pages
    enable_caching yes          # Enable caching of web pages/images
    cache_memory 200            # Amount of memory used for cache, in Megabytes
    max_cacheditem_size 512     # Max size of cached items, in kilobytes
    cache_expires 240           # How long a cached page is valid (in hours)
    enable_dns_caching yes      # Enable caching of dns lookups
    dns_expires 48              # How long cached dns lookups are valid (in hours)
    user nobody                 # User to run server as
    rotatesize 100              # Size (in megabytes) at which to rotate log files
    logformat squid             # Format of access log.  Either apache or squid
    enable_debug no             # Enable debug messages in the errorlog
    accesslog  /var/log/vicompress/accesslog      # Access log file path
    errorlog   /var/log/vicompress/errorlog       # Error log file path
    errorpage  /usr/local/vicompress/etc/errorpage.html # HTML page for errors
    logstats   /usr/local/vicompress/logstats           # Log statistics directory
    opensuse:~ #

  5. Tune the network parameters of the kernel by using the script given by vicompress. This tunes the network parameters like rmem_max, wmem_max, tcp_rmem, tcp_wmem .

    opensuse:~ # /usr/local/vicompress/bin/tune_kernel.sh
    opensuse:~ #
  6. Start the vicompress compression enabled web cache proxy server.

    opensuse:~ # /usr/local/vicompress/bin/vicompress.sh  start
    /usr/local/vicompress/bin/vicompress.sh start: vicompress started
    opensuse:~ #

  7. On the client browsers, change the proxy and check the speed of compression enabled browsing.

  8. vicompress also provides reporting collected by the command
    /usr/local/vicompress/bin/update_log_stats.sh.

Squid Cache proxy – a quick setup guide

This post gives the steps to quickly setup squid proxy and make it up and caching in minutes. There are also some tips & tricks on squid. My squid proxy server runs on OpenSuSE 11.1.


opensuse:~ # cat /etc/SuSE-release
openSUSE 11.1 (i586)
VERSION = 11.1
opensuse:~ #


To download “squid” packages on opensuse, use zypper. The Zypper is a package/patch management tool in Suse, which works similar to yum.


opensuse:~ # zypper install squid3 squidGuard yast2-squid
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following NEW packages are going to be installed:
  libboost_regex1_36_0 squid3 squidGuard yast2-squid

Check the installed Squid rpms.

opensuse:~ # rpm -qa | grep squid
squidGuard-1.3-57.63
squid3-3.0.STABLE10-2.11
yast2-squid-2.17.5-1.11
opensuse:~ #


The default squid configuration file is /etc/squid/squid.conf. The below configuration shows my setup, where I have changed the proxy port number & the cache directory. Make sure you have the cache directory created with the owner as squid.


opensuse:~ # grep -v ^# /etc/squid/squid.conf | grep -v ^$
acl manager proto cache_object
acl localhost src 127.0.0.1/32
<snip>
http_port 8080
hierarchy_stoplist cgi-bin ?
cache_dir ufs /home/cache/squid 200 16 256
access_log /var/log/squid/access.log squid
coredump_dir /var/cache/squid
opensuse:~ #


Start Squid service using the below command.

opensuse:~ # service squid start
Starting WWW-proxy squid                                         done
opensuse:~ #


Now set your client browsers to access internet thorough this proxy. (opensuse:8080)

To check the squid detailed cache information, you can use the command “squidclient”, as described below


opensuse:~ # squidclient -p 8080 cache_object://localhost/info
Squid Object Cache: Version 3.0.STABLE10
Connection information for squid:
        Number of clients accessing cache:      1
        Number of HTTP requests received:       0
        Number of ICP messages received:        0
        Number of ICP messages sent:    0
        Number of queued ICP replies:   0
        Number of HTCP messages received:       0
        Number of HTCP messages sent:   0
        Request failure ratio:   0.00
        Average HTTP requests per minute since start:   -0.0
        Average ICP messages per minute since start:    -0.0
        Select loop called: 23738 times, -143987.578 ms avg
<snip>


opensuse:~ # squidclient -p 8080 cache_object://localhost/info | grep -A10 ^Cache
Cache information for squid:
        Hits as % of all requests:      5min: 0.0%, 60min: 0.0%
        Hits as % of bytes sent:        5min: -0.0%, 60min: -0.0%
        Memory hits as % of hit requests:       5min: 0.0%, 60min: 0.0%
        Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
        Storage Swap size:      6324 KB
        Storage Swap capacity:   3.1% used, 96.9% free
        Storage Mem size:       116 KB
        Storage Mem capacity:    1.4% used, 98.6% free
        Mean Object Size:       14.85 KB
        Requests given to unlinkd:      0
opensuse:~ #


Some more very useful squid tips at : http://sial.org/howto/squid/

TopOfBlogs

hit counter
Technology Blogs - Blog Catalog Blog Directory

  © Free Blogger Templates Columnus by Ourblogtemplates.com 2008

Back to TOP