Back to the story, 32bit to 64bit is a great feature provided. Why so? Because you do not need to perform migration , doing QSM from 32bit aggregate to 64bit aggregate. Getting downtime from the client is another pain too. This is a beautiful feature and I must praise Netapp for doing great job!

Here is my test plan:
1. Get a NAS with Ontap 8.0.2 – 7 Mode
2. Create 32bit “newaggr” and dump in the disk to make it become 15-16TB
3. Upgrade Ontap 8.1 – 7 Mode
4. Perform upgrade to “newaggr” from 32bit to 64bit

Useful Command 

aggr add aggrname
[ -f ]
[ -n ]
[ -g {raidgroup | new | all} ]
[ -c checksum-style ]
[ -64bit-upgrade {check | normal} ] { ndisks[@size]

|

-d disk1 [ disk2 ... ] [ -d diskn [ diskn+1 ... ] ] }

## add an additonal disk to aggregate pfvAggr, use “aggr status” to get group name
aggr status pfvAggr -r
aggr add pfvAggr -g rg0 -64bits-upgrade normal -d v5.25 normal

## Add 4 300GB disk to aggregate aggr1

aggr add aggr1 -64bits-upgrade check 4@300

Adds disks to the aggregate named aggrname. Specify the disks in the same way as for the aggr create command. If the aggregate is mirrored, then the -d argument must be used twice (if at all).

If the size option is used, it specifies the disk size in GB. Disks that are within approximately 20% of the specified size will be selected. If the size option is not specified, existing groups are appended with disks that are the best match by size for the largest disk in the group, i.e., equal or smaller disks are selected first, then larger disks. When starting new groups, disks that are the best match by size for the largest disk in the last raidgroup are selected. The size option is ignored if a specific list of disks is specified.

If the -g option is not used, the disks are added to the most recently created RAID group until it is full, and then one or more new RAID groups are created and the remaining disks are added to new groups. Any other existing RAID groups that are not full remain partially filled.

The -g option allows specification of a RAID group (for example, rg0) to which the indicated disks should be added, or a method by which the disks are added to new or existing RAID groups.

If the -g option is used to specify a RAID group, that RAID group must already exist. The disks are added to that RAID group until it is full. Any remaining disks are ignored.

If the -g option is followed by new, Data ONTAP creates one or more new RAID groups and adds the disks to them, even if the disks would fit into an existing RAID group. Any existing RAID groups that are not full remain partially filled. The name of the new RAID groups are selected automatically. It is not possible to specify the names for the new RAID groups.

If the -g option is followed by all, Data ONTAP adds the specified disks to existing RAID groups first. Disks are added to an existing RAID group until it reaches capacity as defined by raidsize. After all existing RAID groups are full, it creates one or more new RAID groups and adds the remaining disks to the new groups. If the disk type or checksum style or both is specified then the command would operate only on the RAID groups with matching disk type or checksum style or both.

The -n option can be used to display the command that the system will execute, without actually making any changes. This is useful for displaying the automatically selected disks, for example.

The -c checksum-style argument specifies the checksum style of disks to use when adding disks to an existing aggregate. Possible values are: block for Block Checksum and advanced_zoned for Advanced Zoned Checksum (azcs).

If the -64bit-upgrade option is followed by check, Data ONTAP displays a summary of the space impact which would result from upgrading the aggregate to 64-bit. This summary includes the space usage of each contained volume after the volume is upgraded to 64-bit and the amount of space that must be added to the volume to successfully complete the 64-bit upgrade. This option does not result in an upgrade to 64-bit or addition of disks.

If the -64bit-upgrade option is followed by normal, Data ONTAP upgrades the aggregate to 64-bit if the total aggregate size after adding the specified disks exceeds 16TB. This option does not allow Data ONTAP to automatically grow volumes if they run out of space due to the 64-bit upgrade.

By default, the filer fills up one RAID group with disks before starting another RAID group. Suppose an aggregate currently has one RAID group of 12 disks and its RAID group size is 14. If you add 5 disks to this aggregate, it will have one RAID group with 14 disks and another RAID group with 3 disks. The filer does not evenly distribute disks among RAID groups.

You cannot add disks to a mirrored aggregate if one of the plexes is offline.

The disks in a plex are not permitted to span disk pools. This behavior can be overridden with the -f flag when used together with the -d argument to list disks to add. The -f flag, in combination with -d, can also be used to force adding disks that have a rotational speed that does not match that of the majority of existing disks in the aggregate. 

Before Upgrade

mynas02> aggr status
           Aggr State           Status            Options
          aggr0 online          raid_dp, aggr     root
                                32-bit
    sata64_bk11 online          raid_dp, aggr     raidsize=20
                                64-bit
        newaggr online          raid_dp, aggr     raidsize=10
                                32-bit

Perform upgrade check

mynas02> aggr add newaggr -64bit-upgrade check -d 1a.38.22 1a.38.21
File system size 16.97 TB exceeds maximum 15.99 TB
Checking for additional space required to upgrade all writable 32-bit
volumes in aggregate newaggr (Ctrl-C to interrupt)......
 
Adding the specified disks and upgrading the aggregate to
64-bit will add 1489GB of usable space to the aggregate.
 
To initiate the 64-bit upgrade of aggregate newaggr, run this
command with the "normal" option.

After upgrade

mynas02> aggr add newaggr -64bit-upgrade normal -d 1a.38.22 1a.38.21
File system size 16.97 TB exceeds maximum 15.99 TB
Checking for additional space required to upgrade all writable 32-bit
volumes in aggregate newaggr (Ctrl-C to interrupt)......
File system size 16.97 TB exceeds maximum 15.99 TB
Addition of 2 disks to the aggregate has completed.
mynas02> Checking for additional space required to upgrade all writable 32-bit
volumes in aggregate newaggr (Ctrl-C to interrupt)......
Mon MayMon May 14 18:12:35 MYT 14 18: [mynas02:raid.vol.disk.add.done:notice]: Addition of Disk /newaggr/plex0/rg2/1a.38.21 Shelf 38 Bay 21 [NETAPP   X302_HJUPI01TSSM NA02] S/N [N01RBR3L] to aggregate newaggr has completed successfully
12:35 MYT [mynas02:raid.vMon May 14 18:12:35 ol.diskMYT [mynas02:raid.vol.disk.add.done:notice]: Addition of Disk /newaggr/plex0/rg2/1a.38.22 Shelf 38 Bay 22 [NETAPP   X302_HJUPI01TSSM NA02] S/N [N01RGUYL] to aggregate newaggr has completed successfully
.add.done:notice]: Addition of Disk /newaggr/plex0/rg2/1a.38.21 Shelf 38 Bay 21 [NETAPP   X302_HJUPI01TSSM NA02] S/N [N01RBR3L] to aggregate newaggr has completed successfully
Mon May 14 18:12:35 MYT [mynas02Mon May 14 18:12:3:raid.v5 MYT [mynas02:wafl.scan.64bit.upgrade.start:notice]: The 64-bit upgrade scanner has started running on aggregate newaggr.
ol.disk.add.done:notiMon May 14 18:12:35 MYT [mynas02:wafl.scan.start:info]: Startingce]: Ad 64bit upgrade on aggregate newaggr.
dition of Disk /newaggr/plex0/rg2/1a.38.22 Shelf 38 Bay 22 [NETAPP   X302_HJUPI01TSSM NA02] S/N [N01RGUYL] to aggregate newaggr has completed successfully
Mon May 14 18:12:35 MYT [mynas02:wafl.scan.64bit.upgrade.start:notice]: The 64-bit upgrade scanner has started running on aggregate newaggr.
Mon May 14 18:12:35 MYT [mynas02:wafl.scan.start:info]: Starting 64bit upgrade on aggregate newaggr.
 14 18:Mon May 14 18:12:37 MYT [mynas02:wafl.scan.64bit.upgrade.completed:notice]: The 64-bit upgrade scanner has completed running on aggregate newaggr.
12:37 MYT [mynas02:wafl.scan.64bit.upgrade.completed:notice]: The 64-bit upgrade scanner has completed running on aggregate newaggr.
 
mynas02> aggr show_space -h newaggr
Aggregate 'newaggr'
 
    Total space    WAFL reserve    Snap reserve    Usable space       BSR NVLOG           A-SIS          Smtape
           16TB          1738GB             0KB            15TB             0KB             0KB             0KB
 
This aggregate does not contain any volumes
 
Aggregate                       Allocated            Used           Avail
Total space                           0KB             0KB            15TB
Snap reserve                          0KB             0KB             0KB
WAFL reserve                       1738GB          1392KB          1738GB
 
mynas02> aggr status newaggr
           Aggr State           Status            Options
        newaggr online          raid_dp, aggr     raidsize=10
                                64-bit
                Volumes: 
 
                Plex /newaggr/plex0: online, normal, active
                    RAID group /newaggr/plex0/rg0: normal, block checksums
                    RAID group /newaggr/plex0/rg1: normal, block checksums
                    RAID group /newaggr/plex0/rg2: normal, block checksums

So we have seen that the upgrade is very successful and option for “-64bit-upgrade” is only required once, after you got the aggregate to 64bit,  you just keep increasing the disk depending on your budget!

1. Always use snapmirror for volume migration! This is the best way to migrate data in the most consistent way.
2. Avoid using rsync for huge volume with multiple sessions(directory too huge and you have to split it to multiple sessions).
3. Detailed plan and down time needed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71

filer1>snapmirror initialize -S source_filer:/vol/icd_sync/data1 filer1:/vol/dr_temp_data1/data1
 
=========================================================================================
 
filer1>snapmirror status
 
Snapmirror is on.
Source Destination State Lag Status
source_filer:/vol/icd_sync/data1 filer1:/vol/dr_temp_data1/data1 Snapmirrored 21:48:37 Idle
 
 
=========================================================================================
 
filer1>snapmirror update -S source_filer:/vol/icd_sync/data1 filer1:/vol/dr_temp_data1/data1
 
Transfer started.
Monitor progress with 'snapmirror status' or the snapmirror log.
 
=========================================================================================
 
filer1>snapmirror status -l filer1:/vol/dr_temp_data1/data1
Snapmirror is on.
 
Source: source_filer:/vol/icd_sync/data1
Destination: filer1:/vol/dr_temp_data1/data1
Status: Transferring
Progress: 278300 KB
State: Snapmirrored
Lag: 21:54:21
Mirror Timestamp: Thu Mar 15 02:46:55 PDT 2012
Base Snapshot: filer1(1873880973)_dr_temp_data1_data1-dst.2
Current Transfer Type: Update
Current Transfer Error: -
Contents: Transitioning
Last Transfer Type: Initialize
Last Transfer Size: 4967070104 KB
Last Transfer Duration: 14:05:50
Last Transfer From: source_filer:/vol/icd_sync/data1
=========================================================================================
 
filer1>snapmirror status -l filer1:/vol/dr_temp_data1/data1
Snapmirror is on.
 
Source: source_filer:/vol/icd_sync/data1
Destination: filer1:/vol/dr_temp_data1/data1
Status: Idle
Progress: -
State: Snapmirrored
Lag: 01:08:52
Mirror Timestamp: Fri Mar 16 00:49:03 PDT 2012
Base Snapshot: filer1(1873880973)_dr_temp_data1_data1-dst.4
Current Transfer Type: -
Current Transfer Error: -
Contents: Replica
Last Transfer Type: Update
Last Transfer Size: 20705884 KB
Last Transfer Duration: 01:08:10
Last Transfer From: source_filer:/vol/icd_sync/data1
 
=========================================================================================
 
filer1>snapmirror quiesce filer1:/vol/dr_temp_data1/data1
 
snapmirror quiesce: in progress
This can be a long-running operation. Use Control - C (^C) to interrupt.
snapmirror quiesce: /vol/dr_temp_data1/data1 : Successfully quiesced
 
=========================================================================================
filer1>snapmirror break filer1:/vol/dr_temp_data1/data1
 
snapmirror break: Destination /vol/dr_temp_data1/data1 is now writable.

However, the draw back of snapmirror is

• keep the information for last transfer only. No more history more than last transfer.
• NO Alert sent when the job fail due to destination space is full (& others reason) .

 

That is a pain in axx , i have a backup mirror failed for weeks without me realizing it. So i have written a simple script to check on the activity.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

#!/bin/bash
 
 
#Target NAS
nas="nas01"
 
#Target Volume
volume="sync_backup"
 
#Define acceptable threshold , 24 hours is acceptable but 48 hours to play safe in case not able to finish the mirror within 24 hours.
threshold="48"
 
#Email to list.
mail_user_list="group1@testmail.com"
 
#Email CC list
mail_cc_list="group2@testmail.com"
 
#Command to get snapmirror lag time
mirror_hour=`ssh $nas snapmirror status | grep $volume | awk '{print $4}' | cut -f1 -d":"`
 
#Simple mail content.
message='Perform snapmirror status to check from $nas
NAS = $nas
volume = $volume
threshold = $threshold
Lag time of snapmirror = $mirror_hour'
 
if [ $mirror_hour -gt $threshold ]
then
 
echo $message | mail -s "[Alert]snapmirror for syncbackup lag time greater than $threshold hours!" -c $mail_cc_list $mail_user_list -- -r "tiwlim@testmail.com"
fi

My script will eventually check on the “Lag” time of snapmirror status. Lag time = last successful snapmirror activity till now. if you set a daily snapmirror happening once every day, you should not be getting lag time more than 24 hours. If it did, something is wrong and you may want to check more detail using “snapmirror status -l ”

Command:

grep -H ServerMaintenanceMode=dword:1 /<path>/<port_number>/PortRegistry.reg

dword:1 = Normal operation

dword:2 = Set server to read-only mode

dword:3 = block all read and write access to server

Step 1 – Identify OS:

1. Freenas – FreeBSD based , i like FreeBSD because it was the core for Netapp too.

2. Openfiler – CentOS based, customized for NAS.

3. Fedora + Samba

I picked number 3 which is Fedora + Samba , as i still need my NAS OS to be a multi purpose server and i love YUM~

Step2 – Hardware

1. 2 years old HP 110 Netbook

2. 2TB Seagate External Harddisk

Finally…. the combination!

HP 2 years old 110

2TB Seagate External Disk

Fedora 15

Output:
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Collis Drop
e4a 9000 none none 2007724074 1 2991841358 0 0 0

0 is the one you are looking for.

[root@linux tiwlim]# rsh filerDest snapvault stop filerDest:/vol/volume_A_Dataset_new_backup_1/t28lp_ip
Stopping /vol/volume_A_Dataset_new_backup_1/t28lp_ip is permanent.
The secondary qtree will be deleted.
Further incremental updates will be impossible.
Data already stored in snapshots will not be deleted.
This may take a long time to complete.
Are you sure you want to do this? y
Snapvault configuration for the qtree has been deleted.


So the mission for today is to create the write and delete test for my environment.

Local machine  —-COPY—>  Mounted filesystem in NAS

Why i am doing so? to get a baseline for my write and delete operation.I will make use of “time” command to get total runtime for the script to complete the operation. So here goes the simple script(Feel free to provide your code if you have idea, i am a noob in python:) )

BTW, Nanako is just an echo.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

#!/usr/bin/python
import os
import shutil
 
#set source path & file name
src = "/usr2/"
file_name = "testfile.txt"
source_file_path = src + file_name
 
#set destination path
dest = "/data/tingwei/test_copy/"
 
#set number of copies for the copying progress
copies = 1000
dir_copies = 10
 
for x in range(dir_copies):
    dirname = dest + str(x)
    if not os.path.isdir(dirname):
        os.mkdir(dirname)
        print "Nanako: mkdir " + str(dirname)
#start progress
        print "Nanako: Ready to copy " + `copies` + " files."
        for i in range(copies):
                new_file_name = file_name + `i`
                dest_file_path = dirname + "/" + new_file_name
                shutil.copyfile (source_file_path, dest_file_path)
 
print "Nanako: Done copying " + `copies` + " files."
print "Nanako: Preparing to cleanup " + dest
 
#this line is to remove the files & directories created
for x in os.listdir(dest):
   shutil.rmtree(os.path.join(dest,x))
 
#just in case you want to see is there any left over in the destination
def listdir_fullpath(d):
    return [os.path.join(d, f) for f in os.listdir(d)]
 
for i in listdir_fullpath(dest):
    print i

Below is just the example output i have for references:
I am copying from a FAS3170 to FAS960
Directory size: 1.5TB
~45MBps!!!!
Time taken: 9 hours 15 minute
Ndmpcopy: filerA: Log: DUMP: dumping (Pass V) [ACLs]
Ndmpcopy: filerA: Log: DUMP: 1563223738 KB
Ndmpcopy: filerB: Log: RESTORE: RESTORE IS DONE
Ndmpcopy: filerB: Log: RESTORE: The destination path is /vol/vol_restore/re store/
Ndmpcopy: filerB: Notify: restore successful
Ndmpcopy: filerA: Log: DUMP: DUMP IS DONE
Ndmpcopy: filerA: Log: DUMP: Deleting “/vol/fc_loan/../snapshot_for_ backup.195″ snapshot.
Ndmpcopy: filerA: Notify: dump successful
Ndmpcopy: Transfer successful [ 9 hours 15 minutes 47 seconds ]
Ndmpcopy: Done