Netapp 32bit aggregate upgrade to 64bit

Netapp released 8.1GA on May 2012, with quite a number of improvement and the most eye catching is capability to upgrade aggregate from 32bit to 64bit. I have been waiting for 6 months and eager to try with 8.1RC , but RC is not a good choice for production. Always remember not to risk your production NAS!

Back to the story, 32bit to 64bit is a great feature provided. Why so? Because you do not need to perform migration , doing QSM from 32bit aggregate to 64bit aggregate. Getting downtime from the client is another pain too. This is a beautiful feature and I must praise Netapp for doing great job!

Here is my test plan:
1. Get a NAS with Ontap 8.0.2 – 7 Mode
2. Create 32bit “newaggr” and dump in the disk to make it become 15-16TB
3. Upgrade Ontap 8.1 – 7 Mode
4. Perform upgrade to “newaggr” from 32bit to 64bit

 

Useful Command 

aggr add aggrname
[ -f ]
[ -n ]
[ -g {raidgroup | new | all} ]
[ -c checksum-style ]
[ -64bit-upgrade {check | normal} ] { ndisks[@size]

|

-d disk1 [ disk2 … ] [ -d diskn [ diskn+1 … ] ] }

## add an additonal disk to aggregate pfvAggr, use “aggr status” to get group name
aggr status pfvAggr -r
aggr add pfvAggr -g rg0 -64bit-upgrade normal -d v5.25 normal

                    ## Add 4 300GB disk to aggregate aggr1

aggr add aggr1 -64bit-upgrade check 4@300

Adds disks to the aggregate named aggrname. Specify the disks in the same way as for the aggr create command. If the aggregate is mirrored, then the -d argument must be used twice (if at all).

If the size option is used, it specifies the disk size in GB. Disks that are within approximately 20% of the specified size will be selected. If the size option is not specified, existing groups are appended with disks that are the best match by size for the largest disk in the group, i.e., equal or smaller disks are selected first, then larger disks. When starting new groups, disks that are the best match by size for the largest disk in the last raidgroup are selected. The size option is ignored if a specific list of disks is specified.

If the -g option is not used, the disks are added to the most recently created RAID group until it is full, and then one or more new RAID groups are created and the remaining disks are added to new groups. Any other existing RAID groups that are not full remain partially filled.

The -g option allows specification of a RAID group (for example, rg0) to which the indicated disks should be added, or a method by which the disks are added to new or existing RAID groups.

If the -g option is used to specify a RAID group, that RAID group must already exist. The disks are added to that RAID group until it is full. Any remaining disks are ignored.

If the -g option is followed by new, Data ONTAP creates one or more new RAID groups and adds the disks to them, even if the disks would fit into an existing RAID group. Any existing RAID groups that are not full remain partially filled. The name of the new RAID groups are selected automatically. It is not possible to specify the names for the new RAID groups.

If the -g option is followed by all, Data ONTAP adds the specified disks to existing RAID groups first. Disks are added to an existing RAID group until it reaches capacity as defined by raidsize. After all existing RAID groups are full, it creates one or more new RAID groups and adds the remaining disks to the new groups. If the disk type or checksum style or both is specified then the command would operate only on the RAID groups with matching disk type or checksum style or both.

The -n option can be used to display the command that the system will execute, without actually making any changes. This is useful for displaying the automatically selected disks, for example.

The -c checksum-style argument specifies the checksum style of disks to use when adding disks to an existing aggregate. Possible values are: block for Block Checksum and advanced_zoned for Advanced Zoned Checksum (azcs).

If the -64bit-upgrade option is followed by check, Data ONTAP displays a summary of the space impact which would result from upgrading the aggregate to 64-bit. This summary includes the space usage of each contained volume after the volume is upgraded to 64-bit and the amount of space that must be added to the volume to successfully complete the 64-bit upgrade. This option does not result in an upgrade to 64-bit or addition of disks.

If the -64bit-upgrade option is followed by normal, Data ONTAP upgrades the aggregate to 64-bit if the total aggregate size after adding the specified disks exceeds 16TB. This option does not allow Data ONTAP to automatically grow volumes if they run out of space due to the 64-bit upgrade.

By default, the filer fills up one RAID group with disks before starting another RAID group. Suppose an aggregate currently has one RAID group of 12 disks and its RAID group size is 14. If you add 5 disks to this aggregate, it will have one RAID group with 14 disks and another RAID group with 3 disks. The filer does not evenly distribute disks among RAID groups.

You cannot add disks to a mirrored aggregate if one of the plexes is offline.

The disks in a plex are not permitted to span disk pools. This behavior can be overridden with the -f flag when used together with the -d argument to list disks to add. The -f flag, in combination with -d, can also be used to force adding disks that have a rotational speed that does not match that of the majority of existing disks in the aggregate. 

Before Upgrade

mynas02> aggr status
           Aggr State           Status            Options
          aggr0 online          raid_dp, aggr     root
                                32-bit
    sata64_bk11 online          raid_dp, aggr     raidsize=20
                                64-bit
        newaggr online          raid_dp, aggr     raidsize=10
                                32-bit

Perform upgrade check

mynas02> aggr add newaggr -64bit-upgrade check -d 1a.38.22 1a.38.21
File system size 16.97 TB exceeds maximum 15.99 TB
Checking for additional space required to upgrade all writable 32-bit
volumes in aggregate newaggr (Ctrl-C to interrupt)......
 
Adding the specified disks and upgrading the aggregate to
64-bit will add 1489GB of usable space to the aggregate.
 
To initiate the 64-bit upgrade of aggregate newaggr, run this
command with the "normal" option.

After upgrade

mynas02> aggr add newaggr -64bit-upgrade normal -d 1a.38.22 1a.38.21
File system size 16.97 TB exceeds maximum 15.99 TB
Checking for additional space required to upgrade all writable 32-bit
volumes in aggregate newaggr (Ctrl-C to interrupt)......
File system size 16.97 TB exceeds maximum 15.99 TB
Addition of 2 disks to the aggregate has completed.
mynas02> Checking for additional space required to upgrade all writable 32-bit
volumes in aggregate newaggr (Ctrl-C to interrupt)......
Mon MayMon May 14 18:12:35 MYT 14 18: [mynas02:raid.vol.disk.add.done:notice]: Addition of Disk /newaggr/plex0/rg2/1a.38.21 Shelf 38 Bay 21 [NETAPP   X302_HJUPI01TSSM NA02] S/N [N01RBR3L] to aggregate newaggr has completed successfully
12:35 MYT [mynas02:raid.vMon May 14 18:12:35 ol.diskMYT [mynas02:raid.vol.disk.add.done:notice]: Addition of Disk /newaggr/plex0/rg2/1a.38.22 Shelf 38 Bay 22 [NETAPP   X302_HJUPI01TSSM NA02] S/N [N01RGUYL] to aggregate newaggr has completed successfully
.add.done:notice]: Addition of Disk /newaggr/plex0/rg2/1a.38.21 Shelf 38 Bay 21 [NETAPP   X302_HJUPI01TSSM NA02] S/N [N01RBR3L] to aggregate newaggr has completed successfully
Mon May 14 18:12:35 MYT [mynas02Mon May 14 18:12:3:raid.v5 MYT [mynas02:wafl.scan.64bit.upgrade.start:notice]: The 64-bit upgrade scanner has started running on aggregate newaggr.
ol.disk.add.done:notiMon May 14 18:12:35 MYT [mynas02:wafl.scan.start:info]: Startingce]: Ad 64bit upgrade on aggregate newaggr.
dition of Disk /newaggr/plex0/rg2/1a.38.22 Shelf 38 Bay 22 [NETAPP   X302_HJUPI01TSSM NA02] S/N [N01RGUYL] to aggregate newaggr has completed successfully
Mon May 14 18:12:35 MYT [mynas02:wafl.scan.64bit.upgrade.start:notice]: The 64-bit upgrade scanner has started running on aggregate newaggr.
Mon May 14 18:12:35 MYT [mynas02:wafl.scan.start:info]: Starting 64bit upgrade on aggregate newaggr.
 14 18:Mon May 14 18:12:37 MYT [mynas02:wafl.scan.64bit.upgrade.completed:notice]: The 64-bit upgrade scanner has completed running on aggregate newaggr.
12:37 MYT [mynas02:wafl.scan.64bit.upgrade.completed:notice]: The 64-bit upgrade scanner has completed running on aggregate newaggr.
 
mynas02> aggr show_space -h newaggr
Aggregate 'newaggr'
 
    Total space    WAFL reserve    Snap reserve    Usable space       BSR NVLOG           A-SIS          Smtape
           16TB          1738GB             0KB            15TB             0KB             0KB             0KB
 
This aggregate does not contain any volumes
 
Aggregate                       Allocated            Used           Avail
Total space                           0KB             0KB            15TB
Snap reserve                          0KB             0KB             0KB
WAFL reserve                       1738GB          1392KB          1738GB
 
mynas02> aggr status newaggr
           Aggr State           Status            Options
        newaggr online          raid_dp, aggr     raidsize=10
                                64-bit
                Volumes: 
 
                Plex /newaggr/plex0: online, normal, active
                    RAID group /newaggr/plex0/rg0: normal, block checksums
                    RAID group /newaggr/plex0/rg1: normal, block checksums
                    RAID group /newaggr/plex0/rg2: normal, block checksums

So we have seen that the upgrade is very successful and option for “-64bit-upgrade” is only required once, after you got the aggregate to 64bit,  you just keep increasing the disk depending on your budget!

Update 28th May 2012:

WAFL will start to scan 32bit’s volume once Ontap is upgraded to 8.1 . If you have a huge volume, it might take more than 24hours to scan & you are not able to perform the “check” and “normal” command until entire 32bit aggregate is scanned.

mynas03 > aggr add fc_aggr0 -T FCAL -64bit-upgrade check 16@450
 
Note: preparing to add 12 data disks and 4 parity disks.
 
Continue? ([y]es, [n]o, or [p]review RAID layout) y
 
File system size 20.72 TB exceeds maximum 15.99 TB
 
Checking for additional space required to upgrade all writable 32-bit
 
volumes in aggregate fc_aggr0 (Ctrl-C to interrupt)......
 
aggr add: This aggregate has volumes that contain space-reserved
 
files, and the computation of the additional space required to upgrade
 
these files is not yet available. Retry at a later time.

Sample from /etc/messages:

Sun May 27 01:15:05 GMT [mynas03:wafl.scan.start:info]: Starting 64bit space qualifying on volume vol_test_01.
Sun May 27 17:13:00 MYT [mynas03:wafl.scan.64bit.space.done:notice]: The 64-bit space qualifying scanner has completed running on Volume vol_test_01.

Update 20th Feb 2013:

The WAFL scan will not start until the snapmirror relationship is break off. Thanks to locohost & C3 for the info!

I used to turn off snapmirror on the filer before Ontap upgrade, hence my scan is able to proceed. If you have more finding, feel free to post it here 🙂

4 Comments

  1. Thanks for the post

    “Checking for additional space required to upgrade all writable 32-bit
    volumes in aggregate newaggr”

    Hrm, why does it call out writable 32-bit volumes. Does that mean snapmirror destination volumes must be broken first? Hopefully not, maybe it is a reference to snaplock volumes (which would not impact me)

    I upgraded to 8.1.1 a few weeks ago, so we will see how it goes.

  2. The scan does not start until snapmirror break;

    filer> snapmirror break vol_new
    Sat Feb 16 10:14:32 PST [filer:wafl.scan.start:info]: Starting splitting of the inofile on volume vol_new.
    Sat Feb 16 10:14:32 PST [filer:wafl.split.inofile.scan.complete:info]: Volume vol_new Split inofile scan completed.
    Sat Feb 16 10:14:32 PST [filer:wafl.scan.start:info]: Starting 64bit space qualifying on volume vol_new.

  3. TingWei

    February 20, 2013 at 9:53 am

    Hi, locohost/c3,

    I used to turn off snapmirror during the upgrade or maintenance hence i did not know that the scan won’t start until the snapmirror break. Thanks for the info!

    I will append you guys finding on this post.

  4. Great Post – this helped me out a great deal for my 4 DOT updates to 8.1.2

Leave a Reply

Your email address will not be published.

*

© 2018 Thinkway

Theme by Anders NorénUp ↑