AkinsTech

Recently I have been migrating my home virtual server and lab environment from Hyper-V to VMWare ESXi 4.0.  In doing this I found that ESXi has a problem recognizing all of the space on my Dell PERC5 RAID array.  I’m not sure if this is a bug, or just bizarre behavior between the two, but after several days of googling and checking forums I was unable to find a solution.  After troubleshooting the problem a little further, I’ve come up with a workaround.

First off, I have a Dell PERC5 RAID card in my server.  I have 4 Samsung 1TB SATA drives configured in RAID5 which creates a usable disk of 2.73TB.  But when I try to add storage in ESXi, and select this device, it only recognizes 745GB, even though the disk is empty.  See below…

Now I already know that ESX/ESXi (I’ll just use the word “ESX” interchangably to refer to ESX or ESXi) has a 2TB limit on VMFS volumes, as shown in this document.  But instead of presenting me with the ability to create a 2TB vmfs volume, it’s giving me 2.73TB-2TB Limit = .73TB (or 745GB).

I experimented a bit, and rebuilt my array as a 3x1TB, which creates a 1.82GB disk.  Turns out that ESX was able to see all of the space!  So clearly the problem exists when you present ESX an array large2 than the 2TB limit.  But I want the full 2.73TB…and I don’t want to split my array up into a smaller disk set.  So here’s how to get around this.

Note: This will destroy everything on your RAID array, so make sure you back up all your data.

1. Reboot your server and press CTRL-R to enter the PERC5 Configuration Utility.
2. Delete your old RAID array (Virtual Disk).
3. Now create the new one.  Highlight “Controller 0″ and press F2. Select “Create New VD”
4. Select your RAID type. (In my case RAID5)
5. Select the disks. (In my case, all 4 of the 1TB disks)
6. Select your Advanced Settings
   • Element Size: 128K
   • Read Policy: Adaptive Read Ahead
   • Write Policy: Write Back
7. Then, under “Basic Settings” set the VD Size to 2097152MB
8. Then select OK.  This will create a 2TB array just under ESX’s 2TB-512B limit.
9. Now go back and create another virtual disk with your remaining space.
10. Expand “Space Allocation” with the right-arrow
11. You will see “Free Space”.   Highlight this and press enter.  This will create another VD using your free space.

You will now have 2 virtual disks on the controller…one 1.99TB and the other 745GB.

Exit out of the PERC5 Cofiguration utility and reboot into ESX.

Now go to Configuration, Storage, click on “Devices” and you’ll notice we have two disks presented to ESX….one 2.00TB and one 745GB.  Before you do anything, it’s a good idea to rename the devices to something meaningfull.  Right click each one and select rename.  I named mine PERC5 Array 0-0 and PERC5 Array 0-1, respectively.

Now click on Datastores and then click Add Storage.  Select Disk/LUN, and then select the 2TB disk below and click next.

Select your block size, and give it a name…and so on. 

When finished you will have a 2TB datastore.

Now, you can create a second datastore from the other 745GB partition, but I don’t like that idea as it fragments the storage.  I want my full 2.73TB to be one contiguous datastore.  So for that we’ll use an extent. 

But wait, aren’t extents bad?

If you’ve done any research on extents you might find many recommendations in forums and blogs against them.  Unfortunately much of the commentary regarding extents is mired in misconception and fear.  An extent simply distributes a vmfs volume across multiple disks or LUNs.  The basic fear is that a failure in one of those disks or LUNs would cause most or all of that VMFS volume to fail.  I agree with that argument and try to avoid using extents across multiple sets of spindles.

However, in OUR case, our two virtual disks are located on the SAME set of spindles…just as if we had created one large virtual disk.  If we lose disks in our RAID array, both disks disks will go down, no matter how it is configured.  So by using an extent we are not adding or creating any additional risk of failure in our situation. 

To create the extent, simply right-click the 2TB datastore you created, and select properties.

Click the Increase button.  Then select your remaining 745GB disk.

The 745GB disk will be added to the 2.0GB disk.  Now click Close.

Now you’ll see that our data1 datastore shows up as a full 2.73TB.

Hopefully this solution helps others trying to run large arrays on a PERC5 using VMWare ESX.

This entry was posted on Tuesday, March 9th, 2010 at 11:47 am and is filed under Virtualization. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.