Topic 'Re: ESXi PF Exception 14' in forum 'No, I Will Not Fix Your #@$!! Computer' - dslreports.com

Re: ESXi PF Exception 14

Fri, 19 Oct 2012 20:59:13 EDT

JoelC707 posted : These have the two onboard Broadcom 1G chips and a single add-in Intel 1G chip in each. I doubt that is my problem but still good info to have. Thanks!

Re: ESXi PF Exception 14

Fri, 19 Oct 2012 20:56:57 EDT

mkaishar posted : Not sure if it's related to your problem, but I encountered (PF exception 15) last week and it came down to the number of nics...I had too many (4x10G & 4x1G), once I disabled 2 of the 1G nics, then install went smoothly.

Re: ESXi PF Exception 14

Wed, 17 Oct 2012 18:10:24 EDT

JoelC707 posted : Unfortunately I haven't had a chance to play with it yet :(

Re: ESXi PF Exception 14

Wed, 17 Oct 2012 17:24:20 EDT

amungus posted : Quick update:

Fully updated to 5.1 :D

A few issues with the vSphere server itself (low space on C: partition, which was not my fault - will migrate off of this old beast someday), but otherwise smooth sailing.

One really odd issue was encountered with reconnecting the hosts - I got 2/3 of them connected, no prob. The 3rd one came up with an EVC error and said that a VM might be using some incompatible CPU feature. This was odd because all hosts are identical (HP DL 380 G5's). No difference in CPU steppings.
Disabled EVC. Updated all hosts to ESXi 5.1. Enabled EVC. No gripes. Go figure.

VMFS updated on all datastores. No troubles there.

Licensing - all sorted. Nothing crazy there after upgrading online and then applying them.

Still need to do tools/machine version updates. Otherwise, it is a "fully operational deathstar" ...version 5.1... ;)

---

...Any luck on your scratch partition / PF saga, JoelC707?

Re: ESXi PF Exception 14

Tue, 09 Oct 2012 12:26:11 EDT

amungus posted : Right, we are licensed to 2013, and are eligible for "upgrading" the license to 5.x.

My only concern is what this means for the actual ESX hosts themselves. Will they freak out, or will their licenses sit there and happily truck along until I get them upgraded to ESXi 5.1?

Re: ESXi PF Exception 14

Tue, 09 Oct 2012 10:23:28 EDT

JoelC707 posted : The "free" upgrade to v5 sounds basically like SA (Software Assurance) which IIRC VMware does do on the ESX licenses but I've never bought them so I don't know for sure. As for the license check and upgrade migration and what not, that all sounds correct.

Re: ESXi PF Exception 14

Tue, 09 Oct 2012 09:32:13 EDT

amungus posted : Very interesting about the flash disks and "./locker" folder...

I went ahead and updated vCenter last night.

Now comes the fun part of dealing with licenses. Perhaps I don't quite understand this, but, if I "upgrade" licenses, does that not hose me on ESX host licensing??? Their support has been rather useless so far on this front, and I'm kind of confused on how to proceed. I misunderstood that the initial "trial license" was only for new installs, which is completely lame.

Edit: This looks promising, but I'd like to hear from anyone else who's done it... »communities.vmware.com/thread/413129
No "license check" for the hosts? That would be nice... to be able to upgrade existing keys, migrate to 5.1, and call it a day ;)

Re: ESXi PF Exception 14

Thu, 04 Oct 2012 11:09:32 EDT

JoelC707 posted : Alright, so what it looks like it comes down to is the scratch partition or a system BIOS that can't boot GPT partitioned flash disks (that was the only thing in the other thread that wasn't already mentioned, I used gparted from UBCD to wipe the flash disk between attempts).

Scratch partition:
Now that makes sense and I remember something about that. When I installed 5.0 originally as a test, I had no local storage configured and among other things it was bitching about was "no persistent storage for system logs". I didn't care too much about that and it was just a test at the time but I got the same notice with 5.1 with local storage.

So I googled and quickly found this page: »kb.vmware.com/selfservice/micros···=1033696. I followed the vsphere client method, added a /.locker folder on my SAS datastore and told it where to find it. It was happy ever since. Interestingly, 5.0 with local storage did not complain about that and created the /.locker folder and such on it's own.

Now, while confirming all this before this pot I did notice something. I have a /.locker folder on both the SAS and SATA datastores of both hosts. The latest date stamp on the logs is 9/30 in the SAS datastore and 10/4 in the SATA datastore. I'm gonna guess it either didn't want to reuse the old folder or it wanted to put it on the larger datastore (both are entirely possible).

I definately recall creating the /.locker folder on the SAS datastore so I'm going to have to assume the one on the SATA datastore is the one 5.0U1 created during this latest install. I suspect having two is just taking up unnecessary space and not actually causing any problems? I don't remember seeing one on the SATA datastore while trying 5.1 so I don't think two /.locker folders are related to that but I really don't remember (and it doesn't seem to be causing any problems now).

Can't boot GPT formatted disks:
This could make sense to a degree. The servers are from 2008 (Dell 2950 III's) and probably don't support booting from GPT partitions. But I thought 5.0 used GPT partitions too? Or does 5.0 just use it on the datastores and 5.1 attempts to use it on the boot drive as well? Even then, it boots just fine, it's just that it crashes after it boots so if it were really this problem, wouldn't it not boot at all?

Tools first on 5.1 upgrade:
That's one of the first methods I plan to try out in my testing. First, I need to see if I can replicate the full conditions on the Precision 690. I need to make sure it will crash under the same conditions my other hosts are crashing on or else it won't be a worthwhile experiment. I'm even going to try it with HDD boot partition and USB boot and see if that really makes a difference.

Upgrading VMFS:
I've found two documents on that. One is a vmware blog showing the online, non-distruptive upgrade to VMFS-5 as well as seamless transition from MBR to GPT if you upgraded a VMFS-3 datastore and then later expand it beyond 2TB. The other is a PDF that gives considerations for upgrading vs. creating new as some VMFS-3 limitations will remain in an upgraded datastore.
»blogs.vmware.com/vsphere/2011/12···nge.html
»www.vmware.com/files/pdf/techpap···ions.pdf

Re: ESXi PF Exception 14

Thu, 04 Oct 2012 10:07:26 EDT

amungus posted : Cool - I don't think they will as long as we're not going overboard and so OT that it devolves into talking about totally unrelated madness ;)

Actually, I meant "scratch" partition for ESXi itself (should've been more clear.

First, there is this from the vSphere upgrade pdf (page 105):

Partitioning in New ESXi 5.x Installations
In new installations, several new partitions are created for the boot banks, the scratch partition, and the locker.
New ESXi 5.x installations use GUID Partition Tables (GPT) instead of MSDOS-based partitioning.
The partition table is fixed as part of the binary image, and is written to the disk at the time the system is
installed. The ESXi installer leaves the scratch and VMFS partitions blank, and ESXi creates them when the
host is rebooted for the first time after installation or upgrade. The scratch partition is 4GB. The rest of the disk
is formatted as a VMFS5 partition.
NOTE The installer can create multiple VFAT partitions. The VFAT designation does not always indicate that
the partition is a scratch partition. In some cases, a VFAT partition can lie idle.

Second, this might be part of the issue..

page 13:

When installing on USB or SD devices, the installer attempts to allocate a scratch region on an available local disk or
datastore. If no local disk or datastore is found, /scratch is placed on the ramdisk. You should
reconfigure /scratch to use a persistent datastore following the installation

It's possibly breaking somewhere as a "ramdisk" since it's not really meant to use that permanently.

Finally, it could come down to some of the things mentioned in this thead, if you haven't seen it yet - »communities.vmware.com/message/2084277

Modus - that's not too bad really, even with all the errors! I'm hoping to be able to do it in half that, if possible, including reboots for all the VMs themselves (tools, etc.)

Anyone do the tools first, then proceed with everything else??
I'm contemplating that, just so it's out of the way. Also a little scared to update the VMFS versions, but I haven't read any horror stories on that front, so I'll probably just go for it.

Re: ESXi PF Exception 14

Wed, 03 Oct 2012 21:53:10 EDT

Modus posted : It took me about 3 1/2 days mostly because of all the errors i kept running into. Like i said though if you run into any errors just search others have already found the answers :D everything is working a lot better now! I have 3 monster dell poweredge 720's and a new dell compellent san
--
Think Ahead. Learn More. Solve Now!

Re: ESXi PF Exception 14

Wed, 03 Oct 2012 19:12:08 EDT

JoelC707 posted :

said by amungus:

Not to get too far OT, but since the thread is generally discussing 4.x --> 5.x migration trouble, I'm very curious. Page file exceptions are also never cool. It does indeed sound like it's possibly related to flash drive/space - tried another bigger one? I believe you need more space for swap files. Will probably work on your other machine w/more storage.

As long as the mods don't mind, I don't. This is good discussion (and related as far as I'm concerned) and when/if I upgrade again to 5.1 I'm going to need all the insight I can get apparently. And you bet I'll document any issues I run into for others here :)

I haven't tried a different/larger flash drive but I assume you mean VM swap space? If so it's set to store the VM swap files in the same datastore as the VM itself.

I don't know if ESXi itself has a swapfile but 6.8GB of the flash drive is unallocated (and has always been, even when I was on 4.1) so if it wants to use the flash drive for it's own swap space, it didn't anticipate that during install (and there's not much to change during a standard install). That also wouldn't explain why 5.0 works just fine with the same VMs.

Re: ESXi PF Exception 14

Wed, 03 Oct 2012 16:25:18 EDT

amungus posted : Oh frak. Now you all got me worried a bit :(

...looking to do 4.1 ESX to ESXi, across 3 hosts, and also add a brand new DL380 (g8) into the mix once complete (yes, found the 'special' iso for HP already)...

Modus - what happened, and how long did it take to fully migrate? Running everything on 5.1 now and all is good I take it?

Not to get too far OT, but since the thread is generally discussing 4.x --> 5.x migration trouble, I'm very curious. Page file exceptions are also never cool. It does indeed sound like it's possibly related to flash drive/space - tried another bigger one? I believe you need more space for swap files. Will probably work on your other machine w/more storage.

Re: ESXi PF Exception 14

Tue, 02 Oct 2012 21:20:49 EDT

The WeaseL posted : We run View as well, so going to 5.1 wouldn't work anyway (as of last check at their compatibility matrix).
--
How lucky am I to have known someone who is so hard to say good-bye to.

Re: ESXi PF Exception 14

Tue, 02 Oct 2012 19:21:41 EDT

Modus posted : you can upgrade just ready to dig into kb's on vmware's site. I just completed my migration and man i'm just glad it's over! I didn't have to contact support because so many other ran into the issue i had so it wasn't to much of a puzzle
--
Think Ahead. Learn More. Solve Now!

Re: ESXi PF Exception 14

Mon, 01 Oct 2012 11:51:01 EDT

The WeaseL posted : Note to self: Do not upgrade to 5.1 yet.

Re: ESXi PF Exception 14

Mon, 01 Oct 2012 10:53:27 EDT

JoelC707 posted : That's sorta what I did, 4.1 U1 to 5.0 to 5.1. Most VMs had been upgraded to hardware version 8 but some were still on 7 and even after 5.1 I tried version 9 too. Good news on the tools, at least this way I don't have to worry about stripping out all the tools and reinstalling with the 5.0 version (since I'm not 100% sure which ones upgraded and which ones didn't).

So far, downgrading to 5.0 U1 has solved all of my problems. I do want to see about moving to 5.1 though so I'll probably have to take a spare system and run some sandboxed tests. Fortunately my Precision 690 runs ESXi nicely and has the storage space needed :)

Re: ESXi PF Exception 14

Mon, 01 Oct 2012 10:36:47 EDT

amungus posted : From what I've read, the tools can be run prior to upgrade, and are backwards compatible.

Getting ready to move from ESX 4.1 to ESXi 5.1 here..

Re: ESXi PF Exception 14

Sun, 30 Sep 2012 21:00:26 EDT

JoelC707 posted : Yeah I've seen that too but knowing little about ESXi debugging, that meant less to me than a page fault error LOL. I did upgrade the tools after the upgrade, should I upgrade beforehand instead? Is there anything wrong with running the 5.1 tools on 5.0? I know you can download the tools ISO so it's certainly possible to upgrade first, I just never have done it that way (should I?).

Re: ESXi PF Exception 14

Sun, 30 Sep 2012 20:48:45 EDT

izy posted : Not sure if you've seen this yet but is definitely pointing towards a guest issue.

»kb.vmware.com/selfservice/micros···=1019956

Did you update the guest tools "after" you upgraded ESXi to 5.1?
--
"Intellectuals solve problems, geniuses prevent them." Einstein

Re: ESXi PF Exception 14

Sun, 30 Sep 2012 12:36:51 EDT

JoelC707 posted : I was just shutting down VMs on the host that crashed with the not_implemented error again. I got all shutdown but one before it crashed with that error again. I'm gonna downgrade it to 5.0 U1 as well.

Re: ESXi PF Exception 14

Sun, 30 Sep 2012 10:36:20 EDT

JoelC707 posted : You mean after downgrading it to 5.0 U1? If so, yeah to an extent. One VM is a DC and I've tried uninstalling and reinstalling the tools but it still shows "Not running, not installed"; though the mouse, video and vmxnet3 driver are installed.

I'm replacing that DC shortly so I'm not too worried about it. The others either were 5.0 (no updates) and got upgraded, or were 5.1 and haven't complained yet. I should probably remove and reinstall all the tools just to make sure they are on the same version.

Re: ESXi PF Exception 14

Sun, 30 Sep 2012 06:48:39 EDT

izy posted : Did you upgrade the guest tools on the affected VM's after upgrading the host?
--
"Intellectuals solve problems, geniuses prevent them." Einstein

Re: ESXi PF Exception 14

Sat, 29 Sep 2012 22:44:20 EDT

JoelC707 posted : Well.... 5.0.0 U1 on this host. Loaded ALL the VMs one at a time. So far no crashes. I've rebooted a few upgrading VM tools too. Must be some kind of bug in 5.1.

Re: ESXi PF Exception 14

Sat, 29 Sep 2012 20:32:48 EDT

JoelC707 posted : Converting to Hyper-V is going to mean rebuilding a lot of the VMs and that licensing/activation thing I wanted to avoid. I tested conversions on some of the VMs (copies of them) and they generally converted without issue but the apps running on them were deactivated due to hardware changes (a given). If I've got to rebuild most of my VM's I might as well go that route but I'd prefer to get them running as is.

I'm thinking I'm going to downgrade this host in question to 5.0 and see if the original VMs will launch without crashes. If it will, maybe it's some bug in 5.1 and I'll just stick with 5.0 instead.

Re: ESXi PF Exception 14

Sat, 29 Sep 2012 20:26:33 EDT

aguen posted : I wasn't aware from your OP that the other host had already been upgraded, so now you probably need to make a decision as to how quickly you need to get these 2 hosts sorted out I guess. At this point, as you were already considering migrating to hyper-v maybe you should go that route. Otherwise, I'd try to go back to 5.0, if your VM's run ok again, then try patching the 5.0 up to whatever level it currently is and see how that goes. After that, try the upgrade again.

Re: ESXi PF Exception 14

Sat, 29 Sep 2012 20:10:47 EDT

JoelC707 posted : OK maybe I'm barking up the wrong tree here? Maybe it's 5.1 in general? I was shutting down VMs on my other host and got the following crash message:
NOT_IMPLEMENTED bora/vmkernel/sched/memsched.c:17724

Here's the kicker, I got that crash (identical, wrote it down) repeatedly on the original host in question here after the upgrade from 5.0 to 5.1. Every time I booted the host it would crash with that error within minutes. I eventually figured it was something that didn't upgrade properly so I wiped the flash drive and did a fresh install of 5.1 and never saw the error again.

This other host hasn't shown that error before either until now and it was on VM shutdown? This other host was also a fresh install of 5.1 right off the bat (was going to make this host Hyper-V but decided to do ESXi again on it) so there wasn't an "upgrade anomaly" to deal with. If it's not 5.1 in general, maybe it's the flash drive(s)?

Re: ESXi PF Exception 14

Sat, 29 Sep 2012 19:58:20 EDT

JoelC707 posted : I have not tried that yet. I do want to shut down the existing VMs on the other host and upload one of the troublesome VMs and see what it does on that host. I fully expect it to crash as well but it will help narrow things down. Both hosts are on 5.1 already. What's the best way to export/import? All I've really been doing is downloading from the datastore, uploading to the datastore and adding the VMX to inventory or editing the existing VM to see the new vmdk.

What other info would I need to provide to help diagnose this? I've tried some methods found on Google to collect just the crash dump logs but either I'm doing something wrong or it's one of those subtle changes between 4.x/5.0 and 5.1? I do have the full system logs (172MB worth) that I can upload to dropbox or somewhere and post a link to if that would help?

Re: ESXi PF Exception 14

Sat, 29 Sep 2012 19:36:18 EDT

aguen posted : Well then the error message is totally meaningless as it stands.
Can you export one of your VM's running on your other host and then import it into the new server? Maybe there was some not so subtle change in the various attributes between 5.0 and 5.1. Importing into 5.1 "might" correct for that.

Re: ESXi PF Exception 14

Sat, 29 Sep 2012 19:24:23 EDT

JoelC707 posted : Flash drive is 8GB (Patriot Extreme model) and is only a couple of years old. AFAIK the swap space is not on the flash drive and instead it uses one of the datastores. I don't think it really has anything to do with the ESXi OS since I can stand up an entirely new VM and it will run with no problems, it's only the original VMs that cause it to crash.

Re: ESXi PF Exception 14

Sat, 29 Sep 2012 19:16:33 EDT

aguen posted : How big is that flash drive and how old is it? If I'm understanding the writeup correctly, the problem appears to be related specifically with the swap space for the esxi OS. I'm assuming it also located on the flash drive?

ESXi PF Exception 14

Sat, 29 Sep 2012 18:54:22 EDT

JoelC707 posted : I'm having a bit of a problem with one of my ESXi servers. I recently upgraded it from 5.0.0 (no updates) to 5.1.0 (usb flash disk for the host). I downloaded all the VMs prior to the upgrade just in case but apparently whatever happened occurred before the copy. Upon booting up of the new install it proceeds to auto-start my VMs like I've configured and a few minutes into boot up it crashes with a PF 14 error.

This is my first crash ever of ESXi and I'm using the free version so I'm limited in what tools I have available to me. I've used the vsphere client to pull the system logs and I've also done some Google sleuthing on the error. I did come across: »kb.vmware.com/selfservice/micros···=1020181 but I'm not that familiar with debugging ESXi (though I do know what a page fault is).

I did add two 750 GB SATA drives in RAID 1 but it didn't matter what array I ran the VMs from, it would crash the host. Thinking it might be something with the new array, I broke it and ran a self test on both drives. I also ran memtest 86+ on the server (only 1 pass). Both came up with no errors. At this point I've realized it must be something with the VM themselves.

I rebuilt one of the VMs from scratch and attached it's secondary disk (a database server). It didn't crash but upon trying to reattach the database, it tells me it's corrupt. I restore the database from backup and it's been running along happily now for over a week (on the SATA array no less) so I doubt it's the hardware. Any original VM will crash the host.

I've even tried recreating the VM and attaching the disk manually (to rule out any corrupt config file or something) but no luck yet. I could bring up a newly built VM for each one and copy data over (or restore data from backup) but I'd prefer to get these VMs working again so I don't have to go through application activation/licensing issues. Anyone have any ideas? I can post the logs if anyone would like to take a look but they are pretty big.