GREMBLING22
join:2000-10-14
Southbridge, MA
1 edit | Updated VMware / Home Lab setup. Last year I built a ESX rackmount server for my Home lab. As power usage was an issue and Quad core proc. became more powerful I decided it was time for an upgrade.
My old VMWare ESX setup:
2x 2.0Ghz Dual Core Socket F
Tyan HT100M Serverworks Board
Dual Core Dual Proc Socket F Opterons
24GB of ram
3 U160 18GB (ESX Host)
5 U320 15k 73GB (VMFS)
Intel PRO 1000 Dual port
My New Setup:
Intel Quadcore Q9550
ASUS P5BV/SAS (8 SAS/SATA on LSI SAS + 4 SATA on Intel SATA)
8GB of RAM
1 147GB SAS 15K (ESX Host)
4 147GB SAS 15K (VMFS)
Intel Pro 1000 MT (Dual port)
HP P600 512MB Battery backed 8 port SAS
400 Watt 80 Plus certified PSU
Now I just built my intel setup about 2 weeks ago and so far it has been amazing. Currently it is using about 130-150 watts idle versus 213-240 the other one used. 8GB is MAX this motherboard can support. Ill post pics of my rack when I get home and have a chance to add the 4 APC 2U 980Watt UPS's I got yesterday  . | |
|
 GREMBLING22
join:2000-10-14
Southbridge, MA
| Re: Updated VMware / Home Lab setup. While looking around the net i stumbled upon the newer mini-itx motherboards which supported 16GB of ram and a Quad core proc. I was also able to find a 2U Rackmount case that fit 2 mini ITX motherboards.
Im hoping to get vmware esx setup on these and have 2 high powered esx hosts that consumed low power. I would buy a lower wattage Quad core proc. for my curent intel esx server and convert it to run iscsi for the 2 mini itx servers.
I think with the ICH7-M chipset the mini itx computer uses my host drive should be recognized, and with all VMFS going over iscsi to 15K sas drives i should haven't the problem most people have with getting a compatible SATA or SAS controller for the VMFS storage. The MINI-ITX boards only draw about 70 watts idle, which is half oh my Current Intel VMWAre server.
With this setup I will be able to go further into testing ISCSI, VMotion and HA.
What do you guys think of buying the case, 16GB of ram per motherboard (32GB total), a couple of Q9550's, and putting them as esx hosts? | |
|
 | 
IT2_USN
@comcast.net | Re: Updated VMware / Home Lab setup. May I ask what brand and model of mini-itx board you're looking at? Also, which 2u case you found that is made for 2 mini-itx boards? Thanks. | |
|
| | GREMBLING22
join:2000-10-14
Southbridge, MA | Re: Updated VMware / Home Lab setup. After searching for the past couple of days I am now not so sure it was mini-itx. I believe what I was looking at was micro-atx. The case for dual mini-itx can be found on ebay as a no-name by searchin "dual mini-itx rackmount". | |
|
| | | watts3000
join:2002-01-21
Birmingham, AL
1 edit | Re: Updated VMware / Home Lab setup. I'm going to get serious about a esx setup for home over the next few months. However, I plan to do mostly iscsi and nfs for storage so I will be investing in building a decent stotage setup running openfiler. Also I will try to boot esx3i from flash either a usb key or some type of cf adapter thus allowing me to avoid running hard drives in my esx host. I have not decided what I'm going to do for the esx host I've looked at a few asus barebones.
However, I'm going to check out the itx stuff I've run openfiler 2.3 with every flavor of esx 3. and I've never had any problems. I don't want to put out a lot of cash on a hardware raid controller so I'm going to do software raid with filer. I think your plan is great on buying the intel q9550s with 16 to 32 ggs of ram you will be able to test tons of stuff for example exchange 2007, sql server 2005, sql 2008. You will be able to do any clustering setup under the sun and get great performance. Now have you thought about getting a good layer 3 gig switch. They can get expensive so I might get a switch that can do vlans but has no routing ability I will than run a trunk to a monowall box and let monowall route the vlan traffic its amazing what you can do with some intel nics. Overall. I don't want to spend a ton of money on the host imo it's all about the backend storage as long as my host have a good number of pci slots for nics than I'm good | |
|
| | | | GREMBLING22
join:2000-10-14
Southbridge, MA
| Re: Updated VMware / Home Lab setup. Actually my whole Idea for the esx buildout has changed.
I won an auction for (8) 55watt quad core opterons, I also won an action for 4GB x 32 chips of ram.
My buildout will now consist of 4 Dual proc. Socket F motherboards with 2 quad core 1.8Ghz 55watt opterons and 32GB of ram per motherboard (motherboards max for mem).
Each of the 4 Tyan motherboards will recieve a Qlogic 4010c ISCSI hardware initiator to offload ISCSI traffic.
The current Intel ESX server (Q9550 8GB of ram) will be converted into an Openfiler ISCSI box. This box will have (4) SAS 147GB 15K drives in raid 10, and (6) 1.5TB SATA drives in raid 6 (home file storage). The SAS controller is an HP P800 512MB 16port controller which can handle SATA as well. The ISCSI box will also recieve 3 Qlogic 4010 initiators. (two for esx iscsi, one for home network filestorage). | |
|
| | | | | watts3000
join:2002-01-21
Birmingham, AL
| Re: Updated VMware / Home Lab setup. You have a really nice esx setup. So are you going to take the vcp exam or do you already have your vcp. I took a fast track class on 3.5 a few months back so it's time for me to try to get some type of lab together at home so I can prep for the exam. Or I could just use a couple dell 2950's at work to build a lab. I must ask the equipment you have cost a good bit of cash are you paying for this put of pocket or does your company pick up the tab | |
|
| | | | | | GREMBLING22
join:2000-10-14
Southbridge, MA
| Re: Updated VMware / Home Lab setup. All of this has been out of pocket. Believe it or not though this setup of 4 ESX servers and one openfiler box has only set me back a little over $3000. I sold my first Dual Core Dual Proc ESX box for about $1200 so the new servers only cost me about $1800.
You said you were using openfiler? My current tests on the Intel Q9550 with 8GB of ram and dual port Intel Pro 1000MT only gets me a max of 60MB/s on a VM over ISCSI. Have you been able to obtain at least close to line speed of GigEthernet? | |
|
| | | | | | | watts3000
join:2002-01-21
Birmingham, AL
| Re: Updated VMware / Home Lab setup. I have not been able to obtain line speed of gig and I'm running it on a dell poweredge 2950 4 gigs of ram and 6 300 gig 15k sas drives. I plan on doing more testing and tweaks to try to get the speed up. BTW what are you going to do about a layer 3 switch | |
|
| | | | | | | |
grembling222
@charter.com
| Re: Updated VMware / Home Lab setup. what type of speeds are you getting? While the ESX service console supports jumbo frames, the VMKernel does not. Meaning ISCSI on VMWare does not currently support JUMBO frames. There are a few tweaks you can run via command line to enable them on the vswitch and VMKernel.
Do your poweredge have ISCSI and TCP/IP offload engine? | |
|
| | | | | | | | | GREMBLING22
join:2000-10-14
Southbridge, MA | Re: Updated VMware / Home Lab setup. I was finally able to mess around with some settings and was able to get my intel adapter to hit 85MB/s. Im happy with that but im hoping for more. | |
|
|
|
DaMaGeINC
The Lan Man
Premium
join:2002-06-08
Greenville, SC
clubs: | And how much did all of this set you back?? | |
|
GREMBLING22
join:2000-10-14
Southbridge, MA
| The newest intel server only cost about $1325 give or take a little.
Overall I was very impressed with price. It cost me more then twice that to build my AMD server about a year ago.
As i think more and more about converting the current Intel ESX server into ISCSI and making some mini itx esx hosts, it makes more and more sense. I have ordered the mini-itx motherboard, and a couple of Qlogic hardware ISCSI initiators to help offload someone of the resources of converting to ISCSI.
The plan is to start with a Q9550 I have lying around and 2 gigs of DDR3 1600 to see if I can install ESX. If ESX installs I will be ordering another motherboard, 16GB of ram per ITX computer, and another Q9550. | |
|
|
|
DaMaGeINC
The Lan Man
Premium
join:2002-06-08
Greenville, SC
clubs: | Re: Updated VMware / Home Lab setup. You take your servers very seriously I can see. | |
|
| GREMBLING22
join:2000-10-14
Southbridge, MA | I look at it as furthering my education, and you cant put a price on that besides I need some way to justify why i spend more time with the servers that I do my girl! LMFAO | |
|
| |
DaMaGeINC
The Lan Man
Premium
join:2002-06-08
Greenville, SC
clubs: | Re: Updated VMware / Home Lab setup. Words spoken by sooo many of us! | |
|
Bollocks
@comcast.net | Looking for Roomates??? EHEHE | |
|
| GREMBLING22
join:2000-10-14
Southbridge, MA | Re: Updated VMware / Home Lab setup. lol hahah it depends! What can you add to the rack? haha | |
|
| | |
morpheus6d9
join:2003-11-18
Jersey City, NJ | i have the same server case nice setup | |
|
lightspd
join:2005-03-04
UK | Wow, that is an awesome deal you got for all the hardware! So what is the overall power draw of the whole setup? All you need now is a second filer so that you can do storage vmotion | |
|
| GREMBLING22
join:2000-10-14
Southbridge, MA
| Re: Updated VMware / Home Lab setup. My ESX capture with HA and DRS enabled on a cluster of 2 ESX hosts.
I have quite a few pics I was hoping to upload but I broke the mini usb interface on my cam. As soon as the card reader comes in I will be adding the pics of the build. | |
|
GREMBLING22
join:2000-10-14
Southbridge, MA
2 edits | As my setup progresses the whole setup will be moved to a datacenter where she can enjoy 10/Mbps unmetered bandwidth on a multi-homed facility. The best part of the whole thing is it will be racked in the same facility I presently work in. Although not being in the same room as the equipment I monitor, the servers will be a few doors away so if problems do arise i can walk over and check it out.
Overall the power draw is around 570 watts idle, I haven't gone into testing them under full load. Considering the last dual core dual proc. board with 24GB of ram and attached Dell storage array with 8x74GB U320 15K drives drew about 420watts I am more then impressed that 4 dual proc quad core servers draw only a little more power. Keep in mind each server recieved 400 watt 80 + certified PSU's which I swear by... they really do make a difference.
Right now I look at my weak point as being the ISCSI box. All storage is shared and therefor obviously if that box goes down all servers come to a halt. Does openfiler offer some sort or mirrored backup for ISCSI that would kick in if the primary server went down? | |
|
| lightspd
join:2005-03-04
UK
| Re: Updated VMware / Home Lab setup. I don't know anything much about openfiler besides the fact that it is linux-based. If you're lucky that means it supports DRBD which would allow you to have a real-time mirror on a second server. DRBD is effectively network RAID1. You might then be able to use multipath ISCSI to achieve transparent failover if the main storage server dies for some reason.
I don't use vmware myself, but have a DRBD + OCFS2 home-made SAN that works on vaguely similar principles. | |
|
GREMBLING22
join:2000-10-14
Southbridge, MA
| Well after a firmware update to my P800 SAS controller it is no longer supported by openfiler and I cannot use it.
I received my Infiniband cables, switch and hca cards last week. I was able to get the OpenIB over IP drivers installed on ESX. Esx now recognzes 2 storage adapters and 2 network adapters (my cards are dual port) at a network speed of 10000. Openfiler detected the cards but I was unable to install the drivers. I now need to get a copy of Redhat EL install the infinband drivers on that.
Instead of using ISCSI for storage i will be using the infiniband RDMA which does not have the same overhead as iscsi. | |
|
|
|
|
·
