GREMBLING22
join:2000-10-14
Southbridge, MA
| reply to GREMBLING22
Re: Updated VMware / Home Lab setup.
Well after a firmware update to my P800 SAS controller it is no longer supported by openfiler and I cannot use it.
I received my Infiniband cables, switch and hca cards last week. I was able to get the OpenIB over IP drivers installed on ESX. Esx now recognzes 2 storage adapters and 2 network adapters (my cards are dual port) at a network speed of 10000. Openfiler detected the cards but I was unable to install the drivers. I now need to get a copy of Redhat EL install the infinband drivers on that.
Instead of using ISCSI for storage i will be using the infiniband RDMA which does not have the same overhead as iscsi. |
|
lightspd
join:2005-03-04
UK
| reply to GREMBLING22
I don't know anything much about openfiler besides the fact that it is linux-based. If you're lucky that means it supports DRBD which would allow you to have a real-time mirror on a second server. DRBD is effectively network RAID1. You might then be able to use multipath ISCSI to achieve transparent failover if the main storage server dies for some reason.
I don't use vmware myself, but have a DRBD + OCFS2 home-made SAN that works on vaguely similar principles. |
|
GREMBLING22
join:2000-10-14
Southbridge, MA
| reply to lightspd
My ESX capture with HA and DRS enabled on a cluster of 2 ESX hosts.
I have quite a few pics I was hoping to upload but I broke the mini usb interface on my cam. As soon as the card reader comes in I will be adding the pics of the build. |
|
GREMBLING22
join:2000-10-14
Southbridge, MA
2 edits | reply to GREMBLING22
As my setup progresses the whole setup will be moved to a datacenter where she can enjoy 10/Mbps unmetered bandwidth on a multi-homed facility. The best part of the whole thing is it will be racked in the same facility I presently work in. Although not being in the same room as the equipment I monitor, the servers will be a few doors away so if problems do arise i can walk over and check it out.
Overall the power draw is around 570 watts idle, I haven't gone into testing them under full load. Considering the last dual core dual proc. board with 24GB of ram and attached Dell storage array with 8x74GB U320 15K drives drew about 420watts I am more then impressed that 4 dual proc quad core servers draw only a little more power. Keep in mind each server recieved 400 watt 80 + certified PSU's which I swear by... they really do make a difference.
Right now I look at my weak point as being the ISCSI box. All storage is shared and therefor obviously if that box goes down all servers come to a halt. Does openfiler offer some sort or mirrored backup for ISCSI that would kick in if the primary server went down? |
|
lightspd
join:2005-03-04
UK | reply to GREMBLING22
Wow, that is an awesome deal you got for all the hardware! So what is the overall power draw of the whole setup? All you need now is a second filer so that you can do storage vmotion  |
|
GREMBLING22
join:2000-10-14
Southbridge, MA | reply to grembling222
I was finally able to mess around with some settings and was able to get my intel adapter to hit 85MB/s. Im happy with that but im hoping for more. |
|
grembling222
@charter.com
| reply to watts3000
what type of speeds are you getting? While the ESX service console supports jumbo frames, the VMKernel does not. Meaning ISCSI on VMWare does not currently support JUMBO frames. There are a few tweaks you can run via command line to enable them on the vswitch and VMKernel.
Do your poweredge have ISCSI and TCP/IP offload engine? |
|
watts3000
join:2002-01-21
Birmingham, AL
| reply to GREMBLING22
I have not been able to obtain line speed of gig and I'm running it on a dell poweredge 2950 4 gigs of ram and 6 300 gig 15k sas drives. I plan on doing more testing and tweaks to try to get the speed up. BTW what are you going to do about a layer 3 switch |
|
GREMBLING22
join:2000-10-14
Southbridge, MA
| reply to watts3000
All of this has been out of pocket. Believe it or not though this setup of 4 ESX servers and one openfiler box has only set me back a little over $3000. I sold my first Dual Core Dual Proc ESX box for about $1200 so the new servers only cost me about $1800.
You said you were using openfiler? My current tests on the Intel Q9550 with 8GB of ram and dual port Intel Pro 1000MT only gets me a max of 60MB/s on a VM over ISCSI. Have you been able to obtain at least close to line speed of GigEthernet? |
|
watts3000
join:2002-01-21
Birmingham, AL
| reply to GREMBLING22
You have a really nice esx setup. So are you going to take the vcp exam or do you already have your vcp. I took a fast track class on 3.5 a few months back so it's time for me to try to get some type of lab together at home so I can prep for the exam. Or I could just use a couple dell 2950's at work to build a lab. I must ask the equipment you have cost a good bit of cash are you paying for this put of pocket or does your company pick up the tab |
|
GREMBLING22
join:2000-10-14
Southbridge, MA
| reply to watts3000
Actually my whole Idea for the esx buildout has changed.
I won an auction for (8) 55watt quad core opterons, I also won an action for 4GB x 32 chips of ram.
My buildout will now consist of 4 Dual proc. Socket F motherboards with 2 quad core 1.8Ghz 55watt opterons and 32GB of ram per motherboard (motherboards max for mem).
Each of the 4 Tyan motherboards will recieve a Qlogic 4010c ISCSI hardware initiator to offload ISCSI traffic.
The current Intel ESX server (Q9550 8GB of ram) will be converted into an Openfiler ISCSI box. This box will have (4) SAS 147GB 15K drives in raid 10, and (6) 1.5TB SATA drives in raid 6 (home file storage). The SAS controller is an HP P800 512MB 16port controller which can handle SATA as well. The ISCSI box will also recieve 3 Qlogic 4010 initiators. (two for esx iscsi, one for home network filestorage). |
|
watts3000
join:2002-01-21
Birmingham, AL
1 edit | reply to GREMBLING22
I'm going to get serious about a esx setup for home over the next few months. However, I plan to do mostly iscsi and nfs for storage so I will be investing in building a decent stotage setup running openfiler. Also I will try to boot esx3i from flash either a usb key or some type of cf adapter thus allowing me to avoid running hard drives in my esx host. I have not decided what I'm going to do for the esx host I've looked at a few asus barebones.
However, I'm going to check out the itx stuff I've run openfiler 2.3 with every flavor of esx 3. and I've never had any problems. I don't want to put out a lot of cash on a hardware raid controller so I'm going to do software raid with filer. I think your plan is great on buying the intel q9550s with 16 to 32 ggs of ram you will be able to test tons of stuff for example exchange 2007, sql server 2005, sql 2008. You will be able to do any clustering setup under the sun and get great performance. Now have you thought about getting a good layer 3 gig switch. They can get expensive so I might get a switch that can do vlans but has no routing ability I will than run a trunk to a monowall box and let monowall route the vlan traffic its amazing what you can do with some intel nics. Overall. I don't want to spend a ton of money on the host imo it's all about the backend storage as long as my host have a good number of pci slots for nics than I'm good |
|
GREMBLING22
join:2000-10-14
Southbridge, MA | reply to IT2_USN
After searching for the past couple of days I am now not so sure it was mini-itx. I believe what I was looking at was micro-atx. The case for dual mini-itx can be found on ebay as a no-name by searchin "dual mini-itx rackmount". |
|
IT2_USN
@comcast.net | reply to GREMBLING22
May I ask what brand and model of mini-itx board you're looking at? Also, which 2u case you found that is made for 2 mini-itx boards? Thanks. |
|
morpheus6d9
join:2003-11-18
Jersey City, NJ | reply to GREMBLING22
i have the same server case nice setup |
|
DaMaGeINC
The Lan Man
Premium
join:2002-06-08
Greenville, SC
clubs: | reply to GREMBLING22
I got quite a few things. »Video of some new network gear! |
|
GREMBLING22
join:2000-10-14
Southbridge, MA | reply to Bollocks
lol hahah it depends! What can you add to the rack? haha |
|
Bollocks
@comcast.net | reply to GREMBLING22
Looking for Roomates??? EHEHE |
|
DaMaGeINC
The Lan Man
Premium
join:2002-06-08
Greenville, SC
clubs: | reply to GREMBLING22
Words spoken by sooo many of us! |
|
GREMBLING22
join:2000-10-14
Southbridge, MA | reply to GREMBLING22
I look at it as furthering my education, and you cant put a price on that besides I need some way to justify why i spend more time with the servers that I do my girl! LMFAO |
|
