My Dream System (I think)

[joeschmuck]

Well I was going for a single computer to run the firewall and FreeNAS, as well as other things but I can see why someone would need or desire a separate router/firewall and I did have that at one time. I just had an open window of opportunity and I tried to remove the "upsmon" VIB but that failed with the error message that it's in use. Of course that would happen so now I need to stop it's process (need to figure out how to do that first) and try again, but my window of opportunity has closed again when my wife said "Are you doing something to the internet again!". I've spent enough time on the computer today anyway so tomorrow morning I'll connect my backup router and then play with ESXi to get my updates done. I need to document this for myself anyway for the future updates I'll need to install.

 

[Mirfster]

"Are you doing something to the internet again!"

Lol, yeah a separate router would at least keep those comments at bay. ;) I prefer to sneak down very early in the AM to do work. Wife and kids are asleep so no noise or distractions and I can reboot anything my little heart desires. :D

 

[jgreco]

If you want a good "router" that's actually got sufficient software and oomph to actually be considered a Router, look at the Ubiquiti EdgeRouter Lite 3 (~$90) or EdgeRouter X (~$50).

 

[AlainD]

If you want a good "router" that's actually got sufficient software and oomph to actually be considered a Router, look at the Ubiquiti EdgeRouter Lite 3 (~$90) or EdgeRouter X (~$50).

Hi

For me it was and is unclear what the practical difference is between those two. I see higher routing specs for the Ubiquiti EdgeRouter Lite 3, but doubt that they are "feelable" in a home or small SOHO setup. Routing speed for 64byte packets is about 8x times higher for the lite 3 (1million versus 130k 64 bytes pps )

If I'm reading it correct on the X it's possible to use one as an "uplink" router port and the other 4 as a internal line rate switch.

Any info where the lite 3 would be advisable against de X?

 

[joeschmuck]

If you want a good "router" that's actually got sufficient software and oomph to actually be considered a Router, look at the Ubiquiti EdgeRouter Lite 3 (~$90) or EdgeRouter X (~$50).

Nope, not looking for a new router at all. Sophos does what I need and in a pinch, the router I have as a backup is fine even if it doesn't block much. I also have Norton Security on all the computers because I'll never trust a firewall completely.

BTW, the video for the EdgeMax was funny.

 

[Dice]

@joeschmuck What is the logic behind the distribution of RAM among VM's in your box? (a lot unused?)

Cheers,

 

[joeschmuck]

@joeschmuck What is the logic behind the distribution of RAM among VM's in your box? (a lot unused?)

Cheers, Dice

True, there is a lot unused at the moment however the current applications only require a certain amount so I see no benefit of giving them more than they require to do the task. And for my needs, I'm being very generous giving FreeNAS that 16GB of RAM but also that 16GB is locked so it's always allocated to FreeNAS. BTW, I did run FreeNAS without the RAM locked for a few months and I didn't see any issues however I don't want to take any chances with my data so I locked it. Should someone state that it doesn't need to be locked then I might unlock it and allow the RAM to be managed by ESXi. I'm also running a single instance of Ubuntu which runs all the time and periodically a VM of Windoze 7 which I plan to run 24/7 once I get my thoughts down on what I really want to do with it.

So the answer to your question is I only assign the amount of RAM I need to a VM. I did fully populate the RAM in my ESXi system because I had no idea what I would use it for in the future and I do plan to keep this system for at least 10 years, if not longer. Along those same lines I also assign the number of CPUs I desire and of course ESXi will need to share resources as needed.

@Dice out of curiosity, would you have done it differently and if so, why?

 

[jgreco]

Hi

For me it was and is unclear what the practical difference is between those two. I see higher routing specs for the Ubiquiti EdgeRouter Lite 3, but doubt that they are "feelable" in a home or small SOHO setup. Routing speed for 64byte packets is about 8x times higher for the lite 3 (1million versus 130k 64 bytes pps )

If I'm reading it correct on the X it's possible to use one as an "uplink" router port and the other 4 as a internal line rate switch.

Any info where the lite 3 would be advisable against de X?

Sure. The EdgeRouter Lite 3 uses a Cavium processor at 500 MHz that has hardware assisted packet processing capabilities, which means that as long as you stay "in silicon" you can be processing over 1Mpps. I'm not actually clear on which things are hardware assisted, and it's likely that NAT or firewalling or rate limiting has to involve the host OS. However, if you are merely shuffling packets between subnets, you get a high performance router.

The EdgeRouter X has a faster 800 MHz processor without the hardware assist. This is a cheaper platform, despite the additional speed. There are five gigE ports, each of which may be attached either to the host, or to an internal switch. This means you can do anything from making a 5 port switch that happens to have a small Linux host attached to it (maybe good for DHCP/DNS/etc service), to a 4 port switch with NAT upstream (typical home "router"), to a true router with five different subnets.

From a manageability point of view, the EdgeRouter X lacks the out-of-band console port, so you're screwed and have to reset the unit if you have a config-tastrophe.

Neither of them are particularly zippy at things like OpenVPN, where the numbers I hear are usually around 10 megabits per second.

You are not likely to be reaching the packet per second limits on either device unless you have something unusual going on. A hundred machines doing online gaming, high volumes of SIP traffic, being a DDoS target (or having an infected machine and being a DDoS source), stuff like that.

 

[jgreco]

True, there is a lot unused at the moment however the current applications only require a certain amount so I see no benefit of giving them more than they require to do the task.

And that's a good strategy. Make sure you're also assigning CPU cores that way too.

And for my needs, I'm being very generous giving FreeNAS that 16GB of RAM but also that 16GB is locked so it's always allocated to FreeNAS. BTW, I did run FreeNAS without the RAM locked for a few months and I didn't see any issues however I don't want to take any chances with my data so I locked it. Should someone state that it doesn't need to be locked then I might unlock it and allow the RAM to be managed by ESXi. I'm also running a single instance of Ubuntu which runs all the time and periodically a VM of Windoze 7 which I plan to run 24/7 once I get my thoughts down on what I really want to do with it.

If you're running ESXi in an overcommitted memory model, then yes, lock it. However, in general, it isn't a really great idea to run overcommitted unless you have a lot of VM's that are almost always idle or something like that. Memory overcommits poorly because pressure means you have to swap. Also I would think you should be running with unsalted TPS in such a case;

# esxcfg-advcfg -s 0 /Mem/ShareForceSalting

 

[joeschmuck]

Also I would think you should be running with unsalted TPS in such a case;

# esxcfg-advcfg -s 0 /Mem/ShareForceSalting

Are you saying someone should use this setting if they are very over-committed? I had to look up what TPS was (good thing for Google) and see what a setting of "0" (zero, gosh I'd like to have the slash zero font here) does.

And that's a good strategy. Make sure you're also assigning CPU cores that way too.

Yup, I do that as well but I can't help but over-commit on CPU cores, ESXi of course knows how to handle it.

 

[jgreco]

Are you saying someone should use this setting if they are very over-committed? I had to look up what TPS was (good thing for Google) and see what a setting of "0" (zero, gosh I'd like to have the slash zero font here) does.

I'll go so far as to say someone should use this setting unless there's a compelling reason not to use it. The change VMware made to this default several years ago is best described as a security paranoia overreaction, or possibly an attempt to fill the coffers at Intel and memory manufacturers.

http://blogs.vmware.com/apps/2014/10/disabling-tps-vsphere-impact-critical-applications.html

I kid, a bit, at least. It's a good thing to have set, but in a non-cloud environment where you're probably not super-terrified of your neighboring VM's, also not really necessary.

Yup, I do that as well but I can't help but over-commit on CPU cores, ESXi of course knows how to handle it.

Yeah, best to avoid overcommit on memory if possible, but CPU, well, that's hard to avoid. Many of the schemes to do so are varying levels of crazy. Better to keep a bunch of CPU available and just make sure you're not regularly trying to use it all.

 

[AlainD]

Sure. The EdgeRouter Lite 3 uses a Cavium processor at 500 MHz that has hardware assisted packet processing capabilities, which means that as long as you stay "in silicon" you can be processing over 1Mpps. I'm not actually clear on which things are hardware assisted, and it's likely that NAT or firewalling or rate limiting has to involve the host OS. However, if you are merely shuffling packets between subnets, you get a high performance router.

The EdgeRouter X has a faster 800 MHz processor without the hardware assist. This is a cheaper platform, despite the additional speed. There are five gigE ports, each of which may be attached either to the host, or to an internal switch. This means you can do anything from making a 5 port switch that happens to have a small Linux host attached to it (maybe good for DHCP/DNS/etc service), to a 4 port switch with NAT upstream (typical home "router"), to a true router with five different subnets.

From a manageability point of view, the EdgeRouter X lacks the out-of-band console port, so you're screwed and have to reset the unit if you have a config-tastrophe.

Neither of them are particularly zippy at things like OpenVPN, where the numbers I hear are usually around 10 megabits per second.

You are not likely to be reaching the packet per second limits on either device unless you have something unusual going on. A hundred machines doing online gaming, high volumes of SIP traffic, being a DDoS target (or having an infected machine and being a DDoS source), stuff like that.

Thanks, lot's of not known useful info, different than most easy searchable info.

As a home user it's no problem resetting it.

 

[jgreco]

Thanks, lot's of not known useful info, different than most easy searchable info.

Yeah, the practical reality is always a bit different. I come from an environment of both large software routers and 10G switched L3, so in comparison these things look kinda dinky but cool for some applications.

I despise all-in-one home NAT gateways, because they're invariably a multidimensional compromise in favor of price. These days, if I had to suggest something for a friend, I'd probably trend towards an EdgeRouter PoE, then a Unifi AP AC Pro (which can be powered by the ER PoE if you get the 48V brick), and then if you had wired ethernet and needed ports, I do kinda like the Netgear GS108Tv2. That's probably like $400 but it's an uncompromising home networking setup, plus, expandable with another AP AC Pro if needed. Or if you wanted to go towards entry-level 10Gbps, get a Dell 5524 or whatever that newer revision of it is to get yourself two 10G SFP+ ports...

As a home user it's no problem resetting it.

As long as you understand I mean "reset it to factory defaults." Oh and the thing comes by default configured to be 192.168.1.1 on eth0, if I recall correctly, and I don't think it DHCP's, so you actually do need to haul out a laptop and a cable (or other equivalent arrangements). This probably also implies that you want to use eth4 for your WAN/Internet connection so that you're not stuck ripping everything apart. Obviously less of an issue with the EdgeRouters that actually have a console port.

 

[AlainD]

Yeah, the practical reality is always a bit different. I come from an environment of both large software routers and 10G switched L3, so in comparison these things look kinda dinky but cool for some applications.

I despise all-in-one home NAT gateways, because they're invariably a multidimensional compromise in favor of price. These days, if I had to suggest something for a friend, I'd probably trend towards an EdgeRouter PoE, then a Unifi AP AC Pro (which can be powered by the ER PoE if you get the 48V brick), and then if you had wired ethernet and needed ports, I do kinda like the Netgear GS108Tv2. That's probably like $400 but it's an uncompromising home networking setup, plus, expandable with another AP AC Pro if needed. Or if you wanted to go towards entry-level 10Gbps, get a Dell 5524 or whatever that newer revision of it is to get yourself two 10G SFP+ ports...

It's always a compromise in a home situation, especially if you're using corporate stuff at daytime like you. I'm thinking of using the switch part to move some wireless connections to wired connections and adding a UAP-AC-LITE for the remaining wireless use.

As long as you understand I mean "reset it to factory defaults." Oh and the thing comes by default configured to be 192.168.1.1 on eth0, if I recall correctly, and I don't think it DHCP's, so you actually do need to haul out a laptop and a cable (or other equivalent arrangements). This probably also implies that you want to use eth4 for your WAN/Internet connection so that you're not stuck ripping everything apart. Obviously less of an issue with the EdgeRouters that actually have a console port.

I suppose that it's possible to save a config and restore it after a factory reset...

 

[Dice]

@Dice out of curiosity, would you have done it differently and if so, why?

I've played around a lot with RAM settings. I've not squeezed every little ounce of RAM into the VM's, but I've not "reserved" as much as you. 8GB provided for my Win7 VM , 5GB for pfSense, 32GB for FreeNAS.
I locked the Windows VM and FreeNAS's RAM. Heavy memory leaks and crashed applications are 'part of the daily struggle' in my workload on windows. Therefore it needs to be locked too.

Regarding CPU cores, the i3-6100 features 2 of them. I've not dedicated any single core to any of the machines. Instead, I choose to limit by Mhz usage (this option is what I encountered first and therefore tried and tested first). The FreeNAS is dedicated 700Mhz, the Windows Vm is locked to max 2.5Ghz, and the pfSense is not locked at all (yet - I've not managed to set up VPN as of yet. 150/100 Mbit effective bandwidth on VPN will require some CPU juice). The numbers are mostly 'eye-balled' and in testing at the moment.
I locked FreeNAS and Win7's CPU usage is due to the workload. Typically the Win7VM can saturate the entire i3-6100 on its own. When doing so, the FreeNAS shares are heavily stressed and would require some CPU to cope (20-30% from peaks at graphs). These needs led me to locking FreeNAS CPU somewhat, and Win7's CPU most definitely.

Regarding 'strategy' in saving memory for VM's down the line, I feel like it is a waste of resources. Whatever ram that is left over, I'd put into the FreeNAS VM.
I've cut and bumped memory for all VM's without any noticeable impact on the test system.

Cheers,

 

[Rand]

I gotta say I reach memory limits way more often than cpu limits on the three esx hosts I am running atm (2 with freenas).
So it makes a lot of sense to me to be sensible with ram allotment.
On the other hand, i never had a single crash on application side that pointed to memory issues - host horrible slowness when ESX starts swapping...

But I might not be the typical FreeNas on Vm user any more since I dived all in (including running a VDI environment at home now), so ymwv :)

 

[joeschmuck]

(including running a VDI environment at home now)

That is what I'd like to have but my knowledge level isn't quite there to setup a true VDI. If I had an easy to follow "VDI for Dummies" instructions, maybe I could get there.

 

[Rand]

Ah unfortunately I haven't found one... I can give you the short version if you want ?;)

 

[joeschmuck]

Ah unfortunately I haven't found one... I can give you the short version if you want ?;)

Is there a short version? Thanks anyway but I'd need the long version if I were to be successful. I have seen some instructions on the internet but nothing that someone with my limited networking knowledge would easily understand.

 

[gpsguy]

Before you get started - think about licensing. While you can get a 60 day trial license for VDI, they don't offer a perpetual free one. You might want to consider the $200/year VMUG Advantage, which includes their EVALExperience suite of products.


Sent from my iPhone using Tapatalk