mikrotik core router to do direct routing and switching for servers

[uberwebguru]

I am trying to get rid of having to get a server for pfsense and also get a 10G switch, making it 2 things to worry about
So why not get one giant mikrotik core router and connect my servers to it directly?
I only have like 8 servers to connect to with dual 10 SFP+ ports each

Here is the mikrotik router am thinking about CCR2216-1G-12XS-2XQ

CCR2216-1G-12XS-2XQ | MikroTik

The new MikroTik flagship with the power of a whole fleet. Unleash the power of 100 Gigabit networking with L3 Hardware Offloading! This router can be a handy drop-in upgrade for existing CCR1072 setups.

mikrotik.com

Please advice on if this is a good plan or what issues i will have going this route

MikroTik product news: CCR2216-1G-12XS-2XQ

The new MikroTik flagship with the power of a whole fleet. Unleash the power of 100 Gigabit networking with L3 Hardware Offloading! This router can be a hand...

www.youtube.com

 

[uberwebguru]

I am also looking at this mikrotik CRS312-4C+8XG-RM https://mikrotik.com/product/crs312_4c_8xg_rm
i just need at least 8 x 10G either RJ45 or SFP+
and i want 1 device to do both routing and switching
here are my needs

12U colocation
8 servers can be 10G RJ45 or SFP+
for this router lets say each have 10G SFP+ ports

i get 10G uplink WAN from provider for internet(can be RJ45 or SFP+)
Also this connectivity currently is 1G, but can potentially be upgraded to 10G in future

More interested in the 10G for switching internally between servers on network
I just want one device to do both routing and switching because for my setup, it makes no sense to be having 1 device do routing and 1 device do switching and then for redundancy, i am looking at 4 devices; does not make sense for a setup like mine

 

[jgreco]

Sorry I realize we've talked a bit in a different thread somewhere about this, but I find your post to be relatively requirements-free and therefore frustrating to answer.

What's your requirement set?

The solution set is very different depending on your needs.

First off, the use of the word "router" has been corrupted by the home networking crowd to mean "NAT gateway" and usually a feature-poor NAT gateway. Other people expect a "router" to be a traditional packet router but maybe with firewalling capabilities and optional NAT capabilities. Some of us expect a "router" to be able to do actual router stuff such as BGP, which has its own subset of problem issues, such as "how many full tables" and "convergence times".

Unless you are someplace that is carrier neutral and you are buying your own transit and peering via cross-connects (someplace like Equinix/DFT), it is likely that you are in a single-carrier facility, with connectivity provided by the service provider. In such cases, the carrier will often be happy to provide you with a routed IP block for your servers, meaning you don't even need a router, just a switch and possibly a firewall.

None of Mikrotik's products are promising for BGP convergence time, so if you are at a carrier-neutral and buying transit, even the CCR2216-1G-12XS-2XQ is still going to be a bad choice because it's still got a slow CPU (even if it's now ARM 2GHz).

However, as switches, they're serviceable even if the user interface is infuriating.

So the other thing I picked out here was that you had dual SFP+ ports on each server. Are you looking to hook them all up? To a single switch? To two different switches for redundancy? To two different switches so that you can have an internal and external network?

Almost every question I've asked here results in a different solution set.

 

[uberwebguru]

Sorry I realize we've talked a bit in a different thread somewhere about this, but I find your post to be relatively requirements-free and therefore frustrating to answer.

What's your requirement set?

The solution set is very different depending on your needs.

First off, the use of the word "router" has been corrupted by the home networking crowd to mean "NAT gateway" and usually a feature-poor NAT gateway. Other people expect a "router" to be a traditional packet router but maybe with firewalling capabilities and optional NAT capabilities. Some of us expect a "router" to be able to do actual router stuff such as BGP, which has its own subset of problem issues, such as "how many full tables" and "convergence times".

Unless you are someplace that is carrier neutral and you are buying your own transit and peering via cross-connects (someplace like Equinix/DFT), it is likely that you are in a single-carrier facility, with connectivity provided by the service provider. In such cases, the carrier will often be happy to provide you with a routed IP block for your servers, meaning you don't even need a router, just a switch and possibly a firewall.

None of Mikrotik's products are promising for BGP convergence time, so if you are at a carrier-neutral and buying transit, even the CCR2216-1G-12XS-2XQ is still going to be a bad choice because it's still got a slow CPU (even if it's now ARM 2GHz).

However, as switches, they're serviceable even if the user interface is infuriating.

So the other thing I picked out here was that you had dual SFP+ ports on each server. Are you looking to hook them all up? To a single switch? To two different switches for redundancy? To two different switches so that you can have an internal and external network?

Almost every question I've asked here results in a different solution set.

My requirements are pretty simple, i need the router as a firewall really like you said. Provider drops 1G uplink and provide me ipv4/ipv6 address blocks
I will be doing NAT to route public ip to private ip
some firewall policies, and provide switching between servers

each server will get a single 10G connection
So my requirements are pretty basic, all i need is 10G connectivity between servers, WAN is only 1G and more than enough
Even the 10G between servers will be rarely used until i need to migrate VMs and most VMs are not even more than 256GB storage
And rarely will i move VMs between servers

I also plan to throw in NAS server that i can test out for a while before getting serious with it eventually after understanding real life performances and tricks and hacks from usage experience

 

[jgreco]

Honestly you might not need anything more than a Mikrotik CRS309-1G-8S+IN.

 

[uberwebguru]

Honestly you might not need anything more than a Mikrotik CRS309-1G-8S+IN.

I see, yeah i was looking at the 1U version in CRS317-1G-16S+RM https://mikrotik.com/product/crs317_1g_16s_rm
I do see they have CPU for the routing, not the most powerful but they provide what is good enough
Yeah these models are older, but you are right seems they are only ones that do what i need
so i think i will start with that

and later on when price reduces, can upgrade to these CCR2216-1G-12XS-2XQ https://mikrotik.com/product/ccr2216_1g_12xs_2xq

Lastly what do you think about the 1U build [https://www.truenas.com/community/t...-new-truenas-server-for-vms.99154/post-685599] and running pfsense? Will that match up to the performance with CRS317-1G-16S+RM for example?

 

[Ericloewe]

The CRS line might be a bit puny for an edge device at 1 Gb/s, though. Performance quoted by Mikrotik for the CRS317-1G-16S+RM is around 400 Mb/s with 512-byte packets. Might want to, at least, throw in a dedicated edge device to handle firewall duties.

 

[uberwebguru]

The CRS line might be a bit puny for an edge device at 1 Gb/s, though. Performance quoted by Mikrotik for the CRS317-1G-16S+RM is around 400 Mb/s with 512-byte packets. Might want to, at least, throw in a dedicated edge device to handle firewall duties.

challenge is trying to do 1 device for both routing + switching for 10G connectivity to servers
this CCR2116-12G-4S+ https://mikrotik.com/product/ccr2116_12g_4splus would have been a great choice, but thanks to whoever designed it at mikrotik, bunch of 1G RJ45 ports
not sure what the use case for such power but with bunch 1G ports in 2022 especially

CCR2116-12G-4S+ would have been a perfect choice only that it has only 4 x 10G ports(very disappointing)

 

[uberwebguru]

Honestly you might not need anything more than a Mikrotik CRS309-1G-8S+IN.

How about this CCR2004-1G-12S+2XS https://mikrotik.com/product/ccr2004_1g_12s_2xs?
No switching performance results to compare with though

 

[uberwebguru]

Why 1 device for both router and switching?
more than 1 device will be extra power consumption and too much for a tiny network setup with direct 10G to like just 8 servers

 

[jgreco]

Yes, but you still end up getting a "compromise device", one that doesn't do either of its jobs totally well.

 

[uberwebguru]

Yes, but you still end up getting a "compromise device", one that doesn't do either of its jobs totally well.

Is this in regards to CCR2004-1G-12S+2XS or pfsense on the 1U build?

 

[jgreco]

Yes to both. PC's are great at routing, less great at switching. Switches are great at switching, less great at routing.

 

[uberwebguru]

ok decided to go with 2 devices, a dedicated router and switch

router: $595 CCR2004-1G-12S+2XS https://mikrotik.com/product/ccr2004_1g_12s_2xs
switch: $499 CRS326-24S+2Q+RM https://mikrotik.com/product/crs326_24s_2q_rm

 

[uberwebguru]

@jgreco damn these DAC cables make up for the price difference between SFP+ and RJ45 devices
They cost a arm and leg

$39 a piece https://mikrotik.com/product/xs_da0003

People need to STOP saying SFP+ routers/switches are cheaper; they are not!

 

[jgreco]

They cost a arm and leg

That's one of many reasons I don't recommend DAC cables. Usually optics are cheap and fiber is cheap and can be made in to-the-inch lengths.

People need to STOP saying SFP+ routers/switches are cheaper; they are not!

Well, they are cheaper, but you can MAKE it expensive by buying new DAC cables, vendor optics, name brand fiber, etc. Or you can do the cheap option. I can make 10GBase-T expensive easily too, for example by buying Monster Cable Cat6 patch cords.

 

[uberwebguru]

That's one of many reasons I don't recommend DAC cables. Usually optics are cheap and fiber is cheap and can be made in to-the-inch lengths.

How much cheaper? Is it something one can buy direct or i have to cut to my own size myself?
Yeah imagine someone trying to by 42 DAC cables for a 48 port 10G switch
The prices are very serious, especially when compared to other things value for value

 

[jgreco]

The prices are very serious, especially when compared to other things value for value

This really depends on your tolerance for used, generic, or retail parts.

Those of us in the "cheapskate" service provider industry often score legitimate used parts for cheap. As an example, I can get used Intel FTLX8571D3BCV-IT (10G SR) off eBay at $10-$12/ea in bulk, or about $15/ea onesies. What you do is when you buy your device, you keep in mind that the $80 you're paying for an optics-less X520-SR2 card is really more like $110. Then you buy the optics, and leave them in the card forever, and it isn't a problem. New, an Intel X520-SR2 card is currently around $300-$400 (including optics) so I look at it as a 66% discount.

For a switch, I believe the Mikrotiks are very permissive and do not have vendor-locked optics, so it's probably fine just to scrounge whatever cheap optics are on sale on eBay. If you're getting Intel cards, for example, get a bunch of Intel optics and just verify it. Places like fs.com are happy to sell guaranteed-compatible new generic optics at $20/each. https://www.fs.com/products/108387.html

From my perspective, I prefer to consider these parts as integral to the switch, because you really do need SOMETHING in the SFP+ slot. Therefore if I am getting an optics-less switch, I generally budget about $300 for optics for a 24-port switch on top of that $400 I paid for the switch.

Then this comes down to fiber. Fiber can be cheap. I happen to really like fs's BIF fiber product

The BIF stuff is expensive ($14/2m) but awesome for cable management purposes, and can be customized to the inch. but for beginners I recommend conventional OM3 fiber ($5/2m). Armored cable is also available ($11/2m) which is very resistant to the sorts of damage some beginners do to fiber. None of these are the cheapest available.

Basically the thing here is that I consider $5/2m to be around what one might pay for a quality Cat6 patch cable. Pricing scales well; you can get a 30m/100ft OM3 fiber for $19. But you can also get custom to-the-inch lengths for about the same pricing, which is very attractive when doing high density work. OM3 and the BIF stuff are great when you've got a 24- or 48-port 1U switch and lots of links. A bundle of 48 Cat6 is *big*.

 

[uberwebguru]

I ended just buying this one https://www.linktechs.net/productcart/pc/viewPrd.asp?idproduct=3583 for $29 each
$10 off MSRP of $39 a piece

good thing is i only needed about 6
i prefer to use dense servers than many servers
today you can have a single rack more powerful that some peoples rented cages

 

[uberwebguru]

and it gets worse when you need 10G RJ45 => SFP+
https://mikrotik.com/product/s_rj10 $69 per piece