Unable to set static IP for second adapter

[backdoc]

Running "TrueNAS-SCALE-22.02.0".
Motherboard: X11SCH-LN4F

I have successfully changed the IP address on eno1 to 10.0.0.201. But, I am getting an error that says 10.0.0.0/24 is already in use when trying to set eno2 to 10.0.0.202/24. I am at the console because the GUI was also giving me the same error.

Is this a bug or am I doing something wrong?

 

[backdoc]

I very recently built a new 22.02 SCALE box based on a SUPERMICRO MBD-X11SCH-LN4F-O board that has 4 onboard network adapters. I've never owned a server motherboard before (a board with 4 network adapters plus IPMI). So, I'd never given any thought to setting up 4 nics. For now, I only have 2 cables plugged into it. They both obtained IPs via DHCP. One was 10.0.0.48/24 and the other was 10.0.0.49/24, which to me looks like it's on the same network. I would like to make my IPs static. My plan was to set up a VM as a VPN gateway for the house (including services on TrueNAS) using OpenSense, Pfsense or just Linux with firewall rules I currently use on another box for this purpose.

I changed the first adapter to 10.0.0.201/24. But, when I tried to change the second nic to 10.0.0.202/24, I got an error saying that the network 10.0.0.0/24 was already in use. Is that saying that I need to change subnets (if, I'm using that term correctly). Isn't the subnet the description of the network portion of the IP, the /24? That's what it did with DHCP.

Do I need to do something like this instead?

10.0.0.201/24
10.0.10.201/24
10.0.20.201/24
10.0.30.201/24

I originally posted about it here, but never got any replies. My apologies if it is bad form to post the same question twice. I wasn't sure if it was more appropriate to bump my original post or start a new one. I'm trying to do my own research and reading. But, I'm not able to find anything other than a long post by @jgreco that might be answering my question but I don't understand and I read over this conversation involving @jgreco, too. But, I don't understand networking enough to apply it to my question.

 

[sretalla]

2 cables plugged into it. They both obtained IPs via DHCP. One was 10.0.0.48/24 and the other was 10.0.0.49/24, which to me looks like it's on the same network.

Already not a great start. you need to be looking into LAGG or LACP if you want more than one NIC on a subnet as mentioned in @jgreco's post... which also implies you know what you're doing with a managed switch at the other end to make it work properly

Do I need to do something like this instead?

10.0.0.201/24
10.0.10.201/24
10.0.20.201/24
10.0.30.201/24

That's certainly one way to go...

Do you need 4 subnets for your TrueNAS server? (I expect not)

If all you want is to use those NICs to pass into a VM, then do nothing in TrueNAS to set them up, use lspci | grep -i network to find the addresses of your network cards and pass them in as devices in the VM devices section.

You'll then be setting up IP addressing in the VM(s) not in TrueNAS.

 

[jgreco]

That's what it did with DHCP.

Yeah, it does that but that's broken/wrong/weird. DHCP for things like NAS is best used for initial setup. Infrastructure devices should generally get assigned static IP's so they don't vanish if something happens to the DHCP server. Multiple interfaces on a single network does not work the way people "think" they work.

I wasn't sure if it was more appropriate to bump my original post or start a new one.

Eh. We prefer fewer threads. But that's well within moderator abilities. We'll see if I remember to merge these by the end of the post.

My plan was to set up a VM as a VPN gateway

Ok.

might be answering my question

Or overanswering, as the case may be.

So I'm just going to use the term "gateway" to mean "some pfSense, opnSense, or other NAT device often called a router".

This may or may not be what you want to do. But it's handy for discussion:

Let's say that you wanted to have a NAT gateway that serviced your {cable modem, fiber, Starlink} Internet connection. You don't really want the NAS visible to the whole Internet, but if you were running the NAT gateway as a VM on the NAS, then you need to get the packets from the Internet into the VM. That has to go "through" the NAS.

You do this by setting up an interface on the NAS with no IP address, but making sure it is "up" and connected. Then, the process of setting up a VM will create a bridge from that "real" ethernet interface to a virtual ethernet interface offered to your VM. A bridge is just a software "ethernet switch" that moves packets from one interface to another. pfSense would call this the "WAN" ethernet interface.

This still isn't sufficient, because your NAS and your network need to be connected to the gateway. So you have another ("LAN") ethernet interface on the gateway VM, and this can be bridged to your primary network. This is probably the same interface as your TrueNAS has an IP on.

I'm going to stop at that point and let you ponder how this relates to what you envision. Ask questions. Dumb question == the not asked question.

But, I don't understand networking enough to apply it to my question.

It's likely we can get you sorted out.

 

[backdoc]

Thank you both for your replies. Give me a little time to digest it and I'll reply

more than one NIC on a subnet

I probably am not wanting that. I could be asking for the wrong thing or in the wrong way. I just have a basic Netgear router (which I don't love). So, I wouldn't be set up any kind of managed switch.

If all you want is to use those NICs to pass into a VM, then do nothing in TrueNAS to set them up, use lspci | grep -i network to find the addresses of your network cards and pass them in as devices in the VM devices section.

You'll then be setting up IP addressing in the VM(s) not in TrueNAS.

That's probably what I'm thinking, that is, I want some services behind a VPN. LIke I said, I've never had a board with 4 adapters so I was thinking, "this is awesome, every VM can have its own private nic". I'm at work atm. I'll check out lspci when I get home.

 

[backdoc]

We'll see if I remember to merge these by the end of the post.

I can remind you.

You don't really want the NAS visible to the whole Internet

You're right. This may not be what you meant, but, I want to make Plex available outside my network, I'd like to have ssh available to me at work but I'll use keys for that. and finally, I will more than likely start using Backblaze or something for offsite backups. So, the NAS will need to get to the Internet. But, I don't plan to expose the administrative GUI.

You do this by setting up an interface on the NAS with no IP address, but making sure it is "up" and connected. Then, the process of setting up a VM will create a bridge from that "real" ethernet interface to a virtual ethernet interface offered to your VM. A bridge is just a software "ethernet switch" that moves packets from one interface to another. pfSense would call this the "WAN" ethernet interface.

This still isn't sufficient, because your NAS and your network need to be connected to the gateway. So you have another ("LAN") ethernet interface on the gateway VM, and this can be bridged to your primary network. This is probably the same interface as your TrueNAS has an IP on.

I'm going to stop at that point and let you ponder how this relates to what you envision. Ask questions. Dumb question == the not asked question.

Yeah. I think I need to spend some time thinking about what you have said. I think I get it for the most part. And, if I do, I might need to clarify that one of my goals is to create a VPN gateway that some but not all devices in my house will use. I don't plan to make the NAS itself use it. It will have apps (or in another VM) running on it like Plex and Nextcloud that doubt will work well behind a VPN. Some services don't work well with VPNs. So, I just plan to change the gateway on my devices around the house when I want them routing through the gateway.

It's likely we can get you sorted out.

Yeah. I am pretty confident that I'll get there okay with the help of the forums. Quick background, I started using Linux in the middle 90's. It's my passion, so much so that I sold my business and went back to school to get a CS degree. I work as mostly a SQL developer, not in Operations. But, much of my interest lies there. So, I'm not totally inept.

Give me a little time to digest what you and @sretalla have posted and I'll post back sometime over the weekend.

 

[backdoc]

It's taken me longer than I expected to get back to you. I've been plugging away at things. I think I've made some progress toward the suggestions you've made. But, I'm stuck conceptually.

With regards to this....

You do this by setting up an interface on the NAS with no IP address, but making sure it is "up" and connected. Then, the process of setting up a VM will create a bridge from that "real" ethernet interface to a virtual ethernet interface offered to your VM. A bridge is just a software "ethernet switch" that moves packets from one interface to another. pfSense would call this the "WAN" ethernet interface.

I think I misunderstood this the first 20 times I read it. I set up a bridge on my NAS. Then, I added the 2 hardware NICs to the bridge, then I set my VM up to use the bridge on the NAS. I did set up the first hardware NIC with an IP. And, I didn't give an IP to the second NIC. As I'm re-reading your comment now, I think you meant to basically leave the first NIC alone. Then, just get the second one up, without an IP, which I did. But, I should have stopped at that and pointed my VM to the second hardware NIC because the bridge is in the VM. Is that correct?

This still isn't sufficient, because your NAS and your network need to be connected to the gateway. So you have another ("LAN") ethernet interface on the gateway VM, and this can be bridged to your primary network. This is probably the same interface as your TrueNAS has an IP on.

You lose me when you say "your NAS and your network need to be connected to the gateway". I didn't want my whole NAS behind the gateway, if that's what you meant. I want some docker Apps behind the gateway though. And, I want other computers and devices on my home network to be able to utilize the VPN gateway. I did install Opnsense and got it up and running. It was connected to PIA via Wireguard. And, I was able to route my phone through it. I don't recall touching it again and suddenly my phone couldn't route through it anymore. But, based on my bridge misunderstanding, I probably need to nuke that VM, fix my NAS network config and start over with Opnsense.

Ultimately, my goal is to be able to install some docker apps, some that use VPN and some that don't. I think TrueChart makes it harder for me to conceptualize how everything will work. Some of my apps are not TrueChart because they don't have everything I want to install. It looks like TrueChart provides some technique for routing some of their apps through a VPN. But, I don't see how to get my non-TrueChart apps to use their VPN. If I want my docker apps to be behind a VPN (even though I don't want them all behind a VPN), do I have to route my NAS through a VPN? And, if I do, does the OpenVPN built into TrueNAS do that - do I really need the VPN gateway? Also, all of my docker apps are exposed on my NAS IP address with just different ports. Is that the way Apps work? Can I not separate my Apps IP from my NAS IP?

I know that was a lot of words. So, I'll be grateful for whatever limited advice you give, just a point in the right direction would be much appreciated. Oh. And, my proc is a Xeon E-2146G (6 core 12 thread). So, I'm not sure how many VM's I can run before resources start spreading too thin. Originally, I thought I'd run 2 VMs. But, if I can get by with 1 or 0, that would be better.

I could install a lightweight Linux VM, like Arch and just put all my dockers on it that I want behind a VPN, put my VPN on it and configure it as gateway. I already have a Linux box with all the iptables rules set up. I could just copy those.

 

[Nick2253]

From how you're talking, you definitely have some embedded misconceptions about networking, but you are very close. In particular, I believe that your use of the word "gateway" is causing you trouble. I'm not sure if this definition will help, but a gateway is basically a network device that translates between two different networks. The most common gateway that most people are familiar with is the main "router" that plugs into the Internet on one side, and into your local network on the other.

--

Something that helps me conceptualize networking is to ignore virtualization to start with. Imagine how all the services would connect as if each service were on its own server, plugging in to physical switches with physical cables. With that in mind, you can then start to translate various concepts to the hypervisor

Within a hypervisor, it's convenient to think of the host (in this case, the TrueNAS system itself), as one or more super flexible and configurable switches. Can you re-arrange the imagined "physical" layout such that a group of "physical" services (which are in reality VMs) are plugged in to one or more "physical" switches? If so, these become the virtual switches that we need to create in TrueNAS.

So, now think of your VPN Gateway "physical" machine. How many network connections does it have to have? What is on the other end of those connections? And how can you translate that to virtual switches on the hypervisor?

 

[backdoc]

Thank you for taking time to respond.

I believe that your use of the word "gateway" is causing you trouble.

Maybe I'm using the term incorrectly, but I used this YouTube tutorial. And, so, that is what I've been calling it.

But, now that you mention it, I think I have glossed over the networking section of the "Launch Docker App" too quickly. I can probably route my docker app through my VPN "gateway" there.

 

[Jibun no Kage]

You do NOT need to use LACP, you can have two NICs on the same network in any server in any OS. but TrueNAS is breaking this general feature.

Say you want all administrative control to NIC 1, and all storage traffic to NIC 2, and on the same network, any server can do this. You DO NOT need to do anything special. The fact that TrueNAS forces you to use DHCP to do this, and the Web UI breaks this possible configuration, is a LIMITATION of TrueNAS, not networking, server, or OS design in general. Frankly, it is these contribute and arbitrary restrictions that I continue to see hard wired into TrusNAS that are very frustrating and detract from TrueNAS benefit greatly.

IMHO, someone in the TrueNAS development team needs to stop doing this, or at least let the users decide how they want to configure their own environments via advanced configuration options and not lock things down as they have with multiple NICs on the same network segment as is now the case.

 

[sretalla]

you can have two NICs on the same network in any server in any OS. but TrueNAS is breaking this general feature.

Not if you want things to work properly. TrueNAS is just conforming to Unix/Linux routing, so it's far from unique to TrueNAS.

Please read this and reconsider your position:

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...

www.truenas.com

There are many ways to use 1 NIC to have multiple IP addresses that work just fine, but if you're going to use 2 NICs in the same system on the same subnet and not use some form of address abstraction (LACP/LAGG/bond), you're going to get some strange routing outcomes (which you may or may not care about... but if you're using separate NICs for a reason, you would usually expect things that won't happen).

 

[Jibun no Kage]

Yes, I have configured TrueNAS to allow 2 NICs on the same network, but to do this, was NOT simple or straight-forward via the WebUI and even the console method kept throwing errors "A NIC already assigned to segment" foolishness. I had to TRICK TrueNAS to allow 2 NICs on the same network segment.

I am NOT talking about multiple IPs on the SAME NIC. I am talking about 1 NIC say 192.168.1.50, and 1 NIC 192.168.1.60. Then explicit services are assigned to each interface. This is done on servers all the time, to ensure you have traffic 'type' isolation. If you need secure isolation, you then would likely use DIFFERENT physical networks, segments, or even VPNs.

My setup for example:

192.168.1.50
SSH
Web UI
SNMP

192.168.1.60
NFS
SMB

This is not easy to setup with TrueNAS, but with UNRAID, OpenMediaVault, etc., straight-forward. This is STRICTLY a limitation of TrueNAS, and is not simply Linux/Network enforcement, you are incorrect on that point. LACP is completely overkill and not applicable to the issue I am qualifying with the TrueNAS cripping of network configuration via the console or Web UI configuration options.

And no, I will not have strange outcomes, if you know how and why you implemented your design. True, you must know what you are doing to do this, but I have 30+ years experience as an IT Enterprise engineering and distributed server virtualization design architect. Have built firewalls, DMZs, and more, from scatch for global entities. So yes, there are pitfalls and issues with complex network design.

Please don't qualify a TrueNAS design decision as a limit of the underlying OS methods and standards. That is just not accurate. I believe TrueNAS established the basic restrictions to protect users, but giving no option for advanced users, is not a good strategy, or IMHO is an inconsistent strategy, in that some elements of TrueNAS allow extensive advanced configuration, but not with networking? Given the limitations I have outlined above.

 

[sretalla]

OK, so it's clear you didn't read and understand the linked resource and perhaps don't see where the problem is with what you're saying.

If you think that having services "assigned to an interface" means that traffic passing both in and out of the service will use the "assigned" NIC and only that NIC, you're mistaken.

Please read the resource if you care about understanding what's happening and why the GUI is structured to help people not to do things that don't work how they think it will.

 

[Jibun no Kage]

No, I understand the documentation and the design of the core Linux network stack.

The difference is, I want to go OFF the reservation, KNOWING the pitfalls. And TrueNAS does not allow this. TrueNAS is protecting the user. I disagree with this point as a concept. I don't believe TrueNAS should be projecting the user, if the user wants to go OFF the reservation, that should be allowed with the applicable warnings. Of course, violation of Linux network stack is an issue in of its self. But that should be the user call, not TrueNAS, per se, IMO. With the configuration I have now, watching the traffic, there are some interesting things happening, but I digress.

Another example of limitation is, I believe the lack of typical bonding with in TrueNAS? Did I miss where this is allowed?

While looking at the LACP/LAGG support in TrueNAS the question occurred to me, where is support, for classic active/backing bonding. if I recall back, the limits on multiple NIC use, go back to 2012 (that I found), and many are in reference to traffic 'type' isolation, that I noted many have wanted, be it jailing, virtualization, etc. related. I will not discuss the merits of traffic type isolation for virtualization... that gets really into the weeds.

I did a quick google search, planning to pull the TrueNAS documentation, in turn, and discovered that every reference I query about TrueNAS and bonding, ultimately directs back to a LACP/LAGG discussion at some point.

This begs the question, as I see it, what if I just want a simple active/backup bond configuration? This is a core feature of the Linux network stack, never mind many OSes (if not almost all). Again, TrueNAS design limitation? If it can done fine, I just have yet to find the methodology to do so. But a causal search via Google should expose that methodology, no? Unless it is a true limitation of TrueNAS?

 

[Jibun no Kage]

Oh, and yes, service assignment to interface, limiting traffic flow, is a pipe dream under the Linux network stack design. I forgot to mention I agree on that point, that is technical reality. Hence my point if I want to go off the reservation I should be allowed to do.

 

[Jibun no Kage]

Ah... I realized why my Google queries failed, it was 'bonding' versus 'bridging'.... what a difference I word makes.

 

[Jibun no Kage]

Well, reading the documentation, and testing, it really is just a bridge (brX), not a classic bond. Was thinking the terms were mixed up. But not the case. So, typical bonding, say bond 0, active/backup not possible in TrueNAS?

 

[sretalla]

As far as I understand it, if you don't select LACP as the "Link Aggregation Protocol", the other 2 types will result in a bond of the type you're seeking... (FAILOVER in particular seems a match for your desire for active/backup)

Maybe I'm wrong, but don't have the time to test that to be sure.

 

[Jibun no Kage]

Cool, I will give it a shot, see what results. Just testing the limits of TrueNAS design really.

 

[sretalla]

The difference is, I want to go OFF the reservation, KNOWING the pitfalls. And TrueNAS does not allow this. TrueNAS is protecting the user. I disagree with this point as a concept. I don't believe TrueNAS should be projecting the user, if the user wants to go OFF the reservation, that should be allowed with the applicable warnings. Of course, violation of Linux network stack is an issue in of its self. But that should be the user call, not TrueNAS, per se, IMO.

TrueNAS is Open Source, but is designed for its paying customers (Enterprise), who will typically want/appreciate safeguards and the support that's facilitated by those.

Free/community product (CORE/SCALE... for now) users can take what they get... or fork the code and make it how they want.

You can also raise a feature request and see how many other support it (and if that's enough folks, changes will tend to be made).