Unable to set network settings in Console, no LAG being used.

[superpos]

Relatively new user here who has been testing TrueNAS Scale while migrating away from Synology. I have not set up a signature yet, but my server board is a Supermicro X10SRH-CLN4F with 4x1G ethernet on the board, of which I'm only using one. I also have added a Mellanox ConnectX3 10G NIC and I'm using the Mikrotik CRS305 switch.

I'm experiencing an issue where I'm getting 1G speeds with the 10G card. Although when I run iperf3, I do get 9.4gb/s - but that's the only time it seems that it works. I noticed that when I disconnected the 1G cable that the IP address assigned to the 10G card could not be reached (could not be ping'd). So I figured that the 1G NIC is actually being used as the only network interface despite the unique IP addresses created for both.

I then found out from the docs that it's not possible to have multiple NICs in the box on the same subnet, which might explain the weird behaviour described above. Initially I tried setting a new subnet for the 10G interface in the GUI, but I remembered that I was unable to set a static IP for the 10G card in the GUI either. Others have commented on that. I had previously been pinning the dynamic IP in my Firewalla app since the whole LAN is running off a Firewalla device.

So I restarted and went to the setup console through IPMI. I entered a static IP and subnet that matched the subnet I defined in the Mikrotik switch. I got the same error message that others have gotten though:

Code:

"Error: * interface_update.aliases.0: The network 192.0.0.0/8 is already in use by another interface".

In my case, I'm not trying to set up LAG so I'm not sure what to do here. I think at one point early on I did test LAG with the 4x1G connections on the board, but that was ages ago before I added another NIC and of course I've disabled it. So any advice is appreciated!

 

[superpos]

I can't see a way to edit my initial post, so I'm adding that the 1G connections are running off a separate Mikrotik CRS328-24P-4S+RM switch and the smaller CRS305 10G switch is running off that via its 1G management port. So the CRS305 has been set to a different subnet but I'm unable to set the TrueNAS 10G NIC to match it, in the hope of fixing the larger issue of 10G not getting recognised.

 

[jgreco]

I do get 9.4gb/s

Please do not make up units. It's important on a technical forum to be accurate and concise especially when you are describing an attribute of the problem you are having. "gb/s" could easily mean gigabytes/second, etc. Please see the Terminology and Abbreviations Primer, or, if in doubt, just spell it out.

192.0.0.0/8

You have a ridiculously large netmask set somewhere; you do not own net192. One of your existing interfaces is covering other space you are probably trying to allocate. The system won't let you do that. Either reset your networking to default and just start over, or search for the interface that is set to a /8 netmask with a leading octet of 192.

RFC1918 and successors specify that you may use 192.168.0.0/16 or refines as private network space. Other organizations on the Internet own 192.167.0.0/16 and 192.169.0.0/16 so these are not really available for your use. If you need a larger block of private network, use net10 (10.0.0.0/8) of which you are allowed to allocate all of that Class A network block.

 

[superpos]

Please do not make up units. It's important on a technical forum to be accurate and concise especially when you are describing an attribute of the problem you are having. "gb/s" could easily mean gigabytes/second, etc. Please see the Terminology and Abbreviations Primer, or, if in doubt, just spell it out.

Got it - I meant 9.4 gigabits/second, so almost line speed (found the edit post button...).

You have a ridiculously large netmask set somewhere; you do not own net192. One of your existing interfaces is covering other space you are probably trying to allocate. The system won't let you do that. Either reset your networking to default and just start over, or search for the interface that is set to a /8 netmask with a leading octet of 192.

Thank you, I can easily change it to /16 if that makes a difference. I'll take a look when I get back into the office. In case it's not clear, I have manually set the subnet for the Mikrotik CRS305 switch to /8 to conform to what Scale needs. I would have just set it to /32 but it's not in the pop-up list on the subnet options for the Mikrotik switch (there's no manual entry for some reason). So I'll change it to /16 on the switch.

So it is just the switch and three other 10G NICs connected to it that I'm trying to put on to another subnet to make the TrueNAS happy.

The Firewalla device I have for the entire LAN is putting everything in the address space of 192.168.x.x/24, so that is what is dictating that.

 

[superpos]

I could also try disabling the 1G NIC on the mobo and just use the 10G NIC and keep the same subnet as everything else, if that's a thing.

 

[HoneyBadger]

keep the same subnet as everything else

As mentioned by @jgreco earlier in the thread, you appear to be attempting to use 192.0.0.0/8 which is not a private address space. Hosts exist on the routable Internet outside of 192.168.0.0/16 so if you have interfaces or hosts on your network using this space, you will likely "experience unexpected network behavior"

Can you please describe your network, including the IP address ranges that you're using internally?

 

[superpos]

As mentioned by @jgreco earlier in the thread, you appear to be attempting to use 192.0.0.0/8 which is not a private address space. Hosts exist on the routable Internet outside of 192.168.0.0/16 so if you have interfaces or hosts on your network using this space, you will likely "experience unexpected network behavior"

Can you please describe your network, including the IP address ranges that you're using internally?

Okay, maybe /8 is not a good choice for a subnet. I'm attaching a screengrab of the subnet masks I can choose in the Mikrotik CRS305 4x10G switch. This switch is hanging off the main switch, connected via its 1G uplink. For some reason /32 isn't an option so I just picked /8 because it's at the top.

I'm using 192.168.x.x/24 for the main LAN off the firewall, and by extension the main switch (Mikrotik CRS328), and I am only looking at using 192.x.x.x/8 as a different subnet mask for the 10G subnet to solve the issue where TrueNAS needs a different subnet for a different NIC. I have the TrueNAS and two client machines (Mac and Linux) connected to a Mikrotik CRS305 switch.

I'm also talking to the Firewall vendor and they are suggesting setting up a VLAN for the 10G subnet. And as I mentioned, I'm open to just disabling the currently used 1G NIC on the TrueNAS if that is the easiest solution. Currently if I disconnect it I can't access the box at all, but I haven't specifically tried disabling it.

 

[HoneyBadger]

I'm using 192.168.x.x/24 for the main LAN off the firewall

What's the third octet here?

For example, if it's written in the interface as 192.168.0.0/24 then this network has a range of valid IPs from 192.168.0.1 to 192.168.0.255. You can set up 192.168.1.0/24 as a separate, non-overlapping subnet for your 10GbE switch, and that will use 192.168.1.1 through 192.168.1.255.

You can also use only the 10Gbps card, and reset the network interfaces through the console interface.

If you've already got 192.168.0.0/16 listed, then trying to set up 192.0.0.0/8 will attempt to use a range from 192.0.0.1 to 192.255.255.255 - overlapping with your range in the middle, and also conflicting with a big chunk of public IP space.

 

[superpos]

What's the third octet here?

192.168.241.x is what the Firewall (Firewalla) has used on the whole LAN, within the range of about 72-198. In some cases I converted the dynamically assigned IP to a static IP on the device. I've also pinned some of those dynamically assigned IPs within the Firewalla app, but I've disabled that option now for the TrueNAS IPs to avoid conflicts.

Thanks for the additional notes, I can take a look at this later today. Okay, I should look into something like 192.168.242.x/24 for the 10GbE switch and NICs.

So I'll try that and see if I can kill the idea of a different subnet mask and just use /24.

You can also use only the 10Gbps card, and reset the network interfaces through the console interface.

I'll try that if the above doesn't work.

If you've already got 192.168.0.0/16 listed, then trying to set up 192.0.0.0/8 will attempt to use a range from 192.0.0.1 to 192.255.255.255 - overlapping with your range in the middle, and also conflicting with a big chunk of public IP space.

I haven't yet tried /16 as a subnet mask, but if I did it would completely replace /8. And to be clear I have not manually entered 192.0.0.0/8 anywhere. The only place I can assume TrueNAS is getting that from is the fact that I defined that /8 subnet mask in the 10GbE switch - but again, the only reason I defined that is due to the issue of it being required for a second NIC to be on a different subnet. It's not a core feature of my network, just something I tried last night.

 

[jgreco]

And to be clear I have not manually entered 192.0.0.0/8 anywhere.

We don't care about the ".0.0.0" part. The TrueNAS middleware clearly thinks there is already an interface on 192.*/8, however, and you need to find and eliminate this. It is somewhere in the TrueNAS configuration since the middleware is throwing that error. It does not matter (for this error) what you have configured the Mikrotik to, although your network configuration for all devices on a layer 2 network should be configured for the same basic IP network subnet and netmask. So if you have a configuration for an interface that is 192.3.4.5/8 ("192.*/8" or commonly "192/8" among networking people), this is going to be a problem if you also try to create an interface for 192.168.2.3/24 ("192.168.2.*/24"), which is a subnet within the larger 192/8 block. You cannot have this sort of overlap in ethernet networks without causing confusion and other random issues.

 

[superpos]

We don't care about the ".0.0.0" part. The TrueNAS middleware clearly thinks there is already an interface on 192.*/8, however, and you need to find and eliminate this. It is somewhere in the TrueNAS configuration since the middleware is throwing that error. It does not matter (for this error) what you have configured the Mikrotik to, although your network configuration for all devices on a layer 2 network should be configured for the same basic IP network subnet and netmask. So if you have a configuration for an interface that is 192.3.4.5/8 ("192.*/8" or commonly "192/8" among networking people), this is going to be a problem if you also try to create an interface for 192.168.2.3/24 ("192.168.2.*/24"), which is a subnet within the larger 192/8 block. You cannot have this sort of overlap in ethernet networks without causing confusion and other random issues.

Thanks for clarfiying that this is not external to the TrueNAS. I can confirm that I haven't set anything on the NAS itself to 192.x.x.x/8. I was *attempting* to set that subnet mask in the Console in IPMI, but that's where the error message came up stating "The network 192.0.0.0/8 is already in use by another interface".

So in summary, I have two network interfaces that I was trying to use. They were both 192.168.241.x/24. The 1G was set via static IP with an alias and the 10G by DHCP because I was unable to create an alias (as reported by others).

Then, yesterday, per the docs, I realised that I need to set the 10G NIC to a different subnet in order for it to be recognised. I first set the subnet mask in the 10G switch to /8 because it was at the top of the list. I then tried in the TrueNAS GUI to set it to a static IP but I was unable to. So I rebooted and in IPMI I attempted to apply the static IP and /8 alias but it failed. There is no other place in the installation I have added the /8 subnet mask.

 

[jgreco]

I can confirm that I haven't set anything on the NAS itself to 192.x.x.x/8.

It could happen the other way around; if you had an already-configured 192.168.x.y/24 interface, for example. Having overlapping networks is not valid and therefore not allowed.

 

[superpos]

It could happen the other way around; if you had an already-configured 192.168.x.y/24 interface, for example. Having overlapping networks is not valid and therefore not allowed.

I had the gigabit ethernet default interface in place using 192.168.x.y/24 as a static IP with an alias defined. By default, the 10GbE NIC I added was DHCP and picked that address format and subnet mask up. I wasn't able to change that in the GUI or within the Console. I tried deleting those setups a few times as well, on both interfaces.

What I'm currently doing is setting up a VLAN for the 10GbE subnet, so hopefully that will work. Besides the isolation, the Firewalla requires a VLAN anyway in order to use the new subnet in the IP address (where x is changed in 192.168.x.y/24).

 

[jgreco]

Don't mix DHCP and static configuration. DHCP is really only there for bootstrapping your NAS onto the network, or for luddites with trivial single-interface configurations who can't figure out how to properly configure their NAS. The middleware is presented with an impossible challenge when you mix DHCP and static because it can't do the thing that it NEEDS to be able to do to generate a non-broken networking configuration.

I don't know what you mean by "firewall requires a VLAN". A virtual LAN is not something magic. It is just a regular layer 2 LAN with a tag on it, used to allow multiple LANs to coexist peacefully. Nothing should "require" a VLAN because you should be able to use a regular LAN unless you need to share a port to support two or more separate LANs (the entire point of VLANs).

 

[superpos]

Don't mix DHCP and static configuration. DHCP is really only there for bootstrapping your NAS onto the network, or for luddites with trivial single-interface configurations who can't figure out how to properly configure their NAS. The middleware is presented with an impossible challenge when you mix DHCP and static because it can't do the thing that it NEEDS to be able to do to generate a non-broken networking configuration.

I don't know what you mean by "firewall requires a VLAN". A virtual LAN is not something magic. It is just a regular layer 2 LAN with a tag on it, used to allow multiple LANs to coexist peacefully. Nothing should "require" a VLAN because you should be able to use a regular LAN unless you need to share a port to support two or more separate LANs (the entire point of VLANs).

I wasn't trying to mix DHCP and static, I had no choice.

 

[superpos]

I don't know what you mean by "firewall requires a VLAN". A virtual LAN is not something magic. It is just a regular layer 2 LAN with a tag on it, used to allow multiple LANs to coexist peacefully. Nothing should "require" a VLAN because you should be able to use a regular LAN unless you need to share a port to support two or more separate LANs (the entire point of VLANs).

Correct, that is what I tried to explain. I am sharing a single port from the Firewall ("trunking") to support two separate LANs so that I can have the 10G NIC on a separate subnet. The Firewall requires a VLAN created on the switch downstream of it in order to support a different subnet. Then the 10G switch is on the VLAN hanging off the main switch.

 

[jgreco]

I wasn't trying to mix DHCP and static, I had no choice.

You always have a choice. To bootstrap your system, use DHCP. Once you are in the GUI or the console UI, configure the first interface (the DHCP one) for a static IP. Then configure your default gateway. Then configure any additional interfaces you may require.

 

[jgreco]

Correct, that is what I tried to explain. I am sharing a single port from the Firewall ("trunking") to support two separate LANs so that I can have the 10G NIC on a separate subnet. The Firewall requires a VLAN created on the switch downstream of it in order to support a different subnet. Then the 10G switch is on the VLAN hanging off the main switch.

Probably unnecessarily complicated. I'm too tired to draw a picture. You have enough ports on the NAS that you may be better off dropping another interface into the firewall host, hooking that up to the 100/1000 port on the CRS305, and just running two native layer2 networks, one on each switch.

 

[superpos]

You always have a choice. To bootstrap your system, use DHCP. Once you are in the GUI or the console UI, configure the first interface (the DHCP one) for a static IP. Then configure your default gateway. Then configure any additional interfaces you may require.

That's incorrect. The first interface is a static IP, but I'm unable to set the additional 10GbE NIC as a static IP in the GUI due to a bug with TrueNAS. I tried in the Console, but that didn't work either. Got any ideas?

 

[superpos]

Probably unnecessarily complicated. I'm too tired to draw a picture. You have enough ports on the NAS that you may be better off dropping another interface into the firewall host, hooking that up to the 100/1000 port on the CRS305, and just running two native layer2 networks, one on each switch.

I have a single port coming off the firewall host, but it allows setting up a VLAN in concerto with the first switch. I've attached a diagram of what I'm trying to set up based on what I interpret the documentation is as asking for: a subnet for each NIC. If there is a more simple approach, let me know!