Can't ping gateway after reboot - "ping: sendto: Host is down"

[chrisn]

Hi TrueNAS community,

I'm looking for help with a network problem. I have a FreeNAS box that's been in use since 2017. No problems. SuperMicro MB, built in Intel NICs, 16GB RAM, Xeon processor. 3 4TB WD red drives in a mirror configuration. Has served me well.

Recently we had 3 days worth of rolling power outages. I shut the box off to prevent damage. After booting it back up, I can't access it. Does not respond to ping, can't reach the web interface, or the hard disk shares. I have a keyboard, mouse, and monitor hooked up to the console. From the console, I can ping the one active NIC's IP address, but I can't ping the default gateway. I've tried the following: reconfiguring it from static to DHCP, and back to static again. I reconfigured the default gateway. None of this has helped. I can see a link beat light flashing on the back of the workstation and on the GB switch its hardwired into.

In an attempt to rule out damaged NIC hardware, I booted from a Linux Mint DVD and its live session can ping the gateway, can be pinged by another host, and can get out to the internet with a web browser. So there is nothing wrong with the NIC.

In the in the interest of full disclosure, earlier in the year, I hooked up and configured a 2nd NIC port and had it hooked to the LAN. When trying to debug my failed network, I read that was a rookie mistake and have since disconnected the cable and brought the 2nd NIC port down. Now I'm back to the original solo NIC port being active but the network isn't reachable. When I try to ping the gateway I get "ping: sendto: Host is down" on the console.

Can anyone tell me how to debug this?

Thanks in advance.

Chris

 

[Samuel Tai]

A little more information please. Are you using the 10G or the 1G port? What driver are you using with the port? Have you tried flipping to the other port of the same speed? (FreeBSD, upon which TrueNAS is based, doesn't guarantee NIC numbering stability between reboots.)

 

[chrisn]

I'm using the 1st 1G port igb0. I've tried flipping to the other 1G port igb1 but that didn't work. I can't tell exactly what the driver name is, but here's the output of dmesg...

igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.4.3-k> port 0xe020-0xe03f mem 0xfb120000-0x

Here's the output of pciconf...

igb0@pci0:5:0:0. class=0x020000 card=0x152115d9 chip=0x15218086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'I350 Gigabit Network Connection'
class = network
subclass = ethernet

If this doesn't indicate the driver, how do I find that?

Thanks

 

[Samuel Tai]

OK, so you are using the igb driver. This driver is usually pretty reliable. What does ifconfig igb0 show?

 

[chrisn]

It shows...

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TS04,TS06,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether ac:1f:6b:11:17:ac
hwaddr ac:1f:6b:11:17:ac
inet 192.168.4.9 netmask 0xffffff00 broadcast 192.168.4.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active

 

[Samuel Tai]

What happens if you manually blip the NIC off and on via ifconfig igb0 down, then ifconfig igb0 up? There doesn't seem to be anything wrong with the config of your NIC, and it was running fine before the power bumps. This may be a timing issue on boot.

 

[chrisn]

Did not work. Took it down, then back up. First status check status showed no carrier. But on 2nd check status is active. Still can't ping the gateway.

 

[Samuel Tai]

OK, then it's likely an ARP issue. What does arp -a show on both the TrueNAS system AND the gateway? Does each side have the correct MAC address for the other? Can the gateway ping the TrueNAS system?

 

[chrisn]

Gateway is an Eero pro. I don't know how to ping my TrueNAS box from the Eero using the Eero provided iPhone app.

However, arp -a on TrueNAS shows an address for the Eero that doesn't match what the Eero iPhone app says. Eero app shows the following address for the two wired ethernet ports:
30:57:8e:4b:e7:c0
30:57:8e:4b:e7:c1

TrueNAS shows the Eero gateway address as:
30:57:8e:4b:e7:d2

Both my Macbook Pro AND my Linux box show the same address for the gateway that TrueNAS shows, i.e.
30:57:8e:4b:e7:d2

One other observation is that "arp -a" on TrueNAS is slow to respond at first. Unlike Linux and the Macbook, which pretty much respond to the arp command immediately, TrueNAS takes anywhere from 4 to 10 seconds.

One other note about the LAN. I have a 16 port Gigabit tp-link switch and the Linux box, the Eero Pro, and the TrueNAS box are all hardwired to it, all on the same network (collision domain) of 192.168.4.x. The Macbook connects over Wifi. The Linux box and Macbook can both ping the gateway (192.168.4.1) and I can ping the Linux box from the Macbook.

Any ideas?

Thanks

 

[jgreco]

One other observation is that "arp -a" on TrueNAS is slow to respond at first. Unlike Linux and the Macbook, which pretty much respond to the arp command immediately, TrueNAS takes anywhere from 4 to 10 seconds.

That merely means that nameservice resolution is having issues, which is hardly shocking if networking is broken somehow. Try "arp -an" instead.

 

[Samuel Tai]

You didn't mention you had an eero. Those have a known issue with unreliable layer 2 bridging. See https://www.truenas.com/community/threads/jails-eero.59477/.

Unfortunately, you may have to reboot the eero side to get things working again.

 

[chrisn]

arp -an sped it up.

I did reboot the Eero, but that really shouldn't be the issue if I can't ping from TrueNAS to the Linux box, where the TrueNAS is IP 192.168.4.9 and the Linux box is 192.168.4.2 (same network and 255.255.255.0 netmask). This should just involve TruNAS -> Gigabit switch -> Linux. For this particular ping operation, the Eero shouldn't be in the picture should it? Also, the issue is with top-level TrueNAS, not a jail.

Should I clear the TrueNAS ARP cache?

Or should I reboot all three Eeros and TrueNAS? Is there a recommended order?

Thanks

 

[Samuel Tai]

Bounce the switch frist. Reboot the eero, then TrueNAS.

 

[chrisn]

OK, I did that but it didn't work.

Now arp -an shows... (192.168.4.1) at (incomplete) on ix0 expired [ethernet]

Is this a clue that it's trying to talk by default through the ix0 interface instead of the igb0 one?

 

[jgreco]

That's a clue that your ix0 port is up and configured with something that is probably conflicting, yes.

 

[chrisn]

Found it! The ix0 and ix1 ports were configured and up! I deleted the interface for ix0, ix1, and igb1. Now only igb0 remains and the system is working again!

Samuel, jgreco - thank you both so very much for your patience in helping me past a very rookie networking mistake!!!

Solved thanks to you!