Truenas core server keeps shutting down with no log entries

[sqwob]

I've got a truenas server that I upgraded to version TrueNAS-13.0-U5.3 not so long ago.

• Running on a ASROCK B75 PRO3-M, Interl Core i5-3470T with 16gb of non-ecc ram and 8x2TB of storage in 2 RaidZ2 arrays. 500W powersupply, separate gigabit NIC installed
• Hardware has been identical for almost 10 years.
• Boot volume was a usb stick, is now a M.2 SSD in a usb enclosure
• This is a server that isn't running 24/7, but only started up on demand when I want to use Emby or Sabnzb

Issue: Over the past month it started shutting down unexpectedly, sometimes after a few minutes, other days after a few hours.

Things i've tried:
- searching the log files for any relevant info, however the only thing i can see is the next reboot, so nothing is logged when it's shutting down, it just goes "power off" it seems (or i'm looking in the wrong place)
- booting a different usb stick with diagnostic software and running a memorytest & cpu loadtest, both completed succesfully. The machine never shut down during this.
- creating a new SSD for boot medium, clean install and imported config file. the SSD is connected via usb.


Result of this behaviour can look like this:

boot time Thu Jan 4 09:23
boot time Wed Jan 3 21:47
boot time Wed Jan 3 08:42
shutdown time Wed Jan 3 08:40
boot time Wed Jan 3 08:34
boot time Tue Jan 2 21:28
boot time Mon Jan 1 21:55
boot time Sat Dec 30 21:16
shutdown time Wed Dec 27 21:43

Console.log around the timestamp of an unexpected reboot:

Daemon.log

Messages.log

If anyone has any ideas, i'm quite curious to find out what i'm missing!

Kind regards
Rik

 

[sqwob]

hmm... server hasn't powered down since I moved it to the livingroom... i'm contemplating if it possible was a bad power connection to the outlet.

 

[Patrick M. Hausen]

Please copy and paste text for logs and command output. Don't post screenshots.

Like this:

Code:

text in code tags

Thank you.

 

[WI_Hedgehog]

If you're not running the system off of an Uninterruptible Power Supply, uh, you should be. And TrueNAS should be configured to shut down when the UPS capacity starts to run low as to affect a clean shutdown.

Also, check the CPU power filter capacitors on the mainboard and make sure they're not looking a bit dried out (they don't last forever).

 

[sqwob]

Just happened again at about 21h27, so far for hoping it was related to cabling...
we were about 1 hour into a movie being streamed, no other load.

I have highlighted the line where it's being powered on again.

console.log:

Jan 8 20:35:55 freenas Loading kernel modules:
Jan 8 20:35:55 freenas Error: vmware-checkvm must be run inside a virtual machine on a VMware hypervisor product.
Jan 8 20:35:55 freenas add host 127.0.0.1: gateway lo0 fib 0: route already in table
Jan 8 20:35:55 freenas add net default: gateway 10.0.0.1 fib 0: route already in table
Jan 8 20:35:55 freenas add host ::1: gateway lo0 fib 0: route already in table
Jan 8 20:35:55 freenas add net fe80::: gateway ::1
Jan 8 20:35:55 freenas add net ff02::: gateway ::1
Jan 8 20:35:55 freenas add net ::ffff:0.0.0.0: gateway ::1
Jan 8 20:35:55 freenas add net ::0.0.0.0: gateway ::1
Jan 8 20:35:55 freenas Starting ctld.
Jan 8 20:35:55 freenas Updating /var/run/os-release done.
Jan 8 20:35:55 freenas Creating and/or trimming log files.
Jan 8 20:35:55 freenas Updating motd:.
Jan 8 20:35:55 freenas Starting syslog_ng.
Jan 8 20:35:55 freenas Starting watchdogd.
Jan 8 20:35:55 freenas Starting ntpd.
Jan 8 20:35:55 freenas Mounting late filesystems:.
Jan 8 20:35:55 freenas Starting dbus.
Jan 8 20:35:55 freenas Starting rrdcached.
Jan 8 20:35:55 freenas Performing sanity check on openssh configuration.
Jan 8 20:35:55 freenas Starting openssh.
Jan 8 20:35:56 freenas Starting avahi-daemon.
Jan 8 20:35:56 freenas Starting nmbd.
Jan 8 20:35:56 freenas security.bsd.unprivileged_inode_gen: 0 -> 1
Jan 8 20:35:56 freenas Starting smbd.
Jan 8 20:35:56 freenas Starting winbindd.
Jan 8 20:35:56 freenas security.bsd.unprivileged_inode_gen: 1 -> 1
Jan 8 20:35:56 freenas Starting smbd.
Jan 8 20:35:56 freenas starting wsdd
Jan 8 20:35:56 freenas Starting cron.
Jan 8 20:35:56 freenas Starting smartd_daemon.
Jan 8 20:35:56 freenas Performing sanity check on nginx configuration:
Jan 8 20:35:56 freenas nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
Jan 8 20:35:56 freenas nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Jan 8 20:35:56 freenas Starting nginx.
Jan 8 20:35:56 freenas 1 2024-01-08T20:35:56.631323+01:00 freenas.local smartd 1493 - - Device: /dev/ada4, WARNING: A firmware update for this drive may be available,
Jan 8 20:35:56 freenas 1 2024-01-08T20:35:56.631337+01:00 freenas.local smartd 1493 - - see the following Seagate web pages:
Jan 8 20:35:56 freenas 1 2024-01-08T20:35:56.631345+01:00 freenas.local smartd 1493 - - http://knowledge.seagate.com/articles/en_US/FAQ/207931en
Jan 8 20:35:56 freenas 1 2024-01-08T20:35:56.631350+01:00 freenas.local smartd 1493 - - http://knowledge.seagate.com/articles/en_US/FAQ/213915en
Jan 8 20:35:56 freenas Starting collectd_daemon.
Jan 8 20:35:56 freenas Configuring vt: keymap blanktime.
Jan 8 20:35:57 freenas net.inet.carp.allow: 0 -> 1
Jan 8 20:35:57 freenas
Jan 8 20:35:57 freenas Mon Jan 8 20:35:57 CET 2024
Jan 8 21:30:52 freenas Starting devd.
Jan 8 21:30:52 freenas Autoloading module: ichsmb
Jan 8 21:30:52 freenas Starting zfsd.
Jan 8 21:30:52 freenas Setting hostuuid: 59610995-ad9c-11ee-9a0e-6805ca121aa4.
Jan 8 21:30:52 freenas Setting hostid: 0x38f5afa4.
Jan 8 21:30:52 freenas <118>middlewared: starting middlewared: loaded plugin middlewared.plugins.ec2 middlewared: loaded plugin middlewared.plugins.service
Jan 8 21:30:52 freenas <118>middlewared: setting up plugins (enterprise.m_series_nvdimm_middlewared: setting up plugins (device_.devd_events) [49/60middlewared: setting up plugins (kmip) [50/60]
Jan 8 21:30:52 freenas Loading early kernel modules:
Jan 8 21:30:52 freenas Syncing disks...
Jan 8 21:30:52 freenas Waiting on devd connection
Jan 8 21:30:52 freenas Enumerating system disks
Jan 8 21:30:52 freenas Enumerating geom disk XML information
Jan 8 21:30:52 freenas Enumerating disk information from database
Jan 8 21:30:52 freenas Syncing disk 1/9
Jan 8 21:30:52 freenas Syncing disk 2/9
Jan 8 21:30:52 freenas Syncing disk 3/9
Jan 8 21:30:52 freenas Syncing disk 4/9
Jan 8 21:30:52 freenas Syncing disk 5/9
Jan 8 21:30:52 freenas Syncing disk 6/9
Jan 8 21:30:52 freenas Syncing disk 7/9
Jan 8 21:30:52 freenas Syncing disk 8/9
Jan 8 21:30:52 freenas Syncing disk 9/9
Jan 8 21:30:52 freenas Syncing disk 10/9
Jan 8 21:30:52 freenas Syncing disk 11/9
Jan 8 21:30:52 freenas Syncing all disks complete!
Jan 8 21:30:52 freenas Alarm clock
Jan 8 21:30:52 freenas Starting file system checks:
Jan 8 21:30:52 freenas Mounting local filesystems:.
Jan 8 21:30:52 freenas Beginning pools import
Jan 8 21:30:52 freenas Importing Storage

daemon.log

Jan 8 20:36:00 freenas 1 2024-01-08T20:36:00.550710+01:00 freenas.local smartd 1493 - - Device: /dev/ada7, ST2000DM001-1CH164, S/N:W1E24JGW, WWN:5-000c50-05cebb48e, FW:CC24, 2.00 TB
Jan 8 20:36:00 freenas 1 2024-01-08T20:36:00.557541+01:00 freenas.local smartd 1493 - - Device: /dev/ada7, found in smartd database: Seagate Barracuda 7200.14 (AF)
Jan 8 20:36:00 freenas 1 2024-01-08T20:36:00.584230+01:00 freenas.local avahi-daemon 1775 - - Server startup complete. Host name is freenas.local. Local service cookie is 2895768100.
Jan 8 20:36:01 freenas 1 2024-01-08T20:36:01.016495+01:00 freenas.local smartd 1493 - - Device: /dev/ada7, is SMART capable. Adding to "monitor" list.
Jan 8 20:36:01 freenas 1 2024-01-08T20:36:01.016710+01:00 freenas.local smartd 1493 - - Device: /dev/ada5, opened
Jan 8 20:36:01 freenas 1 2024-01-08T20:36:01.041131+01:00 freenas.local smartd 1493 - - Device: /dev/ada5, Hitachi HDS723020BLA642, S/N:MN5220F31SNUNK, WWN:5-000cca-369d8da39, FW:MN6OA5C0, 2.00 TB
Jan 8 20:36:01 freenas 1 2024-01-08T20:36:01.045621+01:00 freenas.local smartd 1493 - - Device: /dev/ada5, found in smartd database: Hitachi Deskstar 7K3000
Jan 8 20:36:01 freenas 1 2024-01-08T20:36:01.457211+01:00 freenas.local avahi-daemon 1775 - - Service "freenas" (/usr/local/etc/avahi/services/SSH.service) successfully established.
Jan 8 20:36:01 freenas 1 2024-01-08T20:36:01.457301+01:00 freenas.local avahi-daemon 1775 - - Service "freenas" (/usr/local/etc/avahi/services/SMB.service) successfully established.
Jan 8 20:36:01 freenas 1 2024-01-08T20:36:01.457356+01:00 freenas.local avahi-daemon 1775 - - Service "freenas" (/usr/local/etc/avahi/services/SFTP_SSH.service) successfully established.
Jan 8 20:36:01 freenas 1 2024-01-08T20:36:01.457430+01:00 freenas.local avahi-daemon 1775 - - Service "freenas" (/usr/local/etc/avahi/services/MIDDLEWARE_SSL.service) successfully established.
Jan 8 20:36:01 freenas 1 2024-01-08T20:36:01.457505+01:00 freenas.local avahi-daemon 1775 - - Service "freenas" (/usr/local/etc/avahi/services/MIDDLEWARE.service) successfully established.
Jan 8 20:36:01 freenas 1 2024-01-08T20:36:01.457627+01:00 freenas.local avahi-daemon 1775 - - Service "freenas" (/usr/local/etc/avahi/services/ISCSITARGET.service) successfully established.
Jan 8 20:36:01 freenas 1 2024-01-08T20:36:01.457714+01:00 freenas.local avahi-daemon 1775 - - Service "freenas" (/usr/local/etc/avahi/services/HTTPS.service) successfully established.
Jan 8 20:36:01 freenas 1 2024-01-08T20:36:01.457808+01:00 freenas.local avahi-daemon 1775 - - Service "freenas" (/usr/local/etc/avahi/services/HTTP.service) successfully established.
Jan 8 20:36:01 freenas 1 2024-01-08T20:36:01.457915+01:00 freenas.local avahi-daemon 1775 - - Service "freenas" (/usr/local/etc/avahi/services/DEV_INFO.service) successfully established.
Jan 8 20:36:02 freenas 1 2024-01-08T20:36:02.022074+01:00 freenas.local daemon 1903 - - 2024-01-08 20:36:02,021:wsdd WARNING(pid 1904): no interface given, using all interfaces
Jan 8 20:36:02 freenas 1 2024-01-08T20:36:02.694097+01:00 freenas.local smartd 1493 - - Device: /dev/ada5, is SMART capable. Adding to "monitor" list.
Jan 8 20:36:02 freenas 1 2024-01-08T20:36:02.694300+01:00 freenas.local smartd 1493 - - Device: /dev/ada1, opened
Jan 8 20:36:02 freenas 1 2024-01-08T20:36:02.703355+01:00 freenas.local smartd 1493 - - Device: /dev/ada1, Hitachi HDS723020BLA642, S/N:MN3220F3316H7E, WWN:5-000cca-369ead340, FW:MN6OA800, 2.00 TB
Jan 8 20:36:02 freenas 1 2024-01-08T20:36:02.707437+01:00 freenas.local smartd 1493 - - Device: /dev/ada1, found in smartd database: Hitachi Deskstar 7K3000
Jan 8 20:36:04 freenas 1 2024-01-08T20:36:04.296666+01:00 freenas.local smartd 1493 - - Device: /dev/ada1, is SMART capable. Adding to "monitor" list.
Jan 8 20:36:04 freenas 1 2024-01-08T20:36:04.296777+01:00 freenas.local smartd 1493 - - Monitoring 8 ATA/SATA, 0 SCSI/SAS and 0 NVMe devices
Jan 8 20:36:06 freenas 1 2024-01-08T20:36:06.449890+01:00 freenas.local dhclient 945 - - em0 link state down -> up
Jan 8 20:36:06 freenas 1 2024-01-08T20:36:06.449991+01:00 freenas.local dhclient 945 - - DHCPREQUEST on em0 to 255.255.255.255 port 67
Jan 8 20:36:09 freenas 1 2024-01-08T20:36:09.451289+01:00 freenas.local dhclient 945 - - DHCPREQUEST on em0 to 255.255.255.255 port 67
Jan 8 20:36:09 freenas 1 2024-01-08T20:36:09.452321+01:00 freenas.local dhclient 945 - - DHCPACK from 10.0.0.1
Jan 8 20:36:09 freenas 1 2024-01-08T20:36:09.454924+01:00 freenas.local avahi-daemon 1775 - - Withdrawing address record for 10.0.0.5 on em0.
Jan 8 20:36:09 freenas 1 2024-01-08T20:36:09.455000+01:00 freenas.local avahi-daemon 1775 - - Leaving mDNS multicast group on interface em0.IPv4 with address 10.0.0.5.
Jan 8 20:36:09 freenas 1 2024-01-08T20:36:09.455019+01:00 freenas.local avahi-daemon 1775 - - Interface em0.IPv4 no longer relevant for mDNS.
Jan 8 20:36:09 freenas 1 2024-01-08T20:36:09.455226+01:00 freenas.local avahi-daemon 1775 - - Joining mDNS multicast group on interface em0.IPv4 with address 10.0.0.5.
Jan 8 20:36:09 freenas 1 2024-01-08T20:36:09.455242+01:00 freenas.local avahi-daemon 1775 - - New relevant interface em0.IPv4 for mDNS.
Jan 8 20:36:09 freenas 1 2024-01-08T20:36:09.455297+01:00 freenas.local avahi-daemon 1775 - - Registering new address record for 10.0.0.5 on em0.IPv4.
Jan 8 20:36:09 freenas 1 2024-01-08T20:36:09.470125+01:00 freenas.local dhclient 945 - - bound to 10.0.0.5 -- renewal in 43200 seconds.
Jan 8 20:36:09 freenas 1 2024-01-08T20:36:09.962967+01:00 freenas.local smartd 1493 - - file /var/run/smartd-daemon.pid written containing PID 1493
Jan 8 21:06:10 freenas 1 2024-01-08T21:06:10.208472+01:00 freenas.local smartd 1493 - - Device: /dev/ada4, SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 111 to 112
Jan 8 21:06:10 freenas 1 2024-01-08T21:06:10.208533+01:00 freenas.local smartd 1493 - - Device: /dev/ada4, SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 68 to 67
Jan 8 21:06:10 freenas 1 2024-01-08T21:06:10.208545+01:00 freenas.local smartd 1493 - - Device: /dev/ada4, SMART Usage Attribute: 194 Temperature_Celsius changed from 32 to 33
Jan 8 21:06:11 freenas 1 2024-01-08T21:06:11.585582+01:00 freenas.local smartd 1493 - - Device: /dev/ada3, SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 99 to 100
Jan 8 21:06:11 freenas 1 2024-01-08T21:06:11.585639+01:00 freenas.local smartd 1493 - - Device: /dev/ada3, SMART Usage Attribute: 194 Temperature_Celsius changed from 171 to 166
Jan 8 21:06:12 freenas 1 2024-01-08T21:06:12.237884+01:00 freenas.local smartd 1493 - - Device: /dev/ada0, SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 62 to 61
Jan 8 21:06:12 freenas 1 2024-01-08T21:06:12.237941+01:00 freenas.local smartd 1493 - - Device: /dev/ada0, SMART Usage Attribute: 194 Temperature_Celsius changed from 38 to 39
Jan 8 21:06:12 freenas 1 2024-01-08T21:06:12.470862+01:00 freenas.local smartd 1493 - - Device: /dev/ada6, SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 105 to 108
Jan 8 21:06:12 freenas 1 2024-01-08T21:06:12.470907+01:00 freenas.local smartd 1493 - - Device: /dev/ada6, SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 69 to 68
Jan 8 21:06:12 freenas 1 2024-01-08T21:06:12.470927+01:00 freenas.local smartd 1493 - - Device: /dev/ada6, SMART Usage Attribute: 194 Temperature_Celsius changed from 31 to 32
Jan 8 21:06:12 freenas 1 2024-01-08T21:06:12.693435+01:00 freenas.local smartd 1493 - - Device: /dev/ada7, SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 64 to 63
Jan 8 21:06:12 freenas 1 2024-01-08T21:06:12.693493+01:00 freenas.local smartd 1493 - - Device: /dev/ada7, SMART Usage Attribute: 194 Temperature_Celsius changed from 36 to 37
Jan 8 21:06:13 freenas 1 2024-01-08T21:06:13.182479+01:00 freenas.local smartd 1493 - - Device: /dev/ada5, SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 94 to 99
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.127590+01:00 freenas.local avahi-daemon 1374 - - Found user 'avahi' (UID 200) and group 'avahi' (GID 200).
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.127831+01:00 freenas.local avahi-daemon 1374 - - Successfully dropped root privileges.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.127863+01:00 freenas.local avahi-daemon 1374 - - avahi-daemon 0.8 starting up.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128032+01:00 freenas.local avahi-daemon 1374 - - Loading service file /usr/local/etc/avahi/services/DEV_INFO.service.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128121+01:00 freenas.local avahi-daemon 1374 - - Loading service file /usr/local/etc/avahi/services/HTTP.service.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128143+01:00 freenas.local avahi-daemon 1374 - - Loading service file /usr/local/etc/avahi/services/HTTPS.service.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128161+01:00 freenas.local avahi-daemon 1374 - - Loading service file /usr/local/etc/avahi/services/ISCSITARGET.service.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128178+01:00 freenas.local avahi-daemon 1374 - - Loading service file /usr/local/etc/avahi/services/MIDDLEWARE.service.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128195+01:00 freenas.local avahi-daemon 1374 - - Loading service file /usr/local/etc/avahi/services/MIDDLEWARE_SSL.service.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128214+01:00 freenas.local avahi-daemon 1374 - - Loading service file /usr/local/etc/avahi/services/SFTP_SSH.service.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128230+01:00 freenas.local avahi-daemon 1374 - - Loading service file /usr/local/etc/avahi/services/SMB.service.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128247+01:00 freenas.local avahi-daemon 1374 - - Loading service file /usr/local/etc/avahi/services/SSH.service.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128406+01:00 freenas.local avahi-daemon 1374 - - Joining mDNS multicast group on interface lo0.IPv4 with address 127.0.0.1.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128432+01:00 freenas.local avahi-daemon 1374 - - New relevant interface lo0.IPv4 for mDNS.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128440+01:00 freenas.local avahi-daemon 1374 - - Joining mDNS multicast group on interface em0.IPv4 with address 10.0.0.5.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128566+01:00 freenas.local avahi-daemon 1374 - - New relevant interface em0.IPv4 for mDNS.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128575+01:00 freenas.local avahi-daemon 1374 - - Network interface enumeration completed.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128604+01:00 freenas.local avahi-daemon 1374 - - Registering new address record for ::1 on lo0.*.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128666+01:00 freenas.local avahi-daemon 1374 - - Registering new address record for 127.0.0.1 on lo0.IPv4.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.128680+01:00 freenas.local avahi-daemon 1374 - - Registering new address record for 10.0.0.5 on em0.IPv4.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.678176+01:00 freenas.local smartd 1443 - - smartd 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.678350+01:00 freenas.local smartd 1443 - - Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.678376+01:00 freenas.local smartd 1443 - - Opened configuration file /usr/local/etc/smartd.conf
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.678855+01:00 freenas.local smartd 1443 - - Configuration file /usr/local/etc/smartd.conf parsed.
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.679119+01:00 freenas.local smartd 1443 - - Device: /dev/ada4, opened
Jan 8 21:30:53 freenas 1 2024-01-08T21:30:53.680209+01:00 freenas.local smartd 1443 - - Device: /dev/ada4, ST32000542AS, S/N:5XW1QHVX, WWN:5-000c50-

messages

Jan 8 20:35:55 freenas GEOM_ELI: Crypto: accelerated software
Jan 8 20:35:55 freenas GEOM_ELI: Device mirror/swap1.eli created.
Jan 8 20:35:55 freenas GEOM_ELI: Encryption: AES-XTS 128
Jan 8 20:35:55 freenas GEOM_ELI: Crypto: accelerated software
Jan 8 20:35:55 freenas CPU: Intel(R) Core(TM) i5-3470T CPU @ 2.90GHz (2893.43-MHz K8-class CPU)
Jan 8 20:35:55 freenas 1 2024-01-08T20:35:55.306072+01:00 freenas.local ntpd 1373 - - ntpd 4.2.8p15-a (1): Starting
Jan 8 20:35:55 freenas 1 2024-01-08T20:35:55.306220+01:00 freenas.local ntpd 1373 - - Command line: /usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f /var/db/ntp/ntpd.drift -g
Jan 8 20:35:55 freenas 1 2024-01-08T20:35:55.306226+01:00 freenas.local ntpd 1373 - - ----------------------------------------------------
Jan 8 20:35:55 freenas 1 2024-01-08T20:35:55.306230+01:00 freenas.local ntpd 1373 - - ntp-4 is maintained by Network Time Foundation,
Jan 8 20:35:55 freenas 1 2024-01-08T20:35:55.306235+01:00 freenas.local ntpd 1373 - - Inc. (NTF), a non-profit 501(c)(3) public-benefit
Jan 8 20:35:55 freenas 1 2024-01-08T20:35:55.306239+01:00 freenas.local ntpd 1373 - - corporation. Support and training for ntp-4 are
Jan 8 20:35:55 freenas 1 2024-01-08T20:35:55.306243+01:00 freenas.local ntpd 1373 - - available at https://www.nwtime.org/support
Jan 8 20:35:55 freenas 1 2024-01-08T20:35:55.306247+01:00 freenas.local ntpd 1373 - - ----------------------------------------------------
Jan 8 20:35:55 freenas Origin="GenuineIntel" Id=0x306a9 Family=0x6 Model=0x3a Stepping=9
Jan 8 20:35:55 freenas Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Jan 8 20:35:55 freenas Features2=0x7fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Jan 8 20:35:55 freenas AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
Jan 8 20:35:55 freenas AMD Features2=0x1<LAHF>
Jan 8 20:35:55 freenas Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS>
Jan 8 20:35:55 freenas Structured Extended Features3=0x9c000400<MD_CLEAR,IBPB,STIBP,L1DFL,SSBD>
Jan 8 20:35:55 freenas XSAVE Features=0x1<XSAVEOPT>
Jan 8 20:35:55 freenas VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Jan 8 20:35:55 freenas TSC: P-state invariant, performance statistics
Jan 8 20:35:55 freenas kernel: em0: link state changed to DOWN
Jan 8 20:35:55 freenas kernel: em0: link state changed to UP
Jan 8 20:35:55 freenas hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI>
Jan 8 20:35:55 freenas Security policy loaded: MAC/ntpd (mac_ntpd)
Jan 8 20:35:56 freenas 1 2024-01-08T20:35:56.631323+01:00 freenas.local smartd 1493 - - Device: /dev/ada4, WARNING: A firmware update for this drive may be available,
Jan 8 20:35:56 freenas 1 2024-01-08T20:35:56.631337+01:00 freenas.local smartd 1493 - - see the following Seagate web pages:
Jan 8 20:35:56 freenas 1 2024-01-08T20:35:56.631345+01:00 freenas.local smartd 1493 - - http://knowledge.seagate.com/articles/en_US/FAQ/207931en
Jan 8 20:35:56 freenas 1 2024-01-08T20:35:56.631350+01:00 freenas.local smartd 1493 - - http://knowledge.seagate.com/articles/en_US/FAQ/213915en
Jan 8 20:35:56 freenas 1 2024-01-08T20:35:56.771245+01:00 freenas.local daemon 1477 - - 2024-01-08 20:35:56,771:wsdd WARNING(pid 1478): no interface given, using all interfaces
Jan 8 20:36:00 freenas kernel: bridge0: Ethernet address: 58:9c:fc:10:ff:bc
Jan 8 20:36:00 freenas kernel: em0: link state changed to DOWN
Jan 8 20:36:00 freenas kernel: bridge0: link state changed to UP
Jan 8 20:36:00 freenas kernel: em0: promiscuous mode enabled
Jan 8 20:36:00 freenas kernel: epair0a: Ethernet address: 02:ec:6e:d4:64:0a
Jan 8 20:36:00 freenas kernel: epair0b: Ethernet address: 02:ec:6e:d4:64:0b
Jan 8 20:36:00 freenas kernel: epair0a: link state changed to UP
Jan 8 20:36:00 freenas kernel: epair0b: link state changed to UP
Jan 8 20:36:00 freenas kernel: epair0a: changing name to 'vnet0.1'
Jan 8 20:36:02 freenas 1 2024-01-08T20:36:02.022074+01:00 freenas.local daemon 1903 - - 2024-01-08 20:36:02,021:wsdd WARNING(pid 1904): no interface given, using all interfaces
Jan 8 20:36:03 freenas kernel: vnet0.1: promiscuous mode enabled
Jan 8 20:36:06 freenas kernel: em0: link state changed to UP
Jan 8 20:36:09 freenas 1 2024-01-08T20:36:09.457133+01:00 freenas.local dhclient 1940 - - New IP Address (em0): 10.0.0.5
Jan 8 20:36:09 freenas 1 2024-01-08T20:36:09.458974+01:00 freenas.local dhclient 1944 - - New Subnet Mask (em0): 255.255.255.0
Jan 8 20:36:09 freenas 1 2024-01-08T20:36:09.460939+01:00 freenas.local dhclient 1948 - - New Broadcast Address (em0): 10.0.0.255
Jan 8 20:36:09 freenas 1 2024-01-08T20:36:09.462680+01:00 freenas.local dhclient 1952 - - New Routers (em0): 10.0.0.1
Jan 8 20:36:13 freenas kernel: lo0: link state changed to UP
Jan 8 20:36:32 freenas kernel: epair1a: Ethernet address: 02:4e:98:cf:78:0a
Jan 8 20:36:32 freenas kernel: epair1b: Ethernet address: 02:4e:98:cf:78:0b
Jan 8 20:36:32 freenas kernel: epair1a: link state changed to UP
Jan 8 20:36:32 freenas kernel: epair1b: link state changed to UP
Jan 8 20:36:32 freenas kernel: epair1a: changing name to 'vnet0.2'
Jan 8 20:36:33 freenas kernel: epair1b: changing name to 'epair0b'
Jan 8 20:36:36 freenas kernel: lo0: link state changed to UP
Jan 8 21:30:52 freenas syslog-ng[1255]: syslog-ng starting up; version='3.35.1'
Jan 8 21:30:52 freenas ---<<BOOT>>---
Jan 8 21:30:52 freenas Copyright (c) 1992-2021 The FreeBSD Project.
Jan 8 21:30:52 freenas Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Jan 8 21:30:52 freenas <------>The Regents of the University of California. All rights reserved.
Jan 8 21:30:52 freenas FreeBSD is a registered trademark of The FreeBSD Foundation.
Jan 8 21:30:52 freenas FreeBSD 13.1-RELEASE-p7 n245428-4dfb91682c1 TRUENAS amd64
Jan 8 21:30:52 freenas FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
Jan 8 21:30:52 freenas VT(vga): text 80x25
Jan 8 21:30:52 freenas CPU: Intel(R) Core(TM) i5-3470T CPU @ 2.90GHz (2893.54-MHz K8-class CPU)
Jan 8 21:30:52 freenas Origin="GenuineIntel" Id=0x306a9 Family=0x6 Model=0x3a Stepping=9
Jan 8 21:30:52 freenas Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Jan 8 21:30:52 freenas Features2=0x7fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Jan 8 21:30:52 freenas AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
Jan 8 21:30:52 freenas AMD Features2=0x1<LAHF>
Jan 8 21:30:52 freenas Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS>
Jan 8 21:30:52 freenas XSAVE Features=0x1<XSAVEOPT>
Jan 8 21:30:52 freenas VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID

 

[Patrick M. Hausen]

So no message whatsoever related to a shutdown or crash. Just like a power cycle followed by a regular startup.

Difficult. Do you have a console monitor connected and also happen to own a dashcam or something similar?

 

[sqwob]

I had a monitor attached during my previous tests, only thing that came up there were ISCSI volume related ping tests that got no reponse and were going to be disconnected ("iqn.something.somthing): no ping reply (NOP-Out) after 5 seconds; dropping connection''. These didn't seem relevant.

I never got it to power down while the screen was connected (also doubt i would be fast enough to see anything before it shuts off)...

Weird that it didn't shut down during the cpu load test & ram tests :/

 

[Patrick M. Hausen]

I never got it to power down while the screen was connected (also doubt i would be fast enough to see anything before it shuts off)...

That's why I hinted at setting up a camera ...

 

[WI_Hedgehog]

This could be anything as obtuse as the CPU paste dried out, CPU overheats under load and bork....

 

[sqwob]

This could be anything as obtuse as the CPU paste dried out, CPU overheats under load and bork....

That would have come up in my cpu load test? I had it running at max cpu load for 40 minutes.
cpu load when i'm using it is <10%

 

[Patrick M. Hausen]

This could be anything as obtuse as the CPU paste dried out, CPU overheats under load and bork....

 

[sqwob]

That's why I hinted at setting up a camera ...

i'll have to get creative so it seems...

Thanks for the input so far!

 

[WI_Hedgehog]

That would have come up in my cpu load test? I had it running at max cpu load for 40 minutes.
cpu load when i'm using it is <10%

Well, that's the thing, it's hard to say; your CPU could be heating up, flexing the mainboard, that bends the CPU just wrong, bork. Though I'd lean more toward memory corruption at that point which is sometimes fixed by re-seating the RAM modules. Years ago I had a CPU that was "over-used" and had electron tunneling, which is an odd thing to diagnose and test for, but it was, and it did odd things...none of which looked like CPU issues (maybe RAM, but not CPU).

Anyway, I'm not saying it's a CPU in your case, rather you've an unusual issue and some system stress testing for a week or so ought to cause it to show up. Re-seating everything is a good start, but the bottom line is your data is possibly at risk, so you might want to ensure your 3-2-1 backup plan works great, then stress-test the whole system at once until the problem can be triggered consistently, then narrow down testing to smaller parts of your system until the source of the problem is located. In an unusual case I've had a drive controller take down the system, power overload issues can do it, tiny shorts can trigger an emergency shutdown in a good power supply (so your system isn't smoked)--on that I had one server power supply system that would output insane amps per PSU yet trip on the smallest short (by design), which is in theory super-safe UNTIL a server went up in flames in front of me--behind my back, actually, but I heard it pop and turned around to see orange flames inside the case and shortly thereafter smoke poured from the unit--anyway, that's all to say I don't trust flaky systems, take the data off them and figure the best way to diagnose the problem before touching anything physically--because I want to find and remove the problem.

 

[MrGuvernment]

M.2 SSD in a usb enclosure

Not ideal, are you short of SATA ports to just use a 2.5 SSD as your boot drive?

 

[Davvo]

No solution here, but a few troubleshooting steps:

• Run memtest86.
• Reseat RAM, CPU, NIC, and power cables.
• Update and reset the BIOS.

This feels more like a power issue than a RAM or driver issue: check whether the PSU fan is spinning or if you can see damage inside it (or smell something strange).

 

[sqwob]

reseated memory, replaced all power cables
memtest was already run as mentioned in initial post
I also swapped out the complete power supply just now for a better model i still had lying around.

curious to see if it had any impact

 

[sqwob]

Not ideal, are you short of SATA ports to just use a 2.5 SSD as your boot drive?

yeah, when i first installed it ten years ago it was actually recommended to load "freenas' from a usb stick, so i started there and equipped all 8 sata ports with hard drives for the data. I only moved the boot volume to msata just now, because i read using usb sticks wasn't recommended anymore.

 

[sqwob]

reseated memory, replaced all power cables
memtest was already run as mentioned in initial post
I also swapped out the complete power supply just now for a better model i still had lying around.

curious to see if it had any impact

And it shut down again about 30 minutes in... :(

 

[Davvo]

Might be a motherboard security feature kicking in, avoiding overheat or the likes. Check BIOS logs, reset, update.

 

[sqwob]

Difficult. Do you have a console monitor connected and also happen to own a dashcam or something similar?

Installed the nas in the livingroom with a camera pointing at the screen this evening, it just plain powered off, no errors on display

I also updated the motherboard bios to the latest verion beforehand, no impact there.
Bios contains no logs that i can find.

Took out half the ram now, will eliminate if this half or the other half might be the cause.
later i'll also swap cpu for an older one, then if it still happens it has to be the motherboard (or software)