4.5TB of data written out of nowhere??

[reykato]

Yesterday, my NAS running TrueNAS-13.0-U4 began to feel sluggish while editing videos from it. I checked the UI and found that it was performing a weekly scrub, so I brushed it off... until I saw that the pool usage was increasing quickly. After stopping the scrub, the used space still kept increasing. I decided to restart the NAS and everything seemed to resolve. After the fact, I have now checked the reporting tab to see that, within ~12 hours, the free space fell from ~7TB to ~2.5TB without any data being written from any machine on my network. Now I have about 4.5 TB of data which is seemingly used by absolutely nothing. All machines on the network have been checked for malware. Checking the pool datasets, nothing seems wrong, and the correct data usages are being shown, except the used and free space account for that new 4TB of 'ghost data'.

The specs of my NAS are as follows:
OS: TrueNAS-13.0-U4
Processor: Intel i5-4570
Memory: 8GB DDR3-1600
Drives:
(6) 4TB HGST Ultrastar 7K4000 HDD
(1) Kingston 120GB A400 SSD
Configured in raid-z2

I am assuming this could have been some freak data leak or glitch in the OS. My main concern is this 'ghost data' taking up 4 terabytes of space on my NAS. I can provide any extra details if necessary. Thanks in advance for your help!

 

[jgreco]

Does du(1) show you where the disk space is? Editing tools sometimes do funky things with temp files. Also, it is entirely possible for UNIX to have a file descriptor open and writing that is getting stored to disk, but has no actual directory entry anywhere. This is usually referred to as an anonymous file. You should be able to reboot the host if something weird like that is happening, though that won't tell you what process was causing the problem.

 

[reykato]

Nothing seems out of the ordinary. I'm assuming the size of iocage is large because of a mount point to DJ, and the size of tyler_acct is large because of a mount point to Tyler.

 

[jgreco]

I'd reboot it. It feels like you have an anonymous file issue of some sort.

 

[Whattteva]

Do you have any snapshots taking up space? What's output of zfs list -t snap look like?

 

[WI_Hedgehog]

Do you have other users on the system? Sometimes out of nowhere one will decide to fill up the drive for fun/curiosity/other and these things happen. It's fun to watch someone write a recursive script and fill your volume completely, then watch it collapse in a matter of seconds back to normal (from the standpoint you've been there before and know what's going on).

Back in the Novell/DOS networking days I watched this happen, the user was quickly discovered and a message box popped up on their screen asking them to clean up completely when they were done playing around. Watching on CCTV the shock of the message was apparent, they terminated the script and quickly removed the data structures and we were functioning normally in under a minute.

That was also when one user would get there early and log in on six PCs to print high-resolution charts from Lotus 1-2-3 on the laser printer, a few pages from each PC. The data from five PCs swamped the network, they did six just to dominate all traffic and get done in under 15 minutes. Every other user was , but it was creative so we let it go...until the day they swamped the print buffer and overflowed the hard-drive-backed cache and crashed the server--then they were asked to stick to 3 PCs or less.

 

[reykato]

Do you have any snapshots taking up space? What's output of zfs list -t snap look like?

This is it. I haven't manually created any snapshots or set up periodic snapshots.

 

[Whattteva]

Those are just the system dataset and they don't take up much anyway. Looks like your phantom space is somewhere else.

 

[reykato]

Do you have other users on the system? Sometimes out of nowhere one will decide to fill up the drive for fun/curiosity/other and these things happen. It's fun to watch someone write a recursive script and fill your volume completely, then watch it collapse in a matter of seconds back to normal (from the standpoint you've been there before and know what's going on).

Back in the Novell/DOS networking days I watched this happen, the user was quickly discovered and a message box popped up on their screen asking them to clean up completely when they were done playing around. Watching on CCTV the shock of the message was apparent, they terminated the script and quickly removed the data structures and we were functioning normally in under a minute.

That was also when one user would get there early and log in on six PCs to print high-resolution charts from Lotus 1-2-3 on the laser printer, a few pages from each PC. The data from five PCs swamped the network, they did six just to dominate all traffic and get done in under 15 minutes. Every other user was , but it was creative so we let it go...until the day they swamped the print buffer and overflowed the hard-drive-backed cache and crashed the server--then they were asked to stick to 3 PCs or less.

The only other user has an automatic backup script run for documents and photos, nothing special.

Hearing about the fun you all had makes me nostalgic for a time when I wasn't even born yet! The only thing in that realm I can share is when my high school gave out these crappy HP laptops. A friend and I spent that night exposing and activating the local Administrator account without making much of a trace. I installed lots of programs and games, and played many of them during class. Briefly, I thought about a Senior prank scheme: breaching network security to gain network admin, then on the last day of class, sending a network-wide shutdown command with a cute little message :) I had a plan built out and everything, but I favored my better judgment.

Something else mildly entertaining from that place... I overheard a conversation between one of the IT guys and a teacher, during which they mentioned the school's secure wireless network password was something related to greek mythology. It took me a few hours of trying during class, and I still can't believe I guessed it. It was just "colossus". No capitals, no numbers or special characters. He said it's been that way for 20 years! (for reference, this was in 2019)

 

[Etorix]

I haven't manually created any snapshots or set up periodic snapshots.

You should. Snapshots are the foundation of backups, and basic protection against malware or user stupidity…

 

[joeschmuck]

user stupidity…

The most common issue of course. Even I have done this, deleted an entire directory by accident.

 

[WI_Hedgehog]

The most common issue of course. Even I have done this, deleted an entire directory by accident.

Same here, a few interruptions in a row and I think I'm somewhere I'm not (on the system), and suddenly there's a progress bar for a simple delete and thousands of files and headed for the dumpster...


Snapshots...smart advice from smart people.

 

[jgreco]

Same here, a few interruptions in a row and I think I'm somewhere I'm not (on the system), and suddenly there's a progress bar for a simple delete and thousands of files and headed for the dumpster...

This also gets to workflow. Your best use case for NAS storage is to see it as permanent storage if you can. A workflow where you create a system of files under /mnt/pool/dataset/dirone/ and they then sit there forever is better than a workflow where you create them in /mnt/pool/tmp/tmpdir/ and then later move them to dataset/dirone/. This has implications for snapshotting, replication, and rsync. A workflow where you are moving things around on the NAS makes rsync based protection much harder to do, for example.

If you buy into this idea, the cool thing is that you can do stuff like taking 15 minute snapshots for a duration like 12 hours, every 6 hours for two weeks, and then weekly for ten years. This gives you massive recoverability in the event of screwups. However, it only works if you are disciplined and are not treating your NAS as a large temporary directory, but more like a "write once" storage device.

 

[Davvo]

The only other user has an automatic backup script run for documents and photos, nothing special.

Hearing about the fun you all had makes me nostalgic for a time when I wasn't even born yet! The only thing in that realm I can share is when my high school gave out these crappy HP laptops. A friend and I spent that night exposing and activating the local Administrator account without making much of a trace. I installed lots of programs and games, and played many of them during class. Briefly, I thought about a Senior prank scheme: breaching network security to gain network admin, then on the last day of class, sending a network-wide shutdown command with a cute little message :) I had a plan built out and everything, but I favored my better judgment.

Something else mildly entertaining from that place... I overheard a conversation between one of the IT guys and a teacher, during which they mentioned the school's secure wireless network password was something related to greek mythology. It took me a few hours of trying during class, and I still can't believe I guessed it. It was just "colossus". No capitals, no numbers or special characters. He said it's been that way for 20 years! (for reference, this was in 2019)

As a fellow (but early) Gen Z I can understand you to a degree... I saw and used VHS for example, as well as analog TV and CRT screens; I loved Windows XP.

Anyway, it's incredible how poor is (was?) the tech safety in schools; ie I was able to easily get psws and usernames of every teacher using a keylog as a browser plugin in the classrom's IWB.

Back to the topic, I strongly suggest you to create a solid snapshot task even if you don't feel the need for it: a second chance is always useful.

 

[Whattteva]

As a fellow (but early) Gen Z I can understand you to a degree... I saw and used VHS for example, as well as analog TV and CRT screens; I loved Windows XP.

Wow Gen Z still got to see VHS and CRT? I thought millennials were the last ones to see those. Did you ever see actual typewriters too? You know, the reason why those keys on your keyboard are called "shift" and "caps lock"

 

[Davvo]

Wow Gen Z still got to see VHS and CRT? I thought millennials were the last ones to see those. Did you ever see actual typewriters too? You know, the reason why those keys on your keyboard are called "shift" and "caps lock"

I did, we had a few from my grandpatents side that I managed to see... One was electrical, the other purely mechanical... was amazing. Sadly I don't know where those are now.
But as I said I'm very early Z, almost late Millennial.

 

[WI_Hedgehog]

I did, we had a few from my grandpatents side that I managed to see... One was electrical, the other purely mechanical...

I have a tredle sewing machine....
...and four flatscreen monitor setup.

I don't know if it's eclectic, but I thinnk it's awesome.

 

[Davvo]

I don't know if it's eclectic, but I thinnk it's awesome.

Doing cardio is good for your heart, you got a nice, healthily setup that strikes the balance between new and old.

 

[Whattteva]

I did, we had a few from my grandpatents side that I managed to see... One was electrical, the other purely mechanical... was amazing. Sadly I don't know where those are now.
But as I said I'm very early Z, almost late Millennial.

Now here's one I don't think you've ever seen. A washing board, often used as a joke about women, but I think most people probably have never even seen one.

 

[joeschmuck]

I think most people probably have never even seen one.

Seen a washboard, never used one. Oh, it was in a museum. However I did use a mechanical typewriter even into high school, then an electric one, and a computer keyboard, and I use to use a punch card machine, IBM 029. Now that was old but it worked.

How did we get so far off topic?