CPU load - uploading vs downloading

[Flybye]

I transferred a 4GB file to FN. Sending the file pegs the server's CPU at an average of about 50%. Receiving the same file from the server pegs the server's CPU at an average of 15%. I tested it about 3 times.

Why the huge differance in server CPU load?

 

[sretalla]

Is the dataset compressed?

What's the ZFS pool layout?

How much RAM do you have?

There's likely an effect from compression, ARC (RAM cache), parity creation (depending on ZFS layout) and tracking the writing itself.

For a number of those points, the re-reading of what was written has an advantage in terms of required work.

 

[ornias]

I transferred a 4GB file to FN. Sending the file pegs the server's CPU at an average of about 50%. Receiving the same file from the server pegs the server's CPU at an average of 15%. I tested it about 3 times.

Why the huge differance in server CPU load?

- Compression, Compression (writing) takes more CPU than DeCompression (reading)
- VDEV: RAIDZ takes more CPU when writing (parity calculations) than reading (no parity calculations if vdev is good)
- ARC-size: If data is cached a read would only need to decompress and wouldn't need to to parity calculations.


Tracking the write itself has such a minimal difference on CPU load profile, that it isn't relevant for describing the 15 vs 50% difference OP asks about.

 

[Flybye]

It is a brand new build with only vanilla options selected. I am still learning a lot about FN, but I will dig up the answers the best I can:

Is the dataset compressed?
Compression level: Inherit (lz4)

What's the ZFS pool layout?
Currently it only has 2 datasets on it. Sync: Standard. Compression level: lz4. Enable Atime: on. ZFS Deduplication: off. I have not touched any other options.

How much RAM do you have?
12GB - Low volume to be mostly used by 1 person and no streaming music/video.

I know I haven't set up raid of anytime. What is ARC-size? I see it in the Reporting > ZFS section, but I don't understand the data quite yet. Currently:

arc : min: 7.89GiB mean: 7.92GiB max: 8.06GiB

I am using a 128GB msata ssd for boot and testing with 1 pool on a 1TB 7200rpm drive if any of this makes any difference.
Compressing using more load make sense. I just didn't think it would be that much of a difference between compressing and uncompressing.

 

[ornias]

Is the dataset compressed?
Compression level: Inherit (lz4)

That would be it (mostly

1 pool on a 1TB 7200rpm

You are aware you can only expand this to a mirror in the future and cant expand it to a raidz vdev, right?
A pool doesn't contain drives btw, A pool contains "vdevs" (raids, for those not into zfs) and each vdev contains a set of disks ;)
Just to be sure... Wouldn't be the first time people hit a snag with zfs...

Compressing using more load make sense. I just didn't think it would be that much of a difference between compressing and uncompressing.

As I did a LOT of research on (de)compression performance for a current ZFS project... most algorithms take multiple times less CPU to decompress than they do to compress, or (the other way around) can decompress multiple times more than the can compress with the same CPU load.

Your CPU is bottlenecked by the single harddrive and/or network, so it makes sense that for the same (lets say) 100MBps, it takes about 3x less CPU to decompress than it does to compress...

For more research on the topic:

Introduce ZSTD compression to ZFS by c0d3z3r0 · Pull Request #9735 · openzfs/zfs

This PR introduces ZSTD compression to ZFS. Many thanks to: The original Proof of Concept by @allanjude see: #9024 The Linux compatibility redesign by @BrainSlayer see: #8941 Spin-off kvmem patch b...

github.com

*edit*
What CPU are you using actually, because for it to reach 50% with one disk it needs to be pretty slow (dualcore?)

 

[Flybye]

That would be it (mostly

You are aware you can only expand this to a mirror in the future and can't expand it to a raidz vdev, right?

I am not. I don't even know what raidz vdev is, or if I would even need it in the future lol. I know raid is used for either mirroring for redundancy or striping for speed, but I won't have the space for either. My little build is going in a tiny mini-ITX box that can only hold either 4 2.5" drives or 2 3.5" and 2 2.5" drives. I have seen a PCI slot bracket that can hold 2 2.5" drives which I may consider later. For backing it up, I was simply going to back up to large USB 3+ drive on my PC and only connecting the drive once or twice a month to backup any changes.

A pool doesn't contain drives btw, A pool contains "vdevs" (raids, for those not into zfs) and each vdev contains a set of disks ;)
Just to be sure... Wouldn't be the first time people hit a snag with zfs...

Yep I've noticed that. I don't understand the reasoning behind having a pool vs just having a folder access to the drive. But I am perfectly sure there are great reasons behind FN being structured this way. I will have to read up on vdevs since you have mentioned it a few times. What I "think" one of the reasons pooling exists is to maybe have like a Raid 0 type setup where the pool/vdev exists across multiple drives to increase performance or have redundancy?

Your CPU is bottlenecked by the single harddrive and/or network, so it makes sense that for the same (lets say) 100MBps, it takes about 3x less CPU to decompress than it does to compress...

Oh, my CPU is most certainly a bottleneck. But that is fine. Again, it's gonna be a low volume NAS. I have a bunch of ISOs to transfer to it, but I won't be constantly transferring them back and forth. I figured well let me test out a 4GB file to see how long it will take and how the CPU load does when transferring the file both ways. The transfer is tolerable on my 1Gb lan.

For more research on the topic:

Introduce ZSTD compression to ZFS by c0d3z3r0 · Pull Request #9735 · openzfs/zfs

This PR introduces ZSTD compression to ZFS. Many thanks to: The original Proof of Concept by @allanjude see: #9024 The Linux compatibility redesign by @BrainSlayer see: #8941 Spin-off kvmem patch b...

github.com

TY. I'll have to leave that readup for later when I can sit down and concentrate on it. :)

What CPU are you using actually, because for it to reach 50% with one disk it needs to be pretty slow (dualcore?)

Yep. Just an AMD A6-6420k 2 core. I was thinking about maxing out this board with a 4 core 880k and upgrading RAM (max 32GB) if need be.

 

[ornias]

I am not. I don't even know what raidz vdev is, or if I would even need it in the future lol.

You definately should because ZFS is not (always) too friendly if you want to change things down the road!

Yep I've noticed that. I don't understand the reasoning behind having a pool vs just having a folder access to the drive

You do have folder access to the drive, pools, vdevs and datasets are not a replacement for folder access (you can click shell in FreeNAS and go to /mnt/*INSERTPOOLNAMEHERE* to get folder access to the pool).

Pools are (simply put) a collection of vdevs, which are striped (like a raid 0).
vdevs are (simply put) combinations of disks that are either mirrored (raid1) or put in RAIDZ1 (raid5), RAIDZ2 (raid6) or RAIDZ3 (raid7)

Simply put:
a RAID 10 equivalent is a single pool with 2xMirror vdevs.

You can view "datasets" as something in-between a folder and a partition.
But most simply put they can be accessed like a folder with special configuration in freenas ;)

What I "think" one of the reasons pooling exists is to maybe have like a Raid 0 type setup where the pool/vdev exists across multiple drives to increase performance or have redundancy?

Existence of pools have a lot of technical itty nitty and the "reason" for existance is not the fact they are de-facto a raid0, but that would be too technical. But yes, they do FUNCTION like a raid*0 where * is another raid (which would be a vdev)

Oh, my CPU is most certainly a bottleneck

Nono, what I tried to explain was: you CPU is not a bottleneck in this case (otherwise it would've been 100% used), I know because I did CPU bound bottleneck testing (as linked)

TY. I'll have to leave that readup for later when I can sit down and concentrate on it. :)

Well, I you don't have basic ZFS experience, I think there are better things to read. The link I said is mostly meant for actual ZFS developers and skips A LOT of explaination. I think you might get all sorts of wrong conclusions if thats your first introduction :P

Yep. Just an AMD A6-6420k 2 core. I was thinking about maxing out this board with a 4 core 880k and upgrading RAM (max 32GB) if need be.

I really hope power is free where you live :P

 

[Flybye]

.....I really hope power is free where you live :p

Lol. Low volume NAS, remember? ;-)

So I read though RAIDZ VDEVS last night in the manual. Great suggestion, but I don't think I will need to go that far. TY though. :)

The only slight worry I have with the CPU is if I add one of those addons that allows you to easily browse pictures. But I can always upgrade to that 4 core.

 

[ornias]

So I read though RAIDZ VDEVS last night in the manual. Great suggestion, but I don't think I will need to go that far. TY though. :)

Thats great, I would suggest at least using mirrors though... Just two drivers carrying a copy of your data. It speeds up reads quite a bit and gives some data safety at (almost) no costs :)

The only slight worry I have with the CPU is if I add one of those addons that allows you to easily browse pictures. But I can always upgrade to that 4 core.

I think there would be more gain to-be-had with a modern dualcore with HT, like the G5420 for example (with a lot less power consumption)... but if you want to stick with current mobo, yeah having a quadcore as a dropin-replacement option is a good thing.

"easy browse pictures"
Mind explaining what you are refering to?

 

[Flybye]

....
"easy browse pictures"
Mind explaining what you are refering to?

Something like the Plex media server but scaled down to only do pictures. I don't need movie capabilities. I haven't done too much reading into it yet since I am still experimenting with FN.

 

[sretalla]

vdevs are (simply put) combinations of disks that are either mirrored (raid1) or put in RAIDZ1 (raid5), RAIDZ2 (raid6) or RAIDZ3 (raid7)

or striped. (although there's no redundancy there, so not recommended most of the time).