SOLVED Scale Internet Performance

[indivision]

I was recently able to upgrade my ISP speeds to 1000 Mb/s.

I'm able to verify that speed with some machines on the network. But, my Scale server is testing much slower, for some reason.

I do need to do some further testing to rule out potential factors outside of that server. But, I thought I would check first whether or not this isn't expected (due to drivers available, etc.)?

With librespeed, I'm seeing 100Mb/s down, 343Mb/s up. Is that normal/expected for Scale?

Hardware:
Supermicro X11SCH-LN4F ( https://www.supermicro.com/en/products/motherboard/X11SCH-LN4F )

[fixed speed abbreviations]

 

[morganL]

I was recently able to upgrade my ISP speeds to 1000 MB/s.

I'm able to verify that speed with some machines on the network. But, my Scale server is testing much slower, for some reason.

I do need to do some further testing to rule out potential factors outside of that server. But, I thought I would check first whether or not this isn't expected (due to drivers available, etc.)?

With librespeed, I'm seeing 100MB/s down, 343MB/s up. Is that normal/expected for Scale?

Hardware:
Supermicro X11SCH-LN4F ( https://www.supermicro.com/en/products/motherboard/X11SCH-LN4F )

I assume 1 Gb/s .. not 1 GigaByte/s ?
Can you test locally.. and then Internet.
We might see whether the issues are IP or TCP related.

 

[indivision]

I assume 1 Gb/s .. not 1 GigaByte/s ?

Yes.

My mistake on the abbreviation.

Can you test locally.. and then Internet.
We might see whether the issues are IP or TCP related.

Transferring large files locally, I hover around 105 MB/s both directions.

 

[Samuel Tai]

Transferring large files locally, I hover around 105 MB/s both directions.

With IP and TCP overhead, that works out to 880 Mbps, very consistent with your Scale system having a 1 Gbps NIC.

 

[indivision]

With IP and TCP overhead, that works out to 880 Mbps, very consistent with your Scale system having a 1 Gbps NIC.

Thank you. Yes. In fact, this would suggest that there isn't anything wrong with the wires or switch on the way to the router.

But, when the server accesses the internet directly, it's much slower than other machines on the same network.

I'm wondering if this is something others are seeing on Scale? Or, if there is anything known that I can do to address it?

 

[Samuel Tai]

But, when the server accesses the internet directly, it's much slower than other machines on the same network.

I'm wondering if this is something others are seeing on Scale? Or, if there is anything known that I can do to address it?

Scale Angelfish is known not to be very optimized, as the developers prioritized safe functionality over fast functionality. I wouldn't worry too much about it.

 

[indivision]

Scale Angelfish is known not to be very optimized, as the developers prioritized safe functionality over fast functionality. I wouldn't worry too much about it.

I would just like to know either way if this is indeed the reason. I'm happy to wait for future optimizations. But, if there is something for me to figure out I'd like to know that too and put some time into it.

 

[Davvo]

Transferring large files locally, I hover around 105 MB/s both directions.

Are you using a SSD to send the test files or a HDD? If it's the latter you are capped by device speeds.

 

[indivision]

Are you using an SSD to send the test files or a HDD? If it's the latter you are capped by device speeds.

For the local network tests I was copying a large file to and from a Z2 pool of WD Red HDD.

[I don't expect more than 105 MB/s locally. It's the internet speed of the server that is unexpected...]

 

[Davvo]

For the local network tests I was copying a large file to and from a Z2 pool of WD Red HDD.

Sorry, I probably wasn't clear.
If you are sending datas from a machine with and HDD, you usually won't get enough throughput to validate your network speeds, as the spinners won't go over 200 MB/s read speed.
That's why you usually want a device capable of reaching higher speeds in order to validate this kind of tests.
It's 4 am in my time zone, so I'm sorry if I overlooked something or stated the obvious.
edit: I'm really sorry, I misread the entire thread.

 

[indivision]

Sorry, I probably wasn't clear.
If you are sending datas from a machine with and HDD, you usually won't get enough throughput to validate your network speeds, as the spinners won't go over 200 MB/s read speed.
That's why you usually want a device capable of reaching higher speeds in order to validate this kind of tests.
It's 4 am in my time zone, so I'm sorry if I overlooked something or stated the obvious.

All good. Thank you for helping.

It looks like the local traffic is saturating the capability of my network. But, when the server directly uploads/downloads from the internet, it is hitting lower speeds.

I'm wondering if others can confirm/verify the same thing or if I may need to crack open the server BIOS to see if there is some adjustment that will help.

 

[morganL]

All good. Thank you for helping.

It looks like the local traffic is saturating the capability of my network. But, when the server directly uploads/downloads from the internet, it is hitting lower speeds.

I'm wondering if others can confirm/verify the same thing or if I may need to crack open the server BIOS to see if there is some adjustment that will help.

It's unlikely to be BIOS... since local speeds are good.

this would imply IP is working well. Ping should behave well. You can verify.

The next level is TCP... there are lots of ways that can be tuned. It depends a little on the target systems.

 

[indivision]

It's unlikely to be BIOS... since local speeds are good.

this would imply IP is working well. Ping should behave well. You can verify.

The next level is TCP... there are lots of ways that can be tuned. It depends a little on the target systems.

Thank you. Is TCP something I would modify via TrueNAS tuning entries? What is a good way to learn more about what is needed?

 

[morganL]

Thank you. Is TCP something I would modify via TrueNAS tuning entries? What is a good way to learn more about what is needed?

Linux man pages and there are lots of research pages on TCP.
1st question is can you confirm the difference between SCALE and some other OS.
Is their IP performance similar, but not TCP performance?
If you can reproduce the issue, then there is something to test and review.

 

[danb35]

Not the most scientific comparison to be sure, but my SCALE box (running in a VM under Proxmox with virtual disks and minimal resources) starts downloading a .iso from GitHub at 85+ MBytes/sec over my gigabit Internet connection. It slows down pretty quickly, which I expect to be a result of a slow virtual disk backend. I don't know of a better way to test Internet connection speed from SCALE, though would be happy to try.

 

[indivision]

1st question is can you confirm the difference between SCALE and some other OS.
Is their IP performance similar, but not TCP performance?

Ok. I will work on testing another OS on same machine and follow up.

Not the most scientific comparison to be sure, but my SCALE box (running in a VM under Proxmox with virtual disks and minimal resources) starts downloading a .iso from GitHub at 85+ MBytes/sec over my gigabit Internet connection. It slows down pretty quickly, which I expect to be a result of a slow virtual disk backend. I don't know of a better way to test Internet connection speed from SCALE, though would be happy to try.

Thank you. What does it slow down to?

I noticed something a little similar on my machine. When the upload part of the speed test runs, it starts off around the same (850 Mb/s or so). But, then it quickly levels out to around 340 Mb/s.

The download part of the test steadily builds up to 100 Mb/s. It seems very attached to that 100 Mb/s mark specifically. It often lands exactly on 100 or is only 1 or 2 off. Which makes me suspicious that it's somehow limited by config/drivers.

 

[danb35]

What does it slow down to?

Around 10 Mbyte/sec.

The speedtest apps I'm seeing for scale only seem to test speed between the client machine and the NAS--which may be useful, but I can't see how to use them to answer your question.

 

[indivision]

Around 10 Mbyte/sec.

The speedtest apps I'm seeing for scale only seem to test speed between the client machine and the NAS--which may be useful, but I can't see how to use them to answer your question.

Ah. If that is the case, I have misunderstood those speed test apps (tried librespeed and openspeedtest)... Are these only testing local network speed? :facepalm:

Even if that is the case... with librespeed, same app install, same machine, no changes at all, I am now getting 553Mbps down and 382Mbps up. So... what the?

I also found a way to test speed from the CLI that I think may be helpful for others running into internet speed questions:

You can run this line in TrueNAS shell:

curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -

That will give you a quick internet speed test in the CLI.

For me, those numbers were much closer to what was expected: 702 Mb/s down, 649 Mb/s up.

Still a little under the 1Gb/s capability. But, I think we can safely conclude from that test that it's NOT a Scale OS issue. IF there is a software issue, it would then seem be related to Kubernetes and/or something about the apps themselves. Or, am I missing something?

 

[danb35]

Are these only testing local network speed? :facepalm:

That appears to be the case--my clues were when (1) the upload speed reported was much higher than my Internet connection gives (I have a gigabit down, but only about 40 Mbit/sec up), and (2) I got very different answers depending on whether I was wired or on WiFi.

curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -

But that's pretty handy. On my OPNsense box, I needed to use | python3 instead, but that should work pretty broadly.