write_throughput: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.25
Starting 8 processes
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)

write_throughput: (groupid=0, jobs=8): err= 0: pid=1226744: Sat May 28 16:35:09 2022
  write: IOPS=355, BW=364MiB/s (381MB/s)(21.3GiB/60016msec); 0 zone resets
    slat (msec): min=3, max=296, avg=22.51, stdev= 8.30
    clat (usec): min=8, max=2258.2k, avg=1396036.86, stdev=224197.89
     lat (msec): min=9, max=2284, avg=1418.31, stdev=225.99
    clat percentiles (msec):
     |  1.00th=[  667],  5.00th=[ 1167], 10.00th=[ 1200], 20.00th=[ 1250],
     | 30.00th=[ 1301], 40.00th=[ 1334], 50.00th=[ 1368], 60.00th=[ 1418],
     | 70.00th=[ 1452], 80.00th=[ 1552], 90.00th=[ 1703], 95.00th=[ 1787],
     | 99.00th=[ 1938], 99.50th=[ 2022], 99.90th=[ 2123], 99.95th=[ 2140],
     | 99.99th=[ 2232]
   bw (  KiB/s): min=194413, max=510684, per=97.83%, avg=364310.15, stdev=7494.00, samples=952
   iops        : min=  189, max=  498, avg=354.76, stdev= 7.37, samples=952
  lat (usec)   : 10=0.01%, 20=0.03%, 100=0.01%
  lat (msec)   : 10=0.01%, 20=0.04%, 50=0.06%, 100=0.09%, 250=0.16%
  lat (msec)   : 500=0.39%, 750=0.51%, 1000=1.28%, 2000=99.17%, >=2000=0.61%
  cpu          : usr=0.31%, sys=3.00%, ctx=190880, majf=0, minf=3300
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,21314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=364MiB/s (381MB/s), 364MiB/s-364MiB/s (381MB/s-381MB/s), io=21.3GiB (22.9GB), run=60016-60016msec
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.25
Starting 1 process
write_iops: Laying out IO file (1 file / 10240MiB)

write_iops: (groupid=0, jobs=1): err= 0: pid=1299335: Sat May 28 16:36:11 2022
  write: IOPS=4378, BW=17.1MiB/s (17.9MB/s)(1027MiB/60001msec); 0 zone resets
    slat (usec): min=8, max=34660, avg=223.49, stdev=276.99
    clat (usec): min=4, max=103639, avg=14388.04, stdev=5893.15
     lat (usec): min=291, max=103713, avg=14612.13, stdev=5977.48
    clat percentiles (usec):
     |  1.00th=[ 3785],  5.00th=[ 5276], 10.00th=[ 6587], 20.00th=[ 9503],
     | 30.00th=[11469], 40.00th=[13173], 50.00th=[14484], 60.00th=[15795],
     | 70.00th=[17171], 80.00th=[18482], 90.00th=[20841], 95.00th=[22938],
     | 99.00th=[28967], 99.50th=[35914], 99.90th=[51643], 99.95th=[63177],
     | 99.99th=[94897]
   bw (  KiB/s): min= 9704, max=46176, per=100.00%, avg=17562.50, stdev=7008.54, samples=119
   iops        : min= 2426, max=11544, avg=4390.62, stdev=1752.10, samples=119
  lat (usec)   : 10=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=1.42%, 10=20.78%, 20=64.60%, 50=13.11%
  lat (msec)   : 100=0.11%, 250=0.01%
  cpu          : usr=2.10%, sys=87.99%, ctx=28273, majf=0, minf=248
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,262738,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=17.1MiB/s (17.9MB/s), 17.1MiB/s-17.1MiB/s (17.9MB/s-17.9MB/s), io=1027MiB (1076MB), run=60001-60001msec
read_throughput: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.25
Starting 8 processes
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)

read_throughput: (groupid=0, jobs=8): err= 0: pid=1732032: Sat May 28 16:42:40 2022
  read: IOPS=346, BW=354MiB/s (372MB/s)(20.8GiB/60064msec)
    slat (usec): min=487, max=544304, avg=23102.36, stdev=48098.67
    clat (usec): min=8, max=4004.5k, avg=1435739.47, stdev=329636.00
     lat (msec): min=10, max=4031, avg=1458.82, stdev=332.83
    clat percentiles (msec):
     |  1.00th=[  659],  5.00th=[  953], 10.00th=[ 1070], 20.00th=[ 1200],
     | 30.00th=[ 1301], 40.00th=[ 1385], 50.00th=[ 1435], 60.00th=[ 1485],
     | 70.00th=[ 1569], 80.00th=[ 1653], 90.00th=[ 1787], 95.00th=[ 1921],
     | 99.00th=[ 2333], 99.50th=[ 2836], 99.90th=[ 3910], 99.95th=[ 3977],
     | 99.99th=[ 4010]
   bw (  KiB/s): min=98263, max=673792, per=97.91%, avg=355407.80, stdev=12733.89, samples=953
   iops        : min=   95, max=  658, avg=346.77, stdev=12.45, samples=953
  lat (usec)   : 10=0.01%, 20=0.03%, 50=0.01%
  lat (msec)   : 20=0.01%, 50=0.01%, 100=0.04%, 250=0.23%, 500=0.41%
  lat (msec)   : 750=0.73%, 1000=5.35%, 2000=92.25%, >=2000=3.35%
  cpu          : usr=0.05%, sys=14.17%, ctx=8826, majf=0, minf=164902
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=20784,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=354MiB/s (372MB/s), 354MiB/s-354MiB/s (372MB/s-372MB/s), io=20.8GiB (22.3GB), run=60064-60064msec
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.25
Starting 1 process
read_iops: Laying out IO file (1 file / 10240MiB)

read_iops: (groupid=0, jobs=1): err= 0: pid=1835852: Sat May 28 16:44:34 2022
  read: IOPS=496, BW=1991KiB/s (2038kB/s)(117MiB/60002msec)
    slat (usec): min=5, max=20066, avg=2009.07, stdev=965.06
    clat (usec): min=4, max=173699, avg=126749.12, stdev=17092.05
     lat (msec): min=2, max=176, avg=128.76, stdev=17.33
    clat percentiles (msec):
     |  1.00th=[   92],  5.00th=[  102], 10.00th=[  105], 20.00th=[  112],
     | 30.00th=[  116], 40.00th=[  123], 50.00th=[  128], 60.00th=[  133],
     | 70.00th=[  138], 80.00th=[  142], 90.00th=[  148], 95.00th=[  155],
     | 99.00th=[  167], 99.50th=[  167], 99.90th=[  171], 99.95th=[  174],
     | 99.99th=[  174]
   bw (  KiB/s): min= 1536, max= 2552, per=99.77%, avg=1986.70, stdev=238.99, samples=120
   iops        : min=  384, max=  638, avg=496.60, stdev=59.80, samples=120
  lat (usec)   : 10=0.01%
  lat (msec)   : 4=0.02%, 10=0.02%, 20=0.02%, 50=0.05%, 100=4.71%
  lat (msec)   : 250=95.39%
  cpu          : usr=0.23%, sys=99.58%, ctx=227, majf=0, minf=402
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=29797,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1991KiB/s (2038kB/s), 1991KiB/s-1991KiB/s (2038kB/s-2038kB/s), io=117MiB (122MB), run=60002-60002msec
