PGGB Performance Tuning

jpizzle · April 30

5 hours ago, rayon said:

As PGGB makes memory to go through limits constantly, Windows ends up in situation in which it needs to start decompressing everything, which means more cpu cycles and also increasing RAM requirements (as it needs room to which it can decompress). Then it starts doing paging and all kinds of stuff to organize all that. When it's turned off, it will just page things that need paging instead of going constantly through the loop of compression, uncompression and paging. Memory compression is designed to avoid paging, but if we are going to need paging anyway, it's just making things heavier. I also noticed that the compressed amount was measured in two gigabytes max during processing so gained potential benefits were very questional.

Oh, very clever! I’m excited to try this myself!

Zaphod Beeblebrox · April 30

v6.1.44 with experimental NVME tuning feature

https://audiophilestyle.com/forums/topic/62699-a-toast-to-pggb-a-heady-brew-of-math-and-magic/?do=findComment&comment=1278428

austinpop · April 30

6 hours ago, rayon said:
Hey, I found a great performance boost! My system used to halt for some three minutes each time I hit the situation when I started writing into paging. It happened two times per track. I now turned off memory compression and that transition is now super smooth. I basically can't even see when that happens. That alone speeds each track by 5-6 minutes directly. It looks like it's faster other times as well.

As PGGB makes memory to go through limits constantly, Windows ends up in situation in which it needs to start decompressing everything, which means more cpu cycles and also increasing RAM requirements (as it needs room to which it can decompress). Then it starts doing paging and all kinds of stuff to organize all that. When it's turned off, it will just page things that need paging instead of going constantly through the loop of compression, uncompression and paging. Memory compression is designed to avoid paging, but if we are going to need paging anyway, it's just making things heavier. I also noticed that the compressed amount was measured in two gigabytes max during processing so gained potential benefits were very questional.

How you can do this:

Open PowerShell as administrator and enter command:
Disable-MMAgent -mc
You can check the status by running:
Get-MMAgent
Note that you need to reboot to make this change effetive.

Awesome! Will try it.

Mista Lova Lova · April 30

8 hours ago, rayon said:

Hey, I found a great performance boost! My system used to halt for some three minutes each time I hit the situation when I started writing into paging. It happened two times per track. I now turned off memory compression and that transition is now super smooth. I basically can't even see when that happens. That alone speeds each track by 5-6 minutes directly. It looks like it's faster other times as well.

As PGGB makes memory to go through limits constantly, Windows ends up in situation in which it needs to start decompressing everything, which means more cpu cycles and also increasing RAM requirements (as it needs room to which it can decompress). Then it starts doing paging and all kinds of stuff to organize all that. When it's turned off, it will just page things that need paging instead of going constantly through the loop of compression, uncompression and paging. Memory compression is designed to avoid paging, but if we are going to need paging anyway, it's just making things heavier. I also noticed that the compressed amount was measured in two gigabytes max during processing so gained potential benefits were very questional.

You may have just solved one of my biggest issues where my PC goes into "hmm, let's take a break and figure out what's next" whenever I run out of RAM, with the OS drive being involved during those few minutes even if I disable the paging file on that drive completely and only use my other drive. I've been wondering what on Earth Windows is doing during that time.

I'll give this solution a try and report back!

Zaphod Beeblebrox · April 30

4 minutes ago, Mista Lova Lova said:

You may have just solved one of my biggest issues where my PC goes into "hmm, let's take a break and figure out what's next" whenever I run out of RAM, with the OS drive being involved during those few minutes even if I disable the paging file on that drive completely and only use my other drive. I've been wondering what on Earth Windows is doing during that time.

I'll give this solution a try and report back!

Yes, I did not have any explanation for the OS drive being involved, but this would be a very good explanation indeed.

Mista Lova Lova · April 30

First experience with the "Experimental" feature of the 6.1.44 version. Same track being processed twice.

With the box ticked - 32 minutes.

With the box unticked - 33 minutes.

In both cases PGGB behaved differently than before. With the box ticked - CPU usage kept fluctuating between the two extremes (100% and approx. 10%) all the way through. With the box unticked - it remained around 40-50% all the way through.

Previously I would get 100% initially which would steadily, almost linearly, drop to around 60-70%. It was probably to do with my NVMEs being utilised more and more and eventually not being able to keep up. With the few tracks that didn't exceed my available RAM, I was able to maintain a steady 100%. The new version seems to behave differently even with the box unticked (although I've only processed 1 track so far).

Zaphod Beeblebrox · April 30

So far, we have got some useful tips:

On PC:

Run in Admin mode.
Disable memory compression.
Use more than one NVME drive and make sure they have good, sustained write speeds.

On Mac:

Try to drop the worker to match the number of physical cores to check if it helps with speed. We have seen it improve even further by dropping some more workers. So, it is worth experimenting to find the right number for your Mac.

PS: I will add a second cache location for Macs to see if that will help some more as it does on PCs.

Zaphod Beeblebrox · April 30

4 minutes ago, Mista Lova Lova said:

Previously I would get 100% initially which would steadily, almost linearly, drop to around 60-70%. It was probably to do with my NVMEs being utilised more and more and eventually not being able to keep up. With the few tracks that didn't exceed my available RAM, I was able to maintain a steady 100%. The new version seems to behave differently even with the box unticked (although I've only processed 1 track so far).

When you say previous, which version is it, because v6.1.42 is a very different processing load and is memory bound. Though it is overall faster than v6.0.42, NVME can really drag it down. So, it is not surprising if your CPU utilization dropped as it is finishing faster and spending more time waiting for data.

kennyb123 · April 30

41 minutes ago, Zaphod Beeblebrox said:

I will add a second cache location for Macs to see if that will help some more as it does on PCs

That would be interesting. I had not been seeing any benefit from a faster disk but maybe this will help achieve that.

Adyc · May 1

8 hours ago, Zaphod Beeblebrox said:

So far, we have got some useful tips:

On PC:

Run in Admin mode.

Disable memory compression.

Use more than one NVME drive and make sure they have good, sustained write speeds.

On Mac:

Try to drop the worker to match the number of physical cores to check if it helps with speed. We have seen it improve even further by dropping some more workers. So, it is worth experimenting to find the right number for your Mac.

PS: I will add a second cache location for Macs to see if that will help some more as it does on PCs.

How do one disable memory compression?

Zaphod Beeblebrox · May 1

2 minutes ago, Adyc said:

How do one disable memory compression?

Check @rayon's clever idea here:

Mista Lova Lova · May 1

12 hours ago, Zaphod Beeblebrox said:

When you say previous, which version is it, because v6.1.42 is a very different processing load and is memory bound. Though it is overall faster than v6.0.42, NVME can really drag it down. So, it is not surprising if your CPU utilization dropped as it is finishing faster and spending more time waiting for data.

This would have been the 6.1.42 although I'm pretty sure that my CPU's behaviour was similar in all recent versions, except this one containing the experimental feature.

It seems to have done really well overall, considering that the aforementioned track involved a 48kHz to 44kHz rate conversion and exceeded my RAM capacity by a fair bit. Just reporting my additional observations.

jeti · May 1

19 hours ago, Zaphod Beeblebrox said:

So far, we have got some useful tips:

On PC:

Run in Admin mode.

Disable memory compression.

Use more than one NVME drive and make sure they have good, sustained write speeds.

On Mac:

Try to drop the worker to match the number of physical cores to check if it helps with speed. We have seen it improve even further by dropping some more workers. So, it is worth experimenting to find the right number for your Mac.

PS: I will add a second cache location for Macs to see if that will help some more as it does on PCs.

Any suggestion for suitable nvme? I guess that Samsung 990 pro is a good one.

Zaphod Beeblebrox · May 1

7 minutes ago, jeti said:

Any suggestion for suitable nvme? I guess that Samsung 990 pro is a good one.

You can check for the specific drive you are interested in; some information was shared here. Also, no need to go overboard because your motherboard may not be able to match the highest speed the drive supports.

rayon · May 2

@Zaphod Beeblebrox and @austinpop I'm now considering either D7-P8510 or Kioxia FL6 as those drives have insane durability and would be great for long term use. Knowing myself, I may want to reusample things later as progress progresses.

Knowing the algorithm and also the bottlenecks when doing 1024fs x 1 (especially from redbook), which one do you think would serve better for paging purposes?

Kioxia FL6
Sustained 4K Random Read: 1,480,000 IOPS
Sustained 4K Random Write: 360,000 IOPS

Solidigm D7-P5810
4K Random Read (QD256): 865,000 IOPS
4K Random Write (QD256): 495,000 IOPS

Solidigm has better random write IOPS, but Kioxia has even bigger advantage in random read. To me it looked like read became bottleneck much faster. I was also able to get 1gbps+ write speed with one nvme, but read has always been more sluggish.

Are these the correct performance numbers to look at or should I instead look at something else (like sequential write + random read or something like that)? Also how important is the latency? Both provide amazing latency, but Kioxia has an edge there as well.

Also the drive size I'm considering is 800GB and idea would be to fill it with page file. To my understanding these drives shouldn't be that sensitive to how full they are, but it may have been a misunderstanding. Here Solidigm 800GB costs 900€ and Kioxia 1400€. The king, Intel P5800X (or P5810X) costs 1800€, but it of course provides even better numbers. Not sure if it actually is worth the extra costs, especially considering that Solidigm is 50% cheaper.

Also on write: these drives can actually sustain that write speed vs. NVMEs like Pro 990 as they don't have separate write cache.

jeti · May 2

19 hours ago, Zaphod Beeblebrox said:

You can check for the specific drive you are interested in; some information was shared here. Also, no need to go overboard because your motherboard may not be able to match the highest speed the drive supports.

From the graph, would Samsung 970pro perform better for running pggb? This is related to rayon's question. Not knowing SSD drives much, just learned that 970pro is on the more expensive side, but much less than rayon's candidates.

rayon · May 2

9 minutes ago, jeti said:

From the graph, would Samsung 970pro perform better for running pggb? This is related to rayon's question. Not knowing SSD drives much, just learned that 970pro is on the more expensive side, but much less than rayon's candidates.

From normal NVME drives, 990 Pro is probably one of the best. I just pasted that pic as I had 660p. However I also am not sure what are the must important parameters afterall. Sustained write and read speeds definitely are important, but also IOPS, sequential vs. random and latency are important parameters.

seeteeyou · May 2

https://audiophilestyle.com/forums/topic/62699-a-toast-to-pggb-a-heady-brew-of-math-and-magic/page/88/#comment-1278544

22 hours ago, rayon said:

The only downside I see in this is that I've started drooling p5800x after seeing those disk write cycles climbing up rapidly :)

https://audiophilestyle.com/forums/topic/62699-a-toast-to-pggb-a-heady-brew-of-math-and-magic/page/88/#comment-1278617

11 hours ago, Sagittarius said:

There seems to be a consensus emerging that upsampling to DSD offers the highest sound quality from PGGB, which is also highly demanding of processing power and time.

First of all, let's compare the differences between the two

https://audiophilestyle.com/forums/topic/69710-pggb-performance-tuning/page/4/#comment-1278456

On 5/1/2024 at 1:00 AM, Zaphod Beeblebrox said:

When you say previous, which version is it, because v6.1.42 is a very different processing load and is memory bound. Though it is overall faster than v6.0.42, NVME can really drag it down. So, it is not surprising if your CPU utilization dropped as it is finishing faster and spending more time waiting for data.

In that particular case, maybe we could check this out

https://www.reddit.com/r/homelab/comments/lx435w/first_homelab_r730_or_r820/

Quote

I use DDR3-1866 and those are almost the same speed as the DDR4-2133, but half the price. V3 and V4 processors adds AVX2 instructions, but are otherwise pretty identical.

Therefore stuff like this wouldn't be such a bad deal since DRAM should be built to last, especially when the prices of relatively large NVMe drives (impressive endurance = impressive costs) could be somewhat prohibitive

https://forums.servethehome.com/index.php?threads/64gb-lrdimm-ddr3-ecc-ram-for-20-or-less.42253/

https://forums.servethehome.com/index.php?threads/2696-v2-for-16-50-2697-v2-for-25-or-less.42059/page-6#post-400357

Quote

64GB DDR3 LRDIMM for $20 or less: Samsung 64GB (1 X 64GB) PC3L-10600L - 8RX4 - LRDIMM Ram Memory (M386B8G70DE0) | eBay
I made offer of $18 and got accepted.

Imagine having 1TB of ram for the price of around $300 with tax and shipping included!

Similar listings of Samsung M386B8G70DE0

https://www.ebay.com/itm/165000303948

https://www.ebay.com/itm/355198389776

https://www.ebay.com/itm/274895534967

When it comes to the server, typically refurbished / used Dell PowerEdge R620 turned out to cost around 100 bucks or so (depending on the configurations / shipping fees etc.) at the moment

https://www.ebay.com/itm/126052485066

https://www.ebay.com/itm/156172043772

https://www.ebay.com/itm/296375188180

https://www.ebay.com/itm/335351919889

https://www.ebay.com/itm/364790321459

https://www.ebay.com/itm/404440146402

https://www.reddit.com/r/homelab/comments/134bcfq/r620_worth_it/

Quote

R620/T620 is a solid unit. I actually have a few T620's at home that I run as my primarys. They're pretty solid and quiet, not too power hungry (100-250w idle depending on what they're filled with.

24 sticks of 64GB DDR3 LRDIMM would provide 1.5 TB of RAM with dual Xeon E5-2600 v2

https://www.dell.com/community/en/conversations/poweredge-hardware-general/r620-lrdimm-support/647fa31cf4ccf8a8de8dbe07

Updates are still available

https://updateyodell.net

https://forum.allenscloud.com/t/dell-poweredge-r620-firmware/60/3

https://www.reddit.com/r/homelab/comments/kspduv/someone_has_rebuilt_and_rehosted_the_dell/

That came with PCIe 3.0-enabled expansion slots, NVMe drives could still be made bootable just like this

https://www.tachytelic.net/2020/10/dell-poweredge-install-boot-pci-nvme/

https://www.hamishmb.com/booting-nvme-older-pc-refind/

While specific drives would also work without Clover EFI bootloader

https://www.reddit.com/r/homelab/comments/ardw09/comment/egoq2xl/

Quote

I've successuly installed and booted from:

- Intel 750 NVMe

- Intel Optane 900p

on 2 Dell R620's (and on ASUS Z9PE-D8 WS, which is also a C602 chipset)

I've installed Windows 10 pro, Ubuntu 16.04 and Proxmox, all multiple times during testing and setting up final configs.

Further readings

https://h30434.www3.hp.com/t5/Business-PCs-Workstations-and-Point-of-Sale-Systems/The-truth-about-z620-z820-NVMe-SSD-boot/m-p/7852981/highlight/true#M34500

400GB U.2 version could be found easily, used ones should be going for $50 a piece

https://www.ebay.com/itm/276168539497

At some point I might somehow find a way to add .NET 8.0 manually so that the entire Win11PE image with PGGB•IT! could be booted into RAM

http://jplay.eu/forum/index.php?/topic/4410-windows-11-pe-audiophile-creation-guide/?p=64022

Quote

The size of Win11PE bare audio is 78.7 MB.

The only caveat of installing 1.5 TB DDR3 LRDIMM should be reducing the maximum speed to PC3-10600 as shown below, though that's exactly what we're getting from Samsung M386B8G70DE0 mentioned above

https://www.dell.com/support/manuals/en-us/poweredge-r620/r620systemownersmanual-v1/system-memory?guid=guid-c93ad857-06a2-4ef8-88d7-4420451c297a

https://www.reddit.com/r/homelab/comments/8wg7oh/ddr3_memory_performance_for_dell_poweredge_r620/

Quote

If you have all 24 slots populated the fastest they will run is at pc3-10600 speeds so it's not a terrible hit.

Hard to beat the price-to-performance ratio of DDR3 with virtually "unlimited" endurance to boot.

rayon · May 2

8 hours ago, seeteeyou said:

https://audiophilestyle.com/forums/topic/62699-a-toast-to-pggb-a-heady-brew-of-math-and-magic/page/88/#comment-1278544

https://audiophilestyle.com/forums/topic/62699-a-toast-to-pggb-a-heady-brew-of-math-and-magic/page/88/#comment-1278617

First of all, let's compare the differences between the two

https://audiophilestyle.com/forums/topic/69710-pggb-performance-tuning/page/4/#comment-1278456

In that particular case, maybe we could check this out

https://www.reddit.com/r/homelab/comments/lx435w/first_homelab_r730_or_r820/

Therefore stuff like this wouldn't be such a bad deal since DRAM should be built to last, especially when the prices of relatively large NVMe drives (impressive endurance = impressive costs) could be somewhat prohibitive

https://forums.servethehome.com/index.php?threads/64gb-lrdimm-ddr3-ecc-ram-for-20-or-less.42253/

https://forums.servethehome.com/index.php?threads/2696-v2-for-16-50-2697-v2-for-25-or-less.42059/page-6#post-400357

Similar listings of Samsung M386B8G70DE0

https://www.ebay.com/itm/165000303948

https://www.ebay.com/itm/355198389776

https://www.ebay.com/itm/274895534967

When it comes to the server, typically refurbished / used Dell PowerEdge R620 turned out to cost around 100 bucks or so (depending on the configurations / shipping fees etc.) at the moment

https://www.ebay.com/itm/126052485066

https://www.ebay.com/itm/156172043772

https://www.ebay.com/itm/296375188180

https://www.ebay.com/itm/335351919889

https://www.ebay.com/itm/364790321459

https://www.ebay.com/itm/404440146402

https://www.reddit.com/r/homelab/comments/134bcfq/r620_worth_it/

24 sticks of 64GB DDR3 LRDIMM would provide 1.5 TB of RAM with dual Xeon E5-2600 v2

https://www.dell.com/community/en/conversations/poweredge-hardware-general/r620-lrdimm-support/647fa31cf4ccf8a8de8dbe07

Updates are still available

https://updateyodell.net

https://forum.allenscloud.com/t/dell-poweredge-r620-firmware/60/3

https://www.reddit.com/r/homelab/comments/kspduv/someone_has_rebuilt_and_rehosted_the_dell/

That came with PCIe 3.0-enabled expansion slots, NVMe drives could still be made bootable just like this

https://www.tachytelic.net/2020/10/dell-poweredge-install-boot-pci-nvme/

https://www.hamishmb.com/booting-nvme-older-pc-refind/

While specific drives would also work without Clover EFI bootloader

https://www.reddit.com/r/homelab/comments/ardw09/comment/egoq2xl/

Further readings

https://h30434.www3.hp.com/t5/Business-PCs-Workstations-and-Point-of-Sale-Systems/The-truth-about-z620-z820-NVMe-SSD-boot/m-p/7852981/highlight/true#M34500

400GB U.2 version could be found easily, used ones should be going for $50 a piece

https://www.ebay.com/itm/276168539497

At some point I might somehow find a way to add .NET 8.0 manually so that the entire Win11PE image with PGGB•IT! could be booted into RAM

http://jplay.eu/forum/index.php?/topic/4410-windows-11-pe-audiophile-creation-guide/?p=64022

The only caveat of installing 1.5 TB DDR3 LRDIMM should be reducing the maximum speed to PC3-10600 as shown below, though that's exactly what we're getting from Samsung M386B8G70DE0 mentioned above

https://www.dell.com/support/manuals/en-us/poweredge-r620/r620systemownersmanual-v1/system-memory?guid=guid-c93ad857-06a2-4ef8-88d7-4420451c297a

https://www.reddit.com/r/homelab/comments/8wg7oh/ddr3_memory_performance_for_dell_poweredge_r620/

Hard to beat the price-to-performance ratio of DDR3 with virtually "unlimited" endurance to boot.

Yes, if I was building the setup from scratch, I think I would consider that route. However, I already have water cooled 13900k based setup that I like. I'm primarily trying to fit in the last pieces. After those performance updates for PGGB, I don't really need more speed. It's already fast enough. Basically these basic NVMe drives will also serve me well for a while. However, this will burn those. Knowing myself, I may later want to even redo at least some part of my library once things progress and new things are found.

The benefit from super high endurance NVMe drives vs. high ram server would be:

1) No need to change the whole setup that I like otherwise (and which is easily reusable for gaming and other purposes at will)

2) I haven't had super low latency drives before - nice new toy and new things to learn

3) Those super low latency drives can be reutilized later easily for all kinds of fun nerdy projects

P.S. As you linked that 400GB U.2 drive, I've also been looking at 800GB P4800X drives. Those pale in comparison to P5800X, but those are so cheap that I could buy two of those (or 2x400GB) and put on a single PCI-E adapter that can carry two and still be at quite reasonable prices. It wouldn't yet approach the performance level of P5800X, but could well be sufficient for this use case. Then I could keep roughly the current upsampling speeds without much hassle and have an absolute peace of mind regarding to drive durability :)

Mista Lova Lova · May 2

@rayon I have now been able to gargle-blast a file with the paging file compression switched off - it works beautifully! Finally, a seamless transition from RAM to NVME which not only saves so much time but also stops my PC from getting "frozen" for a few minutes.

Thanks again for this tip, it's made a huge difference for me! Surely together with the new efficiency improvements in the newest PGGB release. To the point that I'll now do some further testing with different amounts of RAM to check if getting more RAM is definitely worth it (i.e. whether my NVMEs can keep up for the whole duration of the modulation stage).

Do you think that the compression of the paging file would have affected not only data being written onto the NVME (offloaded from RAM) but also being read back from the NVME during modulation (being "uncompressed")?

seeteeyou · May 2

SSDPF21Q400GB01 = Retail

SSDPF21Q400GBT1 = OEM

BTW, I just checked the prices of Optane P5800X here in Hong Kong, 400GB version in the OEM flavor should be HK$3,900 (exactly 500 bucks) right now

https://www.price.com.hk/category.php?c=100055&brand=Intel

https://leshop.hk/product/intel-optane-dc-p5800x-400gb-ssd-2-5″-u-2-pcie-nvme-dell-version-ssdpf21q400gbt1/

DAFS on Google and found that would be either 4,499 or 4,500 RMB (a bit over $600 that is) @ taobao.com for real.

Dell 53M3R could also found on eBay with (somewhat) similar prices

https://www.ebay.com/itm/135022927087

https://www.ebay.com/itm/285817276339

https://www.ebay.com/itm/285826534930

They're indeed providing their own firmware updates

https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=50y36

https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=99wwv

P5801X with E1.S interface

https://www.ebay.com/itm/145694995162

austinpop · May 2

16 hours ago, rayon said:

@Zaphod Beeblebrox and @austinpop I'm now considering either D7-P8510 or Kioxia FL6 as those drives have insane durability and would be great for long term use. Knowing myself, I may want to reusample things later as progress progresses.

Knowing the algorithm and also the bottlenecks when doing 1024fs x 1 (especially from redbook), which one do you think would serve better for paging purposes?

Kioxia FL6
Sustained 4K Random Read: 1,480,000 IOPS
Sustained 4K Random Write: 360,000 IOPS

Solidigm D7-P5810
4K Random Read (QD256): 865,000 IOPS
4K Random Write (QD256): 495,000 IOPS

Solidigm has better random write IOPS, but Kioxia has even bigger advantage in random read. To me it looked like read became bottleneck much faster. I was also able to get 1gbps+ write speed with one nvme, but read has always been more sluggish.

Are these the correct performance numbers to look at or should I instead look at something else (like sequential write + random read or something like that)? Also how important is the latency? Both provide amazing latency, but Kioxia has an edge there as well.

Also the drive size I'm considering is 800GB and idea would be to fill it with page file. To my understanding these drives shouldn't be that sensitive to how full they are, but it may have been a misunderstanding. Here Solidigm 800GB costs 900€ and Kioxia 1400€. The king, Intel P5800X (or P5810X) costs 1800€, but it of course provides even better numbers. Not sure if it actually is worth the extra costs, especially considering that Solidigm is 50% cheaper.

Also on write: these drives can actually sustain that write speed vs. NVMEs like Pro 990 as they don't have separate write cache.

Whoa, you are poised on the precipice of a rabbit hole I have not explored, and truth be told, I do not plan to. That said, if you do, please send regular updates our way! 🙂

To answer your question more seriously: no, I don't know if either of these would make for a better paging drive. To really know this, you would have to first identify the I/O characteristics of PGGB running under load:

Read vs. write IOPs
Average size of IOP
Random vs. sequential behavior
And so on.

Secondly, keep in mind that PGGB is still undergoing performance optimization. You may have noticed that even PGGB DSD went from being a CPU-bound load in 6.0.x to a memory- or disk-bound load in 6.1.x. I would let the dust settle before diving down into any extreme optimizations.

Personally, I am still an advocate of keeping it simple. If you have a Z790 of X570 motherboard for the latest gen Intel or AMD CPUs, you likely already have 3 or more M.2 slots available, and I would just populate these with cheap 500GB or 1TB Gen 4 or Gen 5 (if applicable) of your choice of NVMe drives. This has been paying dividends for me across the big changes in PGGB code.

Your concern about endurance is a valid one. My erstwhile PGGB machine, i7-10700 with 128GB of RAM, has been with me for 3+ years. I have redone my core collection of about 300 albums (that I care enough to PGGB) from my library, at least 4-5 times, so consider that I have done 1500 albums on that machine. Until very recently, this machine had a single Samsung 2TB 970 EVO Gen3 NVMe drive. I just looked at the SMART metrics of that drive with Samsung Magician.

In other words, my drive's useful life is only about half over. BTW - that's just based on the manufacturer's prediction. It could last a lot longer. This tells me 2 things:

Endurance does matter. The I/O generated by PGGB over time does eat into your drive's lifetime
It's not a major concern for machines running PGGB, at least over a machine's usable life span.

All that said, I am half hoping you go down this rabbit hole anyway, as the data should be very useful!

kennyb123 · May 3

59 minutes ago, austinpop said:

Whoa, you are poised on the precipice of a rabbit hole I have not explored, and truth be told, I do not plan to. That said, if you do, please send regular updates our way! 🙂

That made me laugh out loud.

seeteeyou · May 3

11 hours ago, rayon said:

P.S. As you linked that 400GB U.2 drive, I've also been looking at 800GB P4800X drives.

That reminded me this from 2017

https://www.servethehome.com/intel-optane-ssd-dc-p4800x-3d-xpoint-landed/intel-optane-ssd-dc-p4800x-mysql-1-5tb-database-intel-cache-acceleration-software-cas/

CAS is still available for free, though it might be more suitable for accelerating relatively slow storage devices in the first place

https://www.intel.com/content/www/us/en/software/intel-cache-acceleration-software-performance.html

https://registrationcenter-download.intel.com/akdlm/irc_nas/12681/IntelCacheAccelerationSoftware_x64-3.2.2.64_entry.exe

Supposedly it could have the potential to provide more granular control over the caching, though we've gotta figure out which cache mode would work best for PGGB

https://www.intel.com/content/dam/support/us/en/documents/memory-and-storage/ssd-software/CAS_Win_WS_AdminGuide.pdf#page=7

CAS Windows Demo

https://www.youtube.com/watch?v=wHJipCA-9D0

Intel Cache Acceleration Software

https://www.youtube.com/watch?v=zq1-jD2EnZc

Intel Cache Acceleration Software

https://www.youtube.com/watch?v=uMRdzl701t8

Tuesday Tech Tip - 6 OpenCAS Caching Methods

https://www.youtube.com/watch?v=ZxvNvAWX3wI

Tuesday Tech Tip - Intel Optane, Open CAS and Ceph

https://www.youtube.com/watch?v=yfkwI2EhHPk

Windows software caching, an adventure from senselessness up to catastrophe.

https://community.spiceworks.com/t/windows-software-caching-an-adventure-from-senselessness-up-to-catastrophe/781915

Quote

Many others, including myself, have long suspected that the current Windows cache management is responsible for many performance problems in the current Microsoft operating systems. This problem has already been demonstrated to Microsoft in complex processes. However, Microsoft has not yet taken this problem seriously and initially blamed third parties for the drop in performance.

https://community.spiceworks.com/t/windows-software-caching-an-adventure-from-senselessness-up-to-catastrophe/781915/6

Quote

Thus, the situation is now a little clearer. The more RAM you have built into your computer and the smaller files you want to process, the more you get duped by the current Windows cache management.

And I have to listen from Microsoft, no, Alex, that’s not a bug, that’s by design and therefore absolutely correct.

@Microsoft
It’s one of the biggest dung beetles I’ve seen in my 20+ years in IT.

And then there's something else called PrimoCache as well

https://www.romexsoftware.com/en-us/primo-cache/index.html

Not sure if they're actually useful for this particular thread?

seeteeyou · May 3

Recommended AHCI/RAID and NVMe Drivers

https://winraid.level1techs.com/t/recommended-ahci-raid-and-nvme-drivers/28310

Which NVMe Drivers are the best (performance related)?

https://winraid.level1techs.com/t/which-nvme-drivers-are-the-best-performance-related/32785/273

https://www.station-drivers.com/index.php/en/forum/intel-sata-raid-drivers-firmwares-utilities/89-intel-r-nvme-drivers-version-5-3-0-1010#636

GUIDE: NVMe drivers - Which driver for which card?

https://forums.guru3d.com/threads/guide-nvme-drivers-which-driver-for-which-card.422269/

Amazon

https://s3.amazonaws.com/ec2-windows-drivers-downloads/NVMe/Latest/AWSNVMe.zip

Datacenter NVMe* Microsoft Windows* Drivers for Intel® SSDs

https://web.archive.org/web/20230322153431id_/https://sdmsdfwdriver.blob.core.windows.net/files/kba-gcc/drivers-downloads/ka-00095--ddriver/5-3-0-1005/setupnvme.exe

https://web.archive.org/web/20230322153431id_/https://sdmsdfwdriver.blob.core.windows.net/files/kba-gcc/drivers-downloads/ka-00095--ddriver/5-3-0-1005/datacenter-x64.zip

Micron

https://web.archive.org/web/20190130100845id_/https://www.micron.com/-/media/client/global/documents/products/software/nvme/micronnvmepciessddrivers.zip

Microsoft (Intel Corporation - SCSIAdapter - 20.0.0.1038)

https://catalog.s.download.windowsupdate.com/d/msdownload/update/driver/drvs/2024/03/1c24abfd-7ce5-4a75-86f3-1896b6aa4251_d1f044778d33b1668a2ee38c405dd34ad4ec432f.cab

Microsoft (Intel Corporation - SCSIAdapter - 8.5.0.1592)

https://catalog.s.download.windowsupdate.com/c/msdownload/update/driver/drvs/2024/04/5e6bb217-193c-415b-b676-defc1148430c_c553fcb3cb9f6ddbbe8e2170d6cfd15257965ac2.cab

Phison

https://fichiers.touslesdrivers.com/73918/Phison_NVMe_1.5.0.0.zip

https://files1.majorgeeks.com/10afebdbffcd4742c81a3cb0f6ce4092156b4375/drivers/Phison NVMe driver v1.5.0.0.zip

Samsung

https://semiconductor.samsung.com/resources/software-resources/Samsung_NVM_Express_Driver_3.3.exe

update from ms nvme driver to samsung nvme driver

https://winraid.level1techs.com/t/update-from-ms-nvme-driver-to-samsung-nvme-driver/36600/2

BTW, they totally screwed things up since the introduction of Windows 10 Version 1809 / Windows Server 2019

https://community.spiceworks.com/t/windows-server-2019-strange-file-copy/764279/16

Quote

Agree. That is a known issue with Windows Server 2019. There is a workaround that we are practicing to copy large files at a solid, stable speed. The trick is using xcopy or robocopy instead of Windows Explorer with /j switch to bypass disk caching and buffering.

https://community.spiceworks.com/t/windows-server-2019-strange-file-copy/764279/17

Quote

Yes using unbuffered transfers work, it bypasses the FCM (File Cache manager) Which apparently is the issue…using xcopy /j changed the transfer from 2 minutes to 18 sec on a 10 GB file… But most people use Copy/paste from gui and also loading a roaming profile, isn’t capable of using unbuffered (what I’m aware of) So the xcopy /j is a great work-around, but no a solution

https://community.spiceworks.com/t/server-2019-network-performance/724968/647

Quote

Alexander Fuchs The “useless caching” in ram is intentional, has existed since Windows Vista and is called “Ready Boost”. Today it is hardly noticed anymore because it silently loads many OS and app components into RAM on suspicion. Edge, for example, has been loading a large part of itself into RAM at boot for months because many HTML components are constantly used for rendering various windows and apps. This is a kind of “pre fetch”, which improves performance for the vast majority of Windows users. Linux, swiw, also uses almost the entire RAM with cache, unless it is needed for something else. Of course, you as a user are free to adapt this behavior to your needs, as are your customers. In individual cases, the default settings are certainly not optimal - which is why we have experts like you in the Windows “ecosystem” who understand the interrelationships and can apply them for their customers. Please understand that we cannot possibly make an OS that suits all users and scenarios “out of box”. We rely on you for that

https://community.spiceworks.com/t/server-2019-network-performance/724968/648

Quote

According to Passmark, my FSB 2 (W10-1703), equipped with 16 GB DDR3 RAM that runs at 1066 MHz in dual channel, has the following RAM performance.

And here for comparison, the performance values of a high-end server equipped with two Xeon Gold 6254 and Hexachannel DDR4 under Windows Server 2019.

I’m still missing the right words at the moment, so …

Microsoft yanks the Win10 1809 upgrade

https://www.askwoody.com/forums/topic/microsoft-yanks-the-win10-1809-upgrade/

That reminded me the slogan "So easy to LOSE, no wonder it's number one" back in 2001

PGGB Performance Tuning

Recommended Posts

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Create an account or sign in to comment

Create an account

Sign in