High write-rate with Pythons

Message boards : Number crunching : High write-rate with Pythons

To post messages, you must log in.

Previous · 1 · 2

AuthorMessage
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 106702 - Posted: 3 Aug 2022, 18:30:48 UTC - in response to Message 106701.  
Last modified: 3 Aug 2022, 18:34:04 UTC

My computer is not quite 1 year old, the 500GB SSD has 131TB written. I don't know how much of that is R@H VBox jobs but I'm guessing a lot of it.

My Python machine is shut down for an upgrade, but the last time I checked the writes from the operating system were at least 2 TB/day when running 8 to 10 pythons or so.
When I included a write-cache (it was 16 GB with a 4 hour latency), the writes to the SSD itself dropped to a negligible amount of less than about 1 GB/day. So the problem is solved for me.
Even a 4 GB write-cache with a one hour latency would protect the SSD I think.

I managed to kill a 500 GB SSD inadvertently before I learned the correct way to do the write cache as I outlined here:
https://boinc.bakerlab.org/rosetta/forum_thread.php?id=14871&postid=106343#106343
I was doing the pythons for less than a year; maybe six or eight months.

I don't see the point of a second SSD. It would just be killed instead, unless that is OK for you.

I can't see your computers. If you are on Windows, there is PrimoCache.
https://www.romexsoftware.com/en-us/primo-cache/index.html
It is probably no more than the cost of an SSD, and less bother to fix.
ID: 106702 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Paddles

Send message
Joined: 15 Mar 15
Posts: 11
Credit: 5,356,953
RAC: 2,488
Message 106710 - Posted: 4 Aug 2022, 12:07:44 UTC - in response to Message 106702.  

When I included a write-cache (it was 16 GB with a 4 hour latency), the writes to the SSD itself dropped to a negligible amount of less than about 1 GB/day. So the problem is solved for me.
Even a 4 GB write-cache with a one hour latency would protect the SSD I think.

I managed to kill a 500 GB SSD inadvertently before I learned the correct way to do the write cache as I outlined here...

I don't see the point of a second SSD. It would just be killed instead, unless that is OK for you.

I can't see your computers. If you are on Windows, there is PrimoCache.


I am running a Windows machine - and given that BOINC data is on the main Windows system disk/volume (OS and applications; most of my non-BOINC data is on a separate SSD) I'd rather lose a sacrificial disk that's only used for BOINC than the disk that keeps everything going. Adding caching is an interesting idea, but not sure how well it would work on the main system disk.
ID: 106710 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 106713 - Posted: 4 Aug 2022, 13:54:25 UTC - in response to Message 106710.  
Last modified: 4 Aug 2022, 14:03:45 UTC

Adding caching is an interesting idea, but not sure how well it would work on the main system disk.

I have a dedicated Win10 machine (Ryzen 3600) where I run 8 pythons at a time, and use PrimoCache on it.

It runs great; better than Linux, since you don't get the "Vm job unmanageable" ones on Windows, as least if you use VirtualBox 5.2.44 (not 6.x).
However, the problem is that Windows reserves all the memory BOINC allocates for each work unit, even though the pythons do not use it all.
And it allocates all the memory that you set for the cache in addition. Therefore, I have to use 128 GB of memory to run only 8 pythons.
I could use half that memory (64 GB) in Ubuntu and it would work OK with even more pythons; at least 12 at a time is what I usually run on a Ryzen 3900X or 3950X.

On the other hand, I am also running PrimoCache on my main working machine, a Ryzen 5700X running Windows 10.
It is also working great, but I am not running the pythons on it, only other BOINC projects. It has 32 GB of memory, and I am allocating 8 GB (4 hours) for the cache.

But just because it is a system disk won't be a problem when using PrimoCache; it is only a question of whether you have enough memory.
If you run only 2 pythons, that would make the whole process easier, and reduce the write-rate to something more reasonable. I just like to devote an entire machine to them.
ID: 106713 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1994
Credit: 9,611,047
RAC: 9,157
Message 106715 - Posted: 5 Aug 2022, 16:00:17 UTC - in response to Message 106713.  
Last modified: 5 Aug 2022, 16:00:29 UTC

I have a dedicated Win10 machine (Ryzen 3600) where I run 8 pythons at a time, and use PrimoCache on it.
It runs great; better than Linux, since you don't get the "Vm job unmanageable" ones on Windows, as least if you use VirtualBox 5.2.44 (not 6.x).


Thank you for tests and informations.
But i prefer an officially maintained/optimized/debugged app than 3rd party sw
And that's up to the developer team.
ID: 106715 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 106716 - Posted: 5 Aug 2022, 16:20:35 UTC - in response to Message 106715.  
Last modified: 5 Aug 2022, 16:22:52 UTC

But i prefer an officially maintained/optimized/debugged app than 3rd party sw
And that's up to the developer team.

I think what we have is the best they can do. We have to fix up their goofs from there.
But I quite agree that it is their job, should they wish to do it.
ID: 106716 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Paddles

Send message
Joined: 15 Mar 15
Posts: 11
Credit: 5,356,953
RAC: 2,488
Message 106866 - Posted: 5 Sep 2022, 4:43:28 UTC - in response to Message 106710.  
Last modified: 5 Sep 2022, 4:47:09 UTC

When I included a write-cache (it was 16 GB with a 4 hour latency), the writes to the SSD itself dropped to a negligible amount of less than about 1 GB/day. So the problem is solved for me.
Even a 4 GB write-cache with a one hour latency would protect the SSD I think.

I managed to kill a 500 GB SSD inadvertently before I learned the correct way to do the write cache as I outlined here...

I don't see the point of a second SSD. It would just be killed instead, unless that is OK for you.

I can't see your computers. If you are on Windows, there is PrimoCache.


I am running a Windows machine - and given that BOINC data is on the main Windows system disk/volume (OS and applications; most of my non-BOINC data is on a separate SSD) I'd rather lose a sacrificial disk that's only used for BOINC than the disk that keeps everything going. Adding caching is an interesting idea, but not sure how well it would work on the main system disk.


To follow up... I did eventually install another SSD to serve solely for BOINC (might end up doubling as scratch space for anything else that needs it), on the basis that I'd rather lose a secondary SSD than the one that holds Windows and all my software. I've also installed PrimoCache with caching just for that drive (16GB out of 32GB, which should be enough to handle my configured maximum of 4 VBOX jobs, and defer-write set to Infinite).

So far the caching seems to be reducing the amount of data actually written to the SSD to about 40% of what it would otherwise be, and the amount of data written to my primary drive has reduced significantly. It looks like some BOINC jobs complete and and get cleaned up using data in cache, resulting in little or no data being written to disk.

If/when the BOINC/scratch disk dies, I'll just replace it with another low-end SSD, but with PrimoCache as part of the mix this could be 2-3 years away - and who knows what R@H will be doing then?

So thanks to jim and others for suggestions/guidance.
ID: 106866 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 106867 - Posted: 5 Sep 2022, 5:03:57 UTC - in response to Message 106866.  

I think your scratch disk will last forever. The writes are serialized by PrimoCache, and much easier on the SSD than random writes.
And 16 GB is a pretty good size.

You are welcome.
ID: 106867 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Previous · 1 · 2

Message boards : Number crunching : High write-rate with Pythons



©2024 University of Washington
https://www.bakerlab.org