Is it possible to turn on the setting "Max # of simultaneous tasks" ?

Message boards : Number crunching : Is it possible to turn on the setting "Max # of simultaneous tasks" ?
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile rilian
Avatar

Send message
Joined: 20 Mar 24
Posts: 11
Credit: 103,351
RAC: 0
Message 2238 - Posted: 17 Apr 2024, 15:46:09 UTC

Hi, because currently Sidock only gives long tasks that take 1.5-3 days on average computer, and Sidock deadlines are very short, when computer receives multiple tasks this blocks computation of other projects

Many other projects allow to set "Max # of simultaneous tasks" which allows to pick projects more easily for crunchers

Could you please enable this setting ?

Thank you!
I crunch for Ukraine
ID: 2238 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
pututu

Send message
Joined: 24 Oct 20
Posts: 12
Credit: 194,780,236
RAC: 14,732
Message 2239 - Posted: 18 Apr 2024, 21:12:01 UTC - in response to Message 2238.  

Set "Store at least ___ days of work" to 0.00 or 0.01 in your "computing preferences..." under "Options" from the boinc manager window. Also make sure that "Store up to an additional ___ days of works" is set to 0.

This will ensure that if you are running 100% of your cpus (say 8 threads as in the screen shot below), it will only download 8 tasks assuming you only have one active project running from your boinc manager.

ID: 2239 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile rilian
Avatar

Send message
Joined: 20 Mar 24
Posts: 11
Credit: 103,351
RAC: 0
Message 2240 - Posted: 18 Apr 2024, 21:23:40 UTC - in response to Message 2239.  
Last modified: 18 Apr 2024, 21:25:07 UTC

Thank you for reply @pututu. Let me clarify

Besides Sidock, I crunch multiple projects (wcg, rosetta etc) which may have tasks outage periods, that is why i keep setting to cache work for ~1 day

Computer has 8 CPUs and about 40-60 various tasks in the queue, and usually with these projects, the task deadlines are quite short, for example 3 days with Rosetta. Sidock has about 6 days to deadline

Other projects crunch a task in about 1-3 hours, while Sidock crunches in about 2 days

Computer is 80% turned on.

So, when Boinc client requests Sidock, computer usually gets 8 tasks and this immediately puts all tasks at risk cause instead of 1 day cache it now has ~3 days cache and this causes Boinc to juggle the tasks

It would be more convenient to set a limit to avoid such issue
I crunch for Ukraine
ID: 2240 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Bryn Mawr

Send message
Joined: 16 Aug 21
Posts: 40
Credit: 17,532,431
RAC: 28,953
Message 2241 - Posted: 18 Apr 2024, 23:48:00 UTC - in response to Message 2240.  

Thank you for reply @pututu. Let me clarify

Besides Sidock, I crunch multiple projects (wcg, rosetta etc) which may have tasks outage periods, that is why i keep setting to cache work for ~1 day

Computer has 8 CPUs and about 40-60 various tasks in the queue, and usually with these projects, the task deadlines are quite short, for example 3 days with Rosetta. Sidock has about 6 days to deadline

Other projects crunch a task in about 1-3 hours, while Sidock crunches in about 2 days

Computer is 80% turned on.

So, when Boinc client requests Sidock, computer usually gets 8 tasks and this immediately puts all tasks at risk cause instead of 1 day cache it now has ~3 days cache and this causes Boinc to juggle the tasks

It would be more convenient to set a limit to avoid such issue


Set up an appconfig.xml file in the SiDock project directory with a max_concurrent command :-

https://boinc.berkeley.edu/trac/wiki/ClientAppConfig
ID: 2241 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
pututu

Send message
Joined: 24 Oct 20
Posts: 12
Credit: 194,780,236
RAC: 14,732
Message 2242 - Posted: 18 Apr 2024, 23:55:28 UTC - in response to Message 2240.  

If you have multiple active projects, one way to control the number of tasks per project is to setup multiple boinc clients (aka boinc instances). See the guide here on how to set it up: https://www.overclock.net/threads/guide-setting-up-multiple-boinc-instances.1628924/. For each boinc clients or boinc instances, you only run one active projects. For each clients or instances, you can set up your own "Computing Preferences...".

The screenshot below shows that I'm running two boinc instances. One is dedicated for Denis@home and the other is Latinsquares. Both running on a single computure. You have more control over each project like how many days of work you want to fill your work cache and how much cores/threads you want to run for each project. Note that for Denis I'm running 3 tasks and for Latinsquares, I'm running 4 tasks. So there are a total of 7 tasks running on that single computer. Of course optimally, you don't want to run more tasks than your total available cpu cores/threads. If you have 8 threads, then best to run a total of 8 tasks or less in total on two separate boinc instances as shown.

If Denis runs out of work for example, then instead of letting that Denis boinc instance stays idle, you can add Latinsquares as a backup project but you need to set the Latinsquare resource share from 100% to 0% in Latinsquare account preferences (you need to log into Latinsquare website to do this). Since Latinsquare has work available almost 24/7, setting resource share to 0 means there will be 0 day of work cache stored in both boinc instances but that's ok.

ID: 2242 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile rilian
Avatar

Send message
Joined: 20 Mar 24
Posts: 11
Credit: 103,351
RAC: 0
Message 2243 - Posted: 19 Apr 2024, 0:02:16 UTC - in response to Message 2241.  

Set up an appconfig.xml file in the SiDock project directory with a max_concurrent command :-

https://boinc.berkeley.edu/trac/wiki/ClientAppConfig

Thank you @Bryn, this looks like easiest solution, will try

Thanks @pututu
I crunch for Ukraine
ID: 2243 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
pututu

Send message
Joined: 24 Oct 20
Posts: 12
Credit: 194,780,236
RAC: 14,732
Message 2244 - Posted: 19 Apr 2024, 0:27:07 UTC - in response to Message 2243.  
Last modified: 19 Apr 2024, 0:29:24 UTC

Note that the <max_concurrent> tag in the app_config.xml file sets the maximum number of tasks of a specific application to run at a given time. It does not control the work cache size, meaning if you set the max_concurrent to "2" say, it will download tasks to fill whatever cache size and the cpu thread to use that you specified. It will only run 2 tasks irrespective of your cache size. However there is a bug in the boinc manager (not sure what version that is) that could potentially load more tasks that you need and override the cache size. Just fyi.
ID: 2244 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile rilian
Avatar

Send message
Joined: 20 Mar 24
Posts: 11
Credit: 103,351
RAC: 0
Message 2245 - Posted: 19 Apr 2024, 0:46:58 UTC - in response to Message 2244.  

ouch… then it would be great if admin set the setting or make tasks smaller..
I crunch for Ukraine
ID: 2245 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
pututu

Send message
Joined: 24 Oct 20
Posts: 12
Credit: 194,780,236
RAC: 14,732
Message 2246 - Posted: 19 Apr 2024, 1:02:53 UTC - in response to Message 2245.  

Agree. That would be the easiest approach. Not sure if this will be the project admin top priority or working on the research project. Most of us here look for alternative solution such as running multiple boinc clients. BTW, this is commonly used during boinc competition.
ID: 2246 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile rilian
Avatar

Send message
Joined: 20 Mar 24
Posts: 11
Credit: 103,351
RAC: 0
Message 2247 - Posted: 19 Apr 2024, 1:06:10 UTC - in response to Message 2246.  

Agree. That would be the easiest approach. Not sure if this will be the project admin top priority or working on the research project. Most of us here look for alternative solution such as running multiple boinc clients. BTW, this is commonly used during boinc competition.

Thank you

Yes I realize this technique could be helpful for bunkering WUs :)

I left a message to project admin, will see what they reply...
I crunch for Ukraine
ID: 2247 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Bryn Mawr

Send message
Joined: 16 Aug 21
Posts: 40
Credit: 17,532,431
RAC: 28,953
Message 2248 - Posted: 19 Apr 2024, 9:05:33 UTC - in response to Message 2244.  
Last modified: 19 Apr 2024, 9:07:33 UTC

Note that the <max_concurrent> tag in the app_config.xml file sets the maximum number of tasks of a specific application to run at a given time. It does not control the work cache size, meaning if you set the max_concurrent to "2" say, it will download tasks to fill whatever cache size and the cpu thread to use that you specified. It will only run 2 tasks irrespective of your cache size. However there is a bug in the boinc manager (not sure what version that is) that could potentially load more tasks that you need and override the cache size. Just fyi.


The bug has been fixed in the latest versions of the Boinc Client. Even before that, I have been running max_concurrent with a small cache size for several years without a problem - each project (including SiDock and CPDN) working according to their project weighting. (If anything it’s WCG that overfills the cache with respect to its project weighting).
ID: 2248 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile rilian
Avatar

Send message
Joined: 20 Mar 24
Posts: 11
Credit: 103,351
RAC: 0
Message 2249 - Posted: 19 Apr 2024, 13:43:56 UTC - in response to Message 2248.  

Note that the <max_concurrent> tag in the app_config.xml file sets the maximum number of tasks of a specific application to run at a given time. It does not control the work cache size, meaning if you set the max_concurrent to "2" say, it will download tasks to fill whatever cache size and the cpu thread to use that you specified. It will only run 2 tasks irrespective of your cache size. However there is a bug in the boinc manager (not sure what version that is) that could potentially load more tasks that you need and override the cache size. Just fyi.


The bug has been fixed in the latest versions of the Boinc Client. Even before that, I have been running max_concurrent with a small cache size for several years without a problem - each project (including SiDock and CPDN) working according to their project weighting. (If anything it’s WCG that overfills the cache with respect to its project weighting).

Hi Bryn

Yeah i use BOINC 8.0.0 already, so you say that if I set <max_concurrent>1</max_concurrent> will only request 1 task from server ?

I only will be able to try this in few days or so cause I need to crunch multiple SiDock tasks that i already got ..

Thank you
I crunch for Ukraine
ID: 2249 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
pututu

Send message
Joined: 24 Oct 20
Posts: 12
Credit: 194,780,236
RAC: 14,732
Message 2250 - Posted: 19 Apr 2024, 14:12:12 UTC - in response to Message 2248.  
Last modified: 19 Apr 2024, 14:27:32 UTC

Note that the <max_concurrent> tag in the app_config.xml file sets the maximum number of tasks of a specific application to run at a given time. It does not control the work cache size, meaning if you set the max_concurrent to "2" say, it will download tasks to fill whatever cache size and the cpu thread to use that you specified. It will only run 2 tasks irrespective of your cache size. However there is a bug in the boinc manager (not sure what version that is) that could potentially load more tasks that you need and override the cache size. Just fyi.


The bug has been fixed in the latest versions of the Boinc Client. Even before that, I have been running max_concurrent with a small cache size for several years without a problem - each project (including SiDock and CPDN) working according to their project weighting. (If anything it’s WCG that overfills the cache with respect to its project weighting).


Last year I had PCs running older boinc version and they still have the bug, similar to the discussion here with version 7.16: https://github.com/BOINC/boinc/issues/4322.
Have not tested latest version 8.0. Edit: looks like they have fix the bug according to the github post above.

Anyway one of OP main concerns is to limit the number of downloaded tasks other than running a fix number of tasks per project.
ID: 2250 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Bryn Mawr

Send message
Joined: 16 Aug 21
Posts: 40
Credit: 17,532,431
RAC: 28,953
Message 2251 - Posted: 19 Apr 2024, 15:58:39 UTC - in response to Message 2249.  

Note that the <max_concurrent> tag in the app_config.xml file sets the maximum number of tasks of a specific application to run at a given time. It does not control the work cache size, meaning if you set the max_concurrent to "2" say, it will download tasks to fill whatever cache size and the cpu thread to use that you specified. It will only run 2 tasks irrespective of your cache size. However there is a bug in the boinc manager (not sure what version that is) that could potentially load more tasks that you need and override the cache size. Just fyi.


The bug has been fixed in the latest versions of the Boinc Client. Even before that, I have been running max_concurrent with a small cache size for several years without a problem - each project (including SiDock and CPDN) working according to their project weighting. (If anything it’s WCG that overfills the cache with respect to its project weighting).

Hi Bryn

Yeah i use BOINC 8.0.0 already, so you say that if I set <max_concurrent>1</max_concurrent> will only request 1 task from server ?

I only will be able to try this in few days or so cause I need to crunch multiple SiDock tasks that i already got ..

Thank you


As pututu has said, it will limit the client to running 1 task at a time rather than requesting one task at a time but because the scheduler then knows that it only has one core to play with it will restrict the number of tasks requested as well.

Indeed, with a small cache, what tends to happen is that it will only request a new task shortly before the existing task completes.

I’m running 5 projects (of which 3 are currently supplying work) and it works well for me.
ID: 2251 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile rilian
Avatar

Send message
Joined: 20 Mar 24
Posts: 11
Credit: 103,351
RAC: 0
Message 2252 - Posted: 19 Apr 2024, 16:11:54 UTC - in response to Message 2251.  

I have added app_config.xml to sidock project folder

<app_config>
   <app>
      <name>cmdock-l</name>
      <max_concurrent>1</max_concurrent>
   </app>
</app_config>


Will report later if this helped ..
I crunch for Ukraine
ID: 2252 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile rilian
Avatar

Send message
Joined: 20 Mar 24
Posts: 11
Credit: 103,351
RAC: 0
Message 2253 - Posted: 21 Apr 2024, 12:35:44 UTC
Last modified: 21 Apr 2024, 12:45:49 UTC

with the setting above, my 8-core computer just got 28 long sidock tasks while there was a limit to cache 0.8 days :(

i'm going to set this project no more work until admin would implement a max jobs limit setting to avoid losing the work in other projects

UPDATE: i realized that this was a linuz computer without boinc 8.0.0 . For now i set "nomorework" on it for Sidock, and will experiment later

The other computer with boinc 8.0.0 still crunches through 2 sidock tasks and i set the app_config setting to keep it 2. I will report once it will finish crunching and get new work
I crunch for Ukraine
ID: 2253 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
pututu

Send message
Joined: 24 Oct 20
Posts: 12
Credit: 194,780,236
RAC: 14,732
Message 2254 - Posted: 21 Apr 2024, 13:58:58 UTC - in response to Message 2253.  

with the setting above, my 8-core computer just got 28 long sidock tasks while there was a limit to cache 0.8 days :(

i'm going to set this project no more work until admin would implement a max jobs limit setting to avoid losing the work in other projects

UPDATE: i realized that this was a linuz computer without boinc 8.0.0 . For now i set "nomorework" on it for Sidock, and will experiment later

The other computer with boinc 8.0.0 still crunches through 2 sidock tasks and i set the app_config setting to keep it 2. I will report once it will finish crunching and get new work


What boinc version did you install for the PC with more work cache that you specified? I would suggest letting the PC with version 8.0.0 runs for a while to see if the prior bug is fully resolved. I'm still using a range of boinc 7.x versions but I use multiple boinc instances to control work cache if I decide to run more than one boinc project per PC like you are currently doing.
ID: 2254 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
hoarfrost
Volunteer moderator
Project administrator
Project developer

Send message
Joined: 11 Oct 20
Posts: 337
Credit: 25,677,525
RAC: 9,176
Message 2255 - Posted: 21 Apr 2024, 21:20:50 UTC
Last modified: 22 Apr 2024, 5:09:26 UTC

Hello! :)
Presumably, BOINC should balance everything after gathering statistics for tasks calculations. Give it one or two weeks of work.
ID: 2255 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile rilian
Avatar

Send message
Joined: 20 Mar 24
Posts: 11
Credit: 103,351
RAC: 0
Message 2256 - Posted: 22 Apr 2024, 15:07:27 UTC - in response to Message 2255.  
Last modified: 22 Apr 2024, 15:09:18 UTC

Hello! :)
Presumably, BOINC should balance everything after gathering statistics for tasks calculations. Give it one or two weeks of work.

hi @hoarfrost

In theory yes, but this incident happened on "old" computer that crunched Sidock for more than a month. I had to manually set nomorework, allowmorework all the time. All projects on that computer have the same resource share. There is no way that computer could crunch 28 tasks with 8 cores in time..

This bug probably comes from server info which thinks that task average time is 31 hour (from server status page):

CurieMarieDock 0.2.0 long tasks 	14282 	24846 	[b]31.49[/b] (0.62 - 320.84)


-

Anyways, would it be possible to enable the setting to limit max number of jobs on server side?

As I explained, because Sidock has very long tasks, this puts all other projects at risk of missing deadlines and missing credits

Thank you!
I crunch for Ukraine
ID: 2256 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
keputnam
Avatar

Send message
Joined: 12 Mar 23
Posts: 5
Credit: 3,259,016
RAC: 5,659
Message 2268 - Posted: 17 Jun 2024, 23:56:21 UTC - in response to Message 2252.  

This works for me



<app_config>
<project_max_concurrent>3</project_max_concurrent>
</app_config>
ID: 2268 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
1 · 2 · Next

Message boards : Number crunching : Is it possible to turn on the setting "Max # of simultaneous tasks" ?

©2024 SiDock@home Team