197 (0x000000C5) EXIT_TIME_LIMIT_EXCEEDED

Message boards : Number crunching : 197 (0x000000C5) EXIT_TIME_LIMIT_EXCEEDED
Message board moderation

To post messages, you must log in.

AuthorMessage
Yavanius
Avatar

Send message
Joined: 1 Jul 20
Posts: 30
Credit: 69,857
RAC: 0
Message 352 - Posted: 23 Jul 2020, 17:34:32 UTC

Looking through WU history and I got 3 with:

197 (0x000000C5) EXIT_TIME_LIMIT_EXCEEDED


Tasks:
2755896
2751858
2752437
ID: 352 · Report as offensive     Reply Quote
Sergey Kovalchuk

Send message
Joined: 24 Jun 20
Posts: 26
Credit: 1,106,925
RAC: 0
Message 381 - Posted: 26 Jul 2020, 18:59:59 UTC

task kaktwoos_2.04_0148_213454727715161_0
exceeded elapsed time limit 2131.29 (60000000.00G/28151.98G)
GPU Tesla P100 is too weak for this task :-(
ID: 381 · Report as offensive     Reply Quote
Profile Hy
Project administrator
Project developer
Avatar

Send message
Joined: 15 Jun 20
Posts: 74
Credit: 19,537,761
RAC: 0
Message 383 - Posted: 27 Jul 2020, 18:46:21 UTC - in response to Message 381.  
Last modified: 28 Jul 2020, 3:15:46 UTC

A P100 is very decent, and as fast as Vega 56 / 64s for seed mining. I can't tell you why BOINC thinks it should have a timeout, but it's likely not for the reason you expect it to be

I've seen if kaktwoos hangs or doesn't provide updates to BOINC, then this timeout kill will be more likely to happen
ID: 383 · Report as offensive     Reply Quote
Sergey Kovalchuk

Send message
Joined: 24 Jun 20
Posts: 26
Credit: 1,106,925
RAC: 0
Message 384 - Posted: 27 Jul 2020, 20:23:12 UTC - in response to Message 383.  

exceeded elapsed time limit 2131.29 (60000000.00G/28151.98G)

the reason for this is in the number - 28151.98G, each host has its own
task was completed by the hosts with the GPU GTX 1650 and GTX 970
ID: 384 · Report as offensive     Reply Quote
Sergey Kovalchuk

Send message
Joined: 24 Jun 20
Posts: 26
Credit: 1,106,925
RAC: 0
Message 386 - Posted: 28 Jul 2020, 3:51:05 UTC - in response to Message 383.  
Last modified: 28 Jul 2020, 3:52:40 UTC

tested on another host.
at the start there was a performance value

<app_version>
    <app_name>kaktwoos</app_name>
    <flops>14600245329561.041016</flops>

after receiving WU, the value almost doubled

<app_version>
    <app_name>kaktwoos</app_name>
    <flops>23668459158711.167969</flops>

the new value is received from server along with WU (sched_reply.xml)
you greatly overestimate the application performance for some GPUs.
or underestimated the overall difficulty of the task (<rsc_fpops_bound>)

PS. with 10k active tasks, the number of RTS decreases too slowly. probably a lot of resend tasks
ID: 386 · Report as offensive     Reply Quote
Yavanius
Avatar

Send message
Joined: 1 Jul 20
Posts: 30
Credit: 69,857
RAC: 0
Message 392 - Posted: 29 Jul 2020, 3:23:33 UTC

My last 3 all timed out toward the end (I'm not sure how close to the end but definitely last 80-90% completion).

Don't mind helping out, but I could have used those cycles to another project and gotten some points for them too.
ID: 392 · Report as offensive     Reply Quote
Profile chip
Project administrator

Send message
Joined: 14 Jun 20
Posts: 78
Credit: 1,321,619
RAC: 0
Message 411 - Posted: 2 Aug 2020, 17:46:28 UTC - in response to Message 392.  

Timed out tasks still give us the results up to the point they time out. Usually, timeouts are because the computer took an unreasonably long time to complete a task.
I'm planning on upping the limits even further, but they're already pretty enormous.
ID: 411 · Report as offensive     Reply Quote
zombie67 [MM]
Avatar

Send message
Joined: 24 Jun 20
Posts: 21
Credit: 181,142,041
RAC: 0
Message 416 - Posted: 3 Aug 2020, 1:29:20 UTC

I am getting this occasionally:

exceeded elapsed time limit 662.68 (60000000.00G/90272.25G)</message>


Is the limit 662 seconds? That is way too short.

Also, some of use run multiple tasks at a time, in order to maximize GPU utilization. So individual tasks can take 2-3 times as long to run each, but the overall throughput is higher.

I recommend setting the timeout to be something like 4x.
Reno, NV
Team: SETI.USA
ID: 416 · Report as offensive     Reply Quote
Yavanius
Avatar

Send message
Joined: 1 Jul 20
Posts: 30
Credit: 69,857
RAC: 0
Message 417 - Posted: 3 Aug 2020, 5:54:22 UTC - in response to Message 411.  
Last modified: 3 Aug 2020, 6:13:27 UTC

Timed out tasks still give us the results up to the point they time out. Usually, timeouts are because the computer took an unreasonably long time to complete a task.
I'm planning on upping the limits even further, but they're already pretty enormous.


That's rather vague... 5 hours isn't exactly pretty enormous.

Yet here, it's double one of those 3 WU roughly: https://minecraftathome.com/minecrafthome/result.php?resultid=2729912 and it didn't timeout.

I'll try another run and see how it turns out. If it bugs out, then I'll wait till you release some new apps. WCG MCM is studying Sarcoma which is a very personal item for me and I don't want to be using cycles that are just gonna end in error.
ID: 417 · Report as offensive     Reply Quote
Yavanius
Avatar

Send message
Joined: 1 Jul 20
Posts: 30
Credit: 69,857
RAC: 0
Message 424 - Posted: 4 Aug 2020, 4:36:36 UTC - in response to Message 417.  

First of two. Crash and burn.

Second of two. Okay, less than hour left and POOF. There's 10 hours of computing gone.

Well, it was fun initially...
ID: 424 · Report as offensive     Reply Quote

Message boards : Number crunching : 197 (0x000000C5) EXIT_TIME_LIMIT_EXCEEDED