Message boards :
Number crunching :
Pausing tasks causes tast restart
Message board moderation
Author | Message |
---|---|
Send message Joined: 27 Jun 20 Posts: 9 Credit: 788,006 RAC: 0 |
So far, I processed only one task: When initially started yesterday on my NVIDIA GTX1060, I paused computation after about 10 min of computation. When I returned to my machine, run time was set back to zero and upon resuming, the GPU task started from scratch. At least it completed successfully after around 4 hrs and was validated. I have never seen such behaviour before (I am participating 24/7 in distributed computing projects since 2001). Michael. President of Rechenkraft.net - This world's first and largest distributed computing organization. We make those things possible that supercomputers don't. |
Send message Joined: 14 Jun 20 Posts: 78 Credit: 1,321,619 RAC: 0 |
So far, I processed only one task: When initially started yesterday on my NVIDIA GTX1060, I paused computation after about 10 min of computation. When I returned to my machine, run time was set back to zero and upon resuming, the GPU task started from scratch. Hey, checkpointing is available in a development branch, it’ll be pushed shortly to replace the current app version. By the end of the day hopefully. Long story short, we thought it was already sorted, but alas here we are. |
Send message Joined: 24 Jun 20 Posts: 85 Credit: 207,156 RAC: 0 |
Not so sure I would call this checkpointing... 28/06/2020 16:52:47 | minecrafthome | Starting task kaktwoos_1.0.7_0218_268465497051539_0 28/06/2020 16:52:47 | minecrafthome | [cpu_sched] Starting task kaktwoos_1.0.7_0218_268465497051539_0 using kaktwoos version 112 (opencl_amd) in slot 0 28/06/2020 17:32:31 | minecrafthome | [checkpoint] result kaktwoos_1.0.7_0218_268465497051539_0 checkpointed 28/06/2020 17:32:34 | minecrafthome | Computation for task kaktwoos_1.0.7_0218_268465497051539_0 finished That's one checkpoint just seconds before the task stops. Remaining estimate is now 2 hours on entry, Progress gets up to 22% before it's now finished. (My preferences are to checkpoint at most every 65 seconds) |
Send message Joined: 15 Jun 20 Posts: 74 Credit: 19,537,761 RAC: 0 |
See this post for an update on this issue https://minecraftathome.com/minecrafthome/forum_thread.php?id=8&postid=100 |
Send message Joined: 27 Jun 20 Posts: 9 Credit: 788,006 RAC: 0 |
It is not about checkpointing. When a task is paused, I so far assumed it is just halted and kept in memory. The experience I described above, however, suggests that Minecraft discards the entire data after a certain period of time? Which I found baffling. ;-) Michael. President of Rechenkraft.net - This world's first and largest distributed computing organization. We make those things possible that supercomputers don't. |
Send message Joined: 24 Jun 20 Posts: 3 Credit: 1,715,763 RAC: 11 |
It is not about checkpointing. When a task is paused, I so far assumed it is just halted and kept in memory. The experience I described above, however, suggests that Minecraft discards the entire data after a certain period of time? Which I found baffling. ;-) When a GPU task is paused it is removed from the GPU memory. Without checkpointing it has to restart from the beginning. The "Leave in Memory" setting works only for CPU tasks. |
Send message Joined: 27 Jun 20 Posts: 9 Credit: 788,006 RAC: 0 |
When a GPU task is paused it is removed from the GPU memory. Without checkpointing it has to restart from the beginning. Thanks for the information. Michael. President of Rechenkraft.net - This world's first and largest distributed computing organization. We make those things possible that supercomputers don't. |
Send message Joined: 14 Jun 20 Posts: 78 Credit: 1,321,619 RAC: 0 |
This is solved in the latest round of updates, see the last news post. |