OneChunk CPU Pre-Announcement (X86_64 & ARM64)

Message boards : News : OneChunk CPU Pre-Announcement (X86_64 & ARM64)
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Hy
Project administrator
Project developer
Avatar

Send message
Joined: 15 Jun 20
Posts: 64
Credit: 19,241,109
RAC: 47
Message 559 - Posted: 3 May 2021, 15:46:34 UTC
Last modified: 4 May 2021, 17:38:21 UTC

Over on the [email protected] discord, Boysanic ("Boincboi"), Matt (Stronghold simulation coding), other developers and I have been discussing and planning a rollout of our first CPU application. This OneChunk project is "a BOINC project in the works. Our goal is to find all world seeds w/ at least an 11 eye portal in MC Java 1.7.10". So, if this all works out, "you'll finally be able to say you've found an 11 eye portal by yourself!"

Essentially, we are combining parts of the "SSG" (Seed-Speedrun) and 11-eye portal seed-filtering programs together to go through a large dataset. This input file is 9GB in size, but we plan to split it up into many small manageable tasks to be distributed through BOINC, which is where you come in! Going through a 320kb sample input takes a 4.8GHz Ryzen 5800 (the fastest single-core CPU) around 2 minutes, simulating and checking over 5,220 worldseeds a second. A 2GHz Raspberry Pi 4, with 64bit Raspbian manages to compute 1,370 worldseeds a second, making it around 3.8x slower but much more efficient. (45W vs 5W)

Thanks to work by other developers, I have tested and confirmed that both x86_64 ("64bit Windows and Linux") and ARM64 (64bit Linux ARMv8) configurations will work. The OneChunk 'Filter9000' BOINC application is single-threaded, but will likely have multiple tasks running concurrently on your computer to best utilize its resources and scale to your unique hardware configurations. For 'PC' requirements, we suggest Ubuntu 18.04/20.04 64bit or Windows 7 / 10 64bit, running on an CPU such as an AMD FX or Intel 1st Gen Core i5/i7 CPU or newer.

If anyone has a Raspberry Pi 3(B/B+) or 4, we would like to invite you to install the latest 64bit Raspbian on a spare SDcard in preparation. We also expect Nvidia Jetson Nano/TX1/Xavier devices with Ubuntu 18.04 64bit or newer to successfully run our Arm64 tasks, once we configure our BOINC server to provide and schedule tasks for this platform.

Please follow instructions on the Raspberry Pi website, but we suggest that more experienced Pi users join us for this experiment! OS image below:
https://downloads.raspberrypi.org/raspios_arm64/images/raspios_arm64-2021-04-09/2021-03-04-raspios-buster-arm64.zip

Also, check out our Forums for upcoming updates, and to see the first 25-Tall Superflat cactus!
ID: 559 · Report as offensive     Reply Quote
zombie67 [MM]
Avatar

Send message
Joined: 24 Jun 20
Posts: 19
Credit: 106,318,465
RAC: 71
Message 561 - Posted: 4 May 2021, 2:27:32 UTC

I have 5x RP4s attached and ready to go.
Reno, NV
Team: SETI.USA
ID: 561 · Report as offensive     Reply Quote
fzs600

Send message
Joined: 25 Jun 20
Posts: 3
Credit: 23,646,580
RAC: 15,043
Message 562 - Posted: 4 May 2021, 15:38:21 UTC - in response to Message 561.  

I have 5x RP4s attached and ready to go.

I have 1x RP4s attached and ready to go. :-)
ID: 562 · Report as offensive     Reply Quote
Keith Myers
Avatar

Send message
Joined: 8 Mar 21
Posts: 20
Credit: 108,038,899
RAC: 844,018
Message 563 - Posted: 4 May 2021, 15:59:00 UTC

I will be converting over my RPi 3B+ to ARM64 Buster. Card already created and just waiting for the cpu application.
ID: 563 · Report as offensive     Reply Quote
Profile adiquet

Send message
Joined: 12 Apr 21
Posts: 3
Credit: 18,556,631
RAC: 103,749
Message 564 - Posted: 4 May 2021, 16:11:58 UTC

I also have a RP4 - what do I need to do?
I have installed the OS what do I also need to install Boinc? or what else do I need to do?
ID: 564 · Report as offensive     Reply Quote
Profile Hy
Project administrator
Project developer
Avatar

Send message
Joined: 15 Jun 20
Posts: 64
Credit: 19,241,109
RAC: 47
Message 565 - Posted: 4 May 2021, 17:37:33 UTC - in response to Message 564.  
Last modified: 5 May 2021, 15:47:25 UTC

On Windows 10 (for example), you can use Raspberry Pi Image Writer to write the posted Raspbian 64bit OS to a fresh SD card, and then set up your system as usual (plus connect it to the internet). Make sure you have some sort of cooling on your Pi 4 though, like a heatsink or fan, especially if you are overclocking.

Then, you can run " sudo apt-get update && sudo apt-get install boinc boinc-client boinc-manager " to install BOINC, and then you can launch it through either the terminal (boincmgr) or the menu. It is advised though that you have a screen attached to use BOINC via the GUI, but more experienced BOINC/Linux users will be able to use:

https://minecraftathome.com/minecrafthome/weak_auth.php

boinc --daemon --attach_project "minecraftathome.com/minecrafthome/" "weak_key_here"
boinccmd --project minecraftathome.com/minecrafthome/ update
boinccmd --get_tasks
boinccmd --network_available

to connect their screen-free (like SSH) pi to begin crunching when we release the application :)

(Also, code updates can be tracked here: https://github.com/minecrafthome/OneChunk but we have like a dozen things on our checklist to do before its running)
ID: 565 · Report as offensive     Reply Quote
Profile adiquet

Send message
Joined: 12 Apr 21
Posts: 3
Credit: 18,556,631
RAC: 103,749
Message 567 - Posted: 5 May 2021, 0:55:16 UTC - in response to Message 565.  

thanks I was able to get everything installed, BOINC and waiting for new tasks! Thanks everyone
ID: 567 · Report as offensive     Reply Quote
Profile Hy
Project administrator
Project developer
Avatar

Send message
Joined: 15 Jun 20
Posts: 64
Credit: 19,241,109
RAC: 47
Message 568 - Posted: 5 May 2021, 14:28:38 UTC
Last modified: 6 May 2021, 23:23:53 UTC

Small update, we've decided roughly on the input work and are slowly making pieces of the BOINC template files, and task estimates. Keep in mind, here we are using "BOINC Benchmark Whetstone MIPS = OPS"
These break down to:

5,800,000,000 OP/sec per Ryzen 5800X Core Whetstone (what credit is based off of)
12.5 seconds per seed/input line (ie, 10 seeds computed with our current program takes 120-125 seconds on a single 5800X 4.85ghz core)

So, 72,500,000,000 "OPS per seed input line per core" on Zen 3

Thus:
21,750,000,000,000 OPS (2.175e^13) (300 lines of input seeds) of computation done per core, per BOINC task.

If everything works as expected, then CPU task time estimates should be close to reality, being 60-65mins on a Ryzen 5000 CPU, and say, 90-120mins on a Ryzen 1000 CPU due to IPC and frequency differences. Also for fun stats, a 3.2GHz Broadwell laptop core is 2.2X slower (271 seconds for 10 seeds) versus my 5800X
ID: 568 · Report as offensive     Reply Quote
LOVIT

Send message
Joined: 31 Jan 21
Posts: 1
Credit: 8,799
RAC: 0
Message 569 - Posted: 5 May 2021, 15:23:53 UTC

i will try it on Rpi 3B+
SD ready :)
ID: 569 · Report as offensive     Reply Quote
Elspuddy

Send message
Joined: 18 Jul 20
Posts: 1
Credit: 10,320,054
RAC: 12,000
Message 578 - Posted: 8 May 2021, 9:53:30 UTC

hi 1X pi4 8gig ready, il set up the other 10 at some point, i's a bit of a pain imgeing all them ssd's :P
ID: 578 · Report as offensive     Reply Quote
Profile Hy
Project administrator
Project developer
Avatar

Send message
Joined: 15 Jun 20
Posts: 64
Credit: 19,241,109
RAC: 47
Message 582 - Posted: 10 May 2021, 16:10:04 UTC

Thanks for the signs and support and ARM64 hardware being prepared to contribute to this project! We've currently got 4 projects (Kaktwoos-cl update, OneChunk, "new seed video" and Trailer-seed) in progress, but I will keep you updated as we work to close out some of them, and finalize the OneChunk CPU tasks with later testing.
ID: 582 · Report as offensive     Reply Quote
Profile Hy
Project administrator
Project developer
Avatar

Send message
Joined: 15 Jun 20
Posts: 64
Credit: 19,241,109
RAC: 47
Message 585 - Posted: 17 May 2021, 18:10:10 UTC
Last modified: 17 May 2021, 19:17:40 UTC

One of our coders, Matt, is working on some optimizations for OneChunk while we wait for our BOINC-server side code to be finished. So far, he has given my Ryzen 5800X a 12% performance improvement (110 seconds for 10 input seeds, versus 124-125 before). My Pi 4 (doing two seeds with dungeons and lots of output) went from taking 95 seconds to 88 seconds, or an 8% speed improvement.
ID: 585 · Report as offensive     Reply Quote
Profile adiquet

Send message
Joined: 12 Apr 21
Posts: 3
Credit: 18,556,631
RAC: 103,749
Message 586 - Posted: 18 May 2021, 0:14:33 UTC - in response to Message 585.  

Are we expecting work for the Pi's yet? or is that still coming soon?
ID: 586 · Report as offensive     Reply Quote
Profile Hy
Project administrator
Project developer
Avatar

Send message
Joined: 15 Jun 20
Posts: 64
Credit: 19,241,109
RAC: 47
Message 601 - Posted: 5 Jun 2021, 19:21:59 UTC
Last modified: 5 Jun 2021, 19:32:28 UTC

Small update, for now. Sadly things are still in progress as our team's work is divided between multiple projects and IRL (school/work) in-between us and them. I personally have been studying for my next IT certification over the past few weeks, and occasionally have been speaking to the [email protected] team to check in and provide reminders on a few things, and also to post here for updates

Code hasn't been changed for OneChunk/Filter9000 since my previous announcements, and other members are providing input and confirmation on the exact plan on how to generate work and manage the input+output files to keep track of things. The main delay is the work towards our 1 year anniversary of BOINC and [email protected] as its own group as we know it currently. We are preparing scripts, videos and other material for a celebration in the upcoming weeks. A good amount of that is reviewing previous work and projects for any new insights we can make since our discovery of Pack.png last summer. Please check the main page of our site to join us on Discord for any news on it, and to participate with us and the hundreds of users during the celebration!

For updates to Kaktwoos (v2.13), we are waiting for final approvals and merging before sending it to the 'infamous' testbed, which we suggest that you do not get too attached to if you happen to stumble upon it like other BOINCers have.

Additionally, another long-term CPU project (terrain checking) is being theorized, involving a very frustrating "Shot T" which contains nothing but a few grass and dirt blocks, in a pit of some sort. Supposedly, it may take a year or more to find Shot T with 400+ "Core i5" level processors, assuming the code is functional and ported to BOINC after our developers can do as many optimizations as they can. If it indeed takes a long time after we eventually get it onto BOINC, then we may begin cracking "Custom World" example seeds which are another major interest to us.

Various shots for the Trailer seedfinding and eventual recreation project are here:
https://imgur.com/a/ozHEde3

We also recently found the Youtuber 'Seannaners' original Let's Play seed, and I am happy that we did so. Especially after our work on both the recreation and the dungeons, which lead to a relatively quick find after we temporarily paused the project in December.
ID: 601 · Report as offensive     Reply Quote
Profile Hy
Project administrator
Project developer
Avatar

Send message
Joined: 15 Jun 20
Posts: 64
Credit: 19,241,109
RAC: 47
Message 608 - Posted: 2 Jul 2021, 1:23:13 UTC
Last modified: 2 Jul 2021, 5:41:24 UTC

Hello all!

So, remember when I said we had some issues with the work generation, validation, and the immense amount of tasks we expect to store? Well, we've made progress towards making OneChunk more manageable and actually practical on our infrastructure. Some of our admins have recently collaborated with 'Sia', a platform for distributed cloud storage to test out alternative workgen data-input methods. We thought, our BOINC' network is distributed, so why not use another distributed service to solve some problems we kept worrying about?

Boysanic (Boincboi), Matt, I and others on our Discord are discussing some work that has been done the past few days. Boysanic is testing out a basic, but good method of converting our pure-text 9GB seed input file into a 3.5GB binary input file, which we plan to be put on Sia. They have also made adjustments to a validator to easily read our changed data and ensure it compares to the original between conversions. This compression ratio could also apply to the seeds your clients output (or for outputs of future projects with similar data created), but we expect our 1.5 million tasks should not take up much space on your devices, or our servers once this project is officially launched.

Essentially, while generating work (gradually) we'll stream in the data for seeds directly using Sia based file with an offset, bypassing the need to thrash our I/O and RAM storage. It's basically free cost-wise as our file isn't too large, and the many arguments over the method of 'dynamic workgen' can be put to rest I figure.

This may be a week or two, but I am happy to help test and assist with the last touches coding-wise on our BOINC project. Mainly to enable CPU-tasks and add the new platforms (ARM64, ie Raspberry Pi) I mentioned in the original post.
ID: 608 · Report as offensive     Reply Quote
dweeb

Send message
Joined: 25 Apr 21
Posts: 1
Credit: 0
RAC: 0
Message 609 - Posted: 13 Jul 2021, 16:21:17 UTC - in response to Message 608.  

lol, as you mentioned Sia, i thought this is one of the most stupid things i've heard for long.
luckily i'm a dumbass and scrambled it with chia.
because of this review, i got gasping https://www.youtube.com/watch?v=FtXkhIVZBmw
ID: 609 · Report as offensive     Reply Quote

Message boards : News : OneChunk CPU Pre-Announcement (X86_64 & ARM64)