-
-
Notifications
You must be signed in to change notification settings - Fork 133
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test DeskPi Super6C CM4 6-node Mini-ITX cluster board #425
Comments
Board is up on the site now: https://pipci.jeffgeerling.com/boards_cm/cm4-cluster-mini-itx.html Changed the title for this issue so I can test it out soon. |
Guess what just arrived? I believe @danmanners is getting one today too... |
Some notes from a tour around the board:
|
I can confirm that the Chassis-connector power button and LED work as expected, and I've had one node boot up successfully. Power supply shipped with my unit is a 19V 5.26A, or 100W (99.94w, but whatever) and my one Pi CM4 boots up. I haven't hooked up HDMI or network yet, but that's next on the list. Can also confirm that the two Noctua Fans connected or operating as expected, albeit at full speed. The unit powered off (with LEDs on the board on) draws ~2.0 watts according to my Kill-A-Watt, and powered on with a single CM4, Micro SD, and NVMe drive is drawing ~6.3W at peak and is idling at ~4.6W. I'll be able to update more findings over the next few days. EDIT: Just a note (not a gripe), the CM4 heatsink I'm using is definitely a little bit annoying to install with the unit. Not an issue, but definitely makes hot-swapping modules effectively impossible if this is installed in a chassis of any kind. |
@danmanners - Hmm... that's the same one I'm using. It looks like the little flat screws that hold the heatsink on can't but up against the risers in each of the four corners :( D'oh! |
Migrated over my MicroSD and NVMe from one of my Waveshare BaseBoard (B) and have TalosOS up and operational, as well as HDMI via capture card. At least according to my Ubnt EdgeSwitch, it doesn't see any additional MAC Addresses or management IPs, so my immediate guess is that Realtek chip is unmanaged. |
https://wiki.wisdpi.com/raspberry-pi-cm4/rpi-cm4-itx-cluster |
@tltangliang - Oh excellent! Thanks for posting that. Looks like the switch is basically unmanaged, and the two Ethernet ports are just hanging off it in addition to the 6 internal ports then. That site also answers some other questions I had, like:
As @danmanners, myself, and some others get into testing this board, we may have more questions too. Is there a specific support forum or email we can contact? I know I got an email directly from one of the devs, and I'll share feedback there, but it would also be nice to know a central place so other users could also provide feedback or get guidance. Edit: Looks like there's a forum but it's still being set up a bit. Is that the best place to direct users? |
I'll second everything @geerlingguy said. Also going to see if I can control the addressable RGB via Kubernetes somehow, because why not, lol. |
I think the forum can be working now. |
Hmm... this looks awfully similar to 52Pi's DeskPi Super6C, which retails for $199. The Cluster Mini-ITX board is $200 on DWMZone and shipping was $40ish. Shipping is about the same on 52Pi. So the question is, who actually makes the board, and what's the board's most 'official' name? :D |
Great Question. Let's find out..
|
The ''official'' name is "DeskPi Super6C" DWMZone is an authorised agent |
Thanks for clarifying that! I guess we should update the name on the site, then, and also add a reference to both locations in the description. |
Hey guys, would either of you by chance have a 3d Printer? I got this board, and designed an IO shield for the Super6c but don't have a 3D printer to try to print it out just yet. Would you be willing to make sure all of the holes line up? As always, great work, I really appreciate the work you put into your videos. thin_ITX_IO_Shield_Super6c.scad.txt
|
FYI I've purchased a DeskPi Super6C board and am using it to test the SOQuartz module. Booted Manjaro ok, but need to rejig the USB driver. Initial details and picture on the SOQuartz issue here: #336 (comment) |
Hey @drifterdave; my apologies for the delay. I don't have a functional 3D printer, but I'm hoping to get one working in the next 2-3 months, so I should be able to validate everything for you then! |
Updated names and added newer links. for issue geerlingguy#425
Nice! Looks like they're doing a great job taking feedback and adjusting a bit. |
Additionally I'm seeing packet loss of around 25% on all CM4's... wondering if anyone else experienced this. |
Has anyone ran EMMC on this board. I have had nothing but problems Flashed 64-bit piOS lite - regularly fails to boot reporting end kernel panic - not syncing asynchronous error interrupt Tried ubuntu lite and sticks on the rainbow boot screen Spent the day flashing and re-flashing the emmc on this board and a carrier board and all the same results. Have no issues on the carrier board, they boot each and every time |
@geerlingguy @michaelarmstrong @joshuatam Amazing post, video and thread! I am now really thinking about one of these... Do you guys have an idea whether:
Any insights appreciated!!!! |
@rr4444 responding in order:
|
@althas316 sorry, just seeing this now!! I haven't tried any eMMC units on the board, but the fact that DeskPi is selling the eMMC units as related items, it'e be bizarre to not support them. |
Many thanks @danmanners !!
This was my concern.... I'm now wondering about using longer screws, which leads me to ask how tight the gap is at the top between the CM4 heatsinks and the official case? Impossible to know without having one :(
Just for flashing and stuff. Makes sense. BTW can that be used as tty? This will actually be my first foray in rpis!
Booting from something else is ok though. So that link (which hadn't found myself, thank you!) is helpful, cos it confirms that the Rpi does see the Optane as NVME.
Looks official though.... Desk Pi Store I've asked on the Deskpi repo issues, but no answer. |
The official deskpi site and other listings expressly say emmc is supported............ I hope so, that's what I have! |
I have 6x CM4 modules with eMMC on my DeskPi installed and the OS is on eMMCs so it is definitely supported and works just fine. I do have (6) 256GB NVMe SSDs as well installed for storage (not booting from these). |
|
That's great news re eMMC @1201am , thank you! Which NVMEs do you use? And how do your NVME temperatures behave? I'm also wondering about how heatsinks fit in the official case... I was thinking about using longer screws, but then occurred to be that then the ports won't line up with the case :( NVMEs need heatsinks, and I read about the WD Reds, for example, running at 80C! And that's without something like k8s etcd constantly hammering them |
@rr4444: At first, I was testing it with Kioxia (Toshiba) 256GB SSD KBG40ZNV256G and I had nothing than problems: SC4s running too hot even idle, 1 or 2 network ports not working at all (not passing any traffic if all 6 SSDs are installed), SSDs running way too hot (hitting 70C with no I/O operations). I thought that the initial set of (6) SSDs os defective so I purchased (4) more Kioxia but the issue persisted. If I remember correctly, the other brand/model that did not work was SK Hynix HFM256GD3JX013N, don't recall the model of the third one I tried as I was tired of testing and replacing SSDs. What I think is common between all 3 brands that caused the issue is that all 3 of them are single-chip packages and leverage Host Memory Buffer (HMB). The Samsung models utilize separate controller chip along with memory chips. I am not making any conclusions here, just sharing thoughts as I am not sure what is causing the issue here :-). I had a few larger Samsung 970 EVO that I installed just to try and immediately everything started working as it should: no "dead" network ports, CM4 runs relatively cool (I had heatsinks on these), and SSDs are not hot even without cooling. |
@danmanners I had a spare Patriot P300 M.2 PCIe Gen 3 x4 128GB Low-Power Consumption SSD laying around, threw that on and no issues at all Bought another 5 and is now very happily working away. All the CM4 are set up to boot from the nvme drives. Learning Ansible currently, then K3s, then docker, that's the plan |
You'll want to learn Docker before any flavour of Kubernetes. Maybe even Docker Swarm too, which is v limited compared to Kube potential. But perfect for many use cases. Then K3S. You can't really understand Kube without understanding how containers, networking and APIs work. Just my 2c to maximise your learning :) |
There are actually no usable GPIO pins on this cluster board is there? ... |
Does anyone have any idea why NVMe drives wouldn't get detected on this? They're not showing up at all, which is needless to say rather frustrating! |
Okay, this is bleh. So the drives I got are WD Blue SN580s, as I spotted they were discounted for Black Friday. Just nabbed a different drive from another system to try out, and it worked fine. That one.... it's an SN570 !! So something about the newer drive isn't working on the super6c/cm4, grrr. |
As a separate data point I can confirm that SN570 SSDs do work. I got two batches which were made in different factories and they are quite different to each other but both of them work. |
Nice confirmation, thanks. Guess I'll be contacting WD tomorrow, but assuming they can't solve it, I suspect they won't be willing to swap my SN580s for SN570s. Though I'll try. Otherwise I guess I'll be having to return them all, gah. |
Look at my posts from January, I spent many hours until I found that there is a compatibility issue with some SSDs. Samsung PM981a 256GB finally worked as it should. |
When I got mine, I bought 6 Integral M Series drives and none of them
worked in the cluster board. I am using now Patriot P300 (P300P128GM28)
and had no issues
…On Wed, 6 Dec 2023 at 23:31, 1201am ***@***.***> wrote:
Does anyone have any idea why NVMe drives wouldn't get detected on this?
They're not showing up at all, which is needless to say rather frustrating!
Look at my posts from January, I spent many hours until I found that there
is a compatibility issue with some SSDs. Samsung PM981a 256GB finally
worked as it should.
—
Reply to this email directly, view it on GitHub
<#425 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APVMLVJ5O7USKTB6ZDQWS4LYID56NAVCNFSM5TWM7GJ2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBUGM4DKNJTGE3A>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Ugh, this is ridiculous. With my SN570 500GB working fine, and the report from @filipnavara above, I grabbed some 2TB SN570s. And the one I've opened and tried right now isn't working! Tested in a little minipc x86 box that my 500GB is normally in, and the 2TB shows up in the bios fine there, but just as with the SN580s, nothing on the pcie bus at all on the CM4! |
Might try some 1TB drives, see if they work. The 500GB actually shows up on the pcie but listed as 1TB from lspci, though nvme list reports the expected 500GB model. But who knows, maybe an indication that 1TB would work 🤞🏻 |
Well, picked up an SN570 1TB today (ordered more, but didn’t bother grabbing them yet just in case!), and it works 🧡 ETA: So yay for finally having a drive that works, but boo that the 2TB is somehow different in a way that doesn’t work on the CM4/Super6C |
I am interested in buying the Super6c with the case to create a Ceph storage cluster. |
I just completed one. Hoping that it is not deprecated by the CM5 someday. I agree on the switch upgrade though I would like the ability to connect a 2.5Gig port. |
Very interesting board. I wonder if using these M.2 board to USB C adapters and a couple of USB C hubs could be a workaround for the lack of connectivity. The only thing is you may need to make some holes on the back of the case to have access to the adapters IO, or even a rectangle cut to have easy access to the M.2 and SD card slots. Also a DC power jumper on each Pi would be great for easier hot-swap or manual disabling. https://pt.aliexpress.com/item/1005005676467186.html |
I don't think modifying the is that big of a deal. I do prefer to use the NVMe SSDs rather than the sdcard. An interesting thought those is to use an NVMe on the first CM4, and then the adapters on the rest and make CM4 #2-6 network boot from the first one and mount their own FS there over NFS? |
I just heard about the CM4 Cluster Mini ITX board on DWMzone:
It looks like it supports up to 6 CM4 modules, with an M.2 slot and microSD card slot available to each of the boards. It uses a 4-pin ATX CPU power plug, and has two RJ45 Gigabit Ethernet ports connected through an onboard switch. It looks like there is a USB 2.0 hub and two HDMI ports connected through to the first Pi, which is nice for cluster management / debugging purposes.
I just ordered one and will post testing notes here or to a new issue later.
The text was updated successfully, but these errors were encountered: