+- PINE64 (https://forum.pine64.org)
+-- Forum: PINE A64-LTS / SOPINE Compute Module (https://forum.pine64.org/forumdisplay.php?fid=66)
+--- Forum: Clusterboard (https://forum.pine64.org/forumdisplay.php?fid=91)
+--- Thread: (SO)Pine64 Clusterboard (/showthread.php?tid=5461)
(SO)Pine64 Clusterboard - xalius - 12-15-2017
At long last :-)
Thanks to Tl Lim, my clusterboard prototype for SOPine64 modules arrived a couple of days ago and I started to set it up in an old ITX rack mount case. I will try and post some more thoughts and results during the initial bring up and testing.
The board itself is of mini-ITX formfactor and provides slots for 7 SOPine64 modules networked by the onboard GbE switch. Some interfaces of each module are exposed on the board like two USB ports and a header with UART, SPI, I2C and things like RESET, PWR_ON...
Edit 1: 2017/12/16
My current plan to power the board (5V input only, so maybe 15-20A max) is to use the normal FlexATX PSU I used in the case. The Seasonic SS250-SU is a 250W ATX12V 2.3 unit that provides 17A@5V, 22A@12V and 14A@3,3V. Since the topology of those ATX PSU's doesnt allow for low/no load operation each rail has to be loaded with a minimum load specified in the datasheet. For the SS250-SU this is 3.3V - zero load, 5V - 0.5A, 12V-2A. To not waste the 24W on the 12V I will probably use the DC/DC converter in the pictures and go from 12V to 5V for efficiency reasons...
Next step will probably be to completetly rewire the ATX power cable, fans and front panel connectors with some veroboard to get some controls in and add some features like thermal management, power sequencing and safety... the front panel probably also needs some kind of status display...
Edit 2: 2017/12/17
On the software side I will start using mainline firmware/spl/u-boot right away since the nodes will boot from the SOPine' s on-board SPI flash and pull their working images from the network... have to do some research on the state of those components upstream, but from what I heard everything should be already in place. Getting the SPI images on to the Flash can either be done through FEL mode / USB (WIP) or by booting from a sdcard...
Edit 3: Clusterboard 2.1 schematic
Tl Lim provided me with the latest clusterboard hardware schematics for reference:
PINE64 CLUSTER BOARD_2_1.pdf (Size: 572.46 KB / Downloads: 2,470)
Edit 4: 2017/12/17
I built an Armbian mainline image for a first quick test that boots from sdcard, which seems to work fine, but GbE does not seem to work stable yet, after some changes to the dts I at least got a link... I then tried with ayufan's last xenial-minimal image (legacy kernel) and that worked fine out of the box...
first boot...
two nodes...
I ran some quick iperf3 tests between the nodes and to an external box and got between 800-900Mbit/s , so network hardware seems to be OK... next step is to fix the mainline image as I don't really want to work with legacy kernel here...
I ordered some small 14x14x5mm copper heatsinks and two 80x80mm fans now to get some cooling for the cluster.
RE: (SO)Pine64 Clusterboard - dkryder - 12-16-2017
nice to hear that pine64 has finally got some hardware to test out this idea of multiple cpu clusters. best to my knowledge there is nothing else like it so it will be interesting to see where this goes.
RE: (SO)Pine64 Clusterboard - xalius - 12-20-2017
Edit 5: 2017/12/21
Some more parts (heatsinks, sdcards for local storage, fans,....) arrived and I set up the rest of the modules, so we have a fully populated board now...
Running all 7 nodes with simple test images seems to work fine, clusterssh helps a lot with sending commands to all of them at once
ClusterSSH (cssh) login...
Running cpuburn-a53 on all nodes/cores...
RE: (SO)Pine64 Clusterboard - xalius - 12-21-2017
Edit 6: 2017/12/21
While running cpuburn-a53 on all nodes to check out the stability of the system, I also added the two 80mm x 80mm fans... those are needed under full continuous load to prevent frequency throttling of the cores.
80x80mm fans with 2000RPM@12V/0.13A
I need to come up with some fan mounts and maybe some air duct or chimney...
After the tests with cpuburn-a53 I put together a quick distcc cluster setup with one node as master and 6 slaves to build a linux kernel. The time for a build (only Image) went from about 24 minutes on one node to about 7 minutes with the distcc setup... the workload on the nodes was pretty uneven during that build and there is a lot written on how to improve distcc with load-balancing... for comparison my old Thinkpad (i7-2720QM CPU @ 2.20GHz, 32GB RAM, 850PRO SSD) takes about 4 minutes to cross-compile this kernel...
RE: (SO)Pine64 Clusterboard - maya.b - 12-28-2017
Playing catchup - my cluster board only just arrived unfortunately.
Got a Kubernetes book the other day ...
Here's the board with 5 sopines up and running - each sopine has flashing I'm alive LEDs but only one got caught in the photo.
Presenlty mounted in an m350 mini-itx case.
RE: (SO)Pine64 Clusterboard - Luke - 12-28-2017
(12-28-2017, 01:38 PM)maya.b Wrote: Playing catchup - my cluster board only just arrived unfortunately.
Got a Kubernetes book the other day ...
Here's the board with 5 sopines up and running - each sopine has flashing I'm alive LEDs but only one got caught in the photo.
Presenlty mounted in an m350 mini-itx case.
You got to create a custom backplate
RE: (SO)Pine64 Clusterboard - maya.b - 12-29-2017
(12-28-2017, 05:00 PM)Luke Wrote:(12-28-2017, 01:38 PM)maya.b Wrote: Playing catchup - my cluster board only just arrived unfortunately.
Got a Kubernetes book the other day ...
Here's the board with 5 sopines up and running - each sopine has flashing I'm alive LEDs but only one got caught in the photo.
Presenlty mounted in an m350 mini-itx case.
You got to create a custom backplate
Yes, with 40mm PWM fans... might be able to get 3 in (eth and power in get in the way of 4) ... Shouldn't be too hard to laser cut something.
RE: (SO)Pine64 Clusterboard - maya.b - 12-29-2017
(12-28-2017, 01:38 PM)maya.b Wrote: Playing catchup - my cluster board only just arrived unfortunately.
Got a Kubernetes book the other day ...
Here's the board with 5 sopines up and running - each sopine has flashing I'm alive LEDs but only one got caught in the photo.
Presenlty mounted in an m350 mini-itx case.
Got a minimal Kubernetes cluster up and running
- will post details - presently it's 4 nodes (one of my sopine modules decided to melt itself 
)
RE: (SO)Pine64 Clusterboard - tllim - 12-29-2017
(12-29-2017, 02:23 PM)maya.b Wrote:(12-28-2017, 01:38 PM)maya.b Wrote: Playing catchup - my cluster board only just arrived unfortunately.
Got a Kubernetes book the other day ...
Here's the board with 5 sopines up and running - each sopine has flashing I'm alive LEDs but only one got caught in the photo.
Presenlty mounted in an m350 mini-itx case.
Got a minimal Kubernetes cluster up and running
Awesome !
RE: (SO)Pine64 Clusterboard - tllim - 12-31-2017
We just release the Clusterboard schematic capture and PCB job source file. Developers are welcome to use for their cluster board implementation.
http://wiki.pine64.org/index.php/PINE_A64-LTS/SOPine_Main_Page#Pine64_Cluster_Board_Resource
The clusterboard will be available at PINE64 store on late January 2018. We will provide one free SOPine module with clusterboard as promotion :-)