(SO)Pine64 Clusterboard
#1
At long last :-)

Thanks to Tl Lim, my clusterboard prototype for SOPine64 modules arrived a couple of days ago and I started to set it up in an old ITX rack mount case. I will try and post some more thoughts and results during the initial bring up and testing.

   

   

The board itself is of mini-ITX formfactor and provides slots for 7 SOPine64 modules networked by the onboard GbE switch. Some interfaces of each module are exposed on the board like two USB ports and a header with UART, SPI, I2C and things like RESET, PWR_ON...

Edit 1: 2017/12/16

My current plan to power the board (5V input only, so maybe 15-20A max) is to use the normal FlexATX PSU I used in the case. The Seasonic SS250-SU is a 250W ATX12V 2.3 unit that provides 17A@5V, 22A@12V and 14A@3,3V. Since the topology of those ATX PSU's doesnt allow for low/no load operation each rail has to be loaded with a minimum load specified in the datasheet. For the SS250-SU this is 3.3V - zero load, 5V - 0.5A, 12V-2A. To not waste the 24W on the 12V I will probably use the DC/DC converter in the pictures and go from 12V to 5V for efficiency reasons...

Next step will probably be to completetly rewire the ATX power cable, fans and front panel connectors with some veroboard to get some controls in and add some features like thermal management, power sequencing and safety... the front panel probably also needs some kind of status display...

Edit 2: 2017/12/17

On the software side I will start using mainline firmware/spl/u-boot right away since the nodes will boot from the SOPine' s on-board SPI flash and pull their working images from the network... have to do some research on the state of those components upstream, but from what I heard everything should be already in place. Getting the SPI images on to the Flash can either be done through FEL mode / USB (WIP) or by booting from a sdcard...

Edit 3: Clusterboard 2.1 schematic

Tl Lim provided me with the latest clusterboard hardware schematics for reference:

.pdf   PINE64 CLUSTER BOARD_2_1.pdf (Size: 572.46 KB / Downloads: 2,561)

Edit 4: 2017/12/17

I built an Armbian mainline image for a first quick test that boots from sdcard, which seems to work fine, but GbE does not seem to  work stable yet, after some changes to the dts I at least got a link... I then tried with ayufan's last xenial-minimal image (legacy kernel) and that worked fine out of the box...

first boot...
   
two nodes...
   

I ran some quick iperf3 tests between the nodes and to an external box and got between 800-900Mbit/s , so network hardware seems to be OK... next step is to fix the mainline image as I don't really want to work with legacy kernel here...

I ordered some small 14x14x5mm copper heatsinks and two 80x80mm fans now to get some cooling for the cluster.
Come have a chat in the Pine IRC channel >>
#2
nice to hear that pine64 has finally got some hardware to test out this idea of multiple cpu clusters. best to my knowledge there is nothing else like it so it will be interesting to see where this goes.
#3
Edit 5: 2017/12/21

Some more parts (heatsinks, sdcards for local storage, fans,....) arrived and I set up the rest of the modules, so we have a fully populated board now...

   
   
   

Running all 7 nodes with simple test images seems to work fine, clusterssh helps a lot with sending commands to all of them at once Smile

   
ClusterSSH (cssh) login...
   
Running cpuburn-a53 on all nodes/cores...
Come have a chat in the Pine IRC channel >>
#4
Edit 6: 2017/12/21

While running cpuburn-a53 on all nodes to check out the stability of the system, I also added the two 80mm x 80mm fans... those are needed under full continuous load to prevent frequency throttling of the cores.

   
80x80mm fans with 2000RPM@12V/0.13A
   

I need to come up with some fan mounts and maybe some air duct or chimney...

   

After the tests with cpuburn-a53 I put together a quick distcc cluster setup with one node as master and 6 slaves to build a linux kernel. The time for a build (only Image) went from about 24 minutes on one node to about 7 minutes with the distcc setup... the workload on the nodes was pretty uneven during that build and there is a lot written on how to improve distcc with load-balancing... for comparison my old Thinkpad (i7-2720QM CPU @ 2.20GHz, 32GB RAM, 850PRO SSD) takes about 4 minutes to cross-compile this kernel...
Come have a chat in the Pine IRC channel >>
#5
Playing catchup - my cluster board only just arrived unfortunately.

Got a Kubernetes book the other day ...

Here's the board with 5 sopines up and running - each sopine has flashing I'm alive LEDs but only one got caught in the photo.

Presenlty mounted in an m350 mini-itx case.


Attached Files Thumbnail(s)
   
#6
(12-28-2017, 01:38 PM)maya.b Wrote: Playing catchup - my cluster board only just arrived unfortunately.

Got a Kubernetes book the other day ...

Here's the board with 5 sopines up and running - each sopine has flashing I'm alive LEDs but only one got caught in the photo.

Presenlty mounted in an m350 mini-itx case.

You got to create a custom backplate Smile
You can find me on IRC, Discord and Twitter


#7
(12-28-2017, 05:00 PM)Luke Wrote:
(12-28-2017, 01:38 PM)maya.b Wrote: Playing catchup - my cluster board only just arrived unfortunately.

Got a Kubernetes book the other day ...

Here's the board with 5 sopines up and running - each sopine has flashing I'm alive LEDs but only one got caught in the photo.

Presenlty mounted in an m350 mini-itx case.

You got to create a custom backplate Smile

Yes, with 40mm PWM fans... might be able to get 3 in (eth and power in get in the way of 4) ... Shouldn't be too hard to laser cut something.
#8
(12-28-2017, 01:38 PM)maya.b Wrote: Playing catchup - my cluster board only just arrived unfortunately.

Got a Kubernetes book the other day ...

Here's the board with 5 sopines up and running - each sopine has flashing I'm alive LEDs but only one got caught in the photo.

Presenlty mounted in an m350 mini-itx case.

Got a minimal Kubernetes cluster up and running Big Grin - will post details - presently it's 4 nodes (one of my sopine modules decided to melt itself Undecided )
#9
(12-29-2017, 02:23 PM)maya.b Wrote:
(12-28-2017, 01:38 PM)maya.b Wrote: Playing catchup - my cluster board only just arrived unfortunately.

Got a Kubernetes book the other day ...

Here's the board with 5 sopines up and running - each sopine has flashing I'm alive LEDs but only one got caught in the photo.

Presenlty mounted in an m350 mini-itx case.

Got a minimal Kubernetes cluster up and running Big Grin - will post details - presently it's 4 nodes (one of my sopine modules decided to melt itself Undecided )

Awesome !
#10
We just release the Clusterboard schematic capture and PCB job source file. Developers are welcome to use for their cluster board implementation.

http://wiki.pine64.org/index.php/PINE_A6...d_Resource

The clusterboard will be available at PINE64 store on late January 2018. We will provide one free SOPine module with clusterboard as promotion :-)


Possibly Related Threads…
Thread Author Replies Views Last Post
  Managed Ethernet switch on the Clusterboard dsimic 6 6,869 01-16-2024, 02:05 AM
Last Post: hazcam
  Selling Clusterboard + 4 sopine xblack86 7 6,898 09-26-2022, 03:09 AM
Last Post: poVoq
  clusterboard availability? linear 1 1,962 07-17-2022, 06:44 AM
Last Post: poVoq
  Clusterboard not getting IP address after network fix Norlark 14 14,546 08-30-2021, 05:00 PM
Last Post: poVoq
  When would be the Clusterboard back on stock? __kzl__ 5 6,037 08-16-2021, 03:52 AM
Last Post: poVoq
  Schematics for customising the Clusterboard TechDragon 0 2,322 05-25-2021, 10:28 AM
Last Post: TechDragon
Information Clusterboard Guide AZClusterboard 9 25,040 04-06-2021, 06:32 AM
Last Post: poVoq
Brick Clusterboard CAD drawings TeaPack 1 4,178 03-31-2021, 01:38 PM
Last Post: Pine
  New Clusterboard Setup wargfn 5 8,153 01-01-2021, 10:10 AM
Last Post: poVoq
  Clusterboard does not even turn on? poVoq 3 6,816 08-15-2020, 07:31 AM
Last Post: poVoq

Forum Jump:


Users browsing this thread: 1 Guest(s)