[Sept 25] KDE Neon (20180917) - Pinebook / Slackware Aarch64 Miniroot (20180901) - ROCK64 | [Aug 6] Debian Stretch Minimal 64bit (0.7.8) / Ubuntu 18.04 Bionic LXDE Desktop Image (0.7.8) / Ubuntu 18.04 Bionic minimal 64bit / 32bit Image (0.7.8) / Ubuntu 18.04 Bionic Containers Image (0.7.8) / Stretch OpenMediaVault OS Image armhf / Stretch OpenMediaVault OS Image arm64 ( 0.7.8) - ROCK64 | [June 25] Stock Android 8.1 - ROCK64

Project Inspiration | Get Started | IRC Logs | Forum Rules/Policy


(SO)Pine64 Clusterboard
#1
At long last :-)

Thanks to Tl Lim, my clusterboard prototype for SOPine64 modules arrived a couple of days ago and I started to set it up in an old ITX rack mount case. I will try and post some more thoughts and results during the initial bring up and testing.

   

   

The board itself is of mini-ITX formfactor and provides slots for 7 SOPine64 modules networked by the onboard GbE switch. Some interfaces of each module are exposed on the board like two USB ports and a header with UART, SPI, I2C and things like RESET, PWR_ON...

Edit 1: 2017/12/16

My current plan to power the board (5V input only, so maybe 15-20A max) is to use the normal FlexATX PSU I used in the case. The Seasonic SS250-SU is a 250W ATX12V 2.3 unit that provides [email protected][email protected] and [email protected],3V. Since the topology of those ATX PSU's doesnt allow for low/no load operation each rail has to be loaded with a minimum load specified in the datasheet. For the SS250-SU this is 3.3V - zero load, 5V - 0.5A, 12V-2A. To not waste the 24W on the 12V I will probably use the DC/DC converter in the pictures and go from 12V to 5V for efficiency reasons...

Next step will probably be to completetly rewire the ATX power cable, fans and front panel connectors with some veroboard to get some controls in and add some features like thermal management, power sequencing and safety... the front panel probably also needs some kind of status display...

Edit 2: 2017/12/17

On the software side I will start using mainline firmware/spl/u-boot right away since the nodes will boot from the SOPine' s on-board SPI flash and pull their working images from the network... have to do some research on the state of those components upstream, but from what I heard everything should be already in place. Getting the SPI images on to the Flash can either be done through FEL mode / USB (WIP) or by booting from a sdcard...

Edit 3: Clusterboard 2.1 schematic

Tl Lim provided me with the latest clusterboard hardware schematics for reference:

.pdf   PINE64 CLUSTER BOARD_2_1.pdf (Size: 572.46 KB / Downloads: 1,286)

Edit 4: 2017/12/17

I built an Armbian mainline image for a first quick test that boots from sdcard, which seems to work fine, but GbE does not seem to  work stable yet, after some changes to the dts I at least got a link... I then tried with ayufan's last xenial-minimal image (legacy kernel) and that worked fine out of the box...

first boot...
   
two nodes...
   

I ran some quick iperf3 tests between the nodes and to an external box and got between 800-900Mbit/s , so network hardware seems to be OK... next step is to fix the mainline image as I don't really want to work with legacy kernel here...

I ordered some small 14x14x5mm copper heatsinks and two 80x80mm fans now to get some cooling for the cluster.
Come have a chat in the Pine A64 IRC channel >>
Reply
#2
nice to hear that pine64 has finally got some hardware to test out this idea of multiple cpu clusters. best to my knowledge there is nothing else like it so it will be interesting to see where this goes.
Reply
#3
Edit 5: 2017/12/21

Some more parts (heatsinks, sdcards for local storage, fans,....) arrived and I set up the rest of the modules, so we have a fully populated board now...

   
   
   

Running all 7 nodes with simple test images seems to work fine, clusterssh helps a lot with sending commands to all of them at once Smile

   
ClusterSSH (cssh) login...
   
Running cpuburn-a53 on all nodes/cores...
Come have a chat in the Pine A64 IRC channel >>
Reply
#4
Edit 6: 2017/12/21

While running cpuburn-a53 on all nodes to check out the stability of the system, I also added the two 80mm x 80mm fans... those are needed under full continuous load to prevent frequency throttling of the cores.

   
80x80mm fans with [email protected]/0.13A
   

I need to come up with some fan mounts and maybe some air duct or chimney...

   

After the tests with cpuburn-a53 I put together a quick distcc cluster setup with one node as master and 6 slaves to build a linux kernel. The time for a build (only Image) went from about 24 minutes on one node to about 7 minutes with the distcc setup... the workload on the nodes was pretty uneven during that build and there is a lot written on how to improve distcc with load-balancing... for comparison my old Thinkpad (i7-2720QM CPU @ 2.20GHz, 32GB RAM, 850PRO SSD) takes about 4 minutes to cross-compile this kernel...
Come have a chat in the Pine A64 IRC channel >>
Reply
#5
Playing catchup - my cluster board only just arrived unfortunately.

Got a Kubernetes book the other day ...

Here's the board with 5 sopines up and running - each sopine has flashing I'm alive LEDs but only one got caught in the photo.

Presenlty mounted in an m350 mini-itx case.


Attached Files Thumbnail(s)
   
Reply
#6
(12-28-2017, 01:38 PM)maya.b Wrote: Playing catchup - my cluster board only just arrived unfortunately.

Got a Kubernetes book the other day ...

Here's the board with 5 sopines up and running - each sopine has flashing I'm alive LEDs but only one got caught in the photo.

Presenlty mounted in an m350 mini-itx case.

You got to create a custom backplate Smile
  If you manage to click this link you'll join in the IRC channel
Reply
#7
(12-28-2017, 05:00 PM)Luke Wrote:
(12-28-2017, 01:38 PM)maya.b Wrote: Playing catchup - my cluster board only just arrived unfortunately.

Got a Kubernetes book the other day ...

Here's the board with 5 sopines up and running - each sopine has flashing I'm alive LEDs but only one got caught in the photo.

Presenlty mounted in an m350 mini-itx case.

You got to create a custom backplate Smile

Yes, with 40mm PWM fans... might be able to get 3 in (eth and power in get in the way of 4) ... Shouldn't be too hard to laser cut something.
Reply
#8
(12-28-2017, 01:38 PM)maya.b Wrote: Playing catchup - my cluster board only just arrived unfortunately.

Got a Kubernetes book the other day ...

Here's the board with 5 sopines up and running - each sopine has flashing I'm alive LEDs but only one got caught in the photo.

Presenlty mounted in an m350 mini-itx case.

Got a minimal Kubernetes cluster up and running Big Grin - will post details - presently it's 4 nodes (one of my sopine modules decided to melt itself Undecided )
Reply
#9
(12-29-2017, 02:23 PM)maya.b Wrote:
(12-28-2017, 01:38 PM)maya.b Wrote: Playing catchup - my cluster board only just arrived unfortunately.

Got a Kubernetes book the other day ...

Here's the board with 5 sopines up and running - each sopine has flashing I'm alive LEDs but only one got caught in the photo.

Presenlty mounted in an m350 mini-itx case.

Got a minimal Kubernetes cluster up and running Big Grin - will post details - presently it's 4 nodes (one of my sopine modules decided to melt itself Undecided )

Awesome !
Reply
#10
We just release the Clusterboard schematic capture and PCB job source file. Developers are welcome to use for their cluster board implementation.

http://wiki.pine64.org/index.php/PINE_A6...d_Resource

The clusterboard will be available at PINE64 store on late January 2018. We will provide one free SOPine module with clusterboard as promotion :-)
Reply


Possibly Related Threads...
Thread Author Replies Views Last Post
  Ways to boot the Clusterboard khaosgrille 0 51 09-21-2018, 01:26 AM
Last Post: khaosgrille
  Clusterboard dimensions Pine 1 98 08-24-2018, 10:38 PM
Last Post: tllim
  Yet another clusterboard enclosure dcdc 2 321 08-20-2018, 05:57 PM
Last Post: pfeerick
  Clusterboard n00bie... n0ctilucient 7 302 08-18-2018, 06:07 AM
Last Post: pfeerick
  Clusterboard Feature request topic PeterKeuning 4 375 08-14-2018, 07:55 PM
Last Post: dcdc
  eMMC on clusterboard in node 0 aww 3 466 08-14-2018, 07:06 PM
Last Post: dcdc
  Clusterboard Armbian aww 14 1,378 08-13-2018, 11:49 AM
Last Post: SteveG
  Clusterboard Pinouts xalius 2 1,216 08-12-2018, 06:30 AM
Last Post: SteveG
  clusterboard enclosure - mini-itx m350 maya.b 1 320 07-22-2018, 05:30 PM
Last Post: tllim
  Rancheros for clusterboard? ppouliot 0 199 06-05-2018, 10:20 PM
Last Post: ppouliot

Forum Jump:


Users browsing this thread: 1 Guest(s)