cluster computing

[jproffer]

I'm using my pines as a computing cluster with 4 nodes (4 per node), plus one dedicated to delegation.
   
Currently powered by a homemade USB power hub (17 port) and a computer PSU. Re-doing the USB hub to a cleaner implementation, 2 8-port cards
   

[tkaiser]

I'm using my pines as a computing cluster with 4 nodes (4 per node)

Just curious: what kind of workload should run on this cluster?

[hazerty]

you have buy 17 boards?

[jproffer]

you have buy 17 boards?

Yeah I have 17.

---

I'm using my pines as a computing cluster with 4 nodes (4 per node)

Just curious: what kind of workload should run on this cluster?

mainly I'll use it for thermal/stress simulations, and probably a bunch of folding@home projects in its downtime

[tkaiser]

mainly I'll use it for thermal/stress simulations, and probably a bunch of folding@home projects in its downtime

Ok, unfortunately it's already too late but maybe this helps others to improve their cluster setups: We did already extensive research on performance related topics. If you've an hour please read through https://github.com/longsleep/build-pine6...-191984162 where most if not all problems are outlined.

• You won't be able to get decent performance without both heatsinks and some amount of airflow (otherwise throttling will jump in and decrease performance drastically)
  
• Using Micro USB to power the boards is insufficient, you better use the Euler connector (this is especially necessary if you improved heat dissipation and managed to prevent throttling since using optimized code you're able to exceed the Micro USB limitations -- max. 1.8A by design and most likely undervoltage problems way earlier)
  
• If you're after high performance you need active cooling. By controlling the airflow and upright orientation of the boards you could use one or two large (and therefore silent) 120mm fans to increase the amount of air flowing over the heatsink's surfaces (I tested this with a few Allwinner H3 boards a while ago using flexible cardboard stripes between the boards to maximize speed of airflow over SoC/heatsink)
  
• If you don't like fans but want to improve performance tweaking cpufreq/dvfs settings might help: Pine64 is able to use dynamic voltage frequency scaling and by testing all your boards you might be able to decrease VDD_CPUX for the relevant dvfs operating points. Less voltage --> less temperatures --> higher clockspeeds if throttling is an issue. Also adding more cpufreq steps in the range where throttling happens might increase achievable performance by a few percent
  

References: http://linux-sunxi.org/Hardware_Reliability_Tests#CPU and the misused Github 'issue' above

[baryluk]

I am also thinking about creating pine cluster. Probably 17 boards too. Mostly for distributed algorithms development (databases, programming language research, raytracing, etc). But I am still thinking between Pine64+ and RPi3 B. Pi people and Broadcom promised to support PXE booting without sd card (you can already do nfs root over network, but you still need to use a bootloder using small sd card), by just putting few pxe related bits (tftp client, dhcp client, ethernet driver, etc) in the rom/flash itself. I doubt we will see this on Pine. Maybe on next version of the board, which will not happen this year for sure.

[tllim]

I am also thinking about creating pine cluster. Probably 17 boards too. Mostly for distributed algorithms development (databases, programming language research, raytracing, etc). But I am still thinking between Pine64+ and RPi3 B. Pi people and Broadcom promised to support PXE booting without sd card (you can already do nfs root over network, but you still need to use a bootloder using small sd card), by just putting few pxe related bits (tftp client, dhcp client, ethernet driver, etc) in the rom/flash itself. I doubt we will see this on Pine. Maybe on next version of the board, which will not happen this year for sure.

This is an interesting topic, I will check with Allwinner folks but don't expect an answer till next week due to long weekend holiday. Even assume not supported, it is easy to put a small SPI NOR Flash at Pi2 bus as bootloader for booting (A64 support this method).

[jproffer]

mainly I'll use it for thermal/stress simulations, and probably a bunch of folding@home projects in its downtime

Ok, unfortunately it's already too late but maybe this helps others to improve their cluster setups: We did already extensive research on performance related topics. If you've an hour please read through https://github.com/longsleep/build-pine6...-191984162 where most if not all problems are outlined.

• You won't be able to get decent performance without both heatsinks and some amount of airflow (otherwise throttling will jump in and decrease performance drastically)
  
• Using Micro USB to power the boards is insufficient, you better use the Euler connector (this is especially necessary if you improved heat dissipation and managed to prevent throttling since using optimized code you're able to exceed the Micro USB limitations -- max. 1.8A by design and most likely undervoltage problems way earlier)
  
• If you're after high performance you need active cooling. By controlling the airflow and upright orientation of the boards you could use one or two large (and therefore silent) 120mm fans to increase the amount of air flowing over the heatsink's surfaces (I tested this with a few Allwinner H3 boards a while ago using flexible cardboard stripes between the boards to maximize speed of airflow over SoC/heatsink)
  
• If you don't like fans but want to improve performance tweaking cpufreq/dvfs settings might help: Pine64 is able to use dynamic voltage frequency scaling and by testing all your boards you might be able to decrease VDD_CPUX for the relevant dvfs operating points. Less voltage --> less temperatures --> higher clockspeeds if throttling is an issue. Also adding more cpufreq steps in the range where throttling happens might increase achievable performance by a few percent
  

References: http://linux-sunxi.org/Hardware_Reliability_Tests#CPU and the misused Github 'issue' above

Thanks for the info, it's interesting, I hadn't realized the microusb would be limited to 1.8 amps, but it does make sense now that I think about it, and I did confirm its limitation.  Euler it shall be, then.   

Heatsinks are definitely in the plans.  My setup doesn't even have the microsd cards configured yet - just testing out the USB power hub currently -- it took alot of time to build.   Well at least now I have a kickass phone charger, haha! (a 470 ohm resistor between the data pins ensures maximum current draw for devices like the iPhone, for what its worth).

I was planning to set cpufreq to performance (1.2ghz).  I'll have to test it out and see how it performs, and tweak as needed.  You make it sound like the board is a flop -- hopefully I won't be disappointed..  

[tkaiser]

You make it sound like the board is a flop

The hardware is ok if you avoid the design flaws (Micro USB for powering for example). But the project will flop due to the horrible state of documentation.

[jproffer]

You make it sound like the board is a flop

The hardware is ok if you avoid the design flaws (Micro USB for powering for example). But the project will flop due to the horrible state of documentation.

So, I tried powering up a Pine64 board using Euler, connecting +5v to pin 4 and ground to pin 6.  The board's LED did not come on.  Am I missing something?

---

You make it sound like the board is a flop

The hardware is ok if you avoid the design flaws (Micro USB for powering for example). But the project will flop due to the horrible state of documentation.

So, I tried powering up a Pine64 board using Euler, connecting +5v to pin 4 and ground to pin 6.  The board's LED did not come on.  Am I missing something?

never mind, I got it working.  I was using my breadboard to test it with and one of the pins wasnt pushed down far enough.