Best way to set up ubuntu

[pharris430]

So I've been working with my pine for a few months now, trying to get a hadoop server running to support my work as an integration architect. My plan is to use the pine as a linux server, running multiple hadoop clusters, a postgres db and proprietary integration software from my employer. As an experiment I simply installed everything on the micro sd card, but that is not sustainable as far as size, so I experimented with using a usb thumbdrive as a pic. That only worked somewhat since I had to muck around with disk mounts etc, and ultimately as soon as I had it all working a bad shutdown corrupted the thumbdrive sending me back to square one.

I have since invested in a 1tb usb hdd, a powered usb hub, and now need to the optimal way of setting it all up. My first question is is it best to reformat the external hdd to a linux partition or leave it as ntfs. I know that ntfs has limitations with setting privileges, which is a issue with postgres. How would the linux gurus  approach this?

[tkaiser]

Staying with NTFS on Linux will at least ensure that disk access is slow as hell. Combining an USB HDD with database workloads sounds also wrong (fast eMMC or a SATA SSD would be better -- yeah, both not possible with Pine64). The best you could do on Pine64 is to rely on mainline kernel and an UASP capable disk enclosure combined with a fast SSD. Further readings: http://linux-sunxi.org/USB/UAS and
http://forum.armbian.com/index.php/topic...s-on-sbcs/ and maybe also http://forum.armbian.com/index.php/topic...rformance/

[pharris430]

I'm not so much worried about speed as proof in concept. I need to build multiple sample databases of fake companies that I would then build an integration platform that combines the separate databases into on hadoop filesystem, and then from there reporting. That's the idea at least.

[pfeerick]

You're still better of with using a 'proper' linux filesystem such as ext4... you get the journaling features that will make it more likely a unexpected shutdown can be recovered from. NTFS is a journalling filesystem also, but I don't think it is anywhere near as reliable as ext4 is (on linux, anyway). Plus as tkaiser said, performance will be an issue, and since you're working with large data sets, you want the best performance you can get.

I'm not so much worried about speed as proof in concept. I need to build multiple sample databases of fake companies that I would then build an integration platform that combines the separate databases into on hadoop filesystem, and then from there reporting. That's the idea at least.

[pharris430]

You're still better of with using a 'proper' linux filesystem such as ext4... you get the journaling features that will make it more likely a unexpected shutdown can be recovered from. NTFS is a journalling filesystem also, but I don't think it is anywhere near as reliable as ext4 is (on linux, anyway). Plus as tkaiser said, performance will be an issue, and since you're working with large data sets, you want the best performance you can get. 

That's the kind of info I'm looking for!

[MarkHaysHarris777]

I will add that you also want multiple partitions on your drive; several parts of the file-system are better suited to being on their own partition for security, performance and maintenance. The following should be on their own journaled file-system partition preferrably on EXT4:
/
/boot on the SD card
swap
extended
/usr
/opt
/tmp
/var
/usr/local
/home

------------------