Best way to set up ubuntu
#1
So I've been working with my pine for a few months now, trying to get a hadoop server running to support my work as an integration architect. My plan is to use the pine as a linux server, running multiple hadoop clusters, a postgres db and proprietary integration software from my employer. As an experiment I simply installed everything on the micro sd card, but that is not sustainable as far as size, so I experimented with using a usb thumbdrive as a pic. That only worked somewhat since I had to muck around with disk mounts etc, and ultimately as soon as I had it all working a bad shutdown corrupted the thumbdrive sending me back to square one.

I have since invested in a 1tb usb hdd, a powered usb hub, and now need to the optimal way of setting it all up. My first question is is it best to reformat the external hdd to a linux partition or leave it as ntfs. I know that ntfs has limitations with setting privileges, which is a issue with postgres. How would the linux gurus  approach this?
  Reply
#2
Staying with NTFS on Linux will at least ensure that disk access is slow as hell. Combining an USB HDD with database workloads sounds also wrong (fast eMMC or a SATA SSD would be better -- yeah, both not possible with Pine64). The best you could do on Pine64 is to rely on mainline kernel and an UASP capable disk enclosure combined with a fast SSD. Further readings: http://linux-sunxi.org/USB/UAS and
http://forum.armbian.com/index.php/topic...s-on-sbcs/ and maybe also http://forum.armbian.com/index.php/topic...rformance/
  Reply
#3
I'm not so much worried about speed as proof in concept. I need to build multiple sample databases of fake companies that I would then build an integration platform that combines the separate databases into on hadoop filesystem, and then from there reporting. That's the idea at least.
  Reply
#4
You're still better of with using a 'proper' linux filesystem such as ext4... you get the journaling features that will make it more likely a unexpected shutdown can be recovered from. NTFS is a journalling filesystem also, but I don't think it is anywhere near as reliable as ext4 is (on linux, anyway). Plus as tkaiser said, performance will be an issue, and since you're working with large data sets, you want the best performance you can get.

(09-16-2016, 11:09 AM)pharris430 Wrote: I'm not so much worried about speed as proof in concept. I need to build multiple sample databases of fake companies that I would then build an integration platform that combines the separate databases into on hadoop filesystem, and then from there reporting. That's the idea at least.
  Reply
#5
Quote:You're still better of with using a 'proper' linux filesystem such as ext4... you get the journaling features that will make it more likely a unexpected shutdown can be recovered from. NTFS is a journalling filesystem also, but I don't think it is anywhere near as reliable as ext4 is (on linux, anyway). Plus as tkaiser said, performance will be an issue, and since you're working with large data sets, you want the best performance you can get. 

That's the kind of info I'm looking for!
  Reply
#6
I will add that you also want multiple partitions on your drive; several parts of the file-system are better suited to being on their own partition for security, performance and maintenance. The following should be on their own journaled file-system partition preferrably on EXT4:
/
/boot on the SD card
swap
extended
/usr
/opt
/tmp
/var
/usr/local
/home

------------------
marcushh777    Cool

please join us for a chat @  irc.pine64.xyz:6667   or ssl  irc.pine64.xyz:6697

( I regret that I am not able to respond to personal messages;  let's meet on irc! )
  Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Ubuntu server 20.04 request phi0x 2 5,284 12-30-2020, 08:02 PM
Last Post: phi0x
  SPI on ubuntu IlyaM 4 9,084 07-16-2018, 03:00 AM
Last Post: kingflab
  Ubuntu password not working joemassimino 4 8,509 05-18-2018, 03:33 PM
Last Post: Luke
  After flashing Ubuntu- how to get graphical desktop Partymack711 6 10,828 02-01-2018, 09:45 AM
Last Post: Partymack711
  (ARCHIVED) Ubuntu Xenial Image (BSP Kernel) longsleep 337 633,680 01-31-2018, 11:58 AM
Last Post: Luke
  PPTP doesnt work with ubuntu escovedo 2 8,656 08-23-2017, 09:28 AM
Last Post: gilbertotcc
Question HELP Ubuntu Mate password recovery procedure Captain Anni Hilator 5 9,901 08-03-2017, 03:52 PM
Last Post: Luke
  Ubuntu boot very slow with wifi driver remainder 9 20,654 08-03-2017, 07:23 AM
Last Post: victorssantos
  Recovering root access in Ubuntu Scientomancer 3 5,827 07-28-2017, 02:53 PM
Last Post: Scientomancer
  Pine64 2Gb Ubuntu Mono not updating and stuck petec 0 2,873 07-08-2017, 09:34 AM
Last Post: petec

Forum Jump:


Users browsing this thread: 1 Guest(s)