A Rant About Pine, Other Distros, & Still Trying to Get Fedora Booting From eMMC.

[whitecat23]

So the Linux kernel (as of 5.x) has received quite a few new bits for Rockchip devices, making it easier to install a number of distros on the RockPro64 (using official ISO installers now results in some perfectly bootable operating systems). But it's evident that depth of support isn't equal across distros: While some install and boot just fine from the system's microSD reader the very same operating system will not boot from an internal eMMC flash drive, nor from an external SSD. Conversely, other distros employing a 5.x kernel boot from all those storage mediums without any fuss; then there's that part about text not getting printed to the screen (i.e. openSUSE Leap 15.3 and Debian 11), while others have incorporated this very basic (and expected) feature as standard!
As an example, Fedora 35 is a distribution that works beautifully when installed to a microSD card, but installing it onto my eMMC, or SSD, results in several GRUB errors (i.e. "You need to load a kernel first") then going back to the boot menu. Again, everything seems to just work when Fedora is installed to an microSD card. But I need something faster.
Then there are distros I've tested that run just fine off the eMMC, like openSUSE and Debian. Unfortunately those two refuse to print any text to the screen, forcing me into an X session every time (although I can type 'blindly' into a tty session). I need to see text output. I want to see the logs scroll by. Why did they not put the effort in, when Fedora does? In my opinion, Fedora aarch64 is quite simply the best Linux I've used in my limited time with ARM64: It makes for a sane, stable, and responsive system.
Here is what seems to me a general problem: I fail to detect any overt effort on the part of Pine64 to work with distro-makers and get these extremely basic features working, across all releases. It's really looking like a fast-accumulating hodge-podge of distros which might or might not be suitable to task. Things like text console support and booting from recognized storage shoudn't be considered a luxury on any platform. The inconsistencies are staggering.
Still, I'm still hoping for some small breakthrough that'll get Fedora loading from eMMC, or USB 3.0 SSD. Any idea?

[Rocklobster]

So the Linux kernel (as of 5.x) has received quite a few new bits for Rockchip devices, making it easier to install a number of distros on the RockPro64 (using official ISO installers now results in some perfectly bootable operating systems). But it's evident that depth of support isn't equal across distros: While some install and boot just fine from the system's microSD reader the very same operating system will not boot from an internal eMMC flash drive, nor from an external SSD. Conversely, other distros employing a 5.x kernel boot from all those storage mediums without any fuss; then there's that part about text not getting printed to the screen (i.e. openSUSE Leap 15.3 and Debian 11), while others have incorporated this very basic (and expected) feature as standard!
As an example, Fedora 35 is a distribution that works beautifully when installed to a microSD card, but installing it onto my eMMC, or SSD, results in several GRUB errors (i.e. "You need to load a kernel first") then going back to the boot menu. Again, everything seems to just work when Fedora is installed to an microSD card. But I need something faster.
Then there are distros I've tested that run just fine off the eMMC, like openSUSE and Debian. Unfortunately those two refuse to print any text to the screen, forcing me into an X session every time (although I can type 'blindly' into a tty session). I need to see text output. I want to see the logs scroll by. Why did they not put the effort in, when Fedora does? In my opinion, Fedora aarch64 is quite simply the best Linux I've used in my limited time with ARM64: It makes for a sane, stable, and responsive system.
Here is what seems to me a general problem: I fail to detect any overt effort on the part of Pine64 to work with distro-makers and get these extremely basic features working, across all releases. It's really looking like a fast-accumulating hodge-podge of distros which might or might not be suitable to task. Things like text console support and booting from recognized storage shoudn't be considered a luxury on any platform. The inconsistencies are staggering.
Still, I'm still hoping for some small breakthrough that'll get Fedora loading from eMMC, or USB 3.0 SSD. Any idea?

My only thoughts on the matter is as a new board appears from Pine an older one gets abandoned. Software development on older boards is not keeping pace with newer kernel versions. The best kernel version for your board is the last one that worked. I'm afraid that's the way I see things. I reckon you'll be lucky to get past kernel 5.1x. Of course I could be wrong and will stand corrected.

[CounterPillow]

I'm sorry you ran into these issues and feel this way, and I can understand your frustration.

I'd like to offer some explanations and perspectives on the matter. Practically all software work being done for PINE devices is either volunteer work, or time-limited contract work (that may be funded by people other than PINE, I don't know who hires Collabora to work on Rockchip stuff). This means that eventually, the onus of supporting these devices falls onto the community. There is no person hired full-time at PINE64 to work on software because this would be a losing proposition; there's such a wide selection of different OS projects that providing continued software support for them all would be both impossible from a human resources and an organisation perspective.

What you can do when you found an issue in a software image for a PINE64 device is to report it to the project that's behind the software image. If they are keen on fixing the issue, they can even contact PINE64 directly if they need help with acquiring hardware; PINE has in the past sent out devices to volunteers so they could support specific distributions.

In your specific case, the bootloader not showing video output is likely related to the u-boot config; the bootloader needs a video output driver to be able to display something before the kernel starts, and this is controlled by a build-time config option. It's common that these get forgotten, as the people deciding on the distribution's bootloader build configs may not always be knowledgeable enough about the devices or have one to test it with. If the kernel doesn't print text to the screen, that's a bit more worrisome and points towards a module not getting loaded when it should.

As for eMMC, there's currently a problem with u-boot 2021.10. on ROCKPro64 specifically when it comes to eMMC booting. I believe Fedora have a patch for this applied, but I'm unsure about it. Either way, the fix has been posted to the u-boot mailing list, and is awaiting review. People are working on it. I am not entirely certain this specific issue is what you're running into (as Fedora already patched it on their side from what I've heard), but I thought it's worth mentioning.

As for debugging boot-up and such, video output is never the right answer. Get a 3.3V 1.5mbauds capable UART adapter such as the Woodpecker from the PINE store ($1.99), or any other such devices. They can be had for cheap almost anywhere. With UART, you get debug output regardless of whether something manages to initialise and use a very complex video output driver; it's just a text based interface as old as computers themselves.

I'd also like to disagree with

My only thoughts on the matter is as a new board appears from Pine an older one gets abandoned.

In my experience, this isn't true. There may be regressions in newer kernels, but people will fix them if told about it. The whole point of making sure your device runs mainline Linux and mainline u-boot is that it isn't just left by the wayside; you get to enjoy newer kernels for years to come.

As an example, I recently found and reported a regression in lima that was caused by the 5.16 kernel. Lima is the driver used by older ARM GPUs, such as the one found in the ROCK64. The problem was fixed within a day after I ran my git bisect on the kernel, before the regression ever made it into a release.

It's important to both understand how to pinpoint issues precisely and how to test proposed fixes; this allows people without the hardware (in my case, an AMD employee) to come up with a fix and verify that it works. I know this can be hard for people not familiar with all the components of the system, but usually we can help out debugging things like these in the PINE64 chats, where many knowledgeable people reside. They're a bit more real-time and laser-focused than the forums usually are.

In conclusion:

• I get it, I can see how this is frustrating, I've had to deal with it myself as well.
  
• Get a UART adapter.
  
• Ask in the PINE64 chats for help debugging issues; we're the most likely to have the same devices and can quickly look into it if need be. The forums are mostly where people go to complain about their PinePhone orders or whatever.
  
• Communication is key; if a problem is never reported to the responsible component, it'll likely never get fixed. This is a small user base, so it's possible you're the first to run into something.
  
• Maintainers of those components can always reach out to the PINE64 community and PINE64 themselves if they need help improving a certain aspect of their software.
  

[Rocklobster]

Glad to see Fedora is now booting but it wouldn't be first choice of Distro.

[whitecat23]

Glad to see Fedora is now booting but it wouldn't be first choice of Distro.

Thanks, Rocklobster...Just curious to know why Fedora might not be your first choice in distro for the RockPro64?

[Rocklobster]

Sorry, it wouldn't be my first choice of Distro. Probably because I've been running Debian based systems for quite a while, mainly Ubuntu in the x86 world and Armbian laterally with the Arm based hardware although they do say familiarity breathes contempt lol.

[TRS-80]

SBC world are very different to x86, as the latter have a standardized hardware interface. This is one of main reasons you see so much difference across distros.

I learned a long time ago that a distro which specialize in SBCs (like Armbian, or one of its many derivatives) is going to work a lot better than "random distro X I know from x86 world."

May not be what you want to hear if you are a Fedora guy, but that's the way it is (and has been for some years, and likely will be for foreseeable future).

[whitecat23]

SBC world are very different to x86, as the latter have a standardized hardware interface.  This is one of main reasons you see so much difference across distros.

I learned a long time ago that a distro which specialize in SBCs (like Armbian, or one of its many derivatives) is going to work a lot better than "random distro X I know from x86 world."

May not be what you want to hear if you are a Fedora guy, but that's the way it is (and has been for some years, and likely will be for foreseeable future).

Now that I've been using Fedora Linux for a couple months, I can say that the experience has been good, at least for me. Fedpra wasn't my first choice in operating system: I was leaning towards Armbian only to discover that official support for RockPro64 was ending at around the time Fedora 35 got released. It's a very stable and performant system, except that I keep encountering one hiccup related to video playback. And for this reason alone I understand the importance of having a supported distro like Armbian, if only to use as a 'gold standard' when tweaking another system. But I gotta admit, I'm impressed by how well Fedora just works on the RockPro64, and hopefully the next release incorporates some GPU optimizations right out of the box.
In retrospect my "rant" might have been a little too harsh; just a product of my frustration at the time, of trying to find a supported OS that also runs well on my hardware. Ideally, Armbian continues their official support for the the RockPro64. I've read your ongoing thread and it seems the situation regarding Armbian is still up in the air. Even though I'm not an Armbian user (at least not yet), my fingers are crossed this will soon be resolved...

[TRS-80]

Now that I've been using Fedora Linux for a couple months, I can say that the experience has been good, at least for me.

Well, glad to hear it. 

Fedpra wasn't my first choice in operating system: I was leaning towards Armbian only to discover that official support for RockPro64 was ending at around the time Fedora 35 got released.

[...]

Ideally, Armbian continues their official support for the the RockPro64. I've read your ongoing thread and it seems the situation regarding Armbian is still up in the air. Even though I'm not an Armbian user (at least not yet), my fingers are crossed this will soon be resolved...

We (Armbian) have had a lot of trouble around this word 'Supported.'  It's too loaded of a term, and have some specific meanings within Armbian.  Which are not read by most of people.  So you end up with this situation (like you may have seen in that thread) where everyone immediately assume it's the end of the world.  And I have been battling mis-information about it ever since.

Maybe we communicated poorly, chose wrong term(s)?  I have proposed some changes to how we are trying to communicate the underlying realities which are more complex.

Anyway, this removal of 'Supported' status is trying to stop the bleeding on a handful of volunteers trying to support >100 boards for more than 8 years now without any support from vendors (and little from so called 'community').

Having said that, you can still build things just as before, it's just that they are not going to be built automatically, nor hosted on our infrastructure any longer.

Some people are answering the call though, which is nice to see.  We have been meeting with them in IRC and via PM.  And I am, as we speak, working on transferring the (formerly internal / WIP) new list of Board Maintainers to our public Docs.  I will update over in that thread of course after it's posted.

In retrospect my "rant" might have been a little too harsh; just a product of my frustration at the time, of trying to find a supported OS that also runs well on my hardware.

Aah, we've all been there, mate.  lol