U-Boot, video console, and visual artifacts

[dsimic]

Hello,

As probably already known, support for the PineBook Pro's video console has recently been included in the U-Boot packaged by Manjaro ARM.  That's awesome and, for someone coming from the x86 world, makes the PineBook Pro feel much more like a "real" computer that isn't silent early in the boot process.   The support for the video console is currently in form of a patch, but the patch should be merged into one of the next releases of U-Boot.

However, this new U-Boot feature isn't issue free.  In the first couple of seconds after powering up the PineBook Pro, before any text is displayed, there are some weird screen artifacts that last for a split second.  After the U-Boot messages are displayed and right after the kernel image is loaded and started, the same screen artifacts become visible again, this time much better and lasting for a bit longer.  These artifacts are probably caused by some mode switching and the old, gargabe contents of the framebuffer memory.

Before digging a bit deeper into this, I'd like to ask if anyone else sees the same visual artifacts?  The attached pictures show the screen contents right after the kernel image is started, with the very start of the artifacts in the first picture and the "fully developed" artifacts in the second.

[User 22018]

Yes I get this too, also get it if I reboot the laptop.

[dsimic]

Thank you for the report, jms429.  Just to confirm, the behavior is the same when the PineBook Pro is powered on or rebooted.

[User 22018]

Thank you for the report, jms429.  Just to confirm, the behavior is the same when the PineBook Pro is powered on or rebooted.

Yes. I did wonder if it had anything to do with the warning uboot displays about a bad CRC enviroment?

I also thought it might be something to do with the splash screen so I disabled the boot splash screen (https://wiki.pine64.org/index.php/Pinebo...sh_picture) so I just see the traditional text boot. But when switching from uboot to the text, it still performs the full screen green screen effect.

I agree it's some sort of mode switch, but my knowledge of uboot is limited.

[sigmaris]

Pretty sure this is the frame buffer being overwritten with other content when the kernel loads. In short, what happens is U-Boot allocates an area of RAM for it to write its pixel data into, and tells the VOP (video output processor, part of the SoC that handles display output) to read pixel data from that area and display the resulting image on screen. This is the "frame buffer".

While U-Boot is running, it knows that area is for writing data into to display on screen, and doesn't use it for anything else. But when U-Boot loads the kernel and starts it, it doesn't tell the kernel that the frame buffer memory area should be reserved or is special in any way. After control is passed to the kernel, the kernel thinks the frame buffer area is like any other free area of memory and starts using it to store data (e.g. structures related to kernel bookkeeping, or memory used by programs, etc). However the VOP was never told to stop displaying the contents of this area of memory on screen, so what you're seeing is random data structures being interpreted as pixel data and displayed on screen.

When the display driver in the kernel (called "rockchipdrm") loads, it resets the VOP and allocates a new area of RAM for it to use/display as a frame buffer while Linux is running, hence the weird green screen goes away and you start seeing the expected Linux console text and then graphical desktop.

It's probably harmless that the VOP isn't stopped by U-Boot when it hands control to the kernel and is still reading out from random kernel memory, though real problems can happen if the firmware leaves hardware *writing* into a memory area and doesn't tell the kernel about it, as that can corrupt kernel memory structures - for an example of this happening: https://mjg59.dreamwidth.org/11235.html

[hjalfi]

I'm going to assume that this means we're not going to get kernel early messages on the screen, right? Having uboot interactive on the console would still be awesome (I accidentally bricked my machine on holiday and had to wait until I got it home and had access to my serial cable), but it would be nice to have a proper end-to-end console.

[sigmaris]

There are ways to preserve the console through the boot process, U-Boot can tell the kernel about the existing frame buffer via a description in the devicetree, or if booting the kernel via EFI there's a similar protocol in EFI, but they require coordination between U-Boot and the OS, and also the OS needs to properly switch from simple or efifb to the full rockchipdrm driver when it's ready. I think I saw some patches for rockchipdrm to handle this on the linux-rockchip mailing list, which suggests it doesn't work at the moment but someone's working on it.

[dsimic]

My apologies for the delayed response.

I did wonder if it had anything to do with the warning uboot displays about a bad CRC enviroment?

No, that message is just the result of having no U-Boot environment available, because it's never been saved.  The U-Boot environment contains some variables, e.g. they can be used to provide a "static" MAC address for the Ethernet interface.

I also thought it might be something to do with the splash screen so I disabled the boot splash screen (https://wiki.pine64.org/index.php/Pinebo...sh_picture)  so I just see the traditional text boot.  But when switching from uboot to the text, it still performs the full screen green screen effect. 

As a note, I've disabled the splash screen long time ago, and I'm also experiencing the same "plaid" visual artifacts.

Pretty sure this is the frame buffer being overwritten with other content when the kernel loads. In short, what happens is U-Boot allocates an area of RAM for it to write its pixel data into, and tells the VOP (video output processor, part of the SoC that handles display output) to read pixel data from that area and display the resulting image on screen. This is the "frame buffer".

While U-Boot is running, it knows that area is for writing data into to display on screen, and doesn't use it for anything else. But when U-Boot loads the kernel and starts it, it doesn't tell the kernel that the frame buffer memory area should be reserved or is special in any way. After control is passed to the kernel, the kernel thinks the frame buffer area is like any other free area of memory and starts using it to store data (e.g. structures related to kernel bookkeeping, or memory used by programs, etc). However the VOP was never told to stop displaying the contents of this area of memory on screen, so what you're seeing is random data structures being interpreted as pixel data and displayed on screen

I agree that the (mis)handling of the frame buffer, as you've described it, contributes to this issue.  However, there are two key points that clearly show that the framebuffer (mis)handling is not the only cause:

• These visual artifacts are also briefly visible upon powering on or rebooting the laptop, just before the U-Boot messages are displayed.  I.e., the artifacts are not visible only after the control over hardware is passed from U-Boot to the Linux kernel.
  
• The "plaid" pattern visible on the screen is simply too regular and symmetric to be caused by having essentially random data ("garbage") written to the framebuffer, "interpreted" and displayed back by the RK3399 VOP.
  

Based on my research and the way DRM driver in the Linux kernel handles the RK3399 VOP, VOP video outputs, the LCD screen and the LCD backlight, I'm pretty confident that the other causes of this issue are the following:

• Lack of synchronization between the display output and the displaying of the frames, which are handled by the VOP and the display, respectively
  
• The right order of the following operations/steps needs to be determined, in order to avoid different kinds of visual artifacts:
  • Powering up the LCD screen
    
  • Enabling the LCD screen
    
  • Powering up the LCD backlight
    
  • Configuring the LCD backlight
    
  • Configuring the VOP and the eDP video output
    
  • Enabling the eDP video output
    

Of course, the actual cause is the mix of all of the above, including the (mis)handling of the framebuffer.

I've spent more than a few days becoming familiar with the internals of U-Boot, the U-Boot driver model, etc.  As a result, I've implemented and tested out dozens of changes to the way U-Boot video console, including all of its hardware components, is initialized, turned on and off, etc.  Unfortunately, I haven't managed to isolate and eliminate the root cause of this issue (yet).

[hjalfi]

To me that plaid pattern looks very distinctive as a simple memory fill --- increasing integers written to memory. Although I don't know how this can be appearing on the screen _after_ the text has been drawn, unless it's being orred in, which seems odd. It's possible that the video hardware supports multiple framebuffers, overlayed on top of each other, and u-boot is erroneously enabling the overlay but hasn't told Linux about it.

[dsimic]

To me that plaid pattern looks very distinctive as a simple memory fill --- increasing integers written to memory. Although I don't know how this can be appearing on the screen _after_ the text has been drawn, unless it's being orred in, which seems odd. It's possible that the video hardware supports multiple framebuffers, overlayed on top of each other, and u-boot is erroneously enabling the overlay but hasn't told Linux about it.

In theory, it could be that the Linux kernel overwrites the "U-Boot's" framebuffer, without knowing that it should be preserved because U-Boot left the RK3399 VOP active.  As a result, the VOP would briefly display the "garbage" that results from overwriting the framebuffer.  However, why would Linux kernel fill a rather large portion of RAM with such a pattern of integers very early in the boot process?