Boards with no OS and acceleration?

[CounterPillow]

You're in for quite an adventure. Probably your best bet is to rely on TF-A and u-boot to set up the system for you (DRAM training, setting the clocks, etc.) and then have them jump to your custom kernel. If you choose to not go down this route, then graphics acceleration is the least of your worries; unlike microcontrollers like you mentioned, modern SoCs have a lot of complexity to them that are documented in thousands of pages of SoC TRM (Technical Reference Manual) and tens of thousands of pages of architecture documentation from ARM.

As for graphics acceleration, if you don't want to write a gpu driver inside your toy OS, you might be interested in the RGA functionality on Rockchip SoCs such as the RK3399 found on the ROCKPro64. RGA is a bitmap blitter, and can do some other stuff as well (scaling, drawing lines, that sort of stuff).

I recommend that if you want to look into this seriously, you get yourself a copy of Part 1 and Part 3 of the RK3399 TRM (can be found in various places online) and read through some sections. If it doesn't immediately scare the bajeesus out of you, you may then proceed to figuring out how to boot into your custom code from a mainline self-compiled u-boot.