Unfortunately, I cannot tell more specific, because I haven't tried this setup, but you need to look at the uboot environment. also - in the boot.cmd and armbianEnv.txt, I guess. What exactly commands are fed to it in order to get images from a USB mass storage device. Obviously, uboot isn't the bloody edge of firmwares, but there is hope, you can figure out the cause of the problem there and fix it. It looks like either uboot doesn't have the provision to precisely set up the device path to the boot medium or it does have it, but in your current set up, it's "weak" and goes nuts when finds more than one USB mass storage devices. Or rather, it tries USB2 port first. The ports are static, they should be distinguishable, not to mention they are connected to different controllers. It's pretty much the same situation as with BIOS on PCs, that easily can remember an HDD order (through what they are connected to). It's another question if uboot is able to distinguish two USB controllers though. I cannot say if it is. You can get this answer only by luck, because let's face it - their documentation is a very bad joke.
In short, show what boot.cmd contains.
In short, show what boot.cmd contains.
ANT - my hobby OS for x86 and ARM.