How much linux is required for the phone to run?

[Subsentient]

Oh I had no idea there were many hoops to jump through. I thought it'd be fun to program a device that only runs my code with a touchscreen and wifi

Yeah it's depressing on ARM except in the server space where ACPI is starting to appear.

If you want that kind of barebones experience, I'd recommend moving up to raw AVR microcontrollers, like the ATTiny85, my favorite. They're a lot of fun, and they're bare metal entirely, no arduino board to baby you. Just 8K of ROM to put your compiled C into. You can get a row of around 10 of them for like $15, then you can fiddle with them however you like.

You could also consider getting into linux kernel driver development, knowledge which would be useful in both ARM and x86 spaces.