Unless you have the ability to write code for individual hardware you have in your phone to function perfectly with whatever the non-Android OS you are going to port ! - then YES you can.
But nobody would ever dare to do that tedious & next to impossible crazy thing !
Ubuntu, Firefox OS and all other kind of Linux based OS can be ported relatively easily on Android device if you have some experience with android porting, there are also porting guides on Ubuntu wiki. You don't need to write code for drivers, the same drivers work.