If you mean "downgrade" as in "dirty flashing", then no, you shouldn't do that.
If you mean "downgrade" as in "reflashing", then yes, thats absolutely no problem.
No extra steps required, just be sure to use fastboot and run the "_install_and_format_data" script for your OS.