On Android 11 ROMs, you currently can't use a password/pin for unlocking your device if you want to decrypt "/data" partition in TWRP.
So, you can remove your password/pin before booting into TWRP or set a pattern instead.
EDIT: If you have issue to boot in system, try in TWRP => "Reboot" menu > "System". If not working, reflash your current ROM version then reboot your device (you will not lose your data).