If it helps, I noticed that there is a portal near the school area during the midnight phase. It seems that the Attendants can control where to exit from as well, so perhaps we can enter the portal near the school, enter the Velvet Room, and ask them to reopen a portal at the castle.

[ It's... worth a shot?

Maybe. ]