I’ve noticed when running around castles during sieges that some of the guardian towers aren’t where they are supposed to be. Like they teleport to another location in the castle during the siege, then teleport back to it’s normal location once it ends. I’ve seen ones that work fine and others that do this, not sure if it matters in terms of taking a castle but could be something to look into.
So far kent castle can’t be taken since the tower moves off it’s starting location when a siege starts. Can’t destroy the tower probably because it isn’t where it’s supposed to be.