I have looked into the outside visportals in the map.
The visportals on the left/right of the castle are pretty good, I think they do their job well.
The visportals in front of and behind the castle are strange, they don't help rendering in my opinion. Since I doubt they help with sound propagation too... it seems that removing them will not changed anything.
Remove them all. If you have vast space without notable blockers, everything is generally visible from everywhere, and sound travels straight, then there is no reason to put any visportals.
Another problem with these portals is that they seem to be intersecting. I think intersecting portals is a very bad setup in any case.
Now here is the idea of what might actually help.
It is a great luck that your map includes a fence/wall in front of the castle, which reliably blocks the whole castle in the mentioned camera shot. We just need to put portals in such a way that they don't allow light to go through this wall. To achieve it, add two portals in the plane of this wall:
One huge portal just above the wall. From below, it should contact the top of the wall. On the other three sides, let it span over the whole map.
One small portal where player enters the yard (goes through the wall). UPDATE: Actually, there is more than one entering, an each one should have its own portal of course.
With such portal setup, anything which is not behind these two portals will be culled away, which should include the whole castle.
UPDATE: A rough plan: