Take a look at your lin file in DarkRadiant and follow the red zigzagging line.
Basically, if it's possible to go through your visportal, loop back around, and pass through it again without ever having to go through another visportal, it will drop. In this schematic, black is worldspawn brushes, green is a visportal, and red is the .lin file. To avoid looping back through the visportal, I needed to put an additional portal in one, or both, of the other entranceways.
Another very common cause of visportal drops is if there is a gap between your worldspawn and portal. In that case the lin file will look like a tiny red loop or splotch at the portal.