True enough, and his island is really too strong for me to solo hunt so I go there in groups and by the time I get off my boat the wall(s) are probably already knocked down.

Perhaps because exiles keep knocking the walls down? Or maybe they only wish to discourage exiles, but allow them to pass through for business reasons(i.e. Ethereal trainers). I dunno, yi'll have to ask them yiself.

Why don't you? I doubt I'll ever have their ear and I'm sure they would love to work with the Mystic Council to make sure their walls are nice and strong to us exiles while in allows mystic access to the Ethereal Trainers. Maybe you can include their movement to a place that's more easily accessible as part of your deal.

Since they began inhabiting our neighboring islands?

Unwanted neighbors then.