If they don't want exiles in their Keeps they can close and lock the doors so we can't get in.

If they don't want exiles to kill their guards if they do give us access to their keeps, they can tell the guards to not attack exiles. Since the doors to the keeps are open, many assume that it means we can go in. Especially since the Brions have given us Ethereal Amulet trainers and it is sometimes necessary to enter their keeps to speak to these trainers. In addition, there are three slaves that we can do business with (Soul Sword, Ethereal Portal stone, Ethereal Amulet) that are within Tenebrion's Keep so it's neccesary to go in there to get access to them. Since the guards attack us, we are merely defending ourselves.

Unless the Brions created the Sasquatches, Mud Golems, Valley Panthers, Valley Cougars, Undine, rats, vermine, Rippers, etc. that are found in and around the keeps then the Brions have no claim to them and so exiles can kill the beasts freely since they belong to no one.

Therefore, if the Brions are upset at us exiles for any of these acts they only have themselves to blame since there are several actions they can take to remedy them.