You say both that the locals are isolationists, but that there's a big set of businesses catering to adventurers - does this mean there's something of a split between locals who mistrust the outsiders and locals who've done well out of cashing in on them?
Also, more prosaically, what kind of size town/population are you thinking this place is?