Ethan and Tom come and go from the island as well. I'm assuming that Ben did leave the island sometimes, but it wouldn't surprise me if he didn't, given he loves it so much and was the, er, 'administrator'.

I want to know where Abaddon fits in too. I don't think he's a grown up Walt, though - that's too strange and considering how fab they were at finding someone who looked like a young Swoosie Kurtz to play Locke's mum in 'Cabin Fever', he'd be a poor choice to look like a grown up Walt :/