Hmmm, what citizenship are you? Australian? The UK issues work permits, not visas. Wouldn't you become a UK resident then? Your UK work permit means nothing in Switzerland, i.e., UK work permit does not give you rights of an EU citizen, if that is the point you were asking about.
