Unfortunately there is not a simple way to temporarily remove contacts from your lists at this time. The only way to remove a contact would be to delete them from your list.
