Unfortunately, the 'contact information' (i.e. email address and phone number) is stored individually of the billing- or shipping address, as it's user-based, not order-based.
It's too much of a fringe-case to build a custom fix for, sorry!
My solution would be to just change your phone number whenever you move from one country to the next.
