Open regedit and go to: HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\LowCache\Extensible Cache\DOMStore
Open the path shown in CachePath key in Windows Explorer and do a search for *.xml
Inspect the xml files and you'll see names of people you've resolved or searched for in the people picker
Delete the files as necessary.
Alternatively, if you want to disable the cache, in IE settings, go to the Advanced tab and uncheck the checkbox called "Enable DOM Storage".
Close and restart IE for good measure.
Is there a way to disable at the server level? We are writing a custom claims provider and want to restrict audiences based on site collection. I am concerned with the cache causing user confusion.