We use Outlook 2013 in cached mode together with Exchange 2013 CU5.
If somebody send an email from outlook and click on the "TO"-button to select a recipient, the address book appears without delay.
If i do the same, but send the email from word with "send as attachment" and click then on the "TO"-button, it takes sometimes till 5 minutes till the address book appears.
The exact same problem happens also for the users which use Outlook 2010.