MSCRMEmail: Metadata contains a reference that cannot be resolved

Below is an error that can appear when you are running the CRM email router on the same server as your CRM web services. I experienced it after adding a second web site on the CRM server to run a portal that is back-ended by CRM. The error was also accompanied by high CPU and memory usage as well.

Log Name:      Application
Source:        MSCRMEmail
Date:          1/15/2013 8:10:43 AM
Event ID:      26234
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      CrmServer.domain.com
Description:
#26234 - The E-mail Router service could not process a provider work item using assembly: Microsoft.Crm.Tools.EmailProviders.dll and class: Microsoft.Crm.Tools.Email.Providers.SmtpPollingSendEmailProvider. System.InvalidOperationException: Metadata contains a reference that cannot be resolved: 'https://crm.domain.com/organization/XrmServices/2011/Discovery.svc?wsdl'. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
   at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.ConnectStream.WriteHeaders(Boolean async)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.GetResponse()
   at System.ServiceModel.Description.MetadataExchangeClient.MetadataLocationRetriever.DownloadMetadata(TimeoutHelper timeoutHelper)
   at System.ServiceModel.Description.MetadataExchangeClient.MetadataRetriever.Retrieve(TimeoutHelper timeoutHelper)
   --- End of inner exception stack trace ---
   at Microsoft.Crm.ServiceProxyCache`1.GetNewServiceProxy(Uri serviceUrl, Credential credentials, Uri homeRealmUrl, String passportEnvironment, String onlineServiceEnvironment)
   at Microsoft.Crm.Tools.Email.Providers.Utility.GetOrganizationUrl(Uri discoveryServiceUrl, Credential credentials, String organizationName)
   at Microsoft.Crm.Tools.Email.Providers.Utility.GetCrmService(Uri discoveryUri, String authMode, String userName, String password)
   at Microsoft.Crm.Tools.Email.Providers.CrmPollingSendEmailProvider.Run()
   at Microsoft.Crm.Tools.Email.Agent.ServiceCore.ExecuteProviderWork(Object providerQueueRequestObject)
Event Xml:

  
    
    26234
    2
    0
    0x80000000000000
    
    27728919
    Application
    CrmServer.domain.com
    
  
  
    #26234 - The E-mail Router service could not process a provider work item using assembly: Microsoft.Crm.Tools.EmailProviders.dll and class: Microsoft.Crm.Tools.Email.Providers.SmtpPollingSendEmailProvider. System.InvalidOperationException: Metadata contains a reference that cannot be resolved: 'https://crm.domain.com/organization/XrmServices/2011/Discovery.svc?wsdl'. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
   at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.ConnectStream.WriteHeaders(Boolean async)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.GetResponse()
   at System.ServiceModel.Description.MetadataExchangeClient.MetadataLocationRetriever.DownloadMetadata(TimeoutHelper timeoutHelper)
   at System.ServiceModel.Description.MetadataExchangeClient.MetadataRetriever.Retrieve(TimeoutHelper timeoutHelper)
   --- End of inner exception stack trace ---
   at Microsoft.Crm.ServiceProxyCache`1.GetNewServiceProxy(Uri serviceUrl, Credential credentials, Uri homeRealmUrl, String passportEnvironment, String onlineServiceEnvironment)
   at Microsoft.Crm.Tools.Email.Providers.Utility.GetOrganizationUrl(Uri discoveryServiceUrl, Credential credentials, String organizationName)
   at Microsoft.Crm.Tools.Email.Providers.Utility.GetCrmService(Uri discoveryUri, String authMode, String userName, String password)
   at Microsoft.Crm.Tools.Email.Providers.CrmPollingSendEmailProvider.Run()
   at Microsoft.Crm.Tools.Email.Agent.ServiceCore.ExecuteProviderWork(Object providerQueueRequestObject)
  

The problem stems from having the IIS website that CRM runs listening on a specific IP address, and not "All Unassigned". When you run the site under a specified IP binding, it no longer listens on the loopback IP 127.0.0.1. Consequently, the external email router service can no longer contact the CRM web services on that loopback connection.

Resolution:
To fix it, you have to set the bindings for the web site that CRM 2011 is running on to "All Unassigned" instead of the specified IP address that it's set to. To do that,

1. Open the Internet Information Services (IIS) Manager.
2. Click on Sites.
3. Then click on your CRM web site.
4. Click Bindings.
5. Click Edit.
6. Set the IP address to "All Unassigned".
7. Click OK.

Option#2
Another possible issue is that the account that is running your CRM router service is having authentication issues. Make sure it isn't locked out, expired, or experiencing ADFS issues, and that it has all of the rights it needs in CRM. Check the account in the email router config that is used to connect to CRM too.