I recently posted about the Jabber Problem Report Tool and the csf-unified.log that turned out to be one of my most popular posts. I had some feedback asking for more info on specific troubleshooting after home cluster is found, so here it is.

Service Discovery Failure Codes (csf-unified.log)

IDNameUI MessageDescription
1001ServiceDiscoveryFailureFailed to discover network servicesUnknown discovery failure
1002ServiceDiscoveryAuthenticationFailureYour username or password is not correct.Failed to authenticate with CUCM (9.0+)
1003ServiceDiscoveryCannotConnectToCucmServerCannot communicate with the server. Cannot connect to CUCM (9.0+)
1004ServiceDiscoveryNoCucmConfigurationFailed to discover network services.CUCM server is misconfigured
1005ServiceDiscoveryNoSRVRecordsFoundFailed to discover network services.No SRV records are found
1006ServiceDiscoveryCannotConnectToEdgeCannot communicate with the serverCannot connect to EDGE server
1007ServiceDiscoveryNoNetworkConnectivityCannot communicate with the serverNo network connectivity

Service Discovery Cache File (service-location.xml)

Located in %appdata%\Cisco\Unified Communications\Jabber\CSF\Config

<?xml version=”1.0″?>
<UCServices>
<DomainName>pnslabs.com</DomainName>
<UCService>
<type>CUCM</type>
<connectionInformation>
<name>_cisco-uds</name>
<scope>UNKNOWN</scope>
<address>cucm.pnslabs.com</address>
<protocol>_tcp</protocol>
<port>8443</port>
</connectionInformation>
</UCService>
</UCServices>

IMS Result Codes

Client Profile Agent Logs pulled from IM&P Server EPASSoapXXXXX.log

IMS Result CodeDescription
0Success
1Wrong credentials
2Administratively Locked
3Hack Locked
7End User status inactive in LDAP

MRA (Mobile and Remote Access) is a technology that allows phone registrations without raw sip signaling sent over the internet. I compare it to RPC over HTTPS proxy, aka “Outlook Anywhere.” Email came first, now I’m not sure if anyone remembers having to VPN in for Outlook to work.

After having several MRA projects back to back, I’ve found that most of the lessons I’ve learned were dealing with project management than the technical pieces. No other piece of hardware or software requires so many different groups of people: the datacenter guy, the firewall guy, the certificate guy, the dns guy, the voice guy. Throw in a business analyst and a project manager and it gets messy. I literally had projects ongoing for months and months when I was able to get MRA working in my lab in just a couple hours. Anytime I finish a project I look at how it could’ve been done better, so here it is:

First: It’s better to think of MRA as 5 separate, independent tasks because almost nothing depends on anything else.

  1. VM Deployment
  2. Networking
  3. Certificates
  4. DNS
  5. Firewall

Second: Clearly define roles and responsibilities per task at the onset of the project. The certificate guy doesn’t care about ip addresses or storage so just gather contact information and give role takers hyper-specific information to keep their attention. Limit emails.

Third: Make the configuration as cookie-cutter as possible. I wouldn’t even mention single NIC with some nat reflexion workaround, you use the advanced networking key with an outside and an inside interface. Public Certs on everything. It’s always a cluster, even if its a cluster of one; you never know when they will scale up. I assume inside out, everything is allowed through the firewall; this is usually the case and customers get confused with the lengthy cisco documentation. I usually give something like below:

Purpose

Protocol

Internet Endpoint (source)

Expressway-E (listening)

XMPP

TCP

>=1024

5222

XMPP federation

TCP

>=1024

5269

HTTP proxy (UDS)

TCP

>=1024

8443

SIP media

UDP

>=1024

36002 to 59999

SIP signaling

TLS

>=1024

5061

HTTPS (Unity)

TCP

>=1024

443

Also, I posted before about Flexible Jabber ID, which allows for multiple presence domains. That can cause holdups if you aren’t allowed to sign for these domains for your cup-xmpp certificate. It becomes an all-or-nothing service model. The best thing to do is get ahead of this and remove services from domains that you cannot sign for! You can run a whois lookup to ensure the technical contact has some relationship with the customer.

For a list of all jabber users and their last login time, you can run the following query:


run sql
select e.userid, cd.timelastaccessed from enduser as e, credentialdynamic as cd, credential as cr where e.pkid=cr.fkenduser and e.tkuserprofile=1 and e.primarynodeid is not null and cr.tkcredential=3 and cr.pkid=cd.fkcredential order by cd.timelastaccessed

This date is in something called “epoch” format. To convert to something that makes sense, you’ll need to use the following formula:

=(((B2/60)/60)/24)+DATE(1970;1;1)+(-5/24)

This is for Eastern Standard Time where -5 = UTC -5, you can change this accordingly.

Unlike other “report this problem” buttons, Jabber’s problem report doesn’t just fly into the cloud somewhere never to be seen again.

It actually contains a lot of useful information about things like: how services are discovered and assigned, what policies are defined, what devices are configured, etc. The problem report can be generated, but is located below:

\Users\AppData\Local\Cisco\Unified Communications\Jabber\CSF\Logs\csf-unified.log

One of the biggest questions I get is where configurations exist, and what takes priority ? Before looking at the file, I would have guessed 3 places, but as you can see below its actually 12!

1: TftpOverrideConfigStore [10] jabber-config.xml
2: LocalFileConfigStore [20] AppData\Roaming\Cisco\Unified Communications\Jabber\CSF\Config\jabberLocalConfig.xml
3: Ucm90ConfigStore [25] http://cucm.fqdn.com:6970/SPDefault.cnf.xml
4: PresenceAuthenticatorStore [28]
5: ServiceDiscoveryConfigStore [29]
6: PresenceConfigStore [30] Presence Server TFTP+CCMCIP Settings
7: RegistryConfigStore [31] HKCU\Software\Cisco Systems, Inc.\JabberWerxCpp
8: PresenceProductModeStore [35]
9: UrlProvisioningConfigStore [499]
10: BootstrapConfigStore [500] ProgramData\Cisco Systems\Cisco Jabber\jabber-bootstrap.properties
11: InMemoryConfigStore [600]
12: jabber-config-defaults.xml [2147483647] Program Files (x86)\Cisco Systems\Cisco Jabber\jabber-config-defaults.xml

Each of these stores contain configuration elements in the form of <tags>, retrieved from either the local machine, or through HTTP GET requests

When Jabber is launched, Jabber first tries to read jabberLocalConfig.xml for cached settings, if any exists. It then attempts to reach the domain controller of the machine you are running on for SSO. If both of those fail, only then will it give you a login prompt. If no cached username is found, email is prompted for and DISCOVERY begins.

retrieve email address and parse domain:

Getting LastLoggedInUserName from ConfigFeatureSet
Getting servicesDomain from ConfigFeatureSet
Getting PresenceDomain from ConfigFeatureSet
ServicesDomain key not found, parsing email address.
Retrieved Email Address jhalpert@car.pnslabs.com
Retrieved DNS Domain ‘car.pnslabs.com’ from ‘Email Address from UserInput’
Domain is set to: car.pnslabs.com

check for webex im:

makeCasLookupForDomain result is ‘Code: NOT_WEBEX_CUSTOMER; CasUrl: http://loginp.webexconnect.com/cas/FederatedSSO?org=car.pnslabs.com;

check SRV records for on-prem and mra:

*—–* DNS query _cisco-uds._tcp.car.pnslabs.com. has succeeded.
*—–* DNS query _collab-edge._tls.car.pnslabs.com. has failed: DNS name does not exist.

determine home cluster:

*—–* Making HTTP request to: https://cucm.car.pnslabs.com:8443/cucm-uds/clusterUser?email=jhalpert@car.pnslabs.com

fetch user data:

https://cucm.car.pnslabs.com:8443/cucm-uds/user/jhalpert
https://cucm.car.pnslabs.com:8443/cucm-uds/user/jhalpert/devices
https://cucm.car.pnslabs.com:8443/cucm-uds/user/jhalpert/credentials
https://cucm.car.pnslabs.com:8443/cucm-uds/user/jhalpert/extensions/
https://cucm.car.pnslabs.com:8443/cucm-uds/user/jhalpert/subscribedServices
https://cucm.car.pnslabs.com:8443/cucm-uds/user/jhalpert/speedDials
https://cucm.car.pnslabs.com:6970/SPDefault.cnf.xml
https://cucm.car.pnslabs.com:6970/global-settings.xml

set location:

setCurrentLocationOption start: description: Jeffs House ,id: fff8ffb0-1332-4da7-ba78-647b35aa25b2
IsGatewayAddressSubnetExists: gateway address: 00::50::56::E3::E5::50, subnet: 172.16.13.0, locations: 1

The default IM address scheme is to take the userid and append @defaultdomain to the end. This may work for smaller deployments, but caused a lot of confusion when multiple domains or subdomains were used. Flexible Jabber ID allows the IM address scheme to be mapped to Directory URI which is then mapped to either mail or msRTCSIP in LDAP.

jid

Changing the IM address scheme to Flexible JID is simple:

    • Stop the following services:
      Cisco Presence Engine
      Cisco SIP Proxy
      Cisco XCP Router
      Cisco Sync Agent
      Cisco Client Profile Agent
    • Presence > Settings > Advanced Configuration:
      Change IM Address Scheme to Directory URI and save.

imaddress

You can now browse to Presence > Domains to see a list of all domains in your presence environment.

You may find that some should not be there, as a lot of people like to add external contacts in their active directory with an ipPhone field populated. This will cause issues down the road if you deploy Mobile and Remote Access and want to do XMPP Federation. You’ll need to be able to sign for each domain. You’ll want to filter these out either by ldap filter into CUCM, or UC Service Profile. The Default Domain above also gets put into the CSR, so make sure to change that as well.

You can run the sql query below to clear the domains from the CSR once they are removed from the system:

run sql select * from impresencedomains

run sql delete from impresencedomains where pkid=’pkid_from_step_1′

To take advantage of Flexible Jabber ID, you must be running Jabber 10.6.6 or higher !

EDI (Enhanced Directory Integration) can use the logged on user domain credentials to search LDAP directory, or a saved set of credentials in either the Service Profile or jabber-config.xml via the <ConnectionUsername> and <ConnectionPassword> tags. It’s only supported with Jabber for Windows since it uses the built-in Windows API.

BDI (Basic Directory Integration) uses stored credentials in either the Service Profile or jabber-config.xml via the <BDIConnectionUsername> and <BDIConnectionPassword> tags to search LDAP directory. Storing credentials in the service profile is highly recommended, since the jabber-config.xml is plaintext.

When both service profile and jabber-config.xml define a parameter, the service profile will take priority.

UDS (User Directory Service) uses CUCM to provide directory lookup services. UDS is the only supported method when MRA is used, however because CUCM is now providing directory lookups and not LDAP, the load must be considered: a node can support UDS contact service connections for up to 50% of the maximum device registrations supported by the server.

jabber-directory