Thursday, December 1, 2016

Skype4B / Lync Certificate Checker Tool

If you’ve ever installed Skype for Business or Lync before, you will know that the system requires PKI Infrastructure and Certificates to function. The reason for this is that all SIP and Web communications within the Skype for Business environment is secure by design and uses certificates for encrypting data. These communications span between servers, clients, phones, PSTN Gateways, Third Party Video equipment and most other integrations you can think of. So without your certificates being deployed properly, you are going to have a lot of trouble getting your environment up and running.

Skype for Business/Lync Edge servers communicate with each other over Mutual Transport Layer Security (MTLS). When using MTLS connections the server originating a message and the server receiving it exchange certificates from mutually trusted Certificate Authorities. The public certificates presented in either direction prove the identity of each server by being signed by a trusted certificate authority. The main thing here to note here is that both servers need to have root certificates installed from each other’s trusted root certificate authority in order for TLS connections to negotiate successfully. This is also the case for federated connections to other organisations via the Skype for Business Edge server. These connections all rely on MTLS for the successful communication between the servers.

Encryption Used in Skype for Business
Traffic type
Protected by
Server-to-server
MTLS
Client-to-server
TLS
Instant messaging and presence
TLS
Audio and video and desktop sharing of media
SRTP
(No Certificates Used)
Desktop sharing (signaling)
TLS
Web conferencing
TLS
Meeting content download, address book download, distribution group expansion
HTTPS
Mobile Clients (UCWA)
TLS

In many cases you may not have direct access to the other system you are connecting to in order to check whether the certificate it is using is valid, or has been signed by a trusted root certificate Authority. As a result, you may have issues connecting to the server and need to use complex tools like Wireshark to determine what the certificate being presented by the far end looks like. This can take time and involve installing software on servers, so I wanted to create a simple tool that doesn’t require any installation and can be run straight from a Powershell prompt. After doing some coding, that’s exactly what I created, introducing the Skype for Business Certificate Checker Tool…


Skype4B / Lync Certificate Checker Tool




Features:
  • Check the certificate being used by a server using the FQDN/IP and Port number of the server.
  • Check the certificate of a Federation SRV record (_sipfederationtls._tcp.domain.com) simply by entering the SIP domain name and ticking the “FED SRV” checkbox.
  • Check the SIP SRV record (_sip._tls.domain.com) by simply entering the SIP domain name and ticking the “SIP SRV” checkbox.
  • Check the SIP Internal SRV record (_sipinternaltls._tcp.domain.com) by simply entering the SIP domain name and ticking the “SIP INT SRV” checkbox.
  • Select the DNS server you would like to use to resolve DNS from by entering a DNS Server IP address in the “DNS Server” field.
  • “Show Advanced” checkbox will show all of the information in the certificate.
  • The “Show Root Chain” will display the root certificate and all of the intermediate certificates that are applicable in the trust chain for the certificate.
  • The “Test DNSLB Pool” checkbox is on by default and will instruct to the tool to test all of the IP Addresses that are resolved for a DNS Name. In the case of Skype for Business, we nearly always have multiple DNS records per A record for the purposes of DNS load balancing.  Rather than having to look all of the servers yourself, the tool will do this for you. Other servers in pool will be displayed in the Information text box in blue colour and will be tested directly via their IP Address rather than the DNS name.
  • Import multiple DNS name records from a CSV file. This is useful if you want to check a lot of servers in one sitting. See the “Import File Format” section for more details.
  • Save certificate information out to a CSV file. This will save all of the certificate information out in table format that you can open in Excel for record keeping purposes. Note:This export format is different than the one used in conjunction with the “Import” button.
  • Comments section – The comments section will have information in it about things that may be wrong with the certificate to help you troubleshoot your issues.

Available on TechNet Gallery:




Import File Format


You can import a CSV file containing many domains and servers to test if you choose (for example, this may be useful for checking a large list of federated domains). To do this you will first need to create a CSV file with all of the servers and/or domains that you want to test in it. The format of the CSV for each of the record types will look like:
·        
Header row: Domain,Type,Port
Example Federation Record: "microsoft.com","FED","",
Example SIP Record:"microsoft.com","SIP","",
Example SIP Record: "microsoft.com","SIPINT","",
Example direct Record: "sip.microsoft.com","DIR","5061",

Example file:
Domain,Type,Port
"microsoft.com","FED","",
"microsoft.com","SIP","",
"microsoft.com","SIPINT","",
"sip.microsoft.com","DIR","5061"


The Anatomy of a Certificate


The Certificate Checker Tool can display a few different levels of information about the certificate presented by the server. The default basic view of the certificate will be displayed in the tool as follows:


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Checking FQDN:  sipfed.microsoft.com:5061
Checking IP Address: 167.220.67.163:5061

Certificate Response:

Subject: CN=sipfed.microsoft.com, OU=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=WA, C=US
Issuer: CN=Microsoft IT SSL SHA2, OU=Microsoft IT, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
Not Before: 30/04/2015 2:26:22 PM
Not After: 29/04/2017 2:26:22 PM
Serial Number: 5A0000F5B0C7CABB89E4624D1E00010000F5B0
Signature Algorithm: sha256RSA
Thumbprint: 9E1736ACA8C9731798E7FD3496E7D78454A02E80
Version: 3
HasPrivateKey: False

----------------------------------------------------------------------------------
Comments:

- Common Name Match found
- FQDN is in SAN list.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


For a Skype for Business or Lync deployment the most important components here are the Subject name, the Not Before and Not After dates. The “Comments” section is provided by the tool to help you troubleshoot issues with the certificate being displayed. This section will automatically check things like the certificate being out of date, the common name/subject alternate names being correct, if there is a Server EKU, and if the certificate has a CLR list included. These comments should help speed up your troubleshooting of certificate issues. Note: the comments will actually be based on all the advanced certificate details, even though the Advanced checkbox is not ticked by default.

Advanced Settings


All certificate details are displayed when the “Advanced” check box is ticked. When you tick this check box and run the tool you will see information as shown below:


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Checking FQDN:  sipfed.microsoft.com:5061
Checking IP Address: 167.220.67.163:5061

Certificate Response:

Subject: CN=sipfed.microsoft.com, OU=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=WA, C=US
Issuer: CN=Microsoft IT SSL SHA2, OU=Microsoft IT, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
Not Before: 30/04/2015 2:26:22 PM
Not After: 29/04/2017 2:26:22 PM
Serial Number: 5A0000F5B0C7CABB89E4624D1E00010000F5B0
Signature Algorithm: sha256RSA
Thumbprint: 9E1736ACA8C9731798E7FD3496E7D78454A02E80
Version: 3
HasPrivateKey: False
Archived: False

Extension Type: Certificate Policies
Oid Value: 2.5.29.32
Data:
[1]Certificate Policy:
     Policy Identifier=1.3.6.1.4.1.311.42.1
     [1,1]Policy Qualifier Info:
          Policy Qualifier Id=CPS
          Qualifier:
               http://www.microsoft.com/pki/mscorp/cps

Extension Type: Application Policies
Oid Value: 1.3.6.1.4.1.311.21.10
Data:
[1]Application Certificate Policy:
     Policy Identifier=Server Authentication
[2]Application Certificate Policy:
     Policy Identifier=Client Authentication

Extension Type: Key Usage
Oid Value: 2.5.29.15
Data:
Digital Signature, Key Encipherment, Data Encipherment (b0)

Extension Type: Enhanced Key Usage
Oid Value: 2.5.29.37
Data:
Server Authentication (1.3.6.1.5.5.7.3.1)
Client Authentication (1.3.6.1.5.5.7.3.2)

Extension Type: Subject Key Identifier
Oid Value: 2.5.29.14
Data:
df 62 d3 a8 ef 49 3d 2f ed 10 aa 6a 30 3a 6f f9 54 1b 33 39

Extension Type: Authority Key Identifier
Oid Value: 2.5.29.35
Data:
KeyID=51 af 24 26 9c f4 68 22 57 80 26 2b 3b 46 62 15 7b 1e cc a5

Extension Type: CRL Distribution Points
Oid Value: 2.5.29.31
Data:
[1]CRL Distribution Point
     Distribution Point Name:
          Full Name:
               URL=http://mscrl.microsoft.com/pki/mscorp/crl/msitwww2.crl
               URL=http://crl.microsoft.com/pki/mscorp/crl/msitwww2.crl

Extension Type: Authority Information Access
Oid Value: 1.3.6.1.5.5.7.1.1
Data:
[1]Authority Info Access
     Access Method=Certification Authority Issuer (1.3.6.1.5.5.7.48.2)
     Alternative Name:
          URL=http://www.microsoft.com/pki/mscorp/msitwww2.crt
[2]Authority Info Access
     Access Method=On-line Certificate Status Protocol (1.3.6.1.5.5.7.48.1)
     Alternative Name:
          URL=http://ocsp.msocsp.com

Extension Type: Subject Alternative Name
Oid Value: 2.5.29.17
Data:
DNS Name=sipfed.microsoft.com
DNS Name=sipalt.microsoft.com
DNS Name=sip.microsoft.com
DNS Name=ra30.sbweb.microsoft.com
DNS Name=web3.sbweb.microsoft.com
DNS Name=web30.sbweb.microsoft.com
DNS Name=web31.sbweb.microsoft.com


----------------------------------------------------------------------------------
Comments:

- Common Name Match found
- FQDN is in SAN list.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


You will have a full view of all attributes contained within the certificate. Using this information you should be able to troubleshoot most certificate related issues. However, there is one more important piece of information that you might need and that is the certificate chain…

The Certificate Chain


The certificate chain is the hierarchy of Certificate Authority servers from the CA server that issued the certificate through the Intermediate Certificate Authorities to the Root Certificate Authority server. The tool will display the certificate chain as follows:


Certificate Chain:

Certificate Chain Item 1
Chain Subject Name: CN=sipfed.microsoft.com, OU=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=WA, C=US
Chain Issuer name: CN=Microsoft IT SSL SHA2, OU=Microsoft IT, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
Chain Not Before: 11/12/2015 05:36:48
Chain Not After: 11/11/2017 05:36:48
Chain Serial Number: 5A000233C22F738FDBCE9CF8B50001000233C2
Chain Signature Algorithm: sha256RSA
Chain Thumbprint: A710B806065C2187E387635A9F8D7863A63D702A
Chain Version: 3
Chain is valid: True

Certificate Chain Item 2
Chain Subject Name: CN=Microsoft IT SSL SHA2, OU=Microsoft IT, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
Chain Issuer name: CN=Baltimore CyberTrust Root, OU=CyberTrust, O=Baltimore, C=IE
Chain Not Before: 05/08/2014 03:04:09
Chain Not After: 05/08/2018 03:03:30
Chain Serial Number: 0727AA47
Chain Signature Algorithm: sha256RSA
Chain Thumbprint: 97EFF3028677894BDD4F9AC53F789BEE5DF4AD86
Chain Version: 3
Chain is valid: True

Certificate Chain Item 3
Chain Subject Name: CN=Baltimore CyberTrust Root, OU=CyberTrust, O=Baltimore, C=IE
Chain Issuer name: CN=Baltimore CyberTrust Root, OU=CyberTrust, O=Baltimore, C=IE
Chain Not Before: 05/13/2000 04:46:00
Chain Not After: 05/13/2025 09:59:00
Chain Serial Number: 020000B9
Chain Signature Algorithm: sha1RSA
Chain Thumbprint: D4DE20D05E66FC53FE1A50882C78DB2852CAE474
Chain Version: 3
Chain is valid: True

----------------------------------------------------------------------------------
Root Certificates:

- Get Root Certs here: http://cybertrust.omniroot.com/support/sureserver/rootcert_iis.cfm
- Download Root Cert: http://cacert.omniroot.com/bc2025.crt

----------------------------------------------------------------------------------


The tool will show you the Subject Name, Issuer Name (which will be the next server in the list/chain), Element Signature Algorithm (can be important, see below), and whether or not the chain is valid.

In addition to displaying the certificate chain the tool will also, where possible, provide a link to a copy of the root certificate for the root CA being used. The tool knows about the major certificate authorities supported by Microsoft for use with Skype for Business and in most cases will give you a direct link to the root certificate for download.

Root Certificates


One very important thing when configuring external federation with partners or public providers is that MTLS is used for these connections. This means that both ends of the connection need to trust the other’s root certificates. You need to ensure that your edge servers have the root certificates of your partners installed. Fortunately, the Cert Checker Tool has you covered here by showing you where you can download the root certificates for common public certificate authorities. This will appear like shown below:

- Get Root Certs here: http://cybertrust.omniroot.com/support/sureserver/rootcert_iis.cfm
- Download Root Cert: http://cacert.omniroot.com/bc2025.crt

Before you configure a new partner for federation, you can you use the tool to check what certificate authority they are using for their certificates and as a result which root certificates you need installed on your edge servers.

There is also neat trick you can do to automagically install root certificates on a Windows server or PC (post Windows Vista). Note there is a caveat with this process whereby the third party server must be using a Root Certificate Authority that is trusted by Microsoft as part of their Trusted Root Certificate Program (Microsoft supported root CAs can be confirmed on this list). If this is the case then you just need to browse to a web server that is signed by the root certificate authority of choice and Windows will automatically install the root certificate for you! These root certificates are pushed to Window through Windows Update and will be installed only when you try to connect to a website requiring a particular certificate. So connecting to a federated partner's "dialin.domain.com" web page from all of your Edge servers may be enough to download the root certificates for MTLS trust purposes. There is a lot of documentation about this process on TechNet if you would like to know more. A few Skype for Business community have also written about this phenomenon - Chris and Pat.

The Wrap Up


I hope that this new tool finds you well, and I hope that you have many long years of troubleshooting together. Remember, whilst the flame may flicker from time to time, you must stay strong and think fondly of those times in the early days when you hired the car, threw the work laptop in the boot, and drove to the cabin in the woods; not even one bar of 3G internet access could stop you from fixing that server certificate problem. It’s the memory of those times that will keep you on the straight and narrow when that younger and fancier tool with the sexy universal windows app GUI comes along. Your Powershell Certificate Checker will always be faithful, remember that… now get testing!



Read more →

Wednesday, October 26, 2016

Polycom VVX Hold Issue with Sonus SBC1000/2000 Calls

I had an issue recently when using the VVX Music on Hold feature in conjunction with a Sonus SBC1000/2000. The problem was that the VVX phones could not put a trunk call on hold (in this case they were running 5.4.5 software) . The symptom of this issue was the following message being displayed on the VVX screen (“Hold Failed. Moving back to call”):



It should be noted here that the VVX in question had the new Music on Hold feature configured and MoH was working to internal Skype for Business clients and other phones. This was only happening on trunk calls on the Sonus gateway.  This would happen for both Sonus based file hold music (ie. Sonus gateway based hold) or when the Sonus was not being used to supply the hold music (ie. hold passed through from the VVX to the outside party).

Logging on the system showed that the RE-INVITE message being sent by the mediation server to initiate hold was getting rejected by the Sonus SBC. The INVITE message that was being sent looked like this:

RE-INVITE message sent by Mediation Server to Sonus Gateway:
INVITE sip:+61395821300@10.20.1.150:5066;transport=TCP SIP/2.0
FROM: <sip:+61395824500;ext=4500@myskypelab.com;user=phone>;epid=BB69FBB4BE;tag=45afae0d8
TO: <sip:+61395821300@SBC1000GW.myskypelab.com;user=phone>;tag=7b66d712-5fc
CSEQ: 83 INVITE
CALL-ID: a9adec59-5f57-4a9c-ac31-78836eb506a3
MAX-FORWARDS: 70
VIA: SIP/2.0/TCP 10.20.2.104:49796;branch=z9hG4bKbafabbee
CONTACT: <sip:2013ENTFE004.myskypelab.com:5068;transport=Tcp;maddr=10.20.2.104;ms-opaque=7b0cff35b7212cae>
CONTENT-LENGTH: 246
SUPPORTED: timer
SUPPORTED: 100rel
USER-AGENT: RTCC/5.0.0.0 MediationServer
CONTENT-TYPE: application/sdp
Session-Expires: 1800
Min-SE: 90
v=0o=- 1477456907 1477456908 IN IP4 10.22.0.23s=sessionc=IN IP4 10.22.0.23t=0 0a=sendonlym=audio 60034 RTP/AVP 0 101c=IN IP4 10.22.0.23a=rtcp:60035a=sendonlya=rtpmap:0 PCMU/8000a=rtpmap:101 telephone-event/8000a=ptime:80


You will notice in the above invite that the Mediation server is requesting a Payload Size of 80ms (“ptime:80”). The Sonus gateway doesn’t like this though, and rejects the call with a 488 message:

Call setup rejected:
SIP/2.0 488 Not Acceptable Here
FROM: <sip:+61395824500;ext=4500@myskypelab.com;user=phone>;epid=BB69FBB4BE;tag=45afae0d8
TO: <sip:+61395821300@SBC1000GW.myskypelab.com;user=phone>;tag=7b66d712-5fc
CSEQ: 83 INVITE
CALL-ID: a9adec59-5f57-4a9c-ac31-78836eb506a3
VIA: SIP/2.0/TCP 10.20.2.104:49796;branch=z9hG4bKbafabbee
CONTENT-LENGTH: 0
WARNING: 304 "Media type not available"
SERVER: SONUS SBC1000 5.0.4v415 Sonus SBC

When the 488 message gets fed back to the VVX, it displays the “Hold Failed. Moving back to call” message and everyone is very sad :(

Why is this happening?


The Sonus gateway supports a maximum payload sizes of 60ms. This can be seen in INVITE messages sent out of the Sonus like the one below:

v=0
o=SBC 79 1001 IN IP4 10.20.1.150
s=VoipCall
c=IN IP4 10.20.1.150
t=0 0
m=audio 20112 RTP/AVP 0 8 101 13
c=IN IP4 10.20.1.150
a=rtpmap:0 PCMU/8000/1
a=rtpmap:8 PCMA/8000/1
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=rtpmap:13 CN/8000
a=ptime:20
a=maxptime:60
a=sendrecv

As you can see in the SDP message above, the Sonus says that it will only handle a maximum payload size of 60ms. There is also a reference to this in the SBC1k/2k manuals, which state that “If the SBC receives a larger than configured payload size from the peer offer in the re-invite, the SBC rejects with a 488 'Not Acceptable Here' response. The call rolls back to the previous negotiated offer answer.” – Reference: https://support.sonus.net/display/UXDOC60/Creating+and+Modifying+Voice+Codec+Profiles

The Fix!


So the fix turns out to be a VVX phone setting. When initiating hold the VVX the phone doesn’t follow the regular payload size settings for the codec being used (which by default for PCMA/U is 20ms). Instead it has a setting called feature.moh.payload which defaults to 80ms. Fortunately for us, this setting can be changed within the configuration file (it’s not available in the web interface!). The setting for Music on Hold are shown below:

Setting
Default Value
Required Value
feature.moh.enabled
0
1
feature.moh.filename
NULL
Set this to the hold music file name. The file will be served off the config server. The default file is “Polycom-hold.wav”
feature.moh.payload
80
20 or 40 or 60
res.quotas.tone
600
600 – 1024

Note: The default moh file size is 600KB which works fine for the default hold music provided by Polycom. However, if you want to change this to your own file then you can expand the size up to 1024KB.

Polycom has made the payload setting 80ms by default to reduce the processor usage on the phone when it plays the hold file. So selecting 60ms may be the best option from a processor load perspective (which will work with the Sonus max payload size of 60). This way the hold music will work correctly, without putting too much load on the VVX's CPU.

Here is an example of how this might look in a config file:

<feature feature.moh.enabled="1" feature.moh.filename="Polycom-hold.wav" feature.moh.payload="60" />
<res res.quotas.tone="1000" />


The Wrap Up 


Well there you go, another weird integration issue to add to the list… Hopefully it didn’t wreak too much havoc in your deployments before finding this article. Until next time, happy hacking.



Read more →

Thursday, July 14, 2016

Skype for Business and Lync Centralised Event Viewer Tool

Event Viewer logs remain one of the best troubleshooting tools for Lync and Skype for Business servers. An enormous amount of useful information can be found in the Event Viewer Logs, which can then be used to either understand the current state of the system or do root cause analysis on prior issues.

So I decided to build a simple tool that centrally displays all of the Event Logs from Lync or Skype for Business servers or pools within an environment. This allows for a fast one-stop-shop for triaging issues across multiple Lync/Skype for Business servers in your environment. This can be especially handy for easily correlating problems that might have occurred across multiple servers in a pool.


Skype for Business and Lync Centralised Event Log Tool



Features:

  • Start and End Time – You can select Start and End date and time for events (yy/mm/dd hh:mm format). This can be particularly handy if you know the approximate time that an issue occurred on the system.
  • Server – Select specific servers or entire pools for which you would like to see the Events Logs. By default all Lync Front End servers are selected (ie. “ALL FRONTENDS”).
  • Event level selection – Select if you want to see Critical, Error, Warning, and Information event messages.
  • EventID Include/Exclude – By default, these fields can be left blank. However, you can select to either include or exclude specific event numbers when you get events. These fields accept Regular Expression formatting. For example, if you entered “^5”, you will get all events that start with “5”, or you could get specific multiple events using "|" like this “31147|31202”, or your could get a range of events using "[]" like this "620[0-9][0-9]".
  • Find Next / Previous – You can jump forward or backwards through the event list. The find feature checks all cells in each row against the text in the find text box (this is based on regular expression). If you don't put any text in the find textbox the find next and previous buttons can also be used to step through each event in the list.
  • The Get Events button will start the process of getting the specified events from the selected servers.
  • Web Search! – Often the easiest method to find more information on events is to search on the web for them. So the tool gives you the option to Google, Bing, Duck Duck Go or search TechNet Forums for more information.
  • View Event - This button will open a window with the event in larger text box. This helps in some cases when the event message text will not fit within a row in the main window.
  • The Copy button will copy the selected row to the clipboard so you can paste it somewhere, like an email for example.
  • Export Statistics to CSV – The Export Stats button will export per event counts to a CSV file. This will quickly allow you to see which events have been occurring more than others.
  • Export Events to CSV – You may want to have record of these 'momentous' events for future reference.
1.01 Small Update
  • Added standalone mediation servers to the list.
1.02 The Greig Request Update (15/7/2018)
  • Added options for searching "ALL FRONTENDS", "ALL FRONTENDS and SBAs", and "ALL SBAs".
  • Added improved performance (2x-3x improvement) of "Include Event ID" filtering when "41024,41033" format is used (Note, only works for Include and not Exclude Event ID). Using server side filtering for this improvement (also saves CPU cycles on client machine).


Available on the TechNet Gallery:


Tool Requirements


The Centralised Event Viewer Tool must be run on a server with the Lync / Skype for Business module installed on it (as it uses Powershell commands to find the Lync/Skype4B pool information). This would usually be a front end server in the pool. The tool is capable of listing large numbers of events (tens-of-thousands of events), however, getting large numbers of events can take a while to process. The tool will process 1000 events in 2 seconds (this scales fairly linearly). As a rule though it’s usually best to keep searches under a month in length so that the number of events don’t become problematic.

You must enable “Remote Event Log Management (RPC)” on all of your Lync/Skype for Business servers Windows firewalls in order to access these logs from the central server running the tool. This rule is already pre-populated in the Windows Firewall Advanced setting rules. So you simply need to Enabled the rule as shown below:

Open Firewall on all Lync / Skype for Business Servers:



This is a dynamic service rule that opens the required ports automatically. However, the ports that get used in practice are port TCP 135 (RPC) and port TCP 49153 (Remote Event Log). These firewall rules will become more important if you are trying to connect to Edge servers from an internal server, as the firewall between the servers will need to allow these ports.

Once this has been set on the servers that you are getting event logs from, you are set to go!

The Wrap Up


Why are you still reading? Go and get event log troubleshooting!!



Read more →