Sunday, 12 August 2018

Polycom VVX Not Displaying PIN Authentication Option

I had an interesting issue with a Polycom VVX deployment recently that I thought I would share in case others run into the same issue.

The Issue

The symptom of the problem was that after the Polycom VVX had completed booting, including getting an IP Address and downloading software/config files, the PIN Authentication option did not appear on the sign-in options screen. This meant that I was unable to use PIN Authentication at all for signing in the devices which was a problem because we planned on using it for all the phones. Below is an example of what the screen looked like:

Screen without PIN Auth option


There was a series of steps that I went through in troubleshooting this issue. I will take you through all of them so you too can check whether your issue might be solved with some of the earlier steps that I tried before reaching a resolution.

I first confirmed that PIN Authentication was in fact turned on in the configuration file(s) of the phone. To do this I checked that the following setting was not in the configuration files:

<!-- Disable PIN Auth by setting "0" -->
<reg reg.1.auth.usePinCredentials="0" />
Note: The phone can have multiple configuration files that are both manually added by administrators and automatically created by the phone (ie. <MAC>-phone.cfg, <MAC>-web.cfg, etc). You need to check all of the files associated with the phone's MAC address to ensure it’s not being overridden by another file.

I also checked the setting directly in the phone using my VVX Phone Manager Tool to get the active setting out of the phone using the REST interface. In my case this setting was not configured in the config file and it defaults to being on (ie. set to "1"). So this wasn't the problem.

I checked that PIN Authentication was actually enabled on the Skype for Business server. This can be done in the Control Panel > Security > Web Services > Pin Authentication Enabled:

This was also enabled - so in this case it wasn't the problem.

I tested the PIN Authentication process on the server by running Test-CsPhoneBootstrap PowerShell command on the system. This worked just fine:

PS C:\ > Test-CsPhoneBootstrap -PhoneOrExtension 4500 -PIN 12345 -TargetFqdn -TargetUri

Target Fqdn   :
Target Uri    :
Result        : Success
Latency       : 00:00:01.2333041
Error Message :
Diagnosis     :

In this deployment there was a centralised Windows Server that was serving DHCP to all the client subnets. On the central DHCP server I confirmed that all of the DHCP options were correct using my Skype4B/Lync DHCP Config Tool. This tool parses the byte format Vendor Options and displays them as readable text, and if it is unable to parse the byte format it will display an error:

This is an example image from my lab

In this case, all settings were displayed and no encoding issues were detected by the tool, which means this wasn’t the issue. So I checked that there was no DHCP server on a closer subnet (ie. a switch or router) that was responding to DHCP before the central Window DHCP server. This also wasn’t the case as I could see that the central DHCP server had logged the address lease for the Polycom VVX with the particular MAC Address of the test device.

At this point this was starting to look like a more complex problem so I took to the lab to see if I could reproduce such behaviour.  I noticed that after a factory default the phone initially didn’t display the Pin Authentication option for a couple of seconds - it appeared belatedly. This indicated to me that there was some additional check that was being done by the VVX before it would display this option. So this begged the question: what is required for PIN Authentication to function on the VVX? The most important thing that is required is that the phone gets the DHCP Options which tell it where the Cert Provisioning services resides, so it can communicate with the web services required for PIN Authentication.

Given that the VVX phones were being issued IP Addresses via DHCP, it didn’t seem likely to be a connectivity issue between the VVX and the DHCP server. However, I looked into the traffic flows to confirm this and found something interesting. In this Wireshark capture, you can see that the DHCP Options get sent out in response to an INFORM message that the VVX sends. The INFORM message is a special DHCP message that is outside of the initial DHCP IP Address discover process (DISCOVER > OFFER > REQUEST > ACK). The interesting thing about the INFORM message is that the ACK for this message from the DHCP server gets sent as a Unicast response directly back VVX itself rather than to the DHCP Relay IP Address, unlike all the other messages. The screen shot below also shows this from the DHCP server perspective - you can see the final ACK message has a Destination IP Address of the VVX instead of the DHCP relay IP Address:

The highlighted INFORM ACK message in Wireshark shows that it contains all the additional Microsoft specific Vendor Class Options (Certificate Provisioning Service details). It’s the packet that has the information that the VVX needs to get PIN Authentication working.

In this case, because a centralised DHCP was being used, the broadcast DHCP messages on the local subnet were being changed into unicast messages by the local router and sent over to the central DHCP server. There was also a firewall in between this local router and the centralised DHCP server. This meant that because the returning INFORM ACK message was sent directly back to the phone (which is part of the DHCP specification and is correct operation) the firewall had not created a UDP flow for it and the packet gets blocked. The diagram below shows how the DHCP traffic flow works with a DHCP Relay in place and where the issue resides:

As you can see from the diagram above, the firewall appears to be allowing traffic from the DCHP relay through to the DHCP server. After transiting the DCHP relay, the DHCP traffic flows from source port 67 to destination port 67. Then the INFORM ACK message then gets sent back from source port 67 on the DHCP server to destination port 68 on the VVX -  which the firewall did not have an existing flow for and it dropped the packet. As a result, the VVX didn’t receive its required Cert provisioning service URL and because of this didn't display the PIN Authentication sign-in button.

The Solution

So the solution here, as it often is, is firewall related. In this case we had to allow port 68 from the DHCP server IP Address to all the VVX phone subnets. After this was done the INFORM ACK messages could flow as required for the VVX to get its Vendor Class options.

If you don’t have access to the firewall or you need a quick solution to the problem, you can hard code the data contained in the Vendor Class options into the phone. This was added as a config option in software version 5.3. The configuration item is shown below:

<dhcp dhcp.option43.override.stsUri="" />

This can also be set in the web interface of the phone in the Settings > Provisioning Server > DHCP Menu > DHCP Option 43 Override STS-URI:

The Wrap Up

For all the old-school UC people out there, let's finish with a Haiku in the style of the old Lync 2010 powershell blog:

Firewalls drop packets,
This causes many issues,
Switch off all firewalls.

Till next time, see ya! 

Read more →

Monday, 26 March 2018

Building an Edge Server Port Monitor with Azure Function Apps – Part 3

This blog is an expansion on the previous Part 1 and Part 2 posts found here and here. The process of setting up the Function App for this part 3 section is the same as was documented in Part 1 and 2.  I suggest you head over to the first two parts and give them a good read before moving onto this post.

In part 3 we will be adding to the Function App so it can save data over time that we can use to graph and manipulate in Power Bi. To do this we will expand the application to do the following:
  • Save all port test results to Azure Storage tables for analysis.
  • Use Power Bi to connect to Azure Storage Tables and create nice graphs showing the status of Edge servers over a period of time.

Just like in Part 2 we will use the Azure Storage Tables module for Powershell to allow our application to keep both a short term memory for logging errors as well as a long term memory for logging all port testing attempts.

Step 1
To begin with follow Steps 1 through 5 of Part 2.

Step 2 – Download a copy of the script
You can grab a copy of the script I wrote for part 3 from here:

Step 3 - Update variables

Update the Storage Account details in the Powershell Script.

IMPORTANT: These settings are slightly different than in Part 2. In this case we have 2 different storage tables: one of them stores the current state of edge servers (same as part 2) and the other one stores all attempts into a separate table (which we will use for analysis in Power Bi):
$subscriptionName = "Visual Studio Premium with MSDN"  #SUBSCRIPTION NAME
$resourceGroup = "EdgePortTester-Part003"      #RESOURCE GROUP
$storageAccount = "edgeporttesterp8dbf" #STORAGE ACCOUNT NAME
$tableName = "EdgeTesterTablePart3Current"      #CHOOSE A NAME 1
$tableName2 = "EdgeTesterTablePart3Results"      #CHOOSE A NAME 2
$partitionKey = "EdgeTesterStoragePart3Current"      #CHOOSE A NAME 1
$partitionKey2 = "EdgeTesterStoragePart3Results"      #CHOOSE A NAME 2
$storageAccountKey = "7asdkjhasd7KHDKJHAS0dsflasdnnlasd099asdpncsdlknclLJSDLjbadksdjbfa9su9duhoasivRqXA615jQ=="             #STORAGE ACCOUNT > ACCESS KEYS

Don’t forget to fill in your Mail Jet email account information (as you did in Part 1) and add your Skype for Business Edge server's details. See Part 1 for more details. Enter your Mail Jet API Key (Username) and Secret Key (Password) and paste them into the following section of the script:

$emailUsername = "kjh3k23h4kjhkj37573f8f020879dff7"     
$emailPassword = "9898f98fhdjkkdjh46cd418100075a3b"

Edit the Skype for Business Edge server details as required. These are entered as an array of hash tables. The sections highlighted in yellow can be changed. In this case the application is monitoring 2 Edge servers, one in Melbourne and one in Sydney.

Access Edge
Web Conferencing
AV Edge
Access Edge
Web Conferencing
AV Edge
Note: The script only supports testing TCP ports at this time.

$Records = @(@{"Location" = "Melbourne"; "ServerName" = ""; "ServerRole" = "Federation"; "DestinationPort" = "5061"; "Protocol" = "TCP"})
$Records += @(@{"Location" = "Melbourne"; "ServerName" = ""; "ServerRole" = "Access Edge"; "DestinationPort" = "443"; "Protocol" = "TCP"})
$Records += @(@{"Location" = "Melbourne"; "ServerName" = ""; "ServerRole" = "Web Conferencing"; "DestinationPort" = "443"; "Protocol" = "TCP"})
$Records += @(@{"Location" = "Melbourne"; "ServerName" = ""; "ServerRole" = "AV Edge"; "DestinationPort" = "443"; "Protocol" = "TCP"})

$Records += @(@{"Location" = "Sydney"; "ServerName" = ""; "ServerRole" = "Federation"; "DestinationPort" = "5061"; "Protocol" = "TCP"})
$Records += @(@{"Location" = "Sydney"; "ServerName" = ""; "ServerRole" = "Access Edge"; "DestinationPort" = "443"; "Protocol" = "TCP"})
$Records += @(@{"Location" = "Sydney"; "ServerName" = ""; "ServerRole" = "Web Conferencing"; "DestinationPort" = "443"; "Protocol" = "TCP"})
$Records += @(@{"Location" = "Sydney"; "ServerName" = ""; "ServerRole" = "AV Edge"; "DestinationPort" = "443"; "Protocol" = "TCP"})

Step 4 – Parameter Tweaking
This version of the script like part 2 has a few settings that you can tweak. These are how many failures on each port is required before an email gets sent ($RequiredNumberOfFailuresBeforeEmail). There is also a setting for consolidating multiple errors or recoveries into a single email ($consolidateEmailsOnError and $consolidateEmailsOnError). Set these as you like:

#This is the number of required port check failures before an email is sent out
$RequiredNumberOfFailuresBeforeEmail = 3

#Send 1 email rather than one per record
$consolidateEmailsOnError = $true
$consolidateEmailsOnRecover = $true

Step 5 - Download Azure Storage Explorer
Now let your Function Application run for a while and gather some data. At any time you can look into your Function Apps Table Storage using Azure Storage Explorer. This application will show you all of the rows in your storage tables and allow you to see and edit as you see fit. You can download your free copy from here:

Once you have logged into your Azure Account within Azure Storage Explorer you can dig into your storage tables by selecting Storage Accounts > (Storage Resource Group Name) > Tables to see your table data. Note, there will be no table or data until you actually start running the Function App.

Step 6 - Download Power Bi
Now download a copy of Power BI for desktop:

Install the downloaded Power Bi on your PC.

Step 7 - Open Power Bi
Open Power Bi Desktop and you will be greeted with a splash screen and dialog. Click on the “Get Data” button:

Step 8 - Import Data
The Get Data dialog will then be displayed. Select “Azure Table Storage” from the list and click the “Connect” button:

Step 8 - Account URL dialog
Power Bi will now request an Account Name or URL to connect to:

Step 9 - Find Account URL
The account name for the dialog above can be found in the Azure Portal under the storage account Overview > Tables section:

The “Table service endpoint” is the value you will need to fill in the dialog with:

Step 10 - Paste in URL
Enter the Table service endpoint into the “Azure Table Storage” dialog and click “OK”:

Step 11 - Enter Account Key
You will now be asked to enter your “Account Key”:

This can be found under the Storage Account > Access Keys section in the Azure Portal:

Enter the Account Key and click “Connect”:

Step 12 - Load Data
Power Bi will now connect to your Table Storage and list up all of the Tables in there. Select the Results table and click “Load”:

Step 13 - Not all data is displayed
Power Bi will now download your data into the application. You may notice though that all of the columns that you can see in Azure Storage Explorer will not be displayed:

Step 14 - Edit Query
To be able to see all of the columns you need to do a little extra work. On the right hand side of the screen, Right Click on Table name at the top of the top of the column names listed and select “Edit Query”:

Step 15 - Expand content column
You will now see an extended view of the data that includes a “Content” Column:

On the top right of the Column click on the double arrow “expand” button:

You will now get a full list of all of the additional columns available that are stored in Table Storage as a Json blob. Tick the Columns that you want to include in your graphs and data analysis and click OK:

Step 16 - All columns are now available
You will now see the extra data columns:

 Click the “Close and Apply” button from the Home Tab:

The full array of data is now available for you to do as you please with:

Step 18 - Make charts
Back on the "Report" tab you can now put together some nice looking graphs of your data. Here is an example of a Pie Chart and a Bar Chart showing information about the number of errors for each role:

To create these graphs you use the following settings:

From here you can play with the data in Power Bi and make whatever graph you like (I included location in the data so you can even plot your Edge servers on a map). This is what makes Power Bi so powerful!

The Wrap Up

This post ends my series on creating an Edge port monitor with Azure Function Apps. I hope that in addition to helping you monitor your edge servers, this has been informative and taught you some new skills that might help in the future when making your own Function Apps.

Read more →

Popular Posts