Monday, 21 August 2017

Network Assessor for Skype for Business Online and Microsoft Teams

Skype for Business Online and Skype Teams are both real time communications tools that rely heavily on the networking infrastructure that they run on top of. This has always been, and will remain, a challenge in cloud deployments for the foreseeable future. In order to prepare for deploying Skype for Business Online or Microsoft Teams, you need to first ensure that your networking infrastructure will be able to handle the real-time traffic that these applications produce.

To help with this, Microsoft released a tool call the Skype for Business Network Assessment Tool. This tool runs via the command line and uses the same media stack that is used by the Skype for Business client to test Packet Loss, Jitter, Round Trip Latency and Reorder Packet Percentage on connections from your network into Microsoft’s Office 365. The audio stream created by the tool gets sent to Office 365 and then mirrored back from the closest Edge server location to your current physical location. Based on the return stream the tool is able to give accurate statistics on Packet Loss, Jitter, Round Trip Latency and Reorder Packet Percentage. This will give you an idea of the how congested the current network environment is and how well you might expect Skype for Business or Skype Teams conferences to function in your environment.

The Microsoft Network Assessment Tool is useful because it allows you to test a network before an Office 365 tenant has even been deployed and it’s also free! What the tool does lack though is the ability to easily monitor the network over a period of time and produce nicely formatted output that you can show to a manager or customer. So, I thought I would take some time and try to rectify these limitations and build a new front end for the tool…

Network Assessor


Introducing Network Assessor for Skype for Business and Microsoft Teams:



Version 1.00 Features
  • Graphs Packet Loss, Average Jitter, Latency and Reorder Ratio.
  • Zoom graphs in / out / forwards / backwards to view the data clearly.
  • Start / Stop and Pause the network tests at the click of a button.
  • Keep an eye on the status of testing using the tray icon colour by setting breach percentage thresholds. This will easily let you know that things are within your desired operating levels without having to open the interface. There are two levels of threshold breaches; one that changes the icon to orange in colour and the other that changes the icon to red in colour. These percentages are calculated on a per graph basis and if any graph breaches the percentage the colour will change.
  • Graphs will automatically highlight in red points in each of the graphs which are outside of Microsoft’s recommended bounds of operation. There are two levels for these thresholds: Client thresholds and Edge Thresholds. For more details see the Usage section below.
  • The status bar will display PASS/FAIL results for each graph. This calculation follows Microsoft's "ResultsAnalyzer.exe" PASS/FAIL calculation which is based on any graph having more than 10% of test attempts resulting in (Client or Edge) threshold breaches will equal a FAIL result.
  • You can select the frequency at which the tool will run tests. This ranges between 1 and 120 minutes (ie. Run a 17 second long test every 1 minute or up to every 2 hours).
  • Allows for graphs to be saved as PNG images for use in documentation/reports.
  • All the graphs can be shown at once or individual graphs can be selected using the “Window” menu item. Graphs are saved at the resolution that they are displayed, so opening individual graphs and then saving them can offer higher resolutions.
  • Every session that is created by the tool gets recorded in a CSV file for future reference. Sessions will “roll” to a new file based on the “Roll Time (hours)” setting in the Settings dialog. Logs can be rolled between 1 and 12 hours.
  • CSV files can be imported back in to the tool by using the File > Import CSV File menu or dragging and dropping the file onto the interface directly from Windows Explorer.
  • Supports automatic download (note: Internet connection is required for this to work) and installation of the Microsoft Network Assessment Tool and VC++ Redistributable pre-requisite by using the Help > Install Microsoft Tool menu item.

Note: This tool is not intended for load/stress testing!



Prerequisites


Tested with version 6.0.8969.164 (12/14/2016) of the Microsoft Skype for Business Network Assessment Tool (LINK: https://www.microsoft.com/en-us/download/details.aspx?id=53885).

Network Assessor requires the following:
  • At least .NET 4.5 must be installed on the PC to run the tool. (Windows 8 / Windows Server 2012 and above will support this by default. If you are using earlier operating systems the installer will download .net for you)
  • Do not run the tool on a machine with multiple interfaces. Microsoft’s Network Assessment Tool may fail due to sending traffic out the wrong interface.
  • When run on Windows Server, Microsoft’s command line tool needs “Media Foundation” (for Windows 2012, 2012 R2) or “Desktop Experience” (Windows 2008 R2) installed when running on Windows Server. These can be installed via the Server Manager interface or through Powershell:

Windows Server 2008 R2 (Desktop Experience install):
Import-Module ServerManager
Add-WindowsFeature Desktop-Experience

Windows Server 2012 or 2012 R2 (Media Foundation install):
Add-WindowsFeature Server-Media-Foundation
Note: This is not documented in Microsoft’s documentation but is required.

Microsoft’s command line tool and pre-reqs can be can automatically downloaded and installed using the Help > Install Microsoft Tool… menu directly in the Network Assessor interface. For manual offline installation of Microsoft’s Tool, install the following:

Note: Microsoft’s tool should be left with all of the default settings in its config file.


Firewall Requirements:

The tool needs to be able to send traffic to Office 365 in order to work. The following ports are required to be allowed on firewalls between the Network Assessor tool and Office 365:
  • TCP Source Ports: 50000 - 50019
  • TCP Destination Port: 443
  • UDP Source Ports: 50000 - 50019
  • UDP Destination Port: 3478

Network Assessment Procedures:

For more procedural details on how to do Network Assessments, please refer to the Skype Operations Framework. There is a lot of useful documentation in the Determine Network Readiness section of Planning stage within this framework and should give you a good starting point for running Network Readiness assessments.


Usage Details


Walkthrough Video:



Main Window Settings



The “Run Every x Mins” setting controls how often the network assessment tool will be run and log results. This value can be set between 1 minute and 2 hours.

The Thresolds setting refers to two levels of thresholds that are recommend by Microsoft for real time audio for the statistics that are generated by the Network Assessment Tool. The first is the “Edge” values which are recommended for when you are testing from your perimeter network into Office 365. The second is recommended thresholds when testing from a “Client” subnet. The Edge values are lower and more restrictive than the client subnet because is expected that the client subnet will have more hops to get to Office 365 than the Edge.

Customer Edge to Microsoft Edge Thresholds:
Metric
Target
Latency (RTT)
< 60ms
Packet loss
<0.1% during any 15s interval
Note: The value output by the Microsoft tool is not a percentage.
Packet inter-arrival Jitter
<15ms during any 15s interval
Packet reorder
<0.01% out-of-order packets
Note: The value output by the Microsoft tool is not a percentage.

Client to Microsoft Edge Thresholds:
Metric
Target
Latency (RTT or Round-trip Time)
< 100ms
Packet loss
<1% during any 15s interval
Note: The value output by the Microsoft tool is not a percentage.
Packet inter-arrival Jitter
<30ms during any 15s interval
Packet reorder
<0.05% out-of-order packets
Note: The value output by the Microsoft tool is not a percentage.


Settings Window:


The tool has the following settings:



Network Assessment Tool Location: This is the location of the Skype for Business Network Assessment Command Line Tool on the system. If you have manually downloaded the tool and unzipped it to a folder on the system you can enter the location via the Browse button.
Session Log Folder: The Session Logs by default are placed on the system in the “C:\Users\<Username>\AppData\Roaming\NetworkAssessor\SessionLogs” folder. If for some reason you would like to change this folder (perhaps for centralizing the logs on a file share) you can select a new folder here.
Red Level Breach Percentage: This is the percentage of breaches (on a per graph basis) that has to occur for the task bar icon to change to red colour.
Orange Level Breach Percentage: This is the number of breaches (on a per graph basis) that has to occur for the task bar icon to change to orange colour.
Turn Off Minimise Notification: When this check box is ticked it will turn off the task bar notifications.
Auto Scroll Graph: When this check box is ticked it means that the graphs will automatically scroll when new points are added to it graph.
Roll Time (Hours): This is the number of hours of testing that will be logged to CSV until a new session log file is created. When the file rolls the graphs will also be cleared. This is to try and maintain the performance of the graphs. Also, when the graph is cleared the breach counters will also be cleared.

The Wrap Up


Hopefully this post will give you another tool in the toolbox to help you in your preparation for deploying Skype for Business Online or Microsoft Teams in your environment. If you have any comments or issues with the tool please provide feedback below.



Read more →

Monday, 27 February 2017

AudioCodes Mediant Virtual SBC Transcoding Requirements

This blog post covers an interesting issue that I ran into the other day with AudioCodes Mediant Virtual Edition and getting transcoding capabilities working on VMWare. The AudioCodes Virtual Edition SBCs are very capable and are great for SIP Trunk deployments and routing calls between various SIP platforms. They can support a large number of non-transcoded calls as well as supporting transcoding for audio and DTMF streams. In a hardware SBC, transcoding is usually done with a DSP chip that is used to convert one codec to another codec. However, in a Virtual SBC this is done using the CPU cores provided to the virtual machine.

The requirement for transcoding in a deployment can come up when connecting to carriers with mandatory codec requirements like G.729 (a low bandwidth codec) that are not supported by Skype for Business. In these cases, if the carrier decides to send a call inbound with only G.729 (or some other unsupported codec) as the only media type available in the INVITE SDP, rather than reject the call, we can have the SBC transcode between the two different codecs.

So what is required for transcoding to work on the AudioCodes Virtual Edition SBC? Just CPU cores right? Well, not exactly… Continue on, intrepid reader…



CPU Specifications


The AudioCodes manuals are quite clear with their requirements for how many CPU Cores are required to support different numbers of sessions and transcoding capabilities. These range from low spec VMs with only 2 cores up to 8 cores (as of version 7.2). For details of the number of CPU Cores required for each scenario, I suggest you check out the latest Mediant Virtual Edition User Guide (the Channel Capacity section).

In addition to the number of CPU Cores required, the 7.0 and 7.2 manuals for the AudioCodes Virtual Edition both state the following about CPU requirements:

Specifications Resource
Specifications
Processor type
64-bit Intel® CPU of at least 2.5 GHz, with support for hardware virtualization (Intel VT-x) enabled with Advanced Vector Extensions (AVX) and AES-NI support (Sandy-Bridge architecture or newer)

Whilst usually the exact specifications of a CPU that an application vendor lists can be taken as a generic baseline for the systems that they may have done capacity testing on, in this case the CPU capabilities absolutely matter! It seems that when the Virtual Edition was designed, the code used for transcoding was specifically targeted for CPUs that support the AVX feature set introduced in Sandy Bridge CPUs. When the Virtual SBC boots, it checks the CPU capacities and if it is not at least Sandy Bridge level (with AVX support) then the SBC transcoding capabilities will not be available!
This limitation has some additional flow-on effects depending on the hypervisor on which it is being deployed. See the following section for more details of these limitations.

VMWare and CPU Capabilities


In addition to the raw CPU capabilities in the host machines (ie. Sandy Bridge generation of processors with AVX support), a VMWare environment using vMotion also can be configured with a specific level of processor support across all hosts in a vMotion cluster. This is a feature within VMWare that is called Enhanced vMotion Compatibility (EVC) and controls the level of CPU capabilities being emulated at the VM level across multiple machines. This ensures that the same CPU capabilities are presented for all machines in a cluster, and protects against having differing CPU capabilites when VMs are migrated between hosts. This can mean that even if the host machine has a Sandy Bridge CPU, this capability set may not be passing through to the VM on which the SBC is running.

The VMWare vMotion EVC level will be used when the host machines in a cluster have different levels of CPU capabilities. The lowest CPU capability will be the one that is emulated for all machines in the cluster. So, whilst many of the hosts may have Sandy Bridge or above CPUs, there may be lower capability machines included in the cluster that force the EVC baseline in VMWare to be configured to a lower CPU feature set. This setting can directly affect the Virtual Edition SBC's ability to support transcoding, because even if the host has Sandy Bridge capabilities the VMWare environment might be masking these capabilities and presenting a lower CPU level than is actually in the host.

Below is a table that describes the various EVC levels that can be configured within VMWare:

VMWare Description of Intel EVC Baselines:
EVC Level
EVC Baseline
Description
L0
Intel® "Merom" Gen. (Intel® Xeon® Core™ 2)
Applies baseline feature set of Intel® "Merom" Generation (Intel® Xeon® Core™ 2) processors to all hosts in the cluster.
L1
Intel® "Penryn" Gen. (formerly Intel® Xeon® 45nm Core™ 2)
Applies baseline feature set of Intel® "Penryn" Generation (Intel® Xeon® 45nm Core™ 2) processors to all hosts in the cluster.
Compared to the Intel® "Merom" Generation EVC mode, this EVC mode exposes additional CPU features including SSE4.1.
L2
Intel® "Nehalem" Gen. (formerly Intel® Xeon® Core™ i7)
Applies baseline feature set of Intel® "Nehalem" Generation (Intel® Xeon® Core™ i7) processors to all hosts in the cluster.
Compared to the Intel® "Penryn" Generation EVC mode, this EVC mode exposes additional CPU features including SSE4.2 and POPCOUNT.
L3
Intel® "Westmere" Gen. (formerly Intel® Xeon® 32nm Core™ i7)
Applies baseline feature set of Intel® "Westmere" Generation (Intel® Xeon® 32nm Core™ i7) processors to all hosts in the cluster. Compared to the Intel® "Nehalem" Generation mode, this EVC mode exposes additional CPU features including AES and PCLMULQDQ.
L4
Intel® "Sandy Bridge" Generation
Applies baseline feature set of Intel® "Sandy Bridge" Generation processors to all hosts in the cluster. Compared to the Intel® "Westmere" Generation mode, this EVC mode exposes additional CPU features including AVX and XSAVE.
L5
Intel® "Ivy Bridge" Generation
Applies baseline feature set of Intel® "Ivy Bridge" Generation processors to all hosts in the cluster. Compared to the Intel® "Sandy Bridge" Generation EVC mode, this EVC mode exposes additional CPU features including RDRAND, ENFSTRG, FSGSBASE, SMEP, and F16C.
L6
Intel® "Haswell" Generation
Applies baseline feature set of Intel® "Haswell" Generation processors to all hosts in the cluster. Compared to the Intel® "Ivy Bridge" Generation EVC mode, this EVC mode exposes additional CPU features including ABMX2, MOVBE, FMA, PERMD, RORX/MULX, INVPCID, VMFUNC. 


In order for AVX to be supported in a large VMWare environment, the Processor Support needs to be configured at a minimum of L4 Sandy Bridge level. If EVC mode is not being used the EVC mode may display as "N/A" or "disabled", both of these modes will allow for the raw processor capabilities to be passed through to the SBC, which will also allow for transcoding to work if the CPU is Sandy Bridge or above.



The host EVC level can be viewed in the host summary page within vSphere. In the example below the current EVC level is set to Merom Generation or L0 level which would not support transcoding:



Hyper-V Processor Compatibility Mode


I haven’t actually tested this on Hyper-V because transcoding on Hyper-V has only been supported as of 7.2 software. However, Hyper-V also has a CPU Compatibility mode feature which allows for various features that involve moving VMs between machines (eg. Live Migration, Quick Migration and Save/Restore features). This feature is similar to VMWare EVC mode, however, the implementation is slightly different than VMWare. When the Processor Compatibility mode is enabled in Hyper-V the CPU (for Intel and AMD CPUs) feature set gets “normalised” by removing extended feature sets of the each CPU type to lower the capabilities to a baseline.

For the Microsoft documentation, when a VM in processor compatibility mode is started, the following processor features are hidden from the VM:

Host running Intel based processor
SSSE3, SSE4.1, SSE4.2, POPCNT, Misaligned SSE, XSAVE, AVX


You will note that AVX is one of the hidden features, which is one of the features required for the AudioCodes Virtual Edition SBC to work! Uh-oh…

When this feature was released, the following Q&A was documented about this functionality:

From the Hyper-V Compatibility Mode Q and A:

Q: Can I individually select which features should be hidden from VM when put in processor compatibility mode?
A:  No, once you put VM in a processor compatibility mode, Hyper-V will hide the necessary set of features from the VM to ensure a successful migration between Hyper-V enabled processors. 

Q: What happens if a vendor has written an application that uses one of these features that isn’t visible with processor compatibility enabled?
A: Since the feature isn’t exposed to the virtual machine, the application won’t “see it” and it’s up to the application to determine how to proceed; however, there are two likely paths.

So presumably the CPU compatibility feature within Hyper-V is going to cause the AudioCodes Virtual Edition SBC to not be able to see the AVX capabilities of the CPU. As a result, transcoding will likely not work on Hyper-V if processor compatibility mode is enabled.


Hyper-V Processor Compatibility References:


Checking CPU Capabilities after Installation


If the Virtual Edition SBC has been installed within a virtual environment, you can tell if the hardware will support transcoding by checking the following CLI commands:

Working Transcoding example:
Mediant VE SBC> show system hardware
  CPU: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz, total 4 cores avx support(avx)
  Memory: total RAM: 4096 MB
  Chassis: VMware Virtual Platform
  Network:
          VMware VMXNET3 Ethernet Controller (rev 01)
          VMware VMXNET3 Ethernet Controller (rev 01)
  Virtual env: vmware
Mediant VE SBC>

In the example above you can see that the CPU has “avx support (avx)”. Transcoding will work on this server.

Non-supported hardware example:
Mediant VE SBC# show system hardware
  CPU: Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz, total 4 cores avx support(none)
  Memory: total RAM: 8192 MB
  Chassis: VMware Virtual Platform
  Network:
          VMware VMXNET3 Ethernet Controller (rev 01)
          VMware VMXNET3 Ethernet Controller (rev 01)
          VMware VMXNET3 Ethernet Controller (rev 01)
  Virtual env: vmware
Mediant VE SBC#

In the example above you can see that the CPU has “avx support (none)”. Transcoding will not work on this server.


Web Interface Check


The only way to tell from the web interface whether or not the current server hardware will support transcoding is to download the configuration file and check the following:

Transcoding Working Example:
;Board: Mediant VE SBC
;HW Board Type: 73  FK Board Type: 79
;Product Key:
;Slot Number: 1
;Software Version: 7.00A.095.004
;DSP Software Version: SOFTDSP => 700.53
;Board IP Address: 10.20.1.168
;Board Subnet Mask: 255.255.255.0
;Board Default Gateway: 10.20.1.1
;Ram size: 3831M   Flash size: 0M
;Num of DSP Cores: 3  Num DSP Channels: 100
;Profile: NONE
;Client defaults file is being used (file length=352)

In this working example the configuration file displays that it has discovered 3 DSP Cores and has 100 DSP channels available for transcoding.


Transcoding Not Working Example:

In this example, the configuration file displays that it has discovered 0 DSP Cores and has 0 DSP channels available for transcoding. In this case the SBC has detected that the CPU does not have AVX support and as a result will not support transcoding.

;Board: Mediant VE SBC
;HW Board Type: 73  FK Board Type: 79
;Product Key:
;Slot Number: 1
;Software Version: 7.00A.095.004
;DSP Software Version: SOFTDSP => 700.53
;Board IP Address: 10.20.1.165
;Board Subnet Mask: 255.255.255.0
;Board Default Gateway: 10.20.1.1
;Ram size: 7871M   Flash size: 0M
;Num of DSP Cores: 0  Num DSP Channels: 0
;Profile: NONE
;Client defaults file is being used (file length=352)


Example of Transcoding Failure Errors


When the transcoding resources are not available due to a lack of CPU support for AVX, you will see errors complaining about a lack of resources in the log at the time of the SBC trying to apply extension coders or setting up the call:
22:41:25.812  147.76.26.84    local0.notice  [S=4742] [SID=a64ab9:33:279]  (      lgr_flow)(      4645)  #MediaResourcesConnector::AllocateMediaResources
22:41:25.812  147.76.26.84    local0.notice  [S=4743] [SID=a64ab9:33:279]  ( media_connect)(      4646)  ConnectionData::CalculateResourcesForExtTranscoding Leading:DSP Opposite:CODERTRANSCODING MediationLevel:RTP
22:41:25.812  147.76.26.84    local0.notice  [S=4744] [SID=a64ab9:33:279]  ( media_service)(      4647)  ServicesMngr: Allocate Coder Transcoding session. current active: 0 and max is: 250
22:41:25.812  147.76.26.84    local0.notice  [S=4745] [SID=a64ab9:33:279]  ( media_service)(      4648)  ServicesMngr: Allocate Media channel. current active: 0 and max is: 0
22:41:25.812  147.76.26.84    local0.warn    [S=4746] [SID=a64ab9:33:279]  ( media_service)(      4649) !! [ERROR] ServicesMngr: Cannot allocate more Media channel. current active: 0 and max is: 0
22:41:25.812  147.76.26.84    local0.warn    [S=4747] [SID=a64ab9:33:279]  (      lgr_flow)(      4650) !! [ERROR] (#2198)RTPStreamResource::AllocateResource Allocate Resource - cannot allocate DSP. probably lack of resources
22:41:25.812  147.76.26.84    local0.notice  [S=4748] [SID=a64ab9:33:279]  ( media_service)(      4651)  ServicesMngr: Deallocate Coder Transcoding session. current active: 1 and max is: 250
22:41:25.812  147.76.26.84    local0.notice  [S=4749] [SID=a64ab9:33:279]  ( media_service)(      4652)  ServicesMngr: Allocate Coder Transcoding session. current active: 0 and max is: 250
22:41:25.812  147.76.26.84    local0.notice  [S=4750] [SID=a64ab9:33:279]  ( media_service)(      4653)  ServicesMngr: Allocate Media channel. current active: 0 and max is: 0



Licensing


In addition to the CPU requirements, you will also need licensing for transcoding to work. The CODER-TRANSCODING, DspCh  and IPMediaDspCh licences must be available in the licence installed on the SBC. The CODER-TRANSCODING, DspCh and IPMediaDspCh licences will appear in the licence list as shown below:

Key features:
Board Type: Mediant VE SBC
IP Media: VXML
Coders: G723 G729 G728 NETCODER GSM-FR GSM-EFR AMR EVRC-QCELP G727 ILBC EVRC-B AMR-WB G722 EG711 MS_RTA_NB MS_RTA_WB SILK_NB SILK_WB SPEEX_NB SPEEX_WB OPUS_NB OPUS_WB
Channel Type: DspCh=100 IPMediaDspCh=100
HA
Security: IPSEC MediaEncryption StrongEncryption EncryptControlProtocol
DSP Voice features:
DATA features:
Control Protocols: MGCP SIP SBC=10 MSFT FEU=50 CODER-TRANSCODING=50
Default features:
Coders: G711 G726

If you do not have the CODER-TRANSCODING, DspCh and IPMediaDspCh licences then the SBC will still run, however transcoding functionality will not be available (config will show "Num of DSP Cores: 0  Num DSP Channels: 0" as explained earlier).



The Wrap Up


So before jumping in to the Virtual Edition SBC world with your customers, be sure to understand the server and Hypervisor environment that you are deploying the SBC into. Some useful questions to ask up front are:
  • What hypervisor will the SBC be installed on?
  • Have you purchased CODER-TRANSCODING licences for the SBC?
  • Is the hypervisor server host CPU at least Sandy Bridge level and does it support AVX?
  • If vMotion is being used within VMWare: What is the VMWare vMotion EVC baseline of the host that the SBC is deployed on?
  • If Hyper-V is being used, is Processor Capability mode disabled?
  • Have you ensured that the hypervisor is going to have enough dedicated CPU Cores available for the SBC? Also, note that the CPU resource must be reserved for the SBC and not shared.

Armed with this information, you will be able to make an informed assessment on whether or not transcoding is going to be supported on the Virtual Edition SBC! Hope this was helpful. Cheers!



Read more →

Popular Posts