During the last Techdays here in Belgium, I have had the privilege of being able to present a session on Lync. I had chosen as a topic for that session “Deploying Lync: Notes from the Field, and more”, where I talked about the top ten questions I run into when talking to customers about deploying Lync in their environment. I tried to answer the following questions:

• Lync Online – Lync On Premises: where do we go?
• Lync is a Puzzle, really?
• Lync and NAT: Yes or No?
• Whats’s up with the Picture in Lync?
• Are there any tools available to help me design hardware and bandwidth requirements for Lync?

And then the final topic I touched upon was monitoring, and that’s when I was happy to introduce one of my Technology Advisor colleagues, Wesley Backelant, who knows almost everything about PowerPivot and Power View and SQL and BI to the IT Pro Lync audience And we promised to blog about it, so here it is, the details on the web

Monitoring Server in Lync, why would you use it?

In short, the Monitoring Server role in Lync is optional, and can be added after finishing the deployment of all the other server roles within Lync. The Monitoring Server enables you to capture both call detail records (CDR) for Enterprise Voice and Audio/Video conferences and data collections about your Audio/Video Quality of Experience, also incorporating data on  file transfers, application sharing, and remote assistance. It does NOT archive the conversations itself (if you want to keep the IM conversations in your organization, you should consider deploying the Archiving server role).

Some typical questions on the Monitoring Server role in Lync:

• Can the Monitoring server role be combined with any other Lync Server role?

  Yes, as taken from TechNet:

  “A Monitoring Server can be collocated with an Archiving Server, with a SQL Server store of an Enterprise Edition Front End pool, or with a file store of a Front End pool. The Monitoring Server requires a database, but the database can be collocated on the Monitoring Server, with the database server for the Archiving Server, or on the Back End Server of an Enterprise Edition Front End pool. A Monitoring Server cannot be collocated with a Standard Edition server in a production environment.”

• Can you collocate the monitoring database with a Lync back-end database?

The answer is yes, as taken from TechNet

You can collocate each of the following databases on the same database server:

• Back-end database
• Monitoring database
• Archiving database

You can collocate any or any or all of these databases in a single SQL instance or use a separate SQL instances for each, with the following limitations:

Each SQL instance can contain only a single back-end database, single Monitoring database, and single Archiving database.

The database server cannot support more than one Front End pool, one Archiving Server, and one Monitoring Server, but it can support one of each, regardless of whether the databases use the same SQL instance or separate SQL instances.

• Do you require SQL Reporting Server?

  No,  but if you do you will be able to take advantage of the built-in Monitor Server Reports, which you can customize as wanted.

• What’s the advantage of using the Monitoring Server role with Microsoft System Center Operations Manager?

  By installing the Microsoft System Center Operations Manager, which uses the Monitoring CDR and QoE data, you can enable the generation of near real-time alerts showing the health of call reliability and media quality, and define actions upon these.

• Do I need a monitoring server per Lync Pool, and even per Site?

  No, as taken from the same Technet Article linked to before:

  “Multiple central sites can also share any of the following that you deploy in one central site:

  • Archiving Server
  • Monitoring Server
  • Stand-alone Mediation Server or pool
  • Edge Server or Edge pool”

• Any documentation available on how to deploy the Monitoring Server?

  Yes: Microsoft Lync Server 2010 Monitoring Deployment Guide, available for download here: http://www.microsoft.com/download/en/details.aspx?id=8207, which described the different steps envolved:
  • Installing Message Queuing for Monitoring
  • Installing SQL Server (with Reporting Services)
  • Adding a Monitoring Server to the Topology using Topology Builder
  • Installing the Monitoring Server
  • Deploying Monitoring Server Reports (if wanted)
  • Configuring Monitoring Server Settings

• Is there a document that described how to use the Monitoring Server Reports in a Lync Server 2010 deployment?

  Sure there is, there is Microsoft Lync Server 2010: Using Monitoring Server Reports whitepaper available for download here: http://www.microsoft.com/download/en/details.aspx?id=890

Built-In Reports examples:

What does the Dashboard give me?

And yes you can export the reports, and you can customize them…but if you know have the feeling as in “this is not exactly what I want to show my manager”, “I would like to have some more fancy reports” “I’m looking for an easier way to use the gathered data”…then please read along, because now it’s time to dive into PowerPivot and Power View!

Let me start by saying that the solution provided here is obviously for demo purposes.  It is not a perfect solution and we are not responsible for the production use of this file.

Technologies Used

First a couple of words on the technology used in this solution.

PowerPivot is a powerful data mashup and data exploration tool based on xVelocity in-memory technologies providing unmatched analytical performance to process billions of rows at the speed of thought.  It comes with a client part, a free add-in for Excel 2010, which allows you to import and combine data in an easy way.  We used Excel in this particular case to get information from the Lync monitoring databases and create some interesting calculations.  In order to get your hands on PowerPivot please visit the official website. 

The second part of the solution uses SharePoint and Power View to share and visualize the information in a very interactive way.  Sharing PowerPivot files in a scalable and controlled way can be achieved by using the Excel Services functionality of SharePoint.  With Excel Services you have the ability to share Excel files to a broad audience without the need to install Excel 2010 (and in this case PowerPivot) on every machine.  If you need more information about Excel Services please read this.  When integrating SQL Server Reporting Services 2012 in SharePoint 2010, you will get a new powerful data visualization tool called Power View.  Power View is really all about the interactive and stunning experience.  Understanding Power View is actually the easiest by just watching this demo.

Now that you know the technology part let’s dive into some of the details on how we created this.  Of course we started by interviewing customers to understand what they would like to get out of the information available.  Some of the topics that came up were duration distribution, average response time, number of sessions by type, uptake of Lync and a lot more interesting stuff!

Most of the information you need is available in the LcsCDR which is fully documented here.  The leading table is SessionDetails so you better figure out how this table is structured.  One additional set of data you may need is a date table, I have included a hidden sheet with dates but you could also use the DateStream feed from our Windows Azure Marketplace.  A couple of other tables we used are SIP, Users, ClientVersions and UriTypes

The other database we used was rtccab1 to get display names.  This is a bit less straightforward to retrieve but here is how you can extract it:

SELECT avname.Value as 'DisplayName', avuri.Value as 'UserUri'
  FROM AbUserEntry u
  INNER JOIN AbAttributeValue avname ON avname.UserId = u.UserId AND avname.AttrId = 3
  INNER JOIN AbAttributeValue avuri ON avuri.UserId = u.UserId AND avuri.AttrId = 8

The entire model looks like this

Now that we have all the information in place it is time to visualize some of it.  A couple examples of the things we built (remember it was built on a test database too so don’t freak out by the unanswered call rate ).

Taking this PowerPivot file to the server to share it with a broad audience is a matter of saving it on an Excel Services enabled SharePoint Server.  This would give you the following view

But we promised a more compelling and interactive way of visualizing this information and that is exactly what you will do with Power View.  Power View can be used on PowerPivot files published on SharePoint or on solutions deployed to SQL Server Analysis Services.  As a little side note, you can import the PowerPivot model or build this model from scratch in SQL Server Data Tools to make it a real server-side solution running on SQL Server Analysis Services.

The most popular view however is the adoption rate of Lync in the organization, this can be achieved by using the Scatter Chart functionality in Power View.

Please find the PowerPivot file here.  In order to get data from your enviroment go to the PowerPivot window, select “Design” and “Existing Connection”.  Edit both connections to point to your SQL Server and when done use “Refresh All”.  In the Excel Windows go to Data" and select “Refresh All” and you are done.  If you want to use Power View just upload the file to a SharePoint where this feature is enabled.

There is a LOT more you can get out of the Lync databases, so we hope this sparks a whole wave of creativity!  Ilse’s favorite for example is the QoEMetric database to see the call quality for different users and devices.  If I can find some time I’ll see if we can publish this PowerPivot file too. 

UPDATE dd April23rd, 2012

Just a small heads up that I changed the Lync PowerPivot file to make it scale better.  In order to do so I have removed several unique columns from the import and moved some calculations to the query side instead of the client side.

I have placed the new file here and the original file is still available here.

If you are interested in having this as a Tabular Model (for SQL Server Analysis Services) please let me know.  You can easily do this yourself if you like of course by importing the model in the SQL Server Data Tools.

Ilse & Wesley

Yesterday I have had the pleasure of doing a TechNet LiveMeeting here @Microsoft Belgium, entitled “Using the Power of PowerShell to manage your Exchange Online and Exchange On Premises Environment”. The session was not a level 400 deep dive in PowerShell, but its aim was to give some tips on how one can use PowerShell to manage an Exchange Online tenant in Office365.

Tip 1. Set-ExecutionPolicy Unrestricted and -AllowRedirection

Connecting to Exchange Online is easy, and boils down to launching Windows PowerShell, creating a new persistent connection to the remote Exchange Client Access Server, and importing it, like can be seen in the picture below:

Two remarks here:

1. Before you are able to run the cmdlet Import-PSSession, you need to make sure you are allowed to run scripts... When you run Import-PSSession and you get the following error message:

Import-Module : There were errors in loading the format data file:
Microsoft.PowerShell, , C:\Users\Seppe\AppData\Local\Temp\tmp_0740bdd5-5276-437
7-a890-50bb10d3d32b_cuwqdhef.gqv\tmp_0740bdd5-5276-4377-a890-50bb10d3d32b_cuwqd
hef.gqv.format.ps1xml : File skipped because of the following validation except
ion: File C:\Users\Seppe\AppData\Local\Temp\tmp_0740bdd5-5276-4377-a890-50bb10d
3d32b_cuwqdhef.gqv\tmp_0740bdd5-5276-4377-a890-50bb10d3d32b_cuwqdhef.gqv.format
.ps1xml cannot be loaded because the execution of scripts is disabled on this s
ystem. Please see "get-help about_signing" for more details..
At line:3 char:30
+                 Import-Module <<<<  -Name $name -Alias * -Function * -Prefix
$prefix -DisableNameChecking:$disableNameChecking -PassThru -ArgumentList @($se
ssion)
    + CategoryInfo          : InvalidOperation: (:) [Import-Module], RuntimeEx
   ception
    + FullyQualifiedErrorId : FormatXmlUpateException,Microsoft.PowerShell.Com
   mands.ImportModuleCommand

To enable the execution of scripts you can run the cmdlet Set-ExecutionPolicy Unrestricted. This can be enforced btw by using Group Policies, you can download the ADM Group Policy Template for PowerShell here. For more information on Set-ExecutionPolicy, check this link here.

2. Do not forget the parameter AllowRedirection, which will enable redirection to the appropriate Exchange server using different URI.

Tip 2. Connecting to Exchange Online and Exchange On Premises at the same time, use –Prefix

When you launch the Exchange Management Shell, you are using Remote PowerShell to connect to a Client Access Server in your Exchange On Premises environment, as you can see by running Get-PSSession after launching Exchange Management Shell

If you would then create a new persistent connection to Exchange Online using the directions mentioned above, you would get the following warning when importing the PowerShell Session:

“WARNING: Proxy creation has been skipped for the following command: …., because it would shadow an existing local command. Use the AllowClobber parameter if you want to shadow existing local commands.”

If you would use the parameter AllowClobber, you would indeed shadow the existing commands, meaning, you would hide or replace the original commands: eg. running Get-Mailbox would retrieve the Exchange Online mailboxes, but you wouldn’t be able to retrieve the ones in your Exchange On Premises organization anymore in this EMS Session!

Solution: Use the Prefix parameter, which will add the given prefix to the nouns in the names of the imported commands.

Running the following Import-PSSession $Session –Prefix o365 will import all the commands, but will prefix all the nouns with o365, running Get-DistributionGroup will return a list of all Distribution Groups in my On Premises Exchange Organization, where-as running Get-o365DistributionGroup will return a list of Distribution Groups in my Exchange Online environment:

Tip 3. Remove-PSSession

Looking at the definition of Remove-PSSession on TechNet

The Remove-PSSession cmdlet closes Windows PowerShell sessions (PSSessions) in the current session. It stops any commands that are running in the PSSessions, ends the PSSession, and releases the resources that the PSSession was using. If the PSSession is connected to a remote computer, Remove-PSSession also closes the connection between the local and remote computers.

Why would you do this for your Exchange Online? Because if you do not close the Windows PowerShell window without disconnecting from the server-side session, your connection will remain open for 15 minutes. And you have a limit of three connections to the server-side session at one time per account.

Tip 4. Use Profiles

If you want to know more about Windows PowerShell Profiles, please head over here @MSDN, where you can dive into the wonders of profiles. The reason it might be useful to use profiles here is that you don’t need to type everything every single time you want to connect to your Exchange Online environment. By entering the New-PSSession and Import-PSSession lines in any of the 4 profiles, you can create functions that you can call upon when you want to connect to your Office365 Exchange Online tenant by simply entering Connect-ExchangeOnline.

In the TechNet LiveMeeting I created my Windows PowerShell Profile, the process on how to create this is clearly described here, and here’s what it looks like after creating it:

By using the so-called Windows PowerShell user profile, this will only work for the currently logged on user, and only for the Microsoft.PowerShell shell, if I launch ISE for example, you won’t have the functions Connect-ExchangeOnline and Disconnect-ExchangeOnline:

Question: is it possible to schedule a PowerShell script against Exchange Online?

Answer: Sure

In the below example I will schedule a task to when run will create a excel file with an overview of the mailbox sizes, the tricky part is the passing of your credentials. I have chosen to first create a passwordfile, which will contain the password of the user that will be used to connect in the script to the Exchange Online tenant:

Read-Host -AsSecureString | ConvertFrom-SecureString | Out-File C:\users\ilvancri\MyPassword.txt

Then I have created a script called “Mailboxsizes.ps1”, that will connect to Office365, create the csv file, and remove the PSSession in the end:

$password = type C:\users\ilvancri\MyPassword.txt | ConvertTo-SecureString
$userid = "ilse@microsoftbelux.onmicrosoft.com"
$cred = New-Object System.Management.Automation.PSCredential $userid,$password
$global:session365 = New-PSSession -configurationname Microsoft.Exchange -connectionuri https://ps.outlook.com/powershell/ -credential $cred -authentication Basic -AllowRedirection
Import-PSSession $global:session365

Get-Mailbox | Get-MailboxStatistics | Select Displayname,TotalItemSize, ItemCount | Export-csv .\mailboxsizes.csv
Get-PSSession | Remove-PSSession

Now it’s time to create a  batch file that when run, will launch and execute the script:

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -command ". 'C:\users\ilvancri\mailboxsize.ps1'

And now you can schedule to run the BAT file when needed.

Ilse