Configuring SPF and Windows Azure Pack for IaaS usage and metering

Configuring SPF and Windows Azure Pack for IaaS usage and metering

  • Comments 23
  • Likes

After configuring VMM and OM for IaaS usage and metering in the previous blog post (here) it’s now time to configuring SPF and Windows Azure Pack to import Usage data from Operations Manager Data Warehouse into Windows Azure Pack (WAP) Usage Database.

The series includes the following posts:

Service Provider Foundation has a usage module that WAP uses to collect data from Operations Manager data warehouse. In this blog post we will go over how to configure the following:

  • Configure SPF to extract data from Operations Manager Data Warehouse
  • Configure WAP to connect to SPF Usage Service
  • Verify that Data is being stored in the WAP usage database

The following things must be configured before starting on this guide

The Environment:

Operations Manager Server: OM01.contoso.com
Operations Manager Database Server: DB04.contoso.com
VMM Server VMM01.contoso.com
SPF Server VMM01.contoso.com
WAP Server WAP01.contoso.com
WAP Database Server: DB02.contoso.com
Service Reporting Server VMM01.contoso.com
Service Reporting Database DB06.contoso.com
SharePoint Server SP01.contoso.com

Configure SPF to extract data from Operations Manager Data Warehouse

The usage module in SPF needs to be configured using PowerShell. With SPF there is a set of commands that enables you to configure settings in SPF. By using these commands you can tell SPF where the Operations Manager Server and Data Warehouse database are located.

To configure SPF Usage do the following:

1. Logon to the SPF server as an Administrator

2. Open IIS Manager Console

3. Select Application Pools

4. Check the identity for the Usage Application (eg. Contoso\!spf)

image 

5. Close the IIS console

6. Open Computer Management Console > Local Users Groups > Groups

7. Verify that the user you will later use to query SPF for data from WAP is member of the local SPF_Usage users group, if not add the user to the Group. In this scenario I’m using SPF Usage Application Identity user. Any other user can be used as long as it’s member of the Usage security Group.

    image

8. Open PowerShell console with administrative privilege

9. Run the following commands (change values in “<>” with your own values):

    Import-module spfadmin

    # Provide server name to the OM DW SQL instance

    $OMDWSqlServer = "<SCOM DW DB SQL Server>”

    Example: $OMDWSqlServer = "DB04.contoso.com"

    # Provide server name to the OM RMS Server

    $OMServer = "<SCOM RMS Server Name>"

    Example: $OMServer = "OM01.contoso.com"

    # Register the SCOM Data Warehouse instance to SPF usage metering

    $stamp = Get-SCSPFStamp;

    $server = New-SCSPFServer -Name $OMServer -ServerType OMDW -Stamps $stamp[0];

    For SPF Preview use the following command:

    $setting = New-SCSPFSetting -Name $OMDWSqlServer -SettingString "Data Source=$OMDWSqlServer;Initial Catalog=OperationsManagerDW;Integrated Security=True" -SettingType DatabaseConnectionString –Server $server

    For SPF RTM use the following command:

    $setting = New-SCSPFSetting -Name $OMDWSqlServer –Value "Data Source=$OMDWSqlServer;Initial Catalog=OperationsManagerDW;Integrated Security=True" -SettingType DatabaseConnectionString –Server $server

    Example:

    image

    To verify if the setting has been applied to SPF run the following command:

    Get-SCSPFSetting

    image

10. Logon to the SQL Server that hosts the SCOM DW DB as SQL Admin

11. Start SQL Management Studio

12. Select Security > Logins

13. Right Click on Logins folder and select “New Login”

14. Add the SPF Usage Application Identity user for Login name. eg. Contoso\!spf

15. Select User Mapping under “Select a page”

16. Select OperationsManagerDW and select OpsMgrReader under “Database role membership”

17. Click Ok

18. Right Click OperationsManagerDW under Database and select Properties

19. Select Permissions and pick the user just added under User and Roles.

20. Verify that the user has Execute permission on the database, if not assign this permission as shown below.

      image

21. Close SQL Management Studio

Configure WAP to connect to SPF Usage Service

The Usage module inside WAP uses SPF to collect data from Operations Manager Data Warehouse. For WAP usage to collect this data we need to tell WAP where the SPF Usage service is located.

To enable this in WAP do the following

1. Start a browser that has access to the WAP Admin Portal and login in as administrator

2. Select VM Clouds from the main menu

3. Select the VM Clouds Configuration from the VM Clouds menu

image

4. Select Register Service Provider Usage

5. Provide information for each field as given in the example below and Click Ok.

   image

Username is the user you added to the SPF_Usage Group under step 7 for Configure SPF to extract data from Operations Manager Data Warehouse.

NB: Remember to add /Usage after the port number for the SPF usage service.

    Verify that Data is being stored in the WAP usage database

    To verify that IaaS data is being stored in the WAP Usage Database do the following:

    1. Logon the WAP Database as Administrator

    2. Start SQL Management Studio

    3. Select Databases > Tables

    4. Right Click on Usage.Records and select “Select Top 1000 Rows”

    5. Verify that the result returns Data with ResourceId “VM Utilization”

    image

    6. Close the SQL Management Studio Console

    Now that we see data in the WAP Usage DB, we have successfully integrated SPF and Windows Azure Pack.

    This will now enable us to configure Service Reporting and using billing adapters against Windows Azure Pack.

    In the next blog post I will go through how Service Reporting can be configured.

    My colleague KR Kandavel is working on a series of blog posts explaining how you can build a billing adapter.

    • Link to blog post coming soon

    See you next time for more blogging about Windows Azure Pack & System Center 2012 R2 :-)

    Your comment has been posted.   Close
    Thank you, your comment requires moderation so it may take a while to appear.   Close
    Leave a Comment
    • Awaiting for the blog post on "Installing & configuring Service Reporting for IaaS usage and metering ". Please let us know when it will be available.

    • Hold tight this will go out tomorrow. Thanks for reading and asking for more.

    • What should i do in order to investigate why Usage.Records table is empty?

      Get-SCSPFSetting returns results which are similar to yours.

      OM report 'Virtual Machine Utilization Report' shows that OM see all my three VMs in the cloud.

      But SpfUsageRecord table of SCSPFDB contain only two VM records.

    • Hi Paul.

      Can I get you to check the following?

      Under what user is the SPF Usage APP Pool running? What rights does this user have in OM and OMDW?

      From your description below it sounds like this part is ok, just want to make sure.

      Can you also check if you have any errors in the Event log on the SPF Server related to SPF.

      Lastly what user did you use to register Usage Service with in WAP? And is this user member of SPF_Admins and SPF_usage Groups on the SPF Server?

      That the last VM is not getting data in SPF sounds wired. It should be all or nothing. Is this VM running and member of a cloud? Was all VMs created by WAP?

      Anders

    • Yep, Usage.Records table is empty...

      App pool for usage starts with contoso\spf

      contoso\spf member of all spf_* groups on SPF server and have SQL permissions as shown in post (only OperationsManagerDW db. right?)

      Event log empty.

      Get-SCSPFSetting output are similar to yours...

    • Hi !

      I also have the same issue ... Followed your step-by-step guide twice (rebuilding from scratch all servers ...) and the same issue ... Data are written in DW DB, SCOM reporting show also VM metrics, VMM to SCOM connection is OK, get-scspfsetting output the same result. What a pain for few graphs ...

      I just noticed a difference : In Scom diagram, "all Clouds" ressource isn't green checked, only green circled, dunno if it can be the cause ...

      I also dug into event logs. Nothing in SPF eventlog but in MS > WAP > MgmtSvc-Usage I have lot or errors :

      Provider read error occurred for provider 'systemcenter' instance id: '{8cece4a2-0321-4445-aa1f-f639d4e3bc1e}'. Error: System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (Not Found).

        at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()

        at Microsoft.WindowsAzure.Server.Usage.DefaultUsageRecordsReader.<GetRecordsAsync>d__0.MoveNext()

      --- End of stack trace from previous location where exception was thrown ---

        at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

        at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

        at Microsoft.WindowsAzure.Server.Usage.CycleRunner.<ProcessProviderInternal>d__19.MoveNext()

      With instanceID GUID corresponding to systemcenter provider entry in WAP mgmtsvc.usage DB.

      I really don't understand why in the previous version aka Windows Azure Pack for Windows server, VM reporting was provided by default, without SCOM. Why a such infrastructure "just for graph" ?!

      Your help to make it work would be appreciated.

      Thanks anyway for your clear and complete guides !

    • Hi Pierre

      Could you check what your settings are for the Usage setting in WAP Admin portal: Error 404 normally means that it can't find what it's looking for. So is the URL correct and have you added /usage to the of the web service pointer and doe sit have the right port. It sounds like SPF is working but the error is when WAP is quering the SPF Web service.

      Anders

    • Hi Anders,

      Thanks for your quick answer !

      I didn't registered the site with FQDN but with short name. When I test it in a browser, shortname = 404, FQDN = forbiden

      I'm trying to update it in admin site, but I receive the following error :

      One or more subscriptions are associated with this resource provider. Any endpoint changes to the resource provider may cause the subscriptions to be unusable

      Do you yjink I need to delete all existing subscriptions to update the serveur URL ?

      Thnaks again !

    • Hi Pierre

      I need a little time to investigate this, as I beleve after the first subscription is registered you cannot query the services in the same way, which might be what you are seeing.

      Again I need to verify this first, so please giv me a little time to investigate this.

      Anders

    • Hi Pierre

      Can I get you to check the following:

      What was the error you were getting below?

      Can you check that the user you are speficying when configuring the Service Provider Usage is member of SPF_ Usage local Groups on the SPF Server.

      Also check that you use the right port (8090) and that you specify / usage eg. https://spf01:8090/Usage

      Your error sounds like it's between WAP and SPF, not further Down in the SPF layer.

      Let me know if this helps

      Anders

    • Hi Anders and thanks a lot for your investigation

      Actually, I figured I filled https://<MyServer>/8090/usage instead of https://<MyServer>:8090/usage ... (/ instead of : ...)

      So the URL is wrong and it needs to be updated

      And yes, the account used is the correct one filled in the local group

      I found few PS commands shipped with azure pack which could update providers but they are not well documented in the help even in technet ...

      Thanks again !

    • Hi Anders.

      Sorry for the late answer, i was on vacation.

      In order to ease the deployment i've used the same DC' admin at every configuration step so the security was not the cause.

      In my case there were several issues, the following need to be checked:

      1. MgmtSvc-UsageCollector Application Pool should be started in IIS where WAP is running. It asks SPF every 10 minutes to provide usage data by REST. In my case it was stopped after a machine has rebooted.

      2. Then check IIS logs where SPF is running to verify there are recurrent records with status 200 to the USAGE endpoint:

      .......   GET /USAGE/usage lastId=XXX&batchSize=10000 ....

      In my case 'lastId' query parameter never changed and didn't point to any value (lastId=&batchSize...)

      Investigation showed there was a problem with last request index and the resolution was just to clear table [scspf].[SpfUsageRecord].

      Hope that will help others.

    • Hi Anders,

      I just wanted to keep you informed about the situation

      I just removed all the subscriptions and I could change the Usage URL. Now the usage.record table is correctly filled with data ! But in the tenant view, VM ressources are still at 0 except for the RAM which keep stuck at the ammount of memory assigned to the VM. Maybe there's a little time for the ressource to be displayed ...

    • Hi Pavel

      I am experiencing the same problem as you; lastId' query parameter never changed and didn't point to any value (lastId=&batchSize...)

      If I clear table [scspf].[SpfUsageRecord] table I see data in the browser when I go to https://spfserver:8090/usage/usage?lastID=0&batchSize=1000

      However, upon refresh of the browser all data disappears again until I delete the table again - were you able to find a permanent resolution for this problem?

      Thanks

    • Hi Anders, I followed your article step by step and everything goes well except the Usage.Records. In my case there is no data in this table. can you please let me know what other thing can I do to check this. Thanks in Advance !!