Troubleshooting Windows Azure Pack, SPF & VMM

Troubleshooting Windows Azure Pack, SPF & VMM

  • Comments 12
  • Likes

Troubleshooting Windows Azure Pack, SPF & VMM

As a follow-up from the troubleshooting Installation of Windows Azure Pack it’s time to look at the next item on our list being troubleshooting Service Provider Foundation (SPF) & Virtual Machine Manager (VMM). In this blog post I will describe how you can:

    • Verify integration between WAP, SPF and VMM is working correctly
    • Verify SPF is working correctly end to end

As SPF is the foundation between WAP and System Center, we want to make sure this is configured correctly. If SPF and the System Center back-end is not working as expected, there is really no need to troubleshoot Windows Azure Pack as SPF & System Center issues will just bubble up and cause trouble in WAP.

Let’s start by looking at how we can verify all the components are configured correctly.

How to verify integration between WAP, SPF and VMM is working correctly

As WAP IaaS is made up from different layers (WAP, SPF and VMM), in this section I will give some guidance on how you can verify step by step if things are correctly configured all the way from VMM to WAP.

At the end of the day it comes down to ensuring that the right credentials are used in the right areas throughout the different layers.

We want to look into the following areas to verify they are working as expected. In particular, you should verify that:

  • The right user is added to VMM administrators used by SPF.
  • SPF server can connect to VMM Using PowerShell.
  • SPF Web Service is running under the right user credentials.
  • The right user for accessing SPF Web services is configured.
  • WAP VM Clouds is registered for SPF using the right user.

Now let me start by describing my environment so there is a common reference:

VMM Server

VMM01.fabrikam.com

SPF Server

SPF01.fabrikam.com

WAP Server

WAP01.fabrikam.com

SPF Service account

Fabrikam\!sc

SPF local account

SPF01\sc

SPF Web Service

https://SPF01.contoso.com:8090

Pre-requisites before starting the troubleshooting

  • Virtual Machine Manager 2012 R2 is running and configured to manage at a minimum one Hyper-V host.
  • SPF is running and configured and have VMM 2012 R2 console with PowerShell installed.
  • WAP is installed and configured.

Verify that the right user is added to VMM administrators used by SPF.

1. Logon to the VMM Server (VMM01.fabrikam.com) as a member of VMM Admins Group.

2. Start VMM 2012 R2 console.

3. Select Settings and expand Security > User Roles

4. Open the Administrator User Role and select Members from the left menu.

5. Under Members verify that your SPF Service account is listed. e.g fabrikam\!sc.

    image

6. Close VMM Console

Verify that the SPF server can connect to VMM using PowerShell

As SPF uses PowerShell to execute commands against VMM we have to make sure SPF can do this successfully.

1. Login to the SPF Server as the SPF service account (e.g Fabrikam\!sc)

2. Click on start and select Virtual Machine Manager Command Shell under Microsoft System Center 2012

3. Type get-vmmserver <VMM Server> e.g. get-vmmserver vmm01

    image

    Verify that get-vmmserver returns data similar to the picture above.

4. Type Get-VM | ft Name and verify that VMs running on the VMM Server is returned in the output.

5. Type Get-SCCloud | ft Name and verify that Clouds on the VMM is returned in the output

6. Close PowerShell console

If this is not returning data you should try reinstalling the VMM Console on the SPF server.

Verify that the SPF Web Service is running under the right user credentials

The way SPF executes commands against VMM will be in the context of the user under which the web service is running. A common mistake that people have is that the SPF Web Service is not running under the right account but instead it will be running as the Network Service account, which has no access in VMM.

To verify that the SPF Web Service is running under the right service account do the following:

1. Login to the SPF server as an administrator

2. Start IIS Manager

3. Expand SPF Server > Sites and verify that SPF shows in the list.

    image

4. Select Applications Pools under connection menu

5. Verify that both the VMM and Provider Application Pools are running under the account (Identity) that was also a member of the VMM Administrators. (e.g fabrikam\!sc) as verified earlier.

  image   image

If this is not the case and VMM Application Pool is running under eg. Network Service it needs to be changed to an account that has administrator role access to VMM

1.  Select VMM Application Pool and Select Advanced Settings from the Action Menu

2. In the Advanced Settings select Identity and click on the … bottom to specify a user

3. Select Custom Account and click Select

4. Specify User name, Domain and Password and click OK

5. Click OK to Advance Settings.

The Web service should now be running under the right credentials to access data in VMM.

Verify that the right user for accessing SPF Web services is configured

SPF is working in such a way that a user can query the SPF Service if it’s member of local defined User Groups. When SPF is installed, four local groups are created: SPF_Admins, SPF_Providers, SPF_VMM and SPF_Usage.

In order to make WAP connect to SPF a local user should to be created on the SPF Server that is a member of all four SPF Groups.

The reason it’s recommended to use a local user and not a domain user is that WAP and SPF servers might not be in the same domain, for this reason SPF uses basic authentication to authenticate the user that accesses the web service.

To verify this do the following:

1. Logon to the SPF Server as Administrator

2. Start Computer Management

3. Select Local User and Groups

4. Locate the user you want to use for SPF or create a new user by right click Users > new user (e.g. SC)

5. Click on the user and select the “Member Of” tab.

6. Make the user member of all Groups starting with SPF_, if not add the remaning groups.

    image

7. Click OK

Verify that WAP is registered for SPF using the right user

1. Open the WAP Admin Portal (e.g https://wap01.fabrikam.com:30091) as an Administrator

2, Select VM Clouds

3. Select the Cloud icon with the lightning

   image

4. Click on the link under “Register System Center Service Provider Foundation”

5. Specify the SPF Web Service location, provide the local user name created earlier, which was added to the SPF local user Groups and provide the password for the user.

    image

6. Next click on the word CLOUDS and verify that you see a VMM Server

   clip_image001

If there is no VMM Server click on register your VMM server by clicking on USE AN EXISTING VIRTUAL..

Provide the FQDN on your VMM Server and Click OK.

Now verify that you see a VMM Server under CLOUDS.

How to verify SPF is working correctly end to end 

        

To understand if SPF is working as expected we are going to pretend we are the Windows Azure Portal and we will query for data in SPF. By asking for specific data in SPF via the REST API we can ensure that we can extract the needed data. In order to do this we want to do the following:

    • Find the user that WAP is using to communicate with SPF
    • Query SPF with the SPF user.
    • Query different types of data from SPF.

Pre-requisites before starting

    • One or more Clouds defined in VMM
    • One or more Hyper-v hosts managed by VMM.
    • One or more VMs running in VMM

Find the user that WAP is using to communicate with SPF

1. Open the WAP Admin Portal (e.g https://wap01.fabrikam.com:30091)

2, Select VM Clouds

3. Select the Cloud icon with the lightning.

 image_thumb[1]

4. Click on the link under Register System Center Service Provider Foundation.

5. Write Down the Service URL and Username.

   image_thumb[2]

         

Query SPF with the SPF user

Now let’s try to see if we can actually talk to SPF and get SPF to return data that WAP would normally ask for.

6. Start a browser on the WAP server

7. Type in the Service URL from VM Clouds configuration and add the following /SC2012R2/VMM/Microsoft.Management.Odata.svc

eg. https://spf01.fabrikam.com:8090/SC2012R2/VMM/Microsoft.Management.Odata.svc

8. Click continue to this website if you are using a self singed certificate.

9. Provide the user from username under VM Cloud Configuration and click Ok

    e.g SPF01\sc

You should hopefully see data on the screen similar to this:

   image

Query different types of data from SPF

To query for data from a sub-category do the following:

1. Scroll Down and find an areas you would like to see data for. In this example we are going to use “VirtualMachines”

2. Now add the VirtualMachines (Case sensitive) to the end of the url

e.g. https://spf01.fabrikam.com:8090/SC2012R2/VMM/Microsoft.Management.Odata.svc/VirtualMachines

If you see a picture like this, you want to disable reading view in Internet Explorer.

image

Open Internet Options, select Content Tab and click on settings under “Feeds and Web Slices”

Uncheck “Turn on feed reading view”

clip_image002

Click OK and refresh the page.

3. You should now see a XML output similar to this.

    image

4. You want to look for entry > content type > ComputerName

   image

    Note: You can also see other information about this VM by looking at the other attributes in the XML.

5. To list another Example try to add Clouds after Microsoft.Management.Odata.svc

    e.g. https://spf01.fabrikam.com:8090/SC2012R2/VMM/Microsoft.Management.Odata.svc/Clouds

6. In my case you can see that I have three Clouds.

   image

    image

    image

7. Open VMM Console as VMM Administrator, select VMs and Services and expand Clouds

   You should see the same list of Clouds as shown in SPF:

    image_thumb[10]

8. Open the WAP Admin Portal (e.g https://wap01.fabrikam.com:30091)

9. Select VM Clouds > Clouds and Expand the VMM Instance to see listed Clouds:

   image

If you can’t see the same data in WAP Admin portal as in SPF there is most likely a problem connecting to SPF or SPF can’t connect to VMM.

Going over the first section making sure no steps were missing hopefully will solve the problem.

Let me know if this was helpful in any way or if any important steps are missing

If you are looking for other areas of troubleshooting WAP, have a look here: Troubleshooting Installation & Configuration of Windows Azure Pack – An Introduction 

Happy Integrating WAP, SPF and VMM.

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • You should do the same for SPF Usage Point.

    I usually test it with:

    spfserver.contoso.com/.../usage

  • Hi Stanislav Zhelyazkov

    You are absolutely right, and I will make sure that it gets added to the blog post for usage, when it's ready.

    Thanks for the tip

  • Hello Anders. I see that you use one account for the 4 IIS app-pools. Is this really valid for a production environment?

    If not, maybe there are a few more considerations to take care of? Any other extra considerations regarding Kerberos, delegation, SPN or such?

  • Hi Jon

    Yes you are right, I would not recommend running all the App-Pools under the same account in production, but this was my test environment and I had to change quite a bit of things to make it look right. Kerberos and deligation depends on the design of WAP.

  • Thanks, nailed a missing step in our configuration due to your guide, was missing one account under SC-VMM Administrator role.

  • mmm. "A common mistake that people have is that the SPF Web Service is not running under the right account but instead it will be running as the Network Service account, which has no access in VMM.". Not sure I'd go along with that. Given that installer asks for service accounts for SPF, which then don't get applied to the IIS component - and that the install instructions on Microsoft's official site make no mention of it - it would seem to Microsoft's mistake rather than its customers...?

  • I would suggest adding in that having your PowerShell execution policy defined in Group Policy will prevent WAP from adding VMM.

  • Hello
    I try to create a stamp using C# program that interact with SPF, but i have this following exception error:

    Une exception non gérée du type ‘System.Data.Services.Client.DataServiceRequestException’ s’est produite dans Microsoft.Data.Services.Client.dll

    Informations supplémentaires : Une erreur s’est produite lors du traitement de cette requête.

    this is my program:

    SpfADMIN.Admin adminService = new SpfADMIN.Admin(new System.Uri(@”https://172.17.1.1:8090/SC2012R2/ADMIN/Microsoft.Management.Odata.svc/”));
    adminService.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;

    SpfADMIN.Stamp stamp = new SpfADMIN.Stamp();
    stamp.Name = “New stamp”;

    adminService.AddToStamps(stamp);
    adminService.SaveChanges();

  • Dear Anders, it looks like I passed all your tests, however, I'm still unable to retrieve VMs from under tenant portal, neither can add subscription and items VIRTUAL MACHINES and NETWORKS are inoperable from tenant portal.

    Details say that:

    ---------
    One or more errors occurred while contacting the underlying resource providers. The operation may be partially completed. Details: Failed to create subscription. Reason: Message : An error occurred while processing this request., InnerMessage: http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> http://www.w3.org/1999/xhtml">

    Request Error

    The server encountered an error processing the request. See server logs for more details.


    -----------

    No clues in server logs. P.S. We already disabled TLS 1.2 on both SPF-SCVMM and WAP

    What could be casing this?