InvokeRestMethod for the Rest of Us

InvokeRestMethod for the Rest of Us

  • Comments 9
  • Likes

Summary: Learn how to use Windows PowerShell to search web pages.

Scripting Guy, Ed Wilson here. Today’s post is written by Doug Finke, a Windows PowerShell MVP, and June Blender, senior programming writer on the Windows Azure Active Directory team.

Take it away, June…

I was bending my brain around the cool new Windows Azure Graph REST APIs  and Windows Azure Storage Services REST APIs, when I noticed that Doug Finke, author of Windows PowerShell for Developers was posting about calling Representational State Transfer (REST) APIs in Windows PowerShell.

To give you a bit of background, a REST API is an interface based on the HTTP protocol. You use one of four operations—Get, Post, Put, or Delete—with a string parameter in HTTP format. The strings look like URIs, but they don't refer to a web page. Instead, they provide the data for one of the four operations.

I didn't know how to call REST APIs in Windows PowerShell, so I asked Doug for help. Before we started, I asked Doug how to determine which websites have web services with REST interfaces. It turns out that this isn't a very straightforward task. You have to search online for the name of your favorite website or service and "REST API." Doug found YouTube and StackOverflow, and he pointed me to the ProgrammableWeb  API search.

Doug suggested that we start with the Invoke-RestMethod cmdlet, which was introduced in Windows PowerShell 3.0, and YouTube, which has a well-documented REST API. I started here: YouTube API v2.0 – API Query Parameters. This page lists the parameters that you can include in a Get or query operation.

The syntax of the query string is:

https://gdata.youtube.com/feeds/api/videos?

…followed by a parameter and parameter value in key=value format.

For example, the instructions recommend that you include the v (version) parameter with a value of 2 to use the latest version of the REST API:

https://gdata.youtube.com/feeds/api/videos?v=2

To specify the query string (the text you are searching for), use a parameter of q (lower-case) and the search text. This query string searches for PowerShell:

https://gdata.youtube.com/feeds/api/videos?q=PowerShell

To search for multiple words, concatenate the words with a plus sign (+).

https://gdata.youtube.com/feeds/api/videos?q=Windows+PowerShell

To use multiple parameters, separate them with an ampersand (&). The following query string specifies v 2 and searches for PowerShell:

https://gdata.youtube.com/feeds/api/videos?v=2&q=Powershell

Now, to run the query in Windows PowerShell, use the Invoke-RestMethod cmdlet. The following command searches YouTube videos with PowerShell in the title.

PS C:\> Invoke-RestMethod -Uri  "https://gdata.youtube.com/feeds/api/videos?v=2&q=PowerShell"

 

etag          : W/"D0UNQH47eCp7I2A9Wh5TFU4."

id            : tag:youtube.com,2008:video:-Ya1dQ1Igkc

published     : 2012-03-27T19:43:26.000Z

updated       : 2013-09-30T10:01:31.000Z

category      : {category, category}

title         : My Final 1-Day PowerShell v2 Workshop

content       : content

link          : {link, link, link, link...}

author        : author

accessControl : {yt:accessControl, yt:accessControl, yt:accessControl, yt:accessControl...}

comments      : comments

group         : group

rating        : {gd:rating, yt:rating}

statistics    : statistics

 

etag          : W/"DU8DRn47eCp7I2A9WhFaGE0."

id            : tag:youtube.com,2008:video:XpHgSHQZNpU

published     : 2011-09-21T15:38:13.000Z

updated       : 2013-09-21T23:57:57.000Z

category      : {category, category}

title         : Hacking Windows Accounts with Powershell

content       : content

link          : {link, link, link, link...}

author        : author

accessControl : {yt:accessControl, yt:accessControl, yt:accessControl, yt:accessControl...}

comments      : comments

hd            :

group         : group

rating        : {gd:rating, yt:rating}

statistics    : statistics

 . . .

Wow, it works!  I'll select the title, author, and the URL of the video:

PS C:\ps-test> Invoke-RestMethod -Uri "https://gdata.youtube.com/feeds/api/videos?v=2&q=PowerShell" | Select-Object Title, Author, Link

 

title                                            author    link

-----                                            ------    ----

My Final 1-Day PowerShell v2 Workshop            author    {link, link, link, link...}

Hacking Windows Accounts with Powershell         author    {link, link, link, link...}

What is PowerShell and how can it make IT ...    author    {link, link, link, link...}

Basics of Powershell P1                          author    {link, link, link, link...}

...

Hmm. That doesn't quite work. The author property is in author.name and the link is in content.src. You can use a calculated property or a PSCustomObject to fix this. Let's use a PSCustomObject.

PS C:\> Invoke-RestMethod -Uri "https://gdata.youtube.com/feeds/api/videos?v=2&q=PowerShell" | foreach {[PSCustomObject]@{Title=$_.Title; Author=$_.Author.name; Link=$_.content.src}} | Format-List

 

Title  : My Final 1-Day PowerShell v2 Workshop

Author : Don Jones

Link   : https://www.youtube.com/v/-Ya1dQ1Igkc?version=3&f=videos&app=youtube_gdata

 

Title  : Hacking Windows Accounts with Powershell

Author : David Hoelzer

Link   : https://www.youtube.com/v/XpHgSHQZNpU?version=3&f=videos&app=youtube_gdata

...

That's much better! There's lots of other useful information, including the publication date and last updated date, comments, statistics, and ratings.

But before I turn you over to Doug to hear why this works, let me share one more tidbit. One of the parameters in the YouTube REST API is Max-Results. Its default value is 25, and its maximum value is 50.

It doesn't seem to work in Windows PowerShell 3.0. By default, it returns the first 13 hits. When we specified values of 5, 10, 30, and 50, we got only half of what we requested: 3, 5, 15, and 25.

PS C:\> (Invoke-RestMethod -Uri "https://gdata.youtube.com/feeds/api/videos?v=2&q=PowerShell").count

13

 

PS C:\> 5,10,30,50 | % {

    $url=https://gdata.youtube.com/feeds/api/videos?v=2&q=Powershell&max-results=$_;

    (Invoke-RestMethod $url).count

}

3

5

15

25

But when we tested it in Windows PowerShell 4.0, it worked perfectly.

PS C:\> (Invoke-RestMethod -Uri "https://gdata.youtube.com/feeds/api/videos?v=2&q=PowerShell").count

25

 

PS C:\> 5,10,30,50 | % {

    $url=https://gdata.youtube.com/feeds/api/videos?v=2&q=Powershell&max-results=$_;

    (Invoke-RestMethod $url).count

}

5

10

30

50

We checked with Windows PowerShell developer, Lee Holmes; and indeed, in Windows PowerShell 4.0, the team fixed a bug that returned only half of the results of Invoke-RestMethod calls.

So, let me turn you over to Doug so you can learn about why all of this works…

Thanks, June. That was a great write-up about how Windows PowerShell integrates with REST APIs.

REST is the predominant Web API. It’s a way to serve standard programmatic access to data over HTTP. So, as June shows, you can get to a wealth of YouTube data by using a Window PowerShell one-liner.

The cool part is that all REST APIs work this way. So, when you learn the fundamentals, you can identify REST sources and apply these techniques again and again.

Invoke-RestMethod

Invoke-RestMethod is great because the Windows PowerShell team has written and tested a ton of script, and I can get super productive with two lines:

PS C:\>$url = "https://gdata.youtube.com/feeds/api/videos?q=PowerShell"

PS C:\> (Invoke-RestMethod $url).title.'#text'

 

My Final 1-Day PowerShell v2 Workshop

Windows PowerShell 2.0 for Beginners Training & Overview - EPC Group

Hacking Windows Accounts with Powershell

Introduction to Windows PowerShell Part 1

What is PowerShell and how can it make IT more efficient? Jason Helmick - Interface

Windows PowerShell Remoting: Definitely NOT Just for Servers

Basics of Powershell P1

O'Reilly Webcast: Getting Started with Windows PowerShell 3.0

Advanced Automation Using Windows PowerShell 3.0

TechEd 2013: Windows PowerShell Unplugged - Jeffrey Snover

My PowerShell Scripts - Systems Administration

[Powershell] Einführung Teil 1

MCITP 70-640: PowerShell

German PowerShell Basis Video Tutorial Teil 1 von 21 Was ist PowerShell, Warum PowerShell

Bootable USB Drive Using Powershell and Windows 8

Инструменты Windows PowerShell 2.0. Часть 1

Bruce Payette - PowerShell Workflows

Powershell. Curso Hispano.

Exploring PowerShell for Beginners Dec 9th

PowerShell: Introduction and Scripting Tutorial

No Tools No Problem Building a PowerShell Botnet Christopher @obscuresec Campbell

Creating HTML Reports with PowerShell

PowerShell Scripting Basics

PowerShell for common Office 365 Operations

Windows Powershell - What are The Top 10 Cmdlets to Start Using Immediately

Take note of how the ­title information is retrieved:

 (Invoke-RestMethod $url).title.'#text'

The title data is stored in the #text property, but if you don’t enclose "#text" in quotation marks, Windows PowerShell interprets it as a comment because it begins with a # sign, and you will receive an error message.

Before Windows PowerShell delivered Invoke-RestMethod, we could reach into the .NET Framework and use the System.Net.WebClient class to grab this data ourselves:

PS C:\>$wc = New-Object System.Net.WebClient

PS C:\>$url = "https://gdata.youtube.com/feeds/api/videos?q=PowerShell"

PS C:\>$youtubedata = [xml]$wc.DownloadString($url)

PS C:\>$youtubedata.feed.entry.title.'#text'

Invoke-RestMethod does a lot for us. It connects and downloads the data from the target URL, converts the returned string into XML, and extracts the feed.entry information so we can use less .NET notation to get at what we want faster. A feed is a data format for providing frequently updated content. Invoke-RestMethod leverages this standard to make our lives easier.

But wait! There’s more…

Earthquake data and JSON

Invoke-RestMethod also automatically detects the type of data the URL returns. While REST is the predominate way to interact with data on the web, JavaScript Object Notation (JSON), a text-based standard for data interchange, is often used too.

Great news! The designers of Invoke-RestMethod also made it easy to interact with JSON REST feeds. Here's an Invoke-RestMethod command that gets earthquake data from Seismi.org. It returns JSON, but you don't need to know that. Windows PowerShell returns custom objects (PSCustomObject) that you can use in your commands.

PS C:\>(Invoke-RestMethod -URI "http://www.seismi.org/api/eqs").earthquakes | Select-Object -First 10 | Format-Table –AutoSize

 

src eqid     timedate            lat      lon       magnitude depth  region                                              

--- ----     --------            ---      ---       --------- -----  ------                                               

us  c000is61 2013-07-29 22:22:48 7.6413   93.6871   4.6       40.90  Nicobar Islands, India region                       

us  c000is4s 2013-07-29 21:52:12 -57.7816 -25.3260  5.2       53.50  South Sandwich Islands region                        

us  c000is3k 2013-07-29 21:33:34 36.6696  71.0615   4.7       234.10 Hindu Kush region, Afghanistan                      

us  c000irvf 2013-07-29 18:27:41 -37.2993 177.2515  4.9       160.50 off the east coast of the North Island of New Zealand

us  c000irpf 2013-07-29 14:53:32 24.5038  62.5255   4.5       24.30  off the coast of Pakistan                           

us  c000irl3 2013-07-29 10:23:35 45.9104  143.0497  4.4       337.70 Hokkaido, Japan region                              

us  c000irjj 2013-07-29 08:21:31 -6.8134  130.2114  4.6       87.80  Banda Sea                                           

us  c000irjf 2013-07-29 08:12:27 -16.9852 -177.0945 5.1       10.00  Fiji region                                         

us  c000irj9 2013-07-29 08:08:00 36.5108  70.5896   5.0       193.10 Hindu Kush region, Afghanistan                      

us  c000irhe 2013-07-29 05:49:35 -4.6368  -104.9759 4.7       10.00  central East Pacific Rise                            

You can use the WebClient class to build a command like this on your own. Prior to Windows PowerShell 3.0, you’d need to use a third-party .NET JSON parser or build your own. Windows PowerShell 3.0 introduced the ConvertFrom-Json cmdlet:

PS C:\>$wc = New-Object System.Net.WebClient

PS C:\>$data = $wc.DownloadString("http://www.seismi.org/api/eqs") | ConvertFrom-Json

PS C:\>$data.earthquakes | select -First 10 | Format-Table -AutoSize

The results are much noisier than the out-of-the-box experience of Invoke-RestMethod. Plus, Invoke-RestMethod automatically detects whether XML or JSON is being returned, and it does the right thing—it returns Windows PowerShell objects for you to easily work on.

Invoke-RestMethod under the covers

Invoke-RestMethod is one of the Windows PowerShell 3.0 core cmdlets. It is written in C# and delivered in the Microsoft.PowerShell.Commands.Utility.dll. To examine it, I used a free tool from JetBrains, called DotPeek. This tool can look into the compiled DLL and decompile it to source code so you can browse it.

After reading the Invoke-RestMethod code, I asked the Windows PowerShell team to confirm my assumptions. Lee Holmes, author of the great book, Windows PowerShell Cookbook, 3rd Edition, replied, “If it looks like an RSS or ATOM feed, we process and return the elements in the feed. If the response says it returns some form of JSON (there are many), we will first try to convert it to JSON. If that fails, we will convert it to XML. If that fails, we return it as text.”

Here is a simplified Windows PowerShell version I came up with that I sent along with the question: 

$url = "http://www.seismi.org/api/eqs"

 

$request = [System.Net.WebRequest]::Create($url)

$request.Method="Get"

$response = $request.GetResponse()

$requestStream = $response.GetResponseStream()

$readStream = New-Object System.IO.StreamReader $requestStream

 

$data=$readStream.ReadToEnd()

 

if($response.ContentType -match "application/xml") {

    $results = [xml]$data

} elseif($response.ContentType -match "application/json") {

    $results = $data | ConvertFrom-Json

} else {

    try {

        $results = [xml]$data

    } catch {

        $results = $data | ConvertFrom-Json

    }

}

 

$results 

These 23 lines of code give the same results for the earthquake data as the one-line that used Invoke-RestMethod. Thank you, Windows PowerShell team!

That’s a very simple overview of how YouTube and Seismi.org serve this information. The better news is that this is a web standard and many sources of data are available to you this way. Even better, Windows PowerShell makes working with this information a breeze.

~Doug

Thanks, June and Doug.

I invite you to follow me on Twitter and Facebook. If you have any questions, send email to me at scripter@microsoft.com, or post your questions on the Official Scripting Guys Forum. See you tomorrow. Until then, peace.

Ed Wilson, Microsoft Scripting Guy

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • Awesome articel!

  • Thank you June an Doug! 2 of my lovliest PowerShell collegs have written again a great PowerShell Article!

    I have learned much Thank you!

    And my "German PowerShell" Youtube channel is also listed dumdidu.. !

    come and watch ;-) www.youtube.com/.../GermanPowerShell

    greets Peter Kriegel

    http://www.admin-source.de

  • Danke, Peter. And, thanks for supporting the German PowerShell community. We all appreciate it.

  • June/Doug, great article!  I just love playing with this to see what it'll do.  :-D

  • Folks - while I like the examples, the part that consistently stimies me with this method is how to capture and pass the session information. I'm working with an API that causes a session to be created in step 1, and I need to capture this session and pass it back in step 2. I've tried these two lines unsuccessfully: Invoke-RestMethod -Uri $url1 -SessionVariable s Invoke-RestMethod -Uri $url2 -WebSession $s -Method Post. however, the command invariably fails with an 'invalid session' message or somesuch. I can use curl and the command prompt to capture and pass the session ID, but I thought the above approach would work. Please clue me in as to what I'm doing wrong - and thank you! Paul L

  • I have the smae issue as anonymous! I am trying to access Nessus data from our corporate scanning infrastructure via powershell. I can make a single login REST connection via the login page successfully. But none of the other REST pages will work for me. Is ther an example somewhere online of a non public interface that shows how to authenticate, generate a cookie, and then perform additional requests on the same connection?

  • To persist a session see the following:
    help Invoke-RestMethod -par session*

  • jrv - thanks for the hint. I have read the help:

    -SessionVariable
    Creates a web request session and saves it in the value of the specified variable. Enter a variable name without
    the dollar sign ($) symbol.

    When you specify a session variable, Invoke-RestMethod creates a web request session object and assigns it to a
    variable with the specified name in your Windows PowerShell session. You can use the variable in your session as
    soon as the command completes.

    Unlike a remote session, the web request session is not a persistent connection. It is an object that contains
    information about the connection and the request, including cookies, credentials, the maximum redirection value,
    and the user agent string. You can use it to share state and data among web requests.

    To use the web request session in subsequent web requests, specify the session variable in the value of the
    WebSession parameter. Windows PowerShell uses the data in the web request session object when establishing the new
    connection. To override a value in the web request session, use a cmdlet parameter, such as UserAgent or
    Credential. Parameter values take precedence over values in the web request session.

    You cannot use the SessionVariable and WebSession parameters in the same command.

    Required? false
    Position? named
    Default value
    Accept pipeline input? false
    Accept wildcard characters? false


    It makes sense to read it, but I can't an example working. Do you know where there is a sample script using this functionality I could look at? I don't seem to actually ever store a cookie when setting the session variable.

  • For help with blog posts it is better to post in the forum:
    http://social.technet.microsoft.com/Forums/en-US/ITCG/threads