Hey, Scripting Guy! How Can I Use Windows PowerShell to Retrieve the Non-Unique Items in a List?

Hey, Scripting Guy! How Can I Use Windows PowerShell to Retrieve the Non-Unique Items in a List?

  • Comments 2
  • Likes

Hey, Scripting Guy! Question

Hey, Scripting Guy! I understand that I can use Windows PowerShell and the Get-Unique cmdlet to determine which items are unique in a collection. However, I want to do the opposite: I want to determine which items are not unique in a collection. How can I do that?

-- YE

SpacerHey, Scripting Guy! AnswerScript Center

Hey, YE. You know, yesterday afternoon, just as he was getting ready to head for home, the Scripting Guy who writes this column received a package from Don Jones and Sapien Technologies. This wasn’t exactly unexpected; Sapien has generously agreed to donate some very cool prizes for us to give away as part of the 2008 Winter Scripting Games. What was unexpected was one of the items the Scripting Guy who writes this column just happened to notice on the packing list:

tshirts, polos, picopads, postits, pens, monkeys, sm. Folders

Talk about cool: Don Jones sent us monkeys to give away during the Scripting Games!

Of course, after our initial excitement subsided we got a little bit worried; after all, the box wasn’t all that big, and Don hadn’t cut any air holes into it. On the one hand, we were concerned that maybe the monkeys might have died in transit. On the other hand, we were even more concerned that maybe the monkeys hadn’t died in transit. If that was the case, then these poor monkeys had just endured a grueling cross-country trip in a tiny box with no air holes, and probably without any food or water, either. (Although, thanks to the pens and Post-It Notes, they would at least have had a chance to express their feelings about the trip.)

Speaking of feelings, although neither of the Scripting Guys are biologists we had a feeling that monkeys probably don’t like to spend their days locked into a tiny box with no air holes. In turn, that might mean that we didn’t have a box full of monkeys after all; instead, we might very well have a box full of angry monkeys. Neither of us was too excited about opening the box and finding out.

Of course, if we were … lucky … we wouldn’t have angry monkeys, either; instead, we’d have a box full of dead monkeys. Thanks, Don. Thanks a lot.

Eventually, though, curiosity got the better of us and we opened the box. And what did we find? As it turns out, we didn’t have any monkeys, at least not any real ones. (Or at least we don’t think they’re real monkeys. Like we said, neither of us are biologists.) Instead, we had something even better than real monkeys: Flingshot Flying Monkeys.

To be honest, we’d never heard of Flingshot Flying Monkeys, either. (Like we keep trying to tell you, we’re not biologists.) As it turns out, though, Flingshot Flying Monkeys are pretty cool: they’re part slingshot, part stuffed animal, and part screaming banshee terror. In other words, the three best things in life, all rolled-up in a single package.

Thanks, Don! Thanks a lot!

Note. For more information about Flingshot Flying Monkeys visit the As Seen On TV Web site. Although, if you value your sanity, you’ll mute your computer speakers before doing so.

Of course, you’re probably wondering, “Flying monkeys? What do flying monkeys have to do with Sapien Technologies?” To tell you the truth, we’re not really sure. However, Sapien no doubt noticed the same thing we did: all of these monkeys were dressed in capes emblazoned with the following slogan:

PrimalScript
WAY Better than Apes!

How did Sapien manage to find an entire herd of flying monkeys who just happened to be big fans of PrimalScript? Beats us; we’re really not biologists, you know. Maybe they just got lucky. Or maybe all monkeys (not just flying ones) prefer PrimalScript. We’ll get back to you on that.

In the meantime, we need to get our minds off flying monkeys and back on the business at hand. YE has a collection of items; let’s assume this is a list of different kinds of pie:

Apple
Peach
Chocolate
Chocolate
Apple
Banana Cream
Apple
Blackberry
Pecan
Lemon Meringue
French Silk
Apple
Pecan
Raspberry
Strawberry
Key Lime
Apple
Pecan

YE would like to know which of these items are not unique; that is, which items occur more than once. How can she use Windows PowerShell to identify the non-unique items? Well, one way would be to have a million flying monkeys sitting in front of a million copies of PrimalScript, hoping that one of them will eventually come up with a solution. Or, you could run this simple little script:

$arrPies = "Apple", "Peach", "Chocolate", "Chocolate", "Apple", "Banana Cream",`
"Apple", "Blackberry", "Pecan", "Lemon Meringue", "French Silk", "Apple", `
"Pecan", "Raspberry", "Strawberry", "Key Lime", "Apple", "Pecan"
$arrPies | Group-Object | Where-Object {$_.Count -gt 1}

Like we said, there isn’t much to this script. In line 1 we’re simply adding the list of pies to an array named $arrPies. In line 2, we’re taking that array of pies and piping it to the Group-Object cmdlet. Group-Object will then take that collection and sort the pies into groups, keeping track of the number of pies in each group. In other words, Group-Object will give us back information similar to this:

Count 	Name                     	 Group
----- ---- -----
5 Apple {Apple, Apple, Apple, Apple...}
1 Peach {Peach}
2 Chocolate {Chocolate, Chocolate}
1 Banana Cream {Banana Cream}
1 Blackberry {Blackberry}
3 Pecan {Pecan, Pecan, Pecan}
1 Lemon Meringue {Lemon Meringue}
1 French Silk {French Silk}
1 Raspberry {Raspberry}
1 Strawberry {Strawberry}
1 Key Lime {Key Lime}

So how do we know which pies are unique? That’s easy; as you can see, the Count property tells us the number of pies in each group. If a group has a Count equal to 1 then that pie is unique; if a group has a Count greater than 1 then that pie is not unique. It’s that simple.

Now, we could have just eyeballed the preceding data and picked out the non-unique pies for ourselves. But why should we do all that work when flying monkeys can do it for us? And why should flying monkeys do all that work when Windows PowerShell can do it for them? That’s why, after Group-Object does its thing, we then pipe the grouped data to the Where-Object cmdlet:

Where-Object {$_.Count -gt 1}

All we’re doing here is filtering out everything except those groups where the Count property is greater than (-gt) 1. What will that give us? That will give us output similar to this:

Count 	Name                      Group
----- ---- -----
5 Apple {Apple, Apple, Apple, Apple...}
2 Chocolate {Chocolate, Chocolate}
3 Pecan {Pecan, Pecan, Pecan}

You know, you’re right: that does look an awful lot like what the Scripting Guy who writes this column had for lunch today, doesn’t it?

That was easy, wasn’t it? Except now you say that all you want is the Name of the non-unique pies; you don’t want the Count and the Group. Okey-doke. With that in mind, let’s take our non-unique pie data and pipe it to the Select-Object cmdlet, like so:

$arrPies | Group-Object | Where-Object {$_.count -gt 1} | Select-Object Name

As you can see, all we’re doing here is asking Select-Object to return the values for a single property: Name. That means that our output will now look like this:

Name
----
Apple
Chocolate
Pecan

That’s more like it. Now suppose we did want the unique, one-of-a-kind pies. In that case, we’d use this line of code:

$arrPies | Group-Object | Where-Object {$_.count -eq 1} | Select-Object Name

This time around we’re asking Where-Object to filter out everything except the groups where the count is equal to 1:

Where-Object {$_.count -eq 1}

And what will that give us? That will give us output like this:

Name
----
Peach
Banana Cream
Blackberry
Lemon Meringue
French Silk
Raspberry
Strawberry
Key Lime

Cool, huh?

We should note that this output is different than the output Get-Unique would generate. (As you might recall, YE mentioned Get-Unique in her question.) The script we just showed you limits the output to groups where the Count is equal to 1; Get-Unique, by contrast, shows you a single instance of each item type regardless of the count. In other words, we have 1 Peach pie, so Get-Unique’s output will include 1 entry for Peach. We have 5 Chocolate pies, so Get-Unique will also have 1 entry for Chocolate. In other words:

Apple
Banana Cream
Blackberry
Chocolate
French Silk
Key Lime
Lemon Meringue
Peach
Pecan
Raspberry
Strawberry

And here’s the code we used to generate that list. Note that Get-Unique requires you to sort the data before passing it along the pipeline:

($arrPies | Sort-Object) | Get-Unique

That should do it, YE; you should now be able to take your collection and parse that information in all sorts of useful and interesting ways. As for the Scripting Guys, we need to get back to work on the Scripting Games; hard as it is to believe, the Games are just a couple weeks away now. What sorts of things do we still have to do to get ready for the Games? Well, one obvious thing is to put together some Sapien Technologies gift bags containing the cool things Don Jones sent us. And yes, that includes the Flingshot Flying Monkeys. We realize that some of you aren’t interested in winning a Dr. Scripto bobblehead doll; scripting books; copies of PrimalScript, PrimalScope, PowerShell Analyzer, PowerShell Plus, or SystemScripter; a fully-licensed copy of SpecOps Command; or one of 50 copies of Windows Vista Ultimate. (See our Prizes page for complete details.) But now that flying monkeys have been added to the mix, well, that changes everything, doesn’t it?

Note. And yes, we do intend to give away all the flying monkeys. After all, these monkeys were intended for Scripting Games competitors and not for the Scripting Guys.

Well, OK, we’ll probably hang on to one monkey that was … badly damaged … in shipping. We’d feel really bad about foisting a damaged monkey on someone, so ….

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • I would like to get a list of non-unique items from a specific column in CSV-file. Can you provide me with an example script how to arrange this? Thanks in advance!

  • @Xypriz

    Let's say you have a CSV ($file) with columns Name, Date, Telephone and want to determine non-unique names:

    Import-CSV $File | ? { $_.Name -eq "Item1" } | Group-Object | ? { $_.Count -gt 1 }