Removing duplicate values from a PowerShell array

How can I remove duplicates from a PowerShell array?

$a = @(1,2,3,4,5,5,6,7,8,9,0,0) 

12 Answers

Use Select-Object (whose alias is select) with the -Unique switch; e.g.:

$a = @(1,2,3,4,5,5,6,7,8,9,0,0) $a = $a | select -Unique 
6

Another option is to use Sort-Object (whose alias is sort, but only on Windows) with the
-Unique switch, which combines sorting with removal of duplicates:

$a | sort -unique 
4

In case you want to be fully bombproof, this is what I would advise:

@('Apples', 'Apples ', 'APPLES', 'Banana') | Sort-Object -Property @{Expression={$_.Trim()}} -Unique 

Output:

Apples Banana 

This uses the Property parameter to first Trim() the strings, so extra spaces are removed and then selects only the -Unique values.

More info on Sort-Object:

Get-Help Sort-Object -ShowWindow 
1
$a | sort -unique 

This works with case-insensitive, therefore removing duplicates strings with differing cases. Solved my problem.

$ServerList = @( "FS3", "HQ2", "hq2" ) | sort -Unique $ServerList 

The above outputs:

FS3 HQ2 
1

This is how you get unique from an array with two or more properties. The sort is vital and the key to getting it to work correctly. Otherwise you just get one item returned.

PowerShell Script:

$objects = @( [PSCustomObject] @{ Message = "1"; MachineName = "1" } [PSCustomObject] @{ Message = "2"; MachineName = "1" } [PSCustomObject] @{ Message = "3"; MachineName = "1" } [PSCustomObject] @{ Message = "4"; MachineName = "1" } [PSCustomObject] @{ Message = "5"; MachineName = "1" } [PSCustomObject] @{ Message = "1"; MachineName = "2" } [PSCustomObject] @{ Message = "2"; MachineName = "2" } [PSCustomObject] @{ Message = "3"; MachineName = "2" } [PSCustomObject] @{ Message = "4"; MachineName = "2" } [PSCustomObject] @{ Message = "5"; MachineName = "2" } [PSCustomObject] @{ Message = "1"; MachineName = "1" } [PSCustomObject] @{ Message = "2"; MachineName = "1" } [PSCustomObject] @{ Message = "3"; MachineName = "1" } [PSCustomObject] @{ Message = "4"; MachineName = "1" } [PSCustomObject] @{ Message = "5"; MachineName = "1" } [PSCustomObject] @{ Message = "1"; MachineName = "2" } [PSCustomObject] @{ Message = "2"; MachineName = "2" } [PSCustomObject] @{ Message = "3"; MachineName = "2" } [PSCustomObject] @{ Message = "4"; MachineName = "2" } [PSCustomObject] @{ Message = "5"; MachineName = "2" } ) Write-Host "Sorted on both properties with -Unique" -ForegroundColor Yellow $objects | Sort-Object -Property Message,MachineName -Unique | Out-Host Write-Host "Sorted on just Message with -Unique" -ForegroundColor Yellow $objects | Sort-Object -Property Message -Unique | Out-Host Write-Host "Sorted on just MachineName with -Unique" -ForegroundColor Yellow $objects | Sort-Object -Property MachineName -Unique | Out-Host 

Output:

Sorted on both properties with -Unique Message MachineName ------- ----------- 1 1 1 2 2 1 2 2 3 1 3 2 4 1 4 2 5 1 5 2 Sorted on just Message with -Unique Message MachineName ------- ----------- 1 1 2 1 3 1 4 1 5 2 Sorted on just MachineName with -Unique Message MachineName ------- ----------- 1 1 3 2 

Source: https://powershell.org/forums/topic/need-to-unique-based-on-multiple-properties/

1

If the list is sorted, you can use the Get-Unique cmdlet:

 $a | Get-Unique 
1

With my method you can completely remove duplicate values, leaving you with values from the array that only had a count of 1. It was not clear if this is what the OP actually wanted however I was unable to find an example of this solution online so here it is.

$array=@' Bananna Apple Carrot Pear Apricot Pear Bananna '@ -split '\r\n' ($array | Group-Object -NoElement | ?{$_.count -eq 1}).Name 
1

Whether you're using SORT -UNIQUE, SELECT -UNIQUE or GET-UNIQUE from Powershell 2.0 to 5.1, all the examples given are on single Column arrays. I have yet to get this to function across Arrays with multiple Columns to REMOVE Duplicate Rows to leave single occurrences of a Row across said Columns, or develop an alternative script solution. Instead these cmdlets have only returned Rows in an Array that occurred ONCE with singular occurrence and dumped everything that had a duplicate. Typically I have to Remove Duplicates manually from the final CSV output in Excel to finish the report, but sometimes I would like to continue working with said data within Powershell after removing the duplicates.

2

A lot of the provided answers give buggy results. select -Unqiue is not case sensitive. sort -Unique gives you sorted results, which you might want in the original order.

Will gave a great answer, but it's flawed as it discards all duplicate results but forgets to keep one of them.

This is a version I created that seems to work perfectly. It gives unique results back and retains the original sort order.

($properties | Group-Object -NoElement).Name | Get-Unique

To get unique items from an array and preserve their order, you can use .NET HashSet:

$Array = @(1, 3, 1, 2) $Set = New-Object -TypeName 'System.Collections.Generic.HashSet[int]' -ArgumentList (,[int[]]$Array) # PS> $Set # 1 # 3 # 2 

Works best with string arrays that contain both uppercase and lowercase items where you need to preserve first occurrence of each item in case-insensitive manner:

$Array = @("B", "b", "a", "A") $Set = New-Object -TypeName 'System.Collections.Generic.HashSet[string]' -ArgumentList ([string[]]$Array, [StringComparer]::OrdinalIgnoreCase) # PS> $Set # B # a 

Works as expected with other types.

Shortened syntax, compatible with PowerShell 5.1 and newer:

$Array = @("B", "b", "a", "A") $Set = [Collections.Generic.HashSet[string]]::new([string[]]$Array, [StringComparer]::OrdinalIgnoreCase) $Array = @(1, 3, 1, 2) $Set = [Collections.Generic.HashSet[int]]::new([int[]]$Array) 
$a = @('D:/', 'two', 'three') $s = "D:/" $null -ne ($a | ? { $s -match $_ }) # Returns $true 

another solution can be used here , u're welcome

1

I do this:

$data = Get-Content $file | ConvertFrom-Json | Select-Object -Property id, name, streetName1, zipCode, city | Sort-Object -Property id -Unique 

ncG1vNJzZmirpJawrLvVnqmfpJ%2Bse6S7zGiorp2jqbawutJoaGxxYW2CdHvRnqSorpmjtG6w1KmjopuRqbJuwsClrJ6rXZu%2FsLmMmmSpp6eav7S0xKWjZpmip666