How can I remove duplicates from a PowerShell array?
$a = @(1,2,3,4,5,5,6,7,8,9,0,0)
12 Answers
Use Select-Object
(whose alias is select
) with the -Unique
switch; e.g.:
$a = @(1,2,3,4,5,5,6,7,8,9,0,0) $a = $a | select -Unique
6Another option is to use Sort-Object
(whose alias is sort
, but only on Windows) with the-Unique
switch, which combines sorting with removal of duplicates:
$a | sort -unique
4In case you want to be fully bombproof, this is what I would advise:
@('Apples', 'Apples ', 'APPLES', 'Banana') | Sort-Object -Property @{Expression={$_.Trim()}} -Unique
Output:
Apples Banana
This uses the Property
parameter to first Trim()
the strings, so extra spaces are removed and then selects only the -Unique
values.
More info on Sort-Object
:
Get-Help Sort-Object -ShowWindow
1$a | sort -unique
This works with case-insensitive, therefore removing duplicates strings with differing cases. Solved my problem.
$ServerList = @( "FS3", "HQ2", "hq2" ) | sort -Unique $ServerList
The above outputs:
FS3 HQ2
1This is how you get unique from an array with two or more properties. The sort is vital and the key to getting it to work correctly. Otherwise you just get one item returned.
PowerShell Script:
$objects = @( [PSCustomObject] @{ Message = "1"; MachineName = "1" } [PSCustomObject] @{ Message = "2"; MachineName = "1" } [PSCustomObject] @{ Message = "3"; MachineName = "1" } [PSCustomObject] @{ Message = "4"; MachineName = "1" } [PSCustomObject] @{ Message = "5"; MachineName = "1" } [PSCustomObject] @{ Message = "1"; MachineName = "2" } [PSCustomObject] @{ Message = "2"; MachineName = "2" } [PSCustomObject] @{ Message = "3"; MachineName = "2" } [PSCustomObject] @{ Message = "4"; MachineName = "2" } [PSCustomObject] @{ Message = "5"; MachineName = "2" } [PSCustomObject] @{ Message = "1"; MachineName = "1" } [PSCustomObject] @{ Message = "2"; MachineName = "1" } [PSCustomObject] @{ Message = "3"; MachineName = "1" } [PSCustomObject] @{ Message = "4"; MachineName = "1" } [PSCustomObject] @{ Message = "5"; MachineName = "1" } [PSCustomObject] @{ Message = "1"; MachineName = "2" } [PSCustomObject] @{ Message = "2"; MachineName = "2" } [PSCustomObject] @{ Message = "3"; MachineName = "2" } [PSCustomObject] @{ Message = "4"; MachineName = "2" } [PSCustomObject] @{ Message = "5"; MachineName = "2" } ) Write-Host "Sorted on both properties with -Unique" -ForegroundColor Yellow $objects | Sort-Object -Property Message,MachineName -Unique | Out-Host Write-Host "Sorted on just Message with -Unique" -ForegroundColor Yellow $objects | Sort-Object -Property Message -Unique | Out-Host Write-Host "Sorted on just MachineName with -Unique" -ForegroundColor Yellow $objects | Sort-Object -Property MachineName -Unique | Out-Host
Output:
Sorted on both properties with -Unique Message MachineName ------- ----------- 1 1 1 2 2 1 2 2 3 1 3 2 4 1 4 2 5 1 5 2 Sorted on just Message with -Unique Message MachineName ------- ----------- 1 1 2 1 3 1 4 1 5 2 Sorted on just MachineName with -Unique Message MachineName ------- ----------- 1 1 3 2
Source: https://powershell.org/forums/topic/need-to-unique-based-on-multiple-properties/
1If the list is sorted, you can use the Get-Unique cmdlet:
$a | Get-Unique
1With my method you can completely remove duplicate values, leaving you with values from the array that only had a count of 1. It was not clear if this is what the OP actually wanted however I was unable to find an example of this solution online so here it is.
$array=@' Bananna Apple Carrot Pear Apricot Pear Bananna '@ -split '\r\n' ($array | Group-Object -NoElement | ?{$_.count -eq 1}).Name
1Whether you're using SORT -UNIQUE
, SELECT -UNIQUE
or GET-UNIQUE
from Powershell 2.0 to 5.1, all the examples given are on single Column arrays. I have yet to get this to function across Arrays with multiple Columns to REMOVE Duplicate Rows to leave single occurrences of a Row across said Columns, or develop an alternative script solution. Instead these cmdlets have only returned Rows in an Array that occurred ONCE with singular occurrence and dumped everything that had a duplicate. Typically I have to Remove Duplicates manually from the final CSV output in Excel to finish the report, but sometimes I would like to continue working with said data within Powershell after removing the duplicates.
A lot of the provided answers give buggy results. select -Unqiue
is not case sensitive. sort -Unique
gives you sorted results, which you might want in the original order.
Will gave a great answer, but it's flawed as it discards all duplicate results but forgets to keep one of them.
This is a version I created that seems to work perfectly. It gives unique results back and retains the original sort order.
($properties | Group-Object -NoElement).Name | Get-Unique
To get unique items from an array and preserve their order, you can use .NET HashSet:
$Array = @(1, 3, 1, 2) $Set = New-Object -TypeName 'System.Collections.Generic.HashSet[int]' -ArgumentList (,[int[]]$Array) # PS> $Set # 1 # 3 # 2
Works best with string arrays that contain both uppercase and lowercase items where you need to preserve first occurrence of each item in case-insensitive manner:
$Array = @("B", "b", "a", "A") $Set = New-Object -TypeName 'System.Collections.Generic.HashSet[string]' -ArgumentList ([string[]]$Array, [StringComparer]::OrdinalIgnoreCase) # PS> $Set # B # a
Works as expected with other types.
Shortened syntax, compatible with PowerShell 5.1 and newer:
$Array = @("B", "b", "a", "A") $Set = [Collections.Generic.HashSet[string]]::new([string[]]$Array, [StringComparer]::OrdinalIgnoreCase) $Array = @(1, 3, 1, 2) $Set = [Collections.Generic.HashSet[int]]::new([int[]]$Array)
$a = @('D:/', 'two', 'three') $s = "D:/" $null -ne ($a | ? { $s -match $_ }) # Returns $true
another solution can be used here , u're welcome
1I do this:
$data = Get-Content $file | ConvertFrom-Json | Select-Object -Property id, name, streetName1, zipCode, city | Sort-Object -Property id -Unique
ncG1vNJzZmirpJawrLvVnqmfpJ%2Bse6S7zGiorp2jqbawutJoaGxxYW2CdHvRnqSorpmjtG6w1KmjopuRqbJuwsClrJ6rXZu%2FsLmMmmSpp6eav7S0xKWjZpmip666