Visualizing PowerShell pipeline

A picture1 is worth a thousand words.

Occasionally, I see people having issues while trying to understand how PowerShell pipeline is executed. Most of them have no problems when Begin/Process/End blocks are in the single function. And if in doubt, I can always point them to the Don Jones’ The Advanced Function Lifecycle article. But when multiple cmdlets are chained into the one single pipeline, things become a little less clear.

Consider this example.

function Use-Begin {
    Begin {
        Write-Host 'Begin'
    }
}

function Use-End {
    End {
        Write-Host 'End'
    }
}

Let’s try to pipe one function into another:

PS C:\Users\beatcracker> Use-Begin | Use-End

Begin
End

So far, so good, nothing unexpected. The Begin block of the Use-Begin function executes first, and the End block of the Use-End function executes last.

But what happens if we swap the functions in our pipeline?

PS C:\Users\beatcracker>  Use-End | Use-Begin

Begin
End

Gotcha! As you can see, nothing changed. Regardless of the position of the cmdlet in the pipeline, Begin blocks are always executed first and End blocks last. This could be a bit counterintuitive, because it’s easy to imagine pipeline like this:

Begin-1 {} -> Process-1 {} -> End-1 {} | Begin-2 {} -> Process-2 {} -> End-2 {}

When in fact, the pipeline goes this way:

Begin-1 {}
Begin-2 {}
    Process-1 {} | Process-2 {}
End-1 {}
End-2 {}

Which is more logical, when you think about it for a second: for every function in the pipeline, the Begin blocks have to be executed once, before the Process block and when Process blocks are finished iterating over every pipeline element, it’s time to finally run the End block. This gives us to the picture above.

To illustrate my point, I’ve created a View-Pipeline function, that generates a chained pipeline of advanced functions with Begin/Process/End blocks and displays their execution order. It makes it easy to visualize pipeline processing and get solid understanding of the pipeline lifecycle.

Here are some visualization examples made with this function:

  • One advanced function with Begin/Process/End blocks
PS C:\Users\beatcracker> View-Pipeline

View-Pipeline-1

[View-Pipeline-1]::Begin
    [View-Pipeline-1]::Process
      In : ""
      Out: "View-Pipeline-1"
[View-Pipeline-1]::End
  • Three advanced functions, each with its own Begin/Process/End blocks, passing one item through the pipeline
PS C:\Users\beatcracker> View-Pipeline -Pipes 3

View-Pipeline-1 | View-Pipeline-2 | View-Pipeline-3

[View-Pipeline-1]::Begin
[View-Pipeline-2]::Begin
[View-Pipeline-3]::Begin
    [View-Pipeline-1]::Process
      In : ""
      Out: "View-Pipeline-1"
        [View-Pipeline-2]::Process
          In : "View-Pipeline-1"
          Out: "View-Pipeline-2"
            [View-Pipeline-3]::Process
              In : "View-Pipeline-2"
              Out: "View-Pipeline-3"
[View-Pipeline-1]::End
[View-Pipeline-2]::End
[View-Pipeline-3]::End
  • Two advanced functions, with only Process/End blocks, passing two items through the pipeline
PS C:\Users\beatcracker> View-Pipeline -Pipes 2 -Items 2 -NoBegin

View-Pipeline-1 | View-Pipeline-2

    [View-Pipeline-1]::Process
      In : ""
      Out: "View-Pipeline-1"
        [View-Pipeline-2]::Process
          In : "View-Pipeline-1"
          Out: "View-Pipeline-2"
        [View-Pipeline-2]::Process
          In : "View-Pipeline-1"
          Out: "View-Pipeline-2"
[View-Pipeline-1]::End
[View-Pipeline-2]::End

That was easy, isn’t it?

I’m hoping that anyone, when shown this post, can get the gist of the PowerShell’s pipeline lifecycle in no time. If not – just let me know and I’ll do my best to improve it.


  1. There are actually no pictures here. Sorry. 

$PSBoundParameters, Pipeline and the ValueFromPipelineByPropertyName parameter attribute

While using and abusing $PSBoundParameters in one of the my functions, I’ve encountered some strange and unexpected behaviour. If you pass an object via pipeline to a function which accepts parameters from an object properties using ValueFromPipelineByPropertyName, Powershell re-binds $PSBoundParameters only for existing object properties. If object property is not exists in the next pipeline loop, $PSBoundParameters will keep the key from the previous iteration.

Example:

Function Test-PSBoundParameters
{
    [CmdletBinding()]
    Param
    (
        [Parameter(Mandatory = $false, ValueFromPipelineByPropertyName = $true)]
        $ParamA,
        [Parameter(Mandatory = $false, ValueFromPipelineByPropertyName = $true)]
        $ParamB
    ) 

    Process
    {
        '_______________________'
        "Bound 'ParamA': $($PSBoundParameters.ParamA)"
        "Variable 'ParamA': $ParamA"
        "Bound 'ParamB': $($PSBoundParameters.ParamB)"
        "Variable 'ParamB': $ParamB"
    }
} 

$Test = @{ParamA = 'ParamA' ; ParamB = 'ParamB'}, @{ParamA = 'ParamA'}

$Test | ForEach-Object {New-Object psobject -Property $_} | Test-PSBoundParameters

What I expect:

_______________________
Bound 'ParamA': ParamA
Variable 'ParamA': ParamA
Bound 'ParamB': ParamB
Variable 'ParamB': ParamB
_______________________
Bound 'ParamA': ParamA
Variable 'ParamA': ParamA
Bound 'ParamB':
Variable 'ParamB':

What really happens:

_______________________
Bound 'ParamA': ParamA
Variable 'ParamA': ParamA
Bound 'ParamB': ParamB
Variable 'ParamB': ParamB
_______________________
Bound 'ParamA': ParamA
Variable 'ParamA': ParamA
Bound 'ParamB': ParamB
Variable 'ParamB':

As you can see, the second object in pipeline doesn’t have the ParamB property. I expected that both ParamB variable an ParamB key in $PSBoundParameters in that case wouldn’t be assigned. But what actually happens, is that PowerShell keeps stale key for ParamB from the previous object in the next pipeline loop.

I’ve found this thread on the Windows PowerShell forums about this issue. One of the proposed solutions was to clear $PSBoundParameters on every iteration using $PSBoundParameters.Clear() or rely solely on corresponding variables. But clearing $PSBoundParameters breaks pipeline and I wanted to be able to use both $PSBoundParameters and variables.

The only solution I was able to come with, is to manually check if $PSBoundParameters keys’ values are the same as corresponding variables and if not – remove those keys. The comaprison uses the Object.Equals method if it’s available:

        $StaleKeys = $PSBoundParameters.GetEnumerator() |
                        ForEach-Object {
                            if($_.Value.PSobject.Methods.Name -match '^Equals$')
                            {
                                # If object has Equals, compare bound key and variable using it
                                if(!$_.Value.Equals((Get-Variable -Name $_.Key -ValueOnly -Scope 0)))
                                {
                                    $_.Key
                                }
                            }
                            else
                            {
                                # If object doesn't has Equals (e.g. $null), fallback to the PowerShell's -ne operator
                                if($_.Value -ne (Get-Variable -Name $_.Key -ValueOnly -Scope 0))
                                {
                                    $_.Key
                                }
                            }

                        }
        $StaleKeys | ForEach-Object {[void]$PSBoundParameters.Remove($_)}

Here is the function with the quirk mitigated:

Function Test-PSBoundParameters{
[CmdletBinding()]
 	Param(
 	    [Parameter(Mandatory = $false, ValueFromPipelineByPropertyName = $true)]
 	    $ParamA,
 	    [Parameter(Mandatory = $false, ValueFromPipelineByPropertyName = $true)]
 	    $ParamB
    ) 
    Process
    {
        $StaleKeys = $PSBoundParameters.GetEnumerator() |
                        ForEach-Object {
                            if($_.Value.PSobject.Methods.Name -match '^Equals$')
                            {
                                # If object has Equals, compare bound key and variable using it
                                if(!$_.Value.Equals((Get-Variable -Name $_.Key -ValueOnly -Scope 0)))
                                {
                                    $_.Key
                                }
                            }
                            else
                            {
                                # If object doesn't has Equals (e.g. $null), fallback to the PowerShell's -ne operator
                                if($_.Value -ne (Get-Variable -Name $_.Key -ValueOnly -Scope 0))
                                {
                                    $_.Key
                                }
                            }

                        }
        $StaleKeys | ForEach-Object {[void]$PSBoundParameters.Remove($_)}

        '_______________________'
        "Stale Keys: $StaleKeys"
        '_______________________'
        "Bound 'ParamA': $($PSBoundParameters.ParamA)"
        "Variable 'ParamA': $ParamA"
        "Bound 'ParamB': $($PSBoundParameters.ParamB)"
        "Variable 'ParamB': $ParamB"
    }
}

$Test = @{ParamA = 'ParamA' ; ParamB = 'ParamB'}, @{ParamA = 'ParamA'}

$Test | ForEach-Object {New-Object psobject -Property $_} | Test-PSBoundParameters

And the result:

_______________________
Stale Keys: 
_______________________
Bound 'ParamA': ParamA
Variable 'ParamA': ParamA
Bound 'ParamB': ParamB
Variable 'ParamB': ParamB
_______________________
Stale Keys: ParamB
_______________________
Bound 'ParamA': ParamA
Variable 'ParamA': ParamA
Bound 'ParamB': 
Variable 'ParamB':

[Updated: 15 May 2015 – My previous approach to comparing $PsBoundParameters keys and variables using PowerShell’s -ne operator failed to work with anything but simple value types (such as strings or integers)]
In effect, comparison of the arrays, hashtables, objects and so on failed. But it seems that I’ve found a much better solution now: the Object.Equals method. Looks like it fits perfectly for this task, but in case I’ve missed something again, please let me know.

Splatting and mandatory parameters

Splat them. For the PowerShell knows those that are His own.

Splatting is very useful PowerShell technique that allows to greatly simplify calls to a functions or cmdlets. Basically, instead of passing arguments one by one, you create a hashtable where keys named as arguments and their values are passed to the function.

I use splatting extensively. Usually I have one big hashtable, where all the settings for the current script are stored. The hashtable itself is read from INI file, which are in my opinion provide optimum balance between hardcoding settings into the script itself and storing them in XML files. So, when I’ve to call a function in my script, I just splat this hashtable on it and PowerShell binds values from the hashtable to the function parameters. Well, at least this should be so in a perfect world.

The problem comes when you decide that you’re grown up enough to write advanced functions and declare parameters using PowerShell’s Param() statement and parameter arguments. Not having to manually validate if a parameter supplied or not removes a great burden from your shoulders. Combined with Parameter Validation attributes it allows to write a more streamlined and easier to read code. Of course it also makes parameter definition section somewhat bloated, but it totally worth it.

One of the first things to try is a Mandatory argument to make a parameter required. Here is example: I’d like to splat a hashtable $UserInfo, which contains details about some user, on the function Test-Password to validate that user’s password is strong enough, before proceeding further. The parameters Password and UserName in this function are declared as mandatory:

$UserInfo = @{
    Computer = 'Wks_1'
    UserName = 'John'
    Password = '!Passw0rd'
    Comment = 'Marketing'
}

function Test-Password
{
    [CmdletBinding()]
    Param
    (
        [Parameter(Mandatory = $true)]
        [string]$Password,

        [Parameter(Mandatory = $true)]
        [string]$UserName
    )

    Process
    {
        if($Password.Length -ge 8 -and $Password -ne $UserName)
        {
            return $true
        }
        else
        {
            return $false
        }
    }
}

Test-Password @UserInfo

But this is what happens when you actually run this code: if any of parameters in your function are declared as Mandatory and you splat hashtable with extra values, PowerShell will reject your futile attempt.

Why? For the heck of it, that’s why.

Test-Password : A parameter cannot be found that matches parameter name 'UserName'.
At C:\Scripts\Test-Splat.ps1:35 char:15
+ Test-Password @UserInfo
+ ~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Test-Password], ParameterBindingException
+ FullyQualifiedErrorId : NamedParameterNotFound,Test-Password

I’ve no reasonable explanation for this behavior. Sure it would be nice to know, that I’ve passed more parameters the function, that it can handle. Because maybe I was wrong about it, so it’s OK to tell me (Write-Warning would do the job). I’m sure, that behind the scenes, there was some reason to make splatting behave this way, but it seems to me that is not valid anymore. I hope, that someone would be able to enlighten me. This excerpt from the about_Functions_CmdletBindingAttribute help looks related, but i doesn’t mention Mandatory argument and doesn’t really explain why this happens:

In functions that have the CmdletBinding attribute, unknown parameters and positional arguments that have no matching positional parameters cause parameter binding to fail.

OK, it’s not a perfect world, alas. I’ve to cope with this and there are several options:

  • Give up the Mandatory attribute.

    Not a real option. Sure I could go back to the wild days of the simple functions and validate attributes manually, but this is boring, error prone and defies progress. Not the things I’d like to do.

  • Pass all attributes as named arguments

    I did this once, and it made my eyes hurt and fingers ache. Even with tab-completion.

    Test-Password -Password $UserInfo.Password -User $UserInfo.UserName
    
  • Write proxy function to strip hashtable of the all the parameters that target function can’t handle.

    Davin K. Tanabe wrote a good one, be sure to check it out. It’s the only sane way to use this kind of one-size-fits-all splatting with built-in cmdlets and 3rd-party functions.

  • Convert hashtable to PSObject and employ ValueFromPipelineByPropertyName

    We can not solve our problems with the same level of thinking that created them.
    – Albert Einstein

    All it takes, is to add ValueFromPipelineByPropertyName to the parameter arguments, convert hashtable to PSObject and pass it to the function via pipeline. Actually, there is no splatting involved at all, but hey, at least you can keep the hashtable!

    $UserInfo = @{
        Computer = 'Wks_1'
        UserName = 'John'
        Password = '!Passw0rd'
        Comment = 'Marketing'
    }
    
    function Test-Password
    {
        [CmdletBinding()]
        Param
        (
            [Parameter(Mandatory = $true, ValueFromPipelineByPropertyName = $true)]
            [string]$Password,
    
            [Parameter(Mandatory = $true, ValueFromPipelineByPropertyName = $true)]
            [string]$UserName
        )
    
        Process
        {
            if($Password.Length -ge 8 -and $Password -ne $UserName)
            {
                return $true
            }
            else
            {
                return $false
            }
        }
    }
    
    # PowerShell 2.0 way:
    New-Object PSObject -Property $UserInfo | Test-Password 
    
    # Powershell 3.0 way:
    [PSCustomObject]$UserInfo | Test-Password
    
  • Harness the magic of ValueFromRemainingArguments

    ValueFromRemaingArguments is not an argument that you encounter often. Here is what PowerShell help has to say about it:

    ValueFromRemainingArguments Argument

    The ValueFromRemainingArguments argument indicates that the parameter accepts all of the parameters values in the command that are not assigned to other parameters of the function.
    The following example declares a ComputerName parameter that is mandatory and accepts all the remaining parameter values that were submitted to the function.

    Param
    (
        [parameter(Mandatory=$true, ValueFromRemainingArguments=$true)]
        [String[]]
        $ComputerName
    )
    

    Sweep it under the carpet and turn a blind eye.

    In short, once declared, it makes your parameter to catch all what’s left after all other parameters were bound. So here is idea: why don’t we let PowerShell bind everything it can from the hashtable and then dump unused values to the dummy parameter $Splat.

    $UserInfo = @{
        Computer = 'Wks_1'
        UserName = 'John'
        Password = '!Passw0rd'
        Comment = 'Marketing'
    }
    
    function Test-Password
    {
        [CmdletBinding()]
        Param
        (
            [Parameter(Mandatory = $true)]
            [string]$Password,
    
            [Parameter(Mandatory = $true)]
            [string]$UserName,
    
            [Parameter(ValueFromRemainingArguments = $true)]
            $Splat
        )
    
        Process
        {
            if($Password.Length -ge 8 -and $Password -ne $UserName)
            {
                return $true
            }
            else
            {
                return $false
            }
        }
    }
    
    Test-Password @UserInfo
    

    This times code runs fine and that’s what we’ve got in the our $Splat variable:

    [DBG]: PS C:\Users\beatcracker>> $Splat
    -Comment:
    Marketing
    -Computer:
    Wks_1
    
    [DBG]: PS C:\Users\beatcracker>> $Splat.GetType()
    IsPublic IsSerial Name      BaseType
    -------- -------- ----      --------
    True     True     ArrayList System.Object

    All our extra arguments are landed safely here, no harm done. Though, I’ll be honest, this is a kludge, but it’s the only way I know to make the PowerShell splatting work as it should.

Actually one can combine the last two approaches: declare all the parameters as ValueFromPipelineByPropertyName and add a dummy parameter with a ValueFromRemainingArguments argument. This is what I’ve ended up with:

$UserInfo = @{
    Computer = 'Wks_1'
    UserName = 'John'
    Password = '!Passw0rd'
    Comment = 'Marketing'
}

function Test-Password
{
    [CmdletBinding()]
    Param
    (
        [Parameter(Mandatory = $true, ValueFromPipelineByPropertyName = $true)]
        [string]$Password,

        [Parameter(Mandatory = $true, ValueFromPipelineByPropertyName = $true)]
        [string]$UserName,

        [Parameter(ValueFromRemainingArguments = $true)]
        $Splat
    )

    Process
    {
        if($Password.Length -ge 8 -and $Password -ne $UserName)
        {
            return $true
        }
        else
        {
            return $false
        }
    }
}

# Splatting:
Test-Password @UserInfo

# PowerShell 2.0 way:
New-Object PSObject -Property $UserInfo | Test-Password 

# Powershell 3.0 way:
[PSCustomObject]$UserInfo | Test-Password

As you can see, now I’m able to do splatting and piping, while keeping my parameters as mandatory as my heart desires.