Visualizing PowerShell pipeline

A picture1 is worth a thousand words.

Occasionally, I see people having issues while trying to understand how PowerShell pipeline is executed. Most of them have no problems when Begin/Process/End blocks are in the single function. And if in doubt, I can always point them to the Don Jones’ The Advanced Function Lifecycle article. But when multiple cmdlets are chained into the one single pipeline, things become a little less clear.

Consider this example.

function Use-Begin {
    Begin {
        Write-Host 'Begin'
    }
}

function Use-End {
    End {
        Write-Host 'End'
    }
}

Let’s try to pipe one function into another:

PS C:\Users\beatcracker> Use-Begin | Use-End

Begin
End

So far, so good, nothing unexpected. The Begin block of the Use-Begin function executes first, and the End block of the Use-End function executes last.

But what happens if we swap the functions in our pipeline?

PS C:\Users\beatcracker>  Use-End | Use-Begin

Begin
End

Gotcha! As you can see, nothing changed. Regardless of the position of the cmdlet in the pipeline, Begin blocks are always executed first and End blocks last. This could be a bit counterintuitive, because it’s easy to imagine pipeline like this:

Begin-1 {} -> Process-1 {} -> End-1 {} | Begin-2 {} -> Process-2 {} -> End-2 {}

When in fact, the pipeline goes this way:

Begin-1 {}
Begin-2 {}
    Process-1 {} | Process-2 {}
End-1 {}
End-2 {}

Which is more logical, when you think about it for a second: for every function in the pipeline, the Begin blocks have to be executed once, before the Process block and when Process blocks are finished iterating over every pipeline element, it’s time to finally run the End block. This gives us to the picture above.

To illustrate my point, I’ve created a View-Pipeline function, that generates a chained pipeline of advanced functions with Begin/Process/End blocks and displays their execution order. It makes it easy to visualize pipeline processing and get solid understanding of the pipeline lifecycle.

Here are some visualization examples made with this function:

  • One advanced function with Begin/Process/End blocks
PS C:\Users\beatcracker> View-Pipeline

View-Pipeline-1

[View-Pipeline-1]::Begin
    [View-Pipeline-1]::Process
      In : ""
      Out: "View-Pipeline-1"
[View-Pipeline-1]::End
  • Three advanced functions, each with its own Begin/Process/End blocks, passing one item through the pipeline
PS C:\Users\beatcracker> View-Pipeline -Pipes 3

View-Pipeline-1 | View-Pipeline-2 | View-Pipeline-3

[View-Pipeline-1]::Begin
[View-Pipeline-2]::Begin
[View-Pipeline-3]::Begin
    [View-Pipeline-1]::Process
      In : ""
      Out: "View-Pipeline-1"
        [View-Pipeline-2]::Process
          In : "View-Pipeline-1"
          Out: "View-Pipeline-2"
            [View-Pipeline-3]::Process
              In : "View-Pipeline-2"
              Out: "View-Pipeline-3"
[View-Pipeline-1]::End
[View-Pipeline-2]::End
[View-Pipeline-3]::End
  • Two advanced functions, with only Process/End blocks, passing two items through the pipeline
PS C:\Users\beatcracker> View-Pipeline -Pipes 2 -Items 2 -NoBegin

View-Pipeline-1 | View-Pipeline-2

    [View-Pipeline-1]::Process
      In : ""
      Out: "View-Pipeline-1"
        [View-Pipeline-2]::Process
          In : "View-Pipeline-1"
          Out: "View-Pipeline-2"
        [View-Pipeline-2]::Process
          In : "View-Pipeline-1"
          Out: "View-Pipeline-2"
[View-Pipeline-1]::End
[View-Pipeline-2]::End

That was easy, isn’t it?

I’m hoping that anyone, when shown this post, can get the gist of the PowerShell’s pipeline lifecycle in no time. If not – just let me know and I’ll do my best to improve it.


  1. There are actually no pictures here. Sorry. 

Parameter validation gotchas

I didn’t fail the test, I just found 100 ways to do it wrong.
— Benjamin Franklin

PowerShell’s parameter validation is a blessing. Validate parameters properly and you’ll never have to write a code that deals with erroneous user input. But sometimes dealing with Validation Attributes, requires a bit more knowledge that built-in help can provide. Here is what I’ve learned so far and want to share with you.

  • You can have more than one Validation Attribute

This may seem trivial, but PowerShell’s help and various online tutorials do not mention this fact (they just imply). You can have as much Validation Attributes as you like for your parameter. For example, this function requires parameter Number to be even and fall in range from 1 to 256:

function Test-MultipleValidationAttributes
{
    [CmdLetBinding()]
    Param
    (
        [Parameter(Mandatory = $true, ValueFromPipeline = $true)]
        [ValidateScript({
                if($_ % 2)
                {
                    throw 'Supply an even number!'
                }
                $true
        })]
        [ValidateRange(1,256)]
        [int]$Number
    )

    Process
    {
        Write-Host "Congratulations, $Number is an even and it's between 1 and 256!"
    }
}
  • You can have more than one instance of specific Validation Attribute

While not especially useful, it still could be used in some practical cases. Imagine, that you have some complex logic behind attribute validation with the ValidateScript attribute. Then it could be split between several ValidateScript attributes to decrease script complexity. Here is the the function from above, modified to validate parameters using two ValidateScript attributes. As a bonus, you can throw more meaningful error messages:

function Test-MultipleSimilarValidationAttributes
{
    [CmdLetBinding()]
    Param
    (
        [Parameter(Mandatory = $true, ValueFromPipeline = $true)]
        [ValidateScript({
                if($_ % 2)
                {
                    throw 'Supply an even number!'
                }
                $true
        })]
        [ValidateScript({
                if($_ -lt 1 -or $_ -gt 256)
                {
                    throw 'Supply number between 1 and 256!'
                }
                $true
        })]
        [int]$Number
    )

    Process
    {
        Write-Host "Congratulations, $Number is an even and it's between 1 and 256!"
    }
}
  • Once parameter is validated, it stays validated

Once you’ve set the rules, they apply to everyone, even to you. Want proof? Check this out:

function Test-PersistentValidation
{
    [CmdLetBinding()]
    Param
    (
        [Parameter(Mandatory = $true, ValueFromPipeline = $true)]
        [ValidateRange(1,256)]
        [int]$Number
    )

    Process
    {
        $Number = 0
    }
}

Let’s see, where our rebellious spirit will lead us:

PS C:\Users\beatcracker> Test-PersistentValidation -Number 1
The variable cannot be validated because the value 0 is not a valid value for the Number variable.
At C:\Scripts\Test-Validation.ps1:13 char:9
+         $Number = 0
+         ~~~~~~~~~~~
    + CategoryInfo          : MetadataError: (:) [], ValidationMetadataException
    + FullyQualifiedErrorId : ValidateSetFailure

Not actually damnation of the soul, but close enough. This happens because PowerShell assigns Validation Attributes to the variable and they will stay there until variable is destroyed. You can view them, and I’ll show you how. Place a breakpoint on the line $Number = 0, call the function and wait for the debugger to pop up:

PS C:\Users\beatcracker> Test-PersistentValidation -Number 1
Hit Line breakpoint on 'C:\Scripts\Test-Validation.ps1:13'
[DBG]: PS C:\Scripts>> $tmp = Get-Variable Number

[DBG]: PS C:\Scripts>> $tmp.Attributes

TypeId
------
System.Management.Automation.ArgumentTypeConverterAttribute
System.Management.Automation.ValidateRangeAttribute
System.Management.Automation.ParameterAttribute

[DBG]: PS C:\Scripts>> $tmp.Attributes[1]

MinRange MaxRange TypeId
-------- -------- ------
       1      256 System.Management.Automation.ValidateRangeAttribute

And there is more to it! Starting with the PowerShell 3.0, you can place an attribute on any variable:

PS C:\Users\beatcracker> [ValidateRange(1,256)][int]$Number = 1

PS C:\Users\beatcracker> $Number = 0
The variable cannot be validated because the value 0 is not a valid value for the Number variable.
At line:1 char:1
+ $Number = 0
+ ~~~~~~~~~~~
    + CategoryInfo          : MetadataError: (:) [], ValidationMetadataException
    + FullyQualifiedErrorId : ValidateSetFailure

Details about this feature: New V3 Language Features, #PSTip Validate your custom objects.

  • Validation Attributes order matters

This the one of those things you learn the hard way. It’s not actually mentioned anywhere as far as I know. Below is a function, that accepts optional parameter Path, that has to be existing folder on disk. Since there is no sense in running validation script if supplied parameter is empty, there is a ValidateNotNullOrEmpty attribute added.

function Test-ValidationAttributesOrder
{
    [CmdLetBinding()]
    Param
    (
        [Parameter(ValueFromPipelineByPropertyName = $true)]
        [ValidateNotNullOrEmpty()]
        [ValidateScript({
            if(!(Test-Path -LiteralPath $_ -PathType Container))
            {
                throw "Folder not found: $_"
            }
            $true
        })]
        [string]$Path
    )

    Process
    {
        if($Path)
        {
            Write-Host "Optional parameter Path is a valid folder!"
        }
    }
}

Hence, when the function is called with the empty parameter, I expect it to throw “The argument is null or empty” error message. But instead, the validation fails where I do not expect it to:

PS C:\Users\beatcracker> Test-ValidationAttributesOrder -Path ''
Test-ValidationAttributesOrder : Cannot validate argument on parameter 'Path'. Cannot bind argument to parameter 'LiteralPath' because it is an empty string.
At line:1 char:38
+ Test-ValidationAttributesOrder -Path ''
+                                      ~~
    + CategoryInfo          : InvalidData: (:) [Test-ValidationAttributesOrder], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,Test-ValidationAttributesOrder

As you see, my empty string passed to the Path parameter, completely missed ValidateNotNullOrEmpty attribute and landed directly at the validation script, where PowerShell rightly failed to bind it to LiteralPath parameter of the Test-Path cmdlet. Makes you wonder, doesn’t it? After some trial and error I was finally able to make sense of it:

PowerShell evaluates Validation Attributes in a Bottom to Top order

So, armed with this knowledge, let’s fix the function above and make it validate parameters in the correct order:

function Test-ValidationAttributesOrder
{
    [CmdLetBinding()]
    Param
    (
        [Parameter(ValueFromPipelineByPropertyName = $true)]
        [ValidateScript({
            if(!(Test-Path -LiteralPath $_ -PathType Container))
            {
                throw "Folder not found: $_"
            }
            $true
        })]
        [ValidateNotNullOrEmpty()]
        [string]$Path
    )

    Process
    {
        if($Path)
        {
            Write-Host "Optional parameter Path is a valid folder!"
        }
    }
}

And test it, passing the empty string to the Path parameter:

PS C:\Users\beatcracker> Test-ValidationAttributesOrder -Path ''
Test-ValidationAttributesOrder : Cannot validate argument on parameter 'Path'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again.
At line:1 char:38
+ Test-ValidationAttributesOrder -Path ''
+                                      ~~
    + CategoryInfo          : InvalidData: (:) [Test-ValidationAttributesOrder], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,Test-ValidationAttributesOrder

Voilà! Now it validates that parameter is not null or empty before passing parameter value to the validation script.