Powershell and writing files (how fast can you write to a file? )

Powershell and writing files (how fast can you write to a file? )

  • Comments 12
  • Likes

Hi there,

I’ve been working for some time on a tool similar to PAL from mike lagase in order to automate the analysis of loadgen runs.

While doing this I had to write large files with PowerShell and was not impressed with the result.

I thought I’d share the problem and solution with the community.

There are at least four ways of writing files with PSH:

·         use the ‘>>’' alias that calls into the out-file cmd-let

·         use export-csv

·         use .Net

So let’s say I want to write 10000 lines into a file.

Method 1: use the >> operator

$s=”some text”

1..10000 | % {
     $s >> ".\test.txt"
}

 

This way you actually open and close the file 1000 times.

Method 2: use out-file to write an Array

1..10000 | % {
    $a+=[Array] $s
}

$a | out-file ".\test.txt"

This way actually writes once, using powershell.

Method 3: use export-csv to write an Array

$a=@()

1..10000 | % {

      $a+=[Array] $s

}

$a | export-csv "t.txt"

Export-csv can also write the array for you

 

Method 4: use .Net StreamWriter to write the file

 

$stream = [System.IO.StreamWriter] "t.txt"

1..10000 | % {

      $stream.WriteLine($s)

}

$stream.close()

The StreamWriter object from .Net also does the work nicely.

Conclusion: how fast ?

I tried all methods on my laptop, and here is what I got:

 Method

Time to completion

‘>>’

29 s

Outfile and [Array]

27 s

export-csv

22 s

StreamWriter

1.5 s

 

Well results speak by themselves, if you are in a hurry, use .Net StreamWriter !

Cheers,

Next, check that with Powerhell v2!

Guillaume

 

Comments
  • Je viens de publier un petit post rapide sur la méthode d’écriture dans un fichier avec PSH : Powershell

  • 1 s with old school language : vbscript ;-)

  • Merci pour ce post, interessant surtout avec le temps d'execution, A+

  • @dak

    > file.txt doesn't accomplish the same thing

  • $b = 1..10000

    $b | Out-File C:\Temp\temp.txt

    2.3 seconds

  • Very interesting. However, you compared a little different between these methods. You use  the StreamWriter to write line by line, and you use the other methods to write in one big chunk. Maybe this affects performance? I made a comparison myself (using the big chunk method), and the >> method and StreamWriter method had the same performance (both 40 seconds).

  • In your example for Stream Writer it does not output anything to the text file...

    You should have $_ in place of your $s

    Example:

    $stream = [System.IO.StreamWriter] "t.txt"

    1..10000 | % {

         $stream.WriteLine($_)

    }

    $stream.close()

  • @Marcus -- The four snippets actually belong in one file, that is why you see reuse of the $s variable in all 4 examples with only one definition in the first.  It would probably have been better to post the 4 examples as a single file, or post 4 complete files with all definitions needed included in each one.

  • Method 3 doesn't actually work for an array; at least it didn't for mine.  Export-CSV looks at properties and values, not array contents.    So as written above, it will write a file out, but when run against an array of 16 character addresses, it will produce a single column with header value of "Length" in row zero and with the "16" length of each entry on every other output line.

    To make that work, more code has to be added.  I've seen solutions where the array is piped into SELECT-OBJECT creating a new PSObject for each value before the export.  I've seen other solutions that claim adding " -Delimiter ";" -NoTypeInformation" to the Export will help, but haven't tested it since methods 1 and 2 already worked fine for me.

    Thanks!

    -Bob

  • only 0.08 sec with the funny language Batch

  • Thanks for this. I was able to take a 500k line out-file -append script and cut it down from well over an hour to about 18 minutes using the streamWriter suggestion.

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment