Upload BIG files via HTTP

Thank you @Stoune, it was last thing that helped to receive finally working solution.

One more, it is need to organize stream file reading and writing to the webrequest buffer. And it possibly to do with that piece of code:

$requestStream = $webRequest.GetRequestStream()$fileStream = [System.IO.File]::OpenRead($file)$chunk = New-Object byte[] $bufSize  while( $bytesRead = $fileStream.Read($chunk,0,$bufsize) )  {    $requestStream.write($chunk, 0, $bytesRead)    $requestStream.Flush()  }

And final script look like this:

$user = "admin"$pass = "admin123"$dir = "C:\Virtual Hard Disks"$fileName = "win2012r2std.vhdx"$file = "$dir/$fileName"$url = "http://nexus.lab.local:8081/nexus/content/sites/myproj/$fileName"$Timeout=10000000$bufSize=10000$cred = New-Object System.Net.NetworkCredential($user, $pass)$webRequest = [System.Net.HttpWebRequest]::Create($url)$webRequest.Timeout = $timeout$webRequest.Method = "POST"$webRequest.ContentType = "application/data"$webRequest.AllowWriteStreamBuffering=$false$webRequest.SendChunked=$true # needed by previous line$webRequest.Credentials = $cred$requestStream = $webRequest.GetRequestStream()$fileStream = [System.IO.File]::OpenRead($file)$chunk = New-Object byte[] $bufSize  while( $bytesRead = $fileStream.Read($chunk,0,$bufsize) )  {    $requestStream.write($chunk, 0, $bytesRead)    $requestStream.Flush()  }$responceStream = $webRequest.getresponse()#$status = $webRequest.statuscode$FileStream.Close()$requestStream.Close()$responceStream.Close()$responceStream$responceStream.GetResponseHeader("Content-Length") $responceStream.StatusCode#$status

By default HttpWebRequest is buffering data in memory.Just set HttpWebRequest.AllowWriteStreamBuffering property to false and you would be able to upload files with almost any size.See more details at msdn

For uploading to Sonatype Nexus3 I used the code below. Took me some time to figure it out, working with uploads and reponse from Nexus3 and uploading and downloading large files (larger than 2GB). We have Apache in front of Nexus3 taking care of the https connections.Using basic authentication made sure that Nexus was responding right, when Apache is in front of it.Sending pre-authentication via HEAD and using chunked upload for large files fixed that uploading large files did not end prematurely.

Downloading large files via Invoke-WebRequest would fail too with different errors. Now I use WebRequest via .Net and it works, see Download-File.

And when the code was run via an automated process (from System Center Orchestrator), https would fail. So we force TLS 1.2 when scheme https is detected.

function New-HttpWebRequest{    <#    .SYNOPSIS    Creates a new [System.Net.HttpWebRequest] ready for file transmission.    .DESCRIPTION    Creates a new [System.Net.HttpWebRequest] ready for file transmission.    The method will be Put. If the filesize is larger than the buffersize,    the HttpWebRequest will be configured for chunked transfer.    .PARAMETER Url    Url to connect to.    .PARAMETER Credential    Credential for authentication at the Url resource.    .EXAMPLE    An example    #>    param(        [Parameter(Mandatory=$true)]        [ValidateNotNullOrEmpty()]        [string]$Url,        [Parameter(Mandatory=$true)]        [ValidateNotNullOrEmpty()]        [pscredential]$Credential,        [Parameter(Mandatory=$true)]        [long]$FileSize,        [Parameter(Mandatory=$true)]        [long]$BufferSize    )    $webRequest = [System.Net.HttpWebRequest]::Create($Url)    $webRequest.Timeout = 600 * 1000;    $webRequest.ReadWriteTimeout = 600 * 1000;    $webRequest.ProtocolVersion = [System.Net.HttpVersion]::Version11;    $webRequest.Method = "PUT";    $webRequest.ContentType = "application/octet-stream";    $webRequest.KeepAlive = $true;    $webRequest.UserAgent = "<I use a specific UserAgent>";    #$webRequest.UserAgent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)';    $webRequest.PreAuthenticate = $true;    $auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName + ":" + $Credential.GetNetworkCredential().Password));    $webRequest.Headers["Authorization"] = "Basic $auth"    if (Get-UseChunkedUpload -FileSize $FileSize -BufferSize $BufferSize)    {        Write-Verbose "FileSize is greater than BufferSize, using chunked transfer.";        $webRequest.AllowWriteStreamBuffering = $false;        $webRequest.SendChunked = $true;    }    else    {        # Filesize is equal to or smaller than the BufferSize. The file will be transferred in one write.        # Chunked cannot be used in this case.        $webRequest.AllowWriteStreamBuffering = $true;        $webRequest.SendChunked = $false;        $webRequest.ContentLength = $FileSize;    }    return $webRequest;}function Get-BufferSize{    <#    .SYNOPSIS    Returns a buffer size that is 1% of ByteLength, rounded in whole MB's or at least AtLeast size.    .DESCRIPTION    Returns a buffer size that is 1% of ByteLength, rounded to whole MB's or if 1% is smaller than AtLeast, then AtLeast size is returned which is 1MB by default.    .PARAMETER ByteLength    Length of the bytes for which to calculate a valid buffer size.    .PARAMETER AtLeast    The minimum required buffer size, default 1MB.    .EXAMPLE    Get-BufferSize 4283304773    Returns 42991616 which is 41MB.    .EXAMPLE    Get-BufferSize 4283304    Returns 1048576 which is 1MB.    .EXAMPLE    Get-BufferSize 4283304 5MB    Returns 5242880 which is 5MB.    #>    param(        [Parameter(Mandatory=$true)]        [long]$ByteLength,        [long]$AtLeast = 1MB    )    [long]$size = $ByteLength / 100;    if ($size -lt $AtLeast)    {        $size = $AtLeast;    }    else    {        $size = [Math]::Round($size / 1MB) * 1MB;    }    return $size;}function Get-UseChunkedUpload{    param(        [Parameter(Mandatory=$true)]        [long]$FileSize,        [Parameter(Mandatory=$true)]        [long]$BufferSize    )    return $FileSize -gt $BufferSize;}function Configure-Tls{    param(        [Parameter(Mandatory=$true)]        [ValidateNotNullOrEmpty()]        [string]$Url    )    [System.Uri]$uri = $Url;    if ($uri.Scheme -eq "https")    {        Write-Verbose "Using TLS 1.2";        [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12;    }}function Send-PreAuthenticate{    param(        [Parameter(Mandatory=$true)]        [ValidateNotNullOrEmpty()]        [string]$Url,        [Parameter(Mandatory=$true)]        [ValidateNotNullOrEmpty()]        [pscredential]$Credential    )    $response = $null;    try    {        [System.Uri]$uri = $Url;        $repositoryAuthority = (($uri.GetLeftPart([System.UriPartial]::Authority)).TrimEnd('/') + '/');        Write-Verbose "Send-PreAuthenticate - Sending HEAD to $repositoryAuthority";        $wr = [System.Net.WebRequest]::Create($repositoryAuthority);        $wr.Method = "HEAD";        $wr.PreAuthenticate = $true;        $auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName + ":" + $Credential.GetNetworkCredential().Password));        $wr.Headers["Authorization"] = "Basic $auth"        $response = $wr.GetResponse();    }    finally    {        if ($response)        {            $response.Close();            $response.Dispose();            $response = $null;        }    }}function Upload-File{    <#    .SYNOPSIS    Uploads a file to the Nexus repository.    .DESCRIPTION    Uploads a file to the Nexus repository.    If the file was uploaded successfully, the url via which the resource can be downloaded is returned.    .PARAMETER Url    The Url where the resource should be created.    Please note that underscores and dots should be encoded, otherwise the Nexus repository does not accept the upload.    .PARAMETER File    The file that should be uploaded.    .PARAMETER Credential    Credential used for authentication at the Nexus repository.    .EXAMPLE    Upload-File -Url https://nexusrepo.domain.com/repository/repo-name/myfolder/myfile%2Eexe -File (Get-ChildItem .\myfile.exe) -Credential (Get-Credential)    .OUTPUTS    If the file was uploaded successfully, the url via which the resource can be downloaded.    #>    param(        [Parameter(Mandatory=$true)]        [ValidateNotNullOrEmpty()]        [string]$Url,        [Parameter(Mandatory=$true)]        [ValidateNotNullOrEmpty()]        [System.IO.FileInfo]$File,        [Parameter(Mandatory=$true)]        [ValidateNotNullOrEmpty()]        [pscredential]$Credential    )    Write-Verbose "Upload-File Url:$Url"    Configure-Tls -Url $Url;    $fileSizeBytes = $File.Length;    #$bufSize = Get-BufferSize $fileSizeBytes;    $bufSize = 4 * 1MB;    Write-Verbose ("FileSize is {0} bytes ({1:N0}MB). BufferSize is {2} bytes ({3:N0}MB)" -f $fileSizeBytes,($fileSizeBytes/1MB),$bufSize,($bufSize/1MB));    if (Get-UseChunkedUpload -FileSize $fileSizeBytes -BufferSize $bufSize)    {        Write-Verbose "Using chunked upload. Send pre-auth first.";        Send-PreAuthenticate -Url $Url -Credential $Credential;    }    $progressActivityMessage = ("Sending file {0} - {1} bytes" -f $File.Name, $File.Length);    $webRequest = New-HttpWebRequest -Url $Url -Credential $Credential -FileSize $fileSizeBytes -BufferSize $bufSize;    $chunk = New-Object byte[] $bufSize;    $bytesWritten = 0;    $fileStream = [System.IO.File]::OpenRead($File.FullName);    $requestStream = $WebRequest.GetRequestStream();    try    {        while($bytesRead = $fileStream.Read($chunk,0,$bufSize))        {            $requestStream.Write($chunk, 0, $bytesRead);            $requestStream.Flush();            $bytesWritten += $bytesRead;            $progressStatusMessage = ("Sent {0} bytes - {1:N0}MB" -f $bytesWritten, ($bytesWritten / 1MB));            Write-Progress -Activity $progressActivityMessage -Status $progressStatusMessage -PercentComplete ($bytesWritten/$fileSizeBytes*100);        }    }    catch    {        throw;    }    finally    {        if ($fileStream)        {            $fileStream.Close();        }        if ($requestStream)        {            $requestStream.Close();            $requestStream.Dispose();            $requestStream = $null;        }        Write-Progress -Activity $progressActivityMessage -Completed;    }    # Read the response.    $response = $null;    try    {        $response = $webRequest.GetResponse();        Write-Verbose ("{0} responded with {1} at {2}" -f $response.Server,$response.StatusCode,$response.ResponseUri);        return $response.ResponseUri;    }    catch    {        if ($_.Exception.InnerException -and ($_.Exception.InnerException -like "*bad request*"))        {            throw ("ERROR: " + $_.Exception.InnerException.Message + " Possibly the file already exists or the content type of the file does not match the file extension. In that case, disable MIME type validation on the server.")        }        throw;    }    finally    {        if ($response)        {            $response.Close();            $response.Dispose();            $response = $null;        }        if ($webRequest)        {            $webRequest = $null;        }    }}function Download-File{    param(        [Parameter(Mandatory=$true)]        [ValidateNotNullOrEmpty()]        [string]$Url,        [Parameter(Mandatory=$true)]        [ValidateNotNullOrEmpty()]        [string]$FileName    )    $SDXDownloadType = @"    using System.IO;    using System.Net;    public class SDXDownload    {        static public void DownloadFile(string Uri, string Filename)        {            HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(Uri);            webRequest.Method = "GET";            using (HttpWebResponse myHttpWebResponse = (HttpWebResponse)webRequest.GetResponse())            using (Stream fileStream = File.OpenWrite(Filename))            using (Stream streamResponse = myHttpWebResponse.GetResponseStream())            {                int bufSize = 64 * 1024;                byte[] readBuff = new byte[bufSize];                int bytesRead = streamResponse.Read(readBuff, 0, bufSize);                while (bytesRead > 0)                {                    fileStream.Write(readBuff, 0, bytesRead);                    bytesRead = streamResponse.Read(readBuff, 0, 256);                }            }        }    }"@    Configure-Tls -Url $Url;    Add-Type -TypeDefinition $SDXDownloadType;    [SDXDownload]::DownloadFile($Url, $FileName);}