(Print this page)

Downloading files from HTTPS
Published date: Tuesday, September 19, 2017
On: Moer & Éric Moreau's web site

One question that we see once in a while in forums is how to download a file from a HTTPS site.

Luckily, .Net has everything we need. No special 3rd party library are required.

The reason I had to execute code like this was to download the status of some backups from CrashPlan Pro – a useful cloud-backup provider which I have been personally using for quite some time (notice that I don’t get any benefits from them!).

This month’s downloadable code

This month solution contains both VB and C# projects. The solution was created using Visual Studio 2017 but the code should work as well in older version of the.Net Framework.

Figure 1: The demo application in action

The WebRequest namespace

It is not the first time I am using the WebRequest namespace in an article.

I used it in August 2010 to consume RSS feeds.

It was also used in March 2004 to download contents from regular HTTP source.

What this code does?

The code here will be very simple. It is a single method. It will connect to a HTTPS URL with all the arguments that you would usually get from clicking the various options on the web site after being authenticated.

The resulting file, a CSV file in my case, will be saved locally in the temp folder.

The trick here is to pass the credentials to the page. That mechanism can vary depending on how the web site was created.

I was lucky that CrashPlan Pro was accepting the credentials (user id and password) to be passed in the headers of the request object once encoded with 64-bits digits.

Private Sub TransferFile()
    Try
        ListBox1.Items.Add("Starting the file transfer")

        'Your variables need to be filled with your values
        Dim strURL As String = "https://www.crashplanpro.com/api/Computer?orgUid=YourORGIDGoesHere&export='csv'&incActivity=true&incBackupUsage=true"
        Dim strUID As String = "YourUserIDGoesHere"
        Dim strPWD As String = "YourPasswordGoesHere"

        'creating a web request object
        Dim req As WebRequest = WebRequest.Create(strURL)

        'creating the credentials adding them to the header
        Dim cre As String = $"{strUID}:{strPWD}"
        Dim bytes() As Byte = Encoding.ASCII.GetBytes(cre)
        Dim base64 As String = Convert.ToBase64String(bytes)
        req.Headers.Add("Authorization", "basic " + base64)

        'query the web site for the response to our request
        Dim resp As WebResponse = req.GetResponse()

        'Create the local file 
        Dim strDest As String = Path.Combine(Path.GetTempPath(), Date.Now.ToString("yyyyMMdd_HHmmss_") + "Crashplan.csv")

        'process the response
        Dim arrBuffer(16384) As Byte
        Dim intBytesRead As Integer
        Using stmLocalFile As FileStream = New FileStream(strDest, FileMode.Create)
            With resp.GetResponseStream
                Do
                    intBytesRead = .Read(arrBuffer, 0, 16384)
                    stmLocalFile.Write(arrBuffer, 0, intBytesRead)
                Loop Until intBytesRead = 0

                .Close()
                'Force all data out of memory and into the file before closing 
                stmLocalFile.Flush()
                stmLocalFile.Close()
            End With
        End Using

        ListBox1.Items.Add("File downloaded")
        ListBox1.Items.Add(strDest)

    Catch ex As Exception
        ListBox1.Items.Add("Something went wrong...")
        ListBox1.Items.Add(ex.Message)
    End Try
End Sub
private void TransferFile()
{
    try
    {
        ListBox1.Items.Add("Starting the file transfer");

        //Your variables need to be filled with your values
        string strURL = "https://www.crashplanpro.com/api/Computer?orgUid=YourORGIDGoesHere&export='csv'&incActivity=true&incBackupUsage=true";
        string strUID = "YourUserIDGoesHere";
        string strPWD = "YourPasswordGoesHere";

        //creating a web request object
        WebRequest req = WebRequest.Create(strURL);

        //creating the credentials adding them to the header
        string cre = $"{strUID}:{strPWD}";
        byte[] bytes = Encoding.ASCII.GetBytes(cre);
        string base64 = Convert.ToBase64String(bytes);
        req.Headers.Add("Authorization", "basic " + base64);

        //query the web site for the response to our request
        WebResponse resp = req.GetResponse();

        //Create the local file 
        string strDest = Path.Combine(Path.GetTempPath(), DateTime.Now.ToString("yyyyMMdd_HHmmss_") + "Crashplan.csv");

        //process the response
        byte[] arrBuffer = new byte[16385];
        int intBytesRead;
        using (FileStream stmLocalFile = new FileStream(strDest, FileMode.Create))
        {
            var respStream = resp.GetResponseStream();
            do
            {
                intBytesRead = respStream.Read(arrBuffer, 0, 16384);
                stmLocalFile.Write(arrBuffer, 0, intBytesRead);
            } while (intBytesRead != 0);

            respStream.Close();
            //Force all data out of memory and into the file before closing 
            stmLocalFile.Flush();
            stmLocalFile.Close();
        }

        ListBox1.Items.Add("File downloaded");
        ListBox1.Items.Add(strDest);
    }
    catch (Exception e)
    {
        ListBox1.Items.Add("Something went wrong...");
        ListBox1.Items.Add(e.Message);
    }            
}

Conclusion

I needed a way to get the status of my backups sent to CrashPlan Pro. Of course, they are offering emails with those statuses but I wanted to integrate the results in a monitoring dashboard showing the “health” of my system.

The WebRequest namespace has everything I needed to connect to a protected HTTPS site and get the report I needed.


(Print this page)