DL a file from the web

[David Schwartz 426]

This could be a fun little challenge ... I got to wondering what is the shortest bit of code that would let you download a file (mp3 in my case, but any file in the general case) given a URL to a public web site?

 

I'm mainly interested in Windows and WebCore, but solutions for other platforms would be interesting to see as well.

 

function DownloadWebFile( const aURL : string ) : TStream;

 

How many different ways can this be done on each platform? It boggles the mind. Which offers the least amount of Delphi code to accomplish it? (I suspect there are some libraries that have this built-in, but I'm lookng for something that uses low-level APIs, not another library.

 

In my case, I'm calling a REST API and it sends back a URL for an MP3 file, and I need to download it and do something with it.

 

 

 

Edited December 2, 2022 by David Schwartz

[Fr0sT.Brutal 900]

THTTPClient is the least amount of code. If you're on Windows only, check WinInet https://github.com/Fr0sT-Brutal/Delphi_OSMMap/blob/master/Source/OSM.NetworkRequest.WinInet.pas

 

1 hour ago, David Schwartz said:

Which offers the least amount of Delphi code to accomplish it? (I suspect there are some libraries that have this built-in, but I'm lookng for something that uses low-level APIs, not another library.

Low level? This contradicts to "least code". Sockets is the most low level but I doubt you want to deal with them 🙂

Edited December 2, 2022 by Fr0sT.Brutal

[programmerdelphi2k 237]

the minor code for me is: open Chrome and paste the url  

[Fr0sT.Brutal 900]

CreateProcess ('curl.exe ' + URL) 🙂

 

 

[Angus Robertson 574]

There are simple ways to download a file with zero error handling, and proper ways. 

 

Without error handling, unexpected things can happen.  My PC crashed out of disk space a few weeks ago.  Eventually I discovered the Sage accounting software had downloaded 11,000 copies of a tmp file, totalling over 400 gigs, over a couple of days, in background.  No error handling and a lousy design, expected better from a company that charges hundreds of pounds to get a new software version that supports TLS/1.2. 

 

Angus

 

[Attila Kovacs 629]

https://chat.openai.com/chat

 

"To download a file from HTTP in Delphi, you can use the TIdHTTP component from the Indy library. This component provides methods for sending HTTP requests and receiving HTTP responses, including the ability to download files from a URL. Here is an example of how you could use TIdHTTP to download a file in Delphi:"

 

uses
  IdHTTP;

var
  http: TIdHTTP;
  url: string;
  filename: string;
  stream: TMemoryStream;
begin
  // Create the HTTP client object
  http := TIdHTTP.Create;

  // Specify the URL of the file you want to download
  url := 'http://www.example.com/myfile.txt';

  // Specify the name of the file where you want to save the downloaded data
  filename := 'C:\myfile.txt';

  // Create a memory stream to hold the downloaded data
  stream := TMemoryStream.Create;

  try
    // Use the HTTP client to download the file from the specified URL
    http.Get(url, stream);

    // Save the downloaded data to the specified file
    stream.SaveToFile(filename);
  finally
    // Clean up
    stream.Free;
    http.Free;
  end;
end.

"In this example, the TIdHTTP.Get method is used to download the file from the specified URL and save the data to the TMemoryStream object. Once the download is complete, the data is saved to the specified file using the TMemoryStream.SaveToFile method."

 

 

2nd run:

 

 

uses
  IdHTTP;

procedure DownloadFile(const URL, FileName: string);
var
  HTTP: TIdHTTP;
  Stream: TFileStream;
begin
  HTTP := TIdHTTP.Create(nil);
  try
    Stream := TFileStream.Create(FileName, fmCreate);
    try
      HTTP.Get(URL, Stream);
    finally
      Stream.Free;
    end;
  finally
    HTTP.Free;
  end;
end;

 

 "download a file from http in delphi in the background":

uses
  IdHTTP, Classes;

type
  TDownloadThread = class(TThread)
  private
    FURL: string;
    FFileName: string;
    FHTTP: TIdHTTP;
    FStream: TFileStream;
  protected
    procedure Execute; override;
  public
    constructor Create(const URL, FileName: string);
    destructor Destroy; override;
  end;

constructor TDownloadThread.Create(const URL, FileName: string);
begin
  inherited Create(False);
  FURL := URL;
  FFileName := FileName;
  FHTTP := TIdHTTP.Create(nil);
  FStream := TFileStream.Create(FileName, fmCreate);
end;

destructor TDownloadThread.Destroy;
begin
  FStream.Free;
  FHTTP.Free;
  inherited Destroy;
end;

procedure TDownloadThread.Execute;
begin
  FHTTP.Get(FURL, FStream);
end;

procedure DownloadFile(const URL, FileName: string);
var
  Thread: TDownloadThread;
begin
  Thread := TDownloadThread.Create(URL, FileName);
  Thread.Start;
end;

 

 

 

 

Edited December 2, 2022 by Attila Kovacs

[David Schwartz 426]

12 hours ago, programmerdelphi2k said:

the minor code for me is: open Chrome and paste the url  

assumes Chrome is installed. And if it's set to "Ask user for destination" (as mine is) then it will keep popping up for each file, right? What if you want it to go to a specific folder -- as in, the program has no way of knowing where Chrome is set to download the files to?
 

 

Edited December 3, 2022 by David Schwartz

[David Schwartz 426]

16 hours ago, Fr0sT.Brutal said:

THTTPClient is the least amount of code. If you're on Windows only, check WinInet https://github.com/Fr0sT-Brutal/Delphi_OSMMap/blob/master/Source/OSM.NetworkRequest.WinInet.pas

 

Low level? This contradicts to "least code". Sockets is the most low level but I doubt you want to deal with them 🙂

That is, it's not dependent on anything else being installed than what the platform comes with. However, an open-source lib installed into Delphi is OK. The thing is, most libs are based on low-level system calls, like Wininet, right? 

 

Anyway, your code is 162 lines. 

Edited December 3, 2022 by David Schwartz

[David Schwartz 426]

9 hours ago, Angus Robertson said:

There are simple ways to download a file with zero error handling, and proper ways. 

 

Without error handling, unexpected things can happen.  My PC crashed out of disk space a few weeks ago.  Eventually I discovered the Sage accounting software had downloaded 11,000 copies of a tmp file, totalling over 400 gigs, over a couple of days, in background.  No error handling and a lousy design, expected better from a company that charges hundreds of pounds to get a new software version that supports TLS/1.2. 

 

Angus

 

I'm looking for code, not a lecture. 🙂 

[programmerdelphi2k 237]

18 hours ago, David Schwartz said:

what is the shortest bit of code that would let you download a file

@David Schwartz

for sure, I think that you understand my fun-session! 🙂

 

now, what you exactly want to say with "what is the shortest bit of code that would let you download a file?"...

 

I see something very intriguing in this question!
 

I explain:
Even if the client (programmer) types just a single word- (command) like "downloadMyFilePlease('Hello.mp3')" (let's not consider the filename for now), what happens behind the scenes is realized by the operating system, at least!!!
And, in this context, the operating system accessed its library of commands to download... and these commands were coded by someone, who wrote tens, or thousands of lines of code so that "such a task" could be carried out successfully or no!!!
 

So, what exactly would be the definition of the smallest amount of bits to perform the file download?

 

• You mean: the smallest amount of bits encoded by you or the operating system as a whole?

 

I ask for the simple reason: if you don't code, then someone else will have to code for you!!! Or am I wrong?

[David Schwartz 426]

well ... to be most accurate, I guess what I'm thinking is ... starting from a fresh Delphi 4.2.2 install (the version I have), what's the least amount of lines of Delphi source code (probably in a function) that could be used for this. I'm not trying to save memory or have ultra-short variable names, but I've seen some fairly long examples that seemed like overkill for this purpose. 

 

Again, I'm calling a REST API call and it's returning a URL to a (mp3) file that I need to download; it's deleted in an hour or so, assuming it was grabbed fairly quickly. My code needs it immediately anyway, so ... I just got curious what the shortest amount of code needed to DL it into a memory stream for further dealings.

 

I'd also accept something in a free component lib. But not dependent on any particular apps in the environment. Windows has Wininet, Mac has something, Linux has something. Whatever may be there. I guess if you could force something like Chrome to be installed, that might simplify things a bit.

[Fred Ahrens 59]

uses
  Vcl.ExtActns;

procedure Download(URL: String; FileName: String);
var
  DownloadAction: TDownloadUrl;
begin
  DownloadAction := TDownloadUrl.Create(nil);
  DownloadAction.Filename:=FileName;
  DownloadAction.URL:=URL;
  DownloadAction.Execute;
  DownloadAction.Free;
end;

But no error handling at all.

[ertank 27]

There maybe other codes shorter. Below should work on FMX and VCL for versions that has NetHTTPClient support.

procedure TForm1.Button1Click(Sender: TObject);
var
  MS: TMemoryStream;
  Response: IHTTPResponse;
begin
  MS := TMemoryStream.Create();
  try
    try
      Response := NetHTTPClient1.Get('my_url', MS);
      if (Response.StatusCode < 200) or (Response.StatusCode > 299) then
      begin
        raise Exception.Create('Throw an exception or log response. Something is wrong');
      end;
    except
      on E: Exception do
      begin
        raise Exception.Create('Throw the exception or log it');
      end;
    end;

    //Do what you need to do with the memory stream
  finally
    MS.Free();
  end;
end;

 

I placed a component on the form itself for above example. If you need to create TNetHTTPClient instance by code, you will need following units in your uses

uses
  System.Net.URLClient,
  System.Net.HttpClient,
  System.Net.HttpClientComponent;

 

That code (having changed necessary parts) supposed to work on Windows, Linux, MacOS as it is without you to install any external libraries like OpenSSL libraries.

If you get rid of error handling, you'll end up with a shorter version. This is something not suggested as you would know.

 

I do not understand why you are after "shortest" code though.

[David Schwartz 426]

8 hours ago, ertank said:

I do not understand why you are after "shortest" code though.

I'm just trying to uncover things like what Fred illustrated two posts above that may be hiding in places few people look. Without looking into Fred's solution, I don't know if it just saves to disk or if I can get back a TMemoryStream, which is what I need in this case. 

 

Most components have LoadFromFile / SaveToFile, and it is odd to me that nothing provides a simple GetFromURL and SaveToURL. Some people don't know that LoadFromFile/SaveToFile even exist on so many compnents and they write code to Assign File ... and whatnot instead.

 

I was applying for a job one time and was given a short "programming test" for Delphi. It was pretty simple: load up a CSV file and display it; then delete a column, sort by the 3rd column and display it again. Another requirement was that you must employ a TStringlist in some way. Most people don't know that TStringlists can load and manipulate CSV files directly, and the guy who ended up hiring me said I was one of only a few people who seemed to know that. It's just something they've done for ages, yet very few people know.

 

That's kind of what I'm trying to uncover here.

[Remy Lebeau 1396]

18 hours ago, Fred Ahrens said:

uses
  Vcl.ExtActns;

procedure Download(URL: String; FileName: String);
var
  DownloadAction: TDownloadUrl;
begin
  DownloadAction := TDownloadUrl.Create(nil);
  DownloadAction.Filename:=FileName;
  DownloadAction.URL:=URL;
  DownloadAction.Execute;
  DownloadAction.Free;
end;

But no error handling at all.

It raises an exception on error. However, internally it uses Windows' Urlmon.URLDownloadToFile() function, which is notoriously buggy, and crappy at reporting errors.

[Fr0sT.Brutal 900]

On 12/3/2022 at 4:56 AM, David Schwartz said:

That is, it's not dependent on anything else being installed than what the platform comes with. However, an open-source lib installed into Delphi is OK. The thing is, most libs are based on low-level system calls, like Wininet, right? 

 

Anyway, your code is 162 lines. 

In this particular case I just can't understand your requirements. If I hadn't know you by previous posts I'd think you're just trolling.

Just go with builtin THTTPClient (using OS facilities for TLS) or Indy (uses OpenSSL).

On 12/3/2022 at 4:56 AM, David Schwartz said:

Anyway, your code is 162 lines. 

procedure TUpdateThread.Download;
var
  buf: array[0..1023] of Byte;
  read: DWORD;
  HFile: HINTERNET;
const
  ErrorPatt = 'Error getting file from %s: %s';
  FLAGS = INTERNET_FLAG_NO_CACHE_WRITE or INTERNET_FLAG_RELOAD or INTERNET_FLAG_PRAGMA_NOCACHE or
          INTERNET_FLAG_NO_COOKIES or INTERNET_FLAG_NO_UI;
begin
  try try
    FBuffer.Clear;
    // Init WinInet
    FHInet := InternetOpen('Foo', INTERNET_OPEN_TYPE_DIRECT, nil, nil, 0);
    if FHInet = nil then
      Error(IntToStr(GetLastError) + '. ' + SysErrorMessage(GetLastError));
    // Open address
    HFile := InternetOpenUrl(FHInet, PChar(FURL), nil, 0, FLAGS, 0);
    if HFile = nil then
      Error(IntToStr(GetLastError) + '. ' + SysErrorMessage(GetLastError));
    // Read the URL
    while InternetReadFile(HFile, @buf, SizeOf(buf), read) do   
    begin
      if read = 0 then Break;
      FBuffer.Write(buf, read);
    end;
    FBuffer.Position := 0;
  finally
    InternetCloseHandle(HFile);
    InternetCloseHandle(FHInet);
  end;
end;

32 LOC. Anyway are we struggling for the most compact solution? Then nothing will beat ShellExecute('curl.exe '+URL)

Edited December 5, 2022 by Fr0sT.Brutal

[David Schwartz 426]

Man, you guys are just relentless at splitting hairs! I'm simply trying to uncover options that may not be obvious and are fairly short solutions. What's the point of spending a ton of time coming up with rules for a truly casual inquiry that people are only going to put under a microscope anyway? Trolling implies some kind of agenda (to me, anyway). I just posted this out of curiosity.

 

Lots of great answers, thanks to all. More are still welcome if anybody cares to add any.

[Remy Lebeau 1396]

On 12/5/2022 at 12:09 AM, Fr0sT.Brutal said:

 

procedure TUpdateThread.Download;
var
  buf: array[0..1023] of Byte;
  read: DWORD;
  HFile: HINTERNET;
const
  ErrorPatt = 'Error getting file from %s: %s';
  FLAGS = INTERNET_FLAG_NO_CACHE_WRITE or INTERNET_FLAG_RELOAD or INTERNET_FLAG_PRAGMA_NOCACHE or
          INTERNET_FLAG_NO_COOKIES or INTERNET_FLAG_NO_UI;
begin
  try try
    FBuffer.Clear;
    // Init WinInet
    FHInet := InternetOpen('Foo', INTERNET_OPEN_TYPE_DIRECT, nil, nil, 0);
    if FHInet = nil then
      Error(IntToStr(GetLastError) + '. ' + SysErrorMessage(GetLastError));
    // Open address
    HFile := InternetOpenUrl(FHInet, PChar(FURL), nil, 0, FLAGS, 0);
    if HFile = nil then
      Error(IntToStr(GetLastError) + '. ' + SysErrorMessage(GetLastError));
    // Read the URL
    while InternetReadFile(HFile, @buf, SizeOf(buf), read) do   
    begin
      if read = 0 then Break;
      FBuffer.Write(buf, read);
    end;
    FBuffer.Position := 0;
  finally
    InternetCloseHandle(HFile);
    InternetCloseHandle(FHInet);
  end;
end;

 

This code has many small mistakes. Too many try's. Undefined behavior calling InternetCloseHandle() on invalid handles if either InternetOpen() or InternetOpenUrl() fail. Misuse of GetLastError(). Not reporting an error if InternetReadFile() fails. Not making sure FBuffer actually writes all of the bytes it is given.

 

[programmerdelphi2k 237]

in short: it's a bug without wings, thinking it flies high!

[Fr0sT.Brutal 900]

7 hours ago, Remy Lebeau said:

This code has many small mistakes

Yep, that's an excerpt from working but old code with irrelevant parts removed.

7 hours ago, Remy Lebeau said:

Undefined behavior calling InternetCloseHandle()

That's insignificant - GetLastError won't be needed on connect failure as the exception will raise

7 hours ago, Remy Lebeau said:

Misuse of GetLastError()

Could you clarify?

7 hours ago, Remy Lebeau said:

Not reporting an error if InternetReadFile() fails

Here you're right! Forgot about it

7 hours ago, Remy Lebeau said:

Not making sure FBuffer actually writes all of the bytes it is given.

Hmm? You mean checking that Write(buf, read)=read? Stream is TMemoryStream there, I think failure in that case will be the least issue the machine has. However for other variant of WinInet downloading this note is actual, thanks!

Edited December 9, 2022 by Fr0sT.Brutal

[Remy Lebeau 1396]

11 hours ago, Fr0sT.Brutal said:

That's insignificant - GetLastError won't be needed on connect failure as the exception will raise

The point I was trying to make is that, if InternetOpen() fails, the value of HFile is left indeterminate, so calling InternetCloseHandle() on it is undefined behavior. Also, InternetCloseHandle() is not documented as accepting null handles, so if either InternetOpen() or InternetOpenUrl() fails, calling InternetCloseHandle() on the failed handle is also undefined behavior. So, best to call InternetCloseHandle() only on successfully opened handles. Which means, using multiple try..finally blocks.

11 hours ago, Fr0sT.Brutal said:

Could you clarify?

Calling GetLastError() multiple times mixed in other system calls risks having the error code wiped out in between the calls. Best to save the error code to a variable as soon as possible, and then use the variable where needed.

11 hours ago, Fr0sT.Brutal said:

Hmm? You mean checking that Write(buf, read)=read?

Yes. Which is why one should generally use WriteBuffer() instead, which will raise if not all bytes are written.

 

[programmerdelphi2k 237]

var
  MyGetLastErrors: TArray<integer>;

... 
procedure MyProcedureToCommands...
begin
  // command...
  MyGetLastErrors := MyGetLastErrors + [GetLastError];
  // ...
  //
  // command...
  MyGetLastErrors := MyGetLastErrors + [GetLastError];
end;
// ...
  for var E in MyGetLastErrors do
    // ...

  if MyHandle <> INVALID_HANDLE_VALUE then // in case will be = null
    CloseHandle(MyHandle);

 

Edited December 9, 2022 by programmerdelphi2k

[Remy Lebeau 1396]

8 hours ago, programmerdelphi2k said:

  if MyHandle <> INVALID_HANDLE_VALUE then // in case will be = null
    CloseHandle(MyHandle);

 

INVALID_HANDLE_VALUE is not null (0), it is -1. Some APIs return 0 on failure, some return -1 instead. Be careful mixing them up. See Why are HANDLE return values so inconsistent?

[programmerdelphi2k 237]

ok

NOTE: sometimes I read Raymon Chen... good mind!

Edited December 10, 2022 by programmerdelphi2k