Hi,

I have written a service application in delphi 2007 to get updates from our Amazon S3 file folders.

Its pretty simple, it has a CHRON style timer so basically when its between certain hours (usually 1am until 6am) it downloads a file from Amazon S3, stops a service (Firebird server), unzips the file and replaces the old one held locally. It then restarts the firebird service.

All simple stuff but I have had a few clients report the same problem. The Service crashes with I/O error 1784. I think its not downloading the file and hence cant save it. It only happens when proxy servers are in place. And like I say, most clients work fine.

The service is logged on as Local System account.

Here is where it gets tricky. I want to use IE as opposed to the third party download components as i have a lot of clients (300+) and if I use my download comps i have to set proxy settings and store them securely for every client that uses a proxy. So if I use IE to do it then usually IE has the proxy setup and of course I can test this by manually downloading my file from Amazon S3 on the clients server using IE.

I used some code from Torry's delphi pages to do this:

uses
  URLMon, ShellApi;

function DownloadFile(SourceFile, DestFile: string): Boolean;
begin
  try
    Result := UrlDownloadToFile(nil, PChar(SourceFile), PChar(DestFile), 0, nil) = 0;
  except
    Result := False;
  end;
end;

However this had a massive problem. IE downloads this to a cache usually (windows version dependant) c:\windows\system 32\config\systemprofile\appdatazlocal\microsoft\temporary internet files/content.ie5 etc.

now the problem was that we give them about 4gb of data a week and the function above kept copies of all the cached files so in 10 weeks we were using 40Gb of temporary files! not good.

So i changed the code.

I had some more code from torrys delpi pages that downloads a text file into a string value without hitting the hard drive as a file (very useful for downloading a small text file of update dates or versions for example).

function DownloadURL_NOCache(const aUrl: string; var s: String): Boolean;
var
  hSession: HINTERNET;
  hService: HINTERNET;
  lpBuffer: array[0..1024 + 1] of Char;
  dwBytesRead: DWORD;
begin
  Result := False;
  s := '';
  // hSession := InternetOpen( 'MyApp', INTERNET_OPEN_TYPE_DIRECT, nil, nil, 0);
  hSession := InternetOpen('MyApp', INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0);
  try
    if Assigned(hSession) then
    begin
      hService := InternetOpenUrl(hSession, PChar(aUrl), nil, 0, INTERNET_FLAG_RELOAD, 0);
      if Assigned(hService) then
        try
          while True do
          begin
            dwBytesRead := 1024;
            InternetReadFile(hService, @lpBuffer, 1024, dwBytesRead);
            if dwBytesRead = 0 then break;
            lpBuffer[dwBytesRead] := #0;
            s := s + lpBuffer;
          end;
          Result := True;
        finally
          InternetCloseHandle(hService);
        end;
    end;
  finally
    InternetCloseHandle(hSession);
  end;
end;



but this doesnt download a file but interestingly has a flag of  INTERNET_FLAG_RELOAD

so i ended up using this code i adpated, it works fine but like i say when there is a proxy involved even though i can download the file with IE (with the proxy settings) it returns I/O  error 1784 because there is no file to save.

function GetInetFile(const fileURL, FileName: String): boolean;
const BufferSize = 1024;
var
hSession, hURL: HInternet;
Buffer: array[1..BufferSize] of Byte;
BufferLen: DWORD;
f: File;
sAppName: string;
begin
Result:=False;
sAppName := 'SyncService.exe';
hSession := InternetOpen(PChar(sAppName),
INTERNET_OPEN_TYPE_PRECONFIG,
nil, nil, 0);
try
hURL := InternetOpenURL(hSession,
PChar(fileURL),
nil,0,INTERNET_FLAG_RELOAD,0);
try
AssignFile(f, FileName);
Rewrite(f,1);
repeat
InternetReadFile(hURL, @Buffer,
SizeOf(Buffer), BufferLen);
BlockWrite(f, Buffer, BufferLen)
until BufferLen = 0;
CloseFile(f);
Result:=True;
finally
InternetCloseHandle(hURL)
end
finally
InternetCloseHandle(hSession)
end
end;

Can anyone see what is wrong with this code, why it might not use the proxy? or does the service have to be logged on by someone with special prvilages?

Any help much appreciated.

PS if anyone knows how to download using IE with proxy straight to the loacation with no cache at all used, even better as this still caches a copy under system profile before discarding.

many thanks 

Andy

Anyone got any ideas how to build this proxy finding/setting code found in Remy's answer into the function that we use?

> http://support.microsoft.com/kb/226473

Hi Remy,

i havent tried it simply because i dont know how to create a bit of pascal from the code on that site, dont get me wrong if we could have our service pickup on the proxy regardless of the user its logged in as then i would be very happy i just didnt really understand how to add the code in that link you sent into my download file function.

Andy wrote:

> The service logs on as Local System account, we found that just
> because a user sets up the proxy in IE that doesn't mean the Local
> Service account has the proxy settings. Hence the file could not be
> downloaded.

Yes, IE settings are stored per-user.

> We found three solutions to this I'm still not sure which is best
> practice but I'll post them here for other devs to see.

Is there a problem using the solution I mentioned to you?

{quote}
If you want to assign the proxy in code instead, look at InternetSetOption() 
and its INTERNET_OPTION_PER_CONNECTION_OPTION and INTERNET_OPTION_PROXY... 
flags. See this for an example:

How to programmatically query and set proxy settings under Internet Explorer
http://support.microsoft.com/kb/226473
{quote}

--
Remy Lebeau (TeamB)

hi Remy,

Many thanks your code worked well and it no longer caches the file which is ideal.

We found the problem with the proxy.

The service logs on as Local System account, we found that just because a user sets up the proxy in IE that doesn't mean the Local Service account has the proxy settings. Hence the file could not be downloaded. Using your function I now handle that in a much better way where the service effectively does nothing but writes a warning to the event viewer to say the Internet was unavailable and not the service no longer crashes and stops working.

We found three solutions to this I'm still not sure which is best practice but I'll post them here for other devs to see.

Fix 1. Use regedit to export the current users reg entry for a user for whom IE is set up with the proxy. The key is normally HKEY_current_user/software/Microsoft/current version/Internet settings

Then edit the key in notepad, replace HKEY_CURRENT_USER with HKEY_USER/S-1-5-18 through out the file then import it back into the registry. This adds all the proxy settings to the LOCAL SYSTEM account.

Fix 2 open CMD as admin type NETSH WINHTTP IMPORT PROXY SOURCE = ie
On 64 bit machines you have to change directory to SystemRoot%\SysWOW64  folder first. This works on 2008/12 server I have no idea how to do it on 2003.

Fix 3 setup an account with full local admin access (in our case because our service stops and starts other services) and make sure the proxy settings for that user are correct and then log the service on as that account. While this might seem simple and the safest option a lot of our clients have password expiry policies which means they would have to remember to change the service log on password frequently, which they won't hence the service won't run. 

Anyway what is odd is the vast majority of clients we have with proxy servers run fine, but we have found a handful that need this fix so I'm guessing it's a group policy thing.

Anyone with any thoughts on a neater solution, let me know. 

Many thanks for the improved code again!

Andy wrote:

> Many thanks, will that code handle the proxy if required too?

It uses INTERNET_OPEN_TYPE_PRECONFIG, so it will use whatever proxy is configured 
in IE.  If you want to assign the proxy in code instead, look at InternetSetOption() 
and its INTERNET_OPTION_PER_CONNECTION_OPTION and INTERNET_OPTION_PROXY... 
flags.  See this for an example:

How to programmatically query and set proxy settings under Internet Explorer
http://support.microsoft.com/kb/226473

--
Remy Lebeau (TeamB)

Andy wrote:

> Many thanks, will that code handle the proxy if required too?

It uses INTERNET_OPEN_TYPE_PRECONFIG, so it will use whatever proxy is configured 
in IE.  If you want to assign the proxy in code instead, look at InternetSetOption() 
and its INTERNET_OPTION_PER_CONNECTION_OPTION and INTERNET_OPTION_PROXY... 
flags.  See this for an example:

How to programmatically query and set proxy settings under Internet Explorer
http://support.microsoft.com/kb/226473

--
Remy Lebeau (TeamB)

Ps I had to change hSession = 0 to hSession = nil and hURL = 0 to hURL = nil as it said that operand was not applicable to that type, is that correct?

Hi Remy,

Many thanks, will that code handle the proxy if required too?

my knowledge of this stuff is pretty limited as I normally develop database apps with Delphi using Dev Express grids menus etc etc So this is a first go at updating my data using a service and its my first service I have written.

Ill give your function a try and see how we get on and report back, but many thanks again.

Andy wrote:

> All simple stuff but I have had a few clients report the same problem.
> The Service crashes with I/O error 1784.

That is ERROR_INVALID_USER_BUFFER, "The supplied user buffer is not valid 
for the requested operation."

That is a old-style Pascal file I/O error message.  Why are you using old-style 
Pascal I/O?

> Here is where it gets tricky. I want to use IE as opposed to the third
> party download components

Why?

> i have a lot of clients (300+) and if I use my download comps i have to
> set proxy settings and store them securely for every client that uses a
> proxy. So if I use IE to do it then usually IE has the proxy setup

Yes, but it is a global setting, shared by every WinInet instance, including 
the IE web browser.  If you want to control the proxy setup just for your 
dowwnload clients, you have to set the proxy values in code on a per-socket 
basis  anyway.  So using WinInet vs another library just to gain proxy support 
really doesn't offer a very big advantage.  You could just store the proxy 
settings in one place, or even query the IE settings, and have each client 
apply the settings to each download when needed.

> I used some code from Torry's delphi pages to do this:

> However this had a massive problem. IE downloads this to a cache

Yup.  The only way to bypass that is to download the file manually so you 
control how the data is read and saved.  If you want to stay with the WinInet 
API, you can use InternetReadFile(), just be sure to specify the INTERNET_FLAG_NO_CACHE_WRITE 
flag when creating the request.

> but this doesnt download a file

Yes, it does.

However, the code as shown will not work in Unicode versions of Delphi due 
to the use of Char, which is an alias for WideChar now.  The code assigns 
that Char and String are Ansi, which means this is an old example.  It is 
also not doing ay error handling on the InternetReadFile() call.

> but interestingly has a flag of INTERNET_FLAG_RELOAD

All that flag does is tells WinInet to force the request to be sent to the 
target HTTP server, skipping the local cache and any intermediate caching 
proxies along the way, so that the latest version of the data is retreived.

> so i ended up using this code i adpated, it works fine but like i say
> when there is a proxy involved even though i can download the file
> with IE (with the proxy settings) it returns I/O  error 1784 because
> there is no file to save.

That is not the reason for the error.  And there is a file saved in the cache, 
because you are not using the INTERNET_FLAG_NO_CACHE_WRITE flag.

Try something more like this instead:

function GetInetFile(const fileURL, FileName: String): boolean;
const
  BufferSize = 1024;
var
  hSession, hURL: HInternet;
  Buffer: array[0..BufferSize-1] of Byte;
  BufferLen: DWORD;
  f: TFileStream;
  sAppName: string;
begin
  Result := False;
  hSession := InternetOpen('SyncService.exe', INTERNET_OPEN_TYPE_PRECONFIG, 
nil, nil, 0);
  if hSession = 0 then Exit;
  try
    hURL := InternetOpenURL(hSession, PChar(fileURL), nil, 0, INTERNET_FLAG_RELOAD 
or INTERNET_FLAG_NO_CACHE_WRITE, 0);
    if hURL = 0 then Exit;
    try
      f := TFileStream.Create(FileName, fmCreate);
      try
        try
          repeat
            if not InternetReadFile(hURL, @Buffer[0], SizeOf(Buffer), BufferLen) 
then RaiseLastOSError;
            if BufferLen = 0 then Break;
            f.WriteBuffer(Buffer[0], BufferLen);
          until False;
        finally
          f.Free;
        end;
      except
        DeleteFile(FileName);
        Exit;
      end;
      Result := True;
    finally
      InternetCloseHandle(hURL);
    end
  finally
    InternetCloseHandle(hSession);
  end
end;
{code}

--
Remy Lebeau (TeamB)