Topic Options
#17059 - 05/11/06 04:34 PM FTP Get
IrishWolverine Offline
Junior Member

Registered: 09/12/05
Posts: 15
Loc: South Bend, Indiana
One of my processes will ftpget a .txt file. This work fine; however, even if I tell it to run once an hour, or every 600 seconds or whatnot... how can i tell it to not get the file until it's completed? I run into the problem where it will grab the file before it's done.
Any suggestions? i'm using PP4.2.0.1637


#17060 - 05/12/06 11:32 AM Re: FTP Get
MarkH Offline
OL Expert

Registered: 05/05/03
Posts: 218
Loc: Hartford, Connecticut
Originally posted by IrishWolverine:
how can i tell it to not get the file until it's completed?
I think the best way to approach this is with the process that creates the file. If your ftpget is looking for FNAME.TXT, have it write out the file as ANOTHERNAME. Then when the process that creates the file is finished it would rename ANOTHERNAME to FNAME.TXT, an action that only takes a tiny amount of time to accomplish.

Alternatively, have the process that creates the file send it via FTP to your Watch server... OR, have the process that creates "FNAME.TXT" create a "flag" file, like "OK.TXT" once it's done creating the file. Have your watch process check to see if OK.TXT exists. If it does, delete OK.TXT then get FNAME.TXT. If there is no OK.TXT just exit.
Mark Hennessey
State of Connecticut/DSS

#17061 - 05/12/06 12:45 PM Re: FTP Get
Yannick Fortin Offline
OL Expert

Registered: 08/25/00
Posts: 351
Loc: Objectif Lune Montréal
If I understand correctly, the server on which the files resides takes X amount of time to complete writing the data to the file. Within this amount of time, PlanetPress Watch grabs the file. Sicne the writing was not completed, the data file processed by Watch at that point is incomplete.

I assume that the server is a UNIX/Linux server (although the explanation might still be true for a Windows server).

As you may know, UNIX does not enforce file locking as we are used to in a Windows world. Traditionally, a Windows program writing to a file will lock it to prevent anyone from reading it. In the UNIX world, every file is always accessible unless it is explicitly marked as locked.

On the other hand, the FTP protocol does not provide the means to verify if a file is being written to or not. This means that even if file locking was enabled, there would be no way for the FtpGet task to know if the file can safely be captured or not.

There are a number of ways to work around this problem, but none very elegant I'm afraid.

1. It might be possible to configure the FTP server to refuse access to files that are already opened. This might result in an error message being logged in Watch, which you could safely ignore.

2. The program writing the file could be modified to use mandatory file locking. This might also result in an error message being logged in Watch, which you could safely ignore.

3. You could modify your Watch Process shcedule to prevent conflicts. For example, if you know that the file takes 10 seconds to write, setting a 10 minutes interval would minimize (but not entirely remove) the chance of getting incomplete data. Alternatively, you could schedule your Watch Process to run at a time where you are sure that the file is not being written to. For example, the file is written between 10:00 AM and 2:00 PM and the Watch Process is scheduled to run between 3:00 PM and 5:00 PM.

4. Create the file in a different directory, and have the server move it to the directory watched by your process when the file is ready to be grabbed.

5. Use another type of input instead of FTP.

Hope this helps. Keep us posted.
Yannick Fortin, Team OL