Cameron Laird's personal notes on FTP [RFC 959?] automation

As one reader wrote in October 1999, "Your page just saved me hours of time transferring files." That's why I keep coming back here to write up more of my notes on the subject.

[Remarks on automation and administration.]

In early 2000, I'm having doubts about the morality of maintaining this page. Should I encourage FTP use, if even by documenting it? The problem, of course, is that FTP's account authentication travels in clear-text; the Internet waters have become so rapacious that few of those visible passwords fail to be snapped up. Frank da Cruz and his kermit-project colleagues discuss secure alternatives to conventional FTP.

Table of Contents

Technologies: Frank da Cruz aptly points out that one might usefully compare solutions or technologies in terms of such dimensions as

.netrc

Dan Culp wrote a nice introduction for The Linux Gurus on precisely this subject.

[Explain.]

[Remember to "chmod 600 .netrc".]

ncftp

[Explain ncftp.]

Re-directed input

Conventional Unix shells and FTP clients are adequate to far more situations than is generally recognized: 
	ftp -i -n <<HERE
		open $HOST
		user $USER $PASSWORD
		mget $FILENAME_PATTERN
		close
		quit
	HERE
	# Remember that "prompt" is an alternative to "... -i ..."
[More examples.]

Kermit

Frank da Cruz offers 

     #!/usr/local/bin/kermit +
     cd /usr/data/daily/
     if fail exit 1 /usr/data/daily/: \v(errstring)
     ftp open /usr:laird foo.bar.com
     if fail exit 1 Can't open FTP connection
     if not \v(ftp_loggedin) FTP login failed
     ftp get /binary \%1
     if fail exit 1 GET \%1 failed: \v(ftp_message)
     exit 0
as an example which illustrates that Details on kermit's scriptability are available through the

lynx

D. J. Hagberg points out that lynx recognizes {ftp,gopher,http}_proxy environment variables, which permits one to separate several security issues from automation scripting. A request to lynx -dump ftp://$HOST/$FILE > $LOCAL, therefore, can be kept invariant even when working through a firewall.

Expect

[Explain.] [Show library examples.] [Note NT availability.] [Refer to example on pages 83-84 of book.]

Python

Andrew Kuchling kindly transcribed this interactive session to illustrate the capabilities of Python's ftplib module.

[Explain Medusa.]

[Explain expy.]

[Explain telnetlib.]

Jeff Bauer offers implementing code.

Tcl

Modern releases of the tcllib library include an ftp client module which entirely answers needs to automate Tcl-based FTP automation. For old installations of Tcl, "Ftp_lib is a loadable package that extends Tcl/Tk8.x with commands to support the File Transfer Protocol ..." NeoSoft mirrors it.

Expect is, of course, a Tcl extension.

urlget

Daniel Stenberg wrote cURL in C.

wget

As its documentation describes,
GNU Wget is a freely available network utility to retrieve files from the World Wide Web using HTTP and FTP, the two most widely used Internet protocols. It works non-interactively, thus enabling work in the background, after having logged off.

The recursive retrieval of HTML pages, as well as FTP sites is supported -- you can use Wget to make mirrors of archives and home pages, or traverse the web like a WWW robot (Wget understands /robots.txt).

Thanks to David Blackman, who pointed this out to me. An example wget command is wget -m ftp://SOMESITE.com/pub/wget/, which ...

GUI applications

GUIfied FTP applications are not, in general, subject to automation. However, I've received so many inquiries that I'll list here a few that I know are available for Unix.

For browsing FTP sites, of course, WWW browsers are hard to beat for their deployability. Beyond these, the ones I know best are

RxFTP is available for OS/2 (and Windows--with Object Rexx as an automation language?).

I'll report on others as I make time for them.

Cameron Laird's personal notes on FTP automation/claird@phaseit.net