Wednesday, December 2, 2015

How to Automate data Transfer using FTP

Often we require to copy data using FTP which we do manually but if we require data transfer in a regular running job we can using this method.
First let us know the useful FTP commands:

Common FTP Commands
Common FTP Commands

mput to copy multiple files from the local machine to the remote machine;
  you are prompted for a y/n answer before transferring each file
open to open a connection with another computer
pwd to find out the pathname of the current directory on the remote machine 
quit to exit the FTP environment (same as bye)
rmdir to to remove (delete) a directory in the current remote directory 
put to copy one file from the local machine to the remote machine 
cd to change directory on the remote machine
close to terminate a connection with another computer

close brubeck closes the current FTP connection with brubeck,
  but still leaves you within the FTP environment.
delete to delete (remove) a file in the current remote directory (same as rm in UNIX)
get to copy one file from the remote machine to the local machine

get ABC DEF copies file ABC in the current remote directory to (or on top of) a file named DEF in your current local directory.

get XYZ copies file XYZ in the current remote directory to (or on top of) a file with the same name, XYZ, in your current local directory.
help to request a list of all available FTP commands
lcd to change directory on your local machine (same as UNIX cd)
ls to list the names of the files in the current remote directory
mkdir to make a new directory within the current remote directory
mget to copy multiple files from the remote machine to the local machine;
  you are prompted for a y/n answer before transferring each file
mget * copies all the files in the current remote directory to your current local directory, using the same filenames. Notice the use of the wild card character, *.

===========================================
As in many situation we require to copy them in a scheduled Job or inside a code using automation.

Here is a code snippet which is written by my end to copy data using FTP from one server to the other.
This is helpful for DBA and Developer who often require to copy data in between multiple servers.
This uses username and password however can be removed if we are using ssh based authentication.
 LOG_TEMP_FILE=/tmp/to_load.tmp
CRITICAL=NO
HOST=`hostname`
FTP_ERROR=0
EXP_ERR=0
FILES_HOOPED=0
FAILURE_CODE=1
GZIP=/usr/bin/gzip
FTP_PROG=/bin/ftp
FTP_ADDR=100.16.120.222
FTP_USER=autouser
FTP_PASSWD=4utxxcccv


FtpData() {
${FTP_PROG} -vin ${FTP_ADDR} << EOLINE > ${LOG_DIR}/ftp.log
user ${FTP_USER} ${FTP_PASSWD}
ascii
prom
hash
mget FILENAME_*${TIME_STAMP}.csv
mdelete to_*
quit
EOLINE

}


Just pass this code in your shell script with appropriate values and start transferring the data between multiple servers.

No comments: