![]() || scp -o ControlMaster=auto -o -o ControlPersist=900 $SECONDARY/.) &Įrrors I got with parallel shell script- channel 24: open failed: administratively prohibited: open failedĬhannel 25: open failed: administratively prohibited: open failedĬhannel 26: open failed: administratively prohibited: open failedĬhannel 28: open failed: administratively prohibited: open failedĬhannel 30: open failed: administratively prohibited: open failed || scp -o ControlMaster=auto -o -o ControlPersist=900 $PRIMARY/.) &įor sl in -o ControlMaster=auto -o -o ControlPersist=900 $SECONDARY/. if & & įor el in -o ControlMaster=auto -o -o ControlPersist=900 $PRIMARY/. Parallel Shell Script which I tried, top portion of shell script is same as shown above. Can I copy 10 files at a time or 5 files at a time in parallel to speed up this process or any other approach? Is there any way to move the files faster in machineA. I am copying PRIMARY_PARTITION files in PRIMARY folder and SECONDARY_PARTITION files in SECONDARY folder in machineA. || scp -o ControlMaster=auto -o -o ControlPersist=900 $SECONDARY/. || scp -o ControlMaster=auto -o -o ControlPersist=900 $PRIMARY/.įor sl in -o ControlMaster=auto -o -o ControlPersist=900 $SECONDARY/. SECONDARY_PARTITION=(1 2 4 6 8) # this will have more file numbers around 200ĭir1=$(ssh -o "StrictHostKe圜hecking no" ls -dt1 "$MEMORY_MAPPED_LOCATION"/ | head -n1)ĭir2=$(ssh -o "StrictHostKe圜hecking no" ls -dt1 "$MEMORY_MAPPED_LOCATION"/ | head -n1)įor el in -o ControlMaster=auto -o -o ControlPersist=900 $PRIMARY/. PRIMARY_PARTITION=(0 3 5 7 9) # this will have more file numbers around 200 Readonly MEMORY_MAPPED_LOCATION=/data/pe_t1_snapshot Readonly FILERS_LOCATION=(machineB machineC) Readonly SECONDARY=/export/home/david/dist/secondary Readonly PRIMARY=/export/home/david/dist/primary I need to copy around 400 files in machineA from machineB and machineC and each file size is 1.5 GB.Ĭurrently I have my below shell script which works fine as I am using scp but somehow it takes ~ 2 hours to copy the 400 files in machineA which is too long for me I guess. So suppose if this is the latest date folder 20140317 inside /data/pe_t1_snapshot then this will be the full path for me - /data/pe_t1_snapshot/20140317įrom where I need to start copying the files in machineB and machineC. So whatever date is the latest date in this format YYYYMMDD inside the above folder - I will pick that folder as the full path from where I need to start copying the files. In machineB and machineC there will be a folder like this YYYYMMDD inside this folder - /data/pe_t1_snapshot So I will try to copy from machineB first, if it is not there in machineB then I will go to machineC to copy the same files. ![]() If the file is not there in machineB, then it should be there in machineC for sure. ![]() I am running my shell script on machineA which copies the files from machineB and machineC to machineA. ![]()
0 Comments
Leave a Reply. |