curl递归下载软件shell脚本
发布时间:2014-12-13 编辑:jiaochengji.com
分享一例shell脚本,curl递归下载软件,对apache的镜像网站的递归扫描。通过文件io的缓存方式记录可下载文件,再针对需要文件的后缀名批量下载软件包备用。
例子,curl递归下载软件的shell脚本。
代码:
复制代码 代码示例:
#!/bin/env bash
path=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
export path
clear
url="http://mirrors.cnnic.cn/apache/"
downlistfile="/tmp/downlist.txt"
downlisttmpfile="/tmp/tmplist.txt"
downfiletype="zip$|gz$"
downlist=""
urlback="$url"
[ ! -f $downlistfile ] && touch $downlistfile || echo > $downlistfile
[ ! -f $downlisttmpfile ] && touch $downlisttmpfile || echo > $downlisttmpfile
curl_urls(){
urls=`curl $urlback |awk -f "a href=\"" '{printf "%s\n",$2}'|awk -f "\"" '{printf "%s\n",$1}'|grep -ve "^$|^\?|^http:\/\/"|^#`
}
url_list(){
curl_urls
for i in $urls ;do
echo "$urlback$i" >> $downlisttmpfile
done
}
recursive_search_url(){
urlbacktmps=`cat $downlisttmpfile`
[[ "$urlbacktmps" == "" ]] && echo "no more page for search" && exit 1
for j in $urlbacktmps ;do
if [[ "${j##*\/}" != "" ]] ;then
echo "$j" >> $downlistfile
else
urlback="$j"
url_list
fi
urltmps=`grep -ve "$j$" $downlisttmpfile`
echo "$urltmps" > $downlisttmpfile
recursive_search_url
done
}
download_file(){
downlist=`grep -e "$downfiletype" $downlistfile`
for k in $downlist ;do
filepath=/tmp/${k#*\/\/}
[ ! -d `dirname $filepath` ] && mkdir -p `dirname $filepath`
[ ! -f $filepath ] && cd `dirname $filepath` && curl -o $k
done
}
url_list $urls
recursive_search_url
path=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
export path
clear
url="http://mirrors.cnnic.cn/apache/"
downlistfile="/tmp/downlist.txt"
downlisttmpfile="/tmp/tmplist.txt"
downfiletype="zip$|gz$"
downlist=""
urlback="$url"
[ ! -f $downlistfile ] && touch $downlistfile || echo > $downlistfile
[ ! -f $downlisttmpfile ] && touch $downlisttmpfile || echo > $downlisttmpfile
curl_urls(){
urls=`curl $urlback |awk -f "a href=\"" '{printf "%s\n",$2}'|awk -f "\"" '{printf "%s\n",$1}'|grep -ve "^$|^\?|^http:\/\/"|^#`
}
url_list(){
curl_urls
for i in $urls ;do
echo "$urlback$i" >> $downlisttmpfile
done
}
recursive_search_url(){
urlbacktmps=`cat $downlisttmpfile`
[[ "$urlbacktmps" == "" ]] && echo "no more page for search" && exit 1
for j in $urlbacktmps ;do
if [[ "${j##*\/}" != "" ]] ;then
echo "$j" >> $downlistfile
else
urlback="$j"
url_list
fi
urltmps=`grep -ve "$j$" $downlisttmpfile`
echo "$urltmps" > $downlisttmpfile
recursive_search_url
done
}
download_file(){
downlist=`grep -e "$downfiletype" $downlistfile`
for k in $downlist ;do
filepath=/tmp/${k#*\/\/}
[ ! -d `dirname $filepath` ] && mkdir -p `dirname $filepath`
[ ! -f $filepath ] && cd `dirname $filepath` && curl -o $k
done
}
url_list $urls
recursive_search_url
您可能感兴趣的文章:
curl递归下载软件shell脚本
shell脚本使用curl批量解析号码归属地
shell神器curl命令的用法 curl用法实例笔记
使用curl获取网站的http的状态码
shell脚本下载网页图片
递归修改目录与文件名统一为小写的shell脚本
linux使用curl监控网页的shell脚本
递归目录复制文件(maven命令执行类)
shell编程技巧小结(二)
php curl参数详解与用法大全
[关闭]