前面我們提到過,參考:使用ebi資料庫直接下載fastq測序數據 , 偶爾會遇到spera提示:ascp: failed to authenticate, exiting. 以及Session Stop (Error: failed to authenticate),其實這個是無解的, 所以只能說放棄 aspera高速下載了,反正就兩三個樣品,直接wget也行。
但是呢,wget就會遇到另外一個問題,下載速度實在是不可恭維,恰好前些天學徒也遇到了一個資料庫下載龜速問題,他通過多線程的mwget命令解決了問題。
下載和安裝wget 下載有點兒慢,有時候沒得科學上網,慢上加慢了。
首先是下載解壓安裝一條龍:
wget http://jaist.dl.sourceforge.net/project/kmphpfm/mwget/0.1/mwget_0.1.0.orig.tar.bz2
tar -jxvf mwget_0.1.0.orig.tar.bz
cd mwget_0.1.0.orig
/.configure這時候configure 可能會發生報錯:
configure: error: Your intltool is too old. You need intltool 0.35.0 or later.切換一下環境,嘗試用conda 安裝這個 intltool :
conda activate mess
conda install intltool再次編譯安裝:
./configure
make'make 又發生報錯了:
ttpplugin.cpp:149:14: error: 『strstr』 was not declared in this scope
149 | filename = strstr(filename, "filename=");
| ^~~~~~
httpplugin.cpp:149:14: note: 『strstr』 is defined in header 『<cstring>』; did you forget to 『#include <cstring>』?
httpplugin.cpp:151:16: error: 『strlen』 was not declared in this scope
151 | filename += strlen("filename=");
| ^~~~~~
httpplugin.cpp:151:16: note: 『strlen』 is defined in header 『<cstring>』; did you forget to 『#include <cstring>』?
make[2]: *** [Makefile:385: httpplugin.o] Error 1
make[2]: Leaving directory '/home/data/yzpeng/3.biosoft/other_soft/mwget_0.1.0.orig/src'
make[1]: *** [Makefile:323: all-recursive] Error 1
make[1]: Leaving directory '/home/data/yzpeng/3.biosoft/other_soft/mwget_0.1.0.orig'參考:https://blog.csdn.net/lihengchang0212/article/details/89946235
主要對兩個文件修改一下:
make[2]: *** [Makefile:385: httpplugin.o] Error 1
Leaving directory '/home/data/yzpeng/3.biosoft/other_soft/mwget_0.1.0.orig/src'
make[1]: *** [Makefile:323: all-recursive] Error 1
make[1]: Leaving directory '/home/data/yzpeng/3.biosoft/other_soft/mwget_0.1.0.orig'在httpplugin.cpp頭文件中添加:
#include <string.h>
#include <string>
using std::string;又報錯了,接著來:
make[2]: *** [Makefile:385: ftpplugin.o] Error 1
make[2]: Leaving directory '/home/data/yzpeng/3.biosoft/other_soft/mwget_0.1.0.orig/src'(後面還經歷了幾次報錯,這裡統一記錄一下)在ftpplugin.cpp, downloader.cpp添加一下:
#include <string.h>
#include <string>
using std::string;終於成功了:
...
file=`echo zh_CN | sed 's,.*/,,'`.gmo \
&& rm -f $file && /home/yzpeng/miniconda3/envs/mess/bin/msgfmt -o $file zh_CN.po
make[2]: Leaving directory '/home/data/yzpeng/3.biosoft/other_soft/mwget_0.1.0.orig/po'
make[2]: Entering directory '/home/data/yzpeng/3.biosoft/other_soft/mwget_0.1.0.orig'
make[2]: Leaving directory '/home/data/yzpeng/3.biosoft/other_soft/mwget_0.1.0.orig'
make[1]: Leaving directory '/home/data/yzpeng/3.biosoft/other_soft/mwget_0.1.0.orig'接著make install 又發生了權限錯誤:
make install
...
/usr/bin/install: cannot create regular file '/usr/local/bin/mwget': Permission denied
make[2]: *** [Makefile:315: install-binPROGRAMS] Error 1
make[2]: Leaving directory '/home/data/yzpeng/3.biosoft/other_soft/mwget_0.1.0.orig/src'
make[1]: *** [Makefile:506: install-am] Error 2
make[1]: Leaving directory '/home/data/yzpeng/3.biosoft/other_soft/mwget_0.1.0.orig/src'
make: *** [Makefile:323: install-recursive] Error 1我們需要把軟體編譯到用戶家目錄的bin 文件夾中:
make install DESTDIR=/home/user/bin成功了~
~/0.bin
$ bin/mwget
mwget: missing URL
Usage: mwget [Options]... [URL]...
Try `mwget --help` for more options,Thanks!我們將它放到bin 目錄中:
cd 0.bin
ln -s ../bin/wget .奇怪啊,為什麼不直接使用conda來安裝多線程的mwget命令?其實如果是原始碼安裝軟體,應該是學習Jimmy老師的視頻:
https://www.bilibili.com/video/BV1cJ411e7UH 生信分析入門環境搭建
1.4萬播放 ·
20彈幕
https://www.bilibili.com/video/BV1ps411M7UZ 生信人應該這樣安裝軟體
1.7萬播放 ·
47彈幕
https://www.bilibili.com/video/BV1XW411d7Bp 生信伺服器配置全攻略
1.3萬播放 ·
67彈幕
``不管那麼多了,先使用看看!
使用$ mwget -h
GNU MWget 0.1.0 ,a non-interactive and multiline network retriever of all POSTIX Systems.
Usage: mwget [Options]... [URL]...
Options:
-b, --debug Show the debug message
-c, --count=num Set the retry count to [num], no limit when "0", the default is "99"
-d, --directory=dir Set the local direcotry to [dir], the default is "."
-f, --file=file Rename the file to [file]
-h, --help A brief summary of all the options
-i, --interval=num Set the ftp retry interval to [num] seconds, the default is "5"
-n, --number=num Use [num] connections instead of the default (4)
-r, --referer=URL Include `Referer: [URL]' header in HTTP request.
-t, --timeout=num Set the connection timeout to [num] seconds, the default is "30"
-v, --version Show the version of the mwget and exit
-x, --proxy=URL Set the proxy [URL]
Mail bug reports and suggestions to <sa@kmlinux.tk><xiao_suo@hotmail.com>
On website bug reports and suggestions to <http://mwget.sourceforge.net/> <http://www.kmlinux.tk/>
We Welcome your BUG REPORT!其實最主要的參數還是這個 -n,設置多線程下載的數目。
嘗試下載一個annovar 上的數據:
nohup mwget -n 16 --file=hg38_1000g2015aug.zip http://www.openbioinformatics.org/annovar/download/hg38_1000g2015aug.zip &比較一下速度:
# mwget
Begin to download: hg38_exac03.txt.idx.gz
FileSize: 6.1M
100% [=========================================>===] [6.1M] [1.9M/s]
# wget
hg38_exac03.txt.idx 36%[======> ] 2.22M 174KB/s eta 21s雖然我設置了 -n 16 ,但是並不代表它可以達到16倍的加速哦,正常情況下是174KB/s ,加速後是 1.9M/s,也是有10倍加速啦,挺滿意的。