如何解决Docker容器的sshfs挂载冻结，但仅在通过Python挂载时冻结
我有一台开发笔记本电脑（Mint 19.3）和一台测试服务器（Ubuntu 18.04.4 LTS）。
笔记本电脑是Docker version 19.03.5,build 633a0ea838，服务器是Docker version 19.03.12,build 48a66213fe
我正在容器内运行Python 3.6代码，该容器使用subprocess（下面的代码）在第三个服务器上创建sshfs挂载，然后python代码遍历挂载的目录。
在我的开发笔记本电脑上一切正常。但是在服务器上，目录已挂载（并且可以通过mount命令看到），但是挂入目录的CD'只是挂起，而Python代码的后续walk也只是挂起。 （注意：python代码永远不会崩溃或出错。它只会永远挂起。）
但是，如果我在容器的命令行上手动使用相同的sshfs命令，则该目录可以正常工作。
我不知道如何解决此问题。
=== 2020-09-25 UPDATE ===
好。由于Python代码使用子进程，因此sshfs挂载显然可用于任何想要使用它的终端窗口。
我尝试从容器内的新终端窗口访问挂载，但是当我cd到挂载时-该窗口冻结了。
好吧，我把所有东西都放在一夜之间-现在当我尝试将CD装入安装架时...可以了。就像坐骑必须坐好几个小时才能工作。
有什么想法吗？
Python代码
@target_dir.setter
def target_dir(self,value):
if value is None:
_tmp = tempfile.mkdtemp()
self._TARGETDIR = _tmp
else:
self._TARGETDIR = value
def mount(self):
# sshfs username@server-worker01:/test_project/ /mnt/test3 -o IdentityFile=/home/username/.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks
command = "sshfs {U}@{S}:{RD} {LD}".format(
U = self.user,S = self.server,RD = self.remote_dir,LD = self.target_dir)
command += " -o IdentityFile={IDF},follow_symlinks,allow_other".format(IDF = key)
cp = subprocess.getoutput(command)
if cp != "":
err = "Something appears to be wrong with the mount. ({})".format(cp)
raise RuntimeError(err)
在容器内部手动安装
root@328100dd78be:/# mkdir /mnt/test
root@328100dd78be:/# sshfs username@server-worker01:/test_project/ /mnt/test -o IdentityFile=/etc/ssl/certs/sshprivkey,follow_symlinks
root@328100dd78be:/# cd /mnt/test
root@328100dd78be:/mnt/test# ls -la
total 40
drwxr-s--- 1 16543 41500 210 Aug 6 07:07 .
drwxr-xr-x 1 root root 4096 Sep 24 14:27 ..
drwxr-s--- 1 16543 41500 187 Jul 27 11:16 configFiles
(...snip...)
drwx--S--- 1 16543 41500 104 May 6 14:03 submission
但是，如果我让代码挂载它，则目录挂起
root@328100dd78be:# ODCFFileCrawl.py
2020-09-24 14:31:16,722 - root - INFO - logger started! with:{'loglevel': '10'}
sshfs username@server-worker01:/test_project/ /tmp/tmp0wwjcg_6 -o IdentityFile=/home/username/.ssh/sshprivkey,allow_other,StrictHostKeyChecking=no
2020-09-24 14:31:16,797 - root - WARNING - Mounted remote filesystem 'server-worker01:/test_project/' to '/tmp/tmp7fig76gx'.
(change to new console)
root@328100dd78be:/# mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/RWED5GN6GYRRGECDFN4LLIVVU7:/var/lib/docker/overlay2/l/7MTLBYSZ4ARQHKQ65TYBY5VSHW:/var/lib/docker/overlay2/l/EJX3L6YHRYB5IICXKAIH4PHT5I:/var/lib/docker/overlay2/l/AFJWYKSAAP6E7RMIU3H2PUMRSN:/var/lib/docker/overlay2/l/CU3AWZFBMNMQLQSBUQSVCHOKWC:/var/lib/docker/overlay2/l/RBJZT34EXSVZD3UYORTLM6L5UX:/var/lib/docker/overlay2/l/OJ7YFP5Z4SEETA2UVO35HQ2K53:/var/lib/docker/overlay2/l/CBU6F6UI47VHQ2BW5NMGJE64UA:/var/lib/docker/overlay2/l/62ZEX3XIFI7VYTLYYJX6XNLJTM:/var/lib/docker/overlay2/l/NBRXSW53NRLL5KRGYQ2BDL2ICZ:/var/lib/docker/overlay2/l/W5DJJWYP2VAYHTNDDHPY2KK2N3:/var/lib/docker/overlay2/l/ZOR2MOYW2NICOX3YVUUY6VXKVW:/var/lib/docker/overlay2/l/WLZEM73R55XKTPNVFCC5TRDCKT:/var/lib/docker/overlay2/l/PARGLNJTE4CM7PTTS5UOLXAIOC:/var/lib/docker/overlay2/l/JDA4NCWU5RN5YNVC63USV4O3VC:/var/lib/docker/overlay2/l/CEGTXJOQH65FTIZLAJNTAPF24R:/var/lib/docker/overlay2/l/B4VV7H463RL3THLES637O57DUO:/var/lib/docker/overlay2/l/KCENZYGKRWT7BRZC2HRUKSB4C5:/var/lib/docker/overlay2/l/EOYAIHOSSFBSX7X5HEWWZQGXS4:/var/lib/docker/overlay2/l/T2ZFALNJHY37UVCYHXD6GC36R5:/var/lib/docker/overlay2/l/4ACYIA6XBKYGFV5BUHYWTOALAY:/var/lib/docker/overlay2/l/PWZLNT3WAZPWPPYFOHZQ2SBHMG:/var/lib/docker/overlay2/l/W7UH56VKQDQH65GOWIGCYAMR3U,upperdir=/var/lib/docker/overlay2/9c2385aa1221d4fbcac321bfb7862dae23677bdf594ddea803315144b5124617/diff,workdir=/var/lib/docker/overlay2/9c2385aa1221d4fbcac321bfb7862dae23677bdf594ddea803315144b5124617/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,size=65536k,mode=755)
devpts on /dev/pts type devpts (rw,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (ro,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (ro,xattr,name=systemd)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,cpu,cpuacct)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (ro,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,cpuset)
cgroup on /sys/fs/cgroup/rdma type cgroup (ro,rdma)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,devices)
mqueue on /dev/mqueue type mqueue (rw,relatime)
/dev/vda1 on /etc/resolv.conf type ext4 (rw,data=ordered)
/dev/vda1 on /etc/hostname type ext4 (rw,data=ordered)
/dev/vda1 on /etc/hosts type ext4 (rw,data=ordered)
shm on /dev/shm type tmpfs (rw,size=65536k)
/dev/vda1 on /etc/ssl/certs type ext4 (rw,data=ordered)
/dev/vdb1 on /opt/vep/.vep type ext4 (rw,data=ordered)
proc on /proc/bus type proc (ro,relatime)
proc on /proc/fs type proc (ro,relatime)
proc on /proc/irq type proc (ro,relatime)
proc on /proc/sys type proc (ro,relatime)
proc on /proc/sysrq-trigger type proc (ro,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime)
tmpfs on /proc/kcore type tmpfs (rw,mode=755)
tmpfs on /proc/keys type tmpfs (rw,mode=755)
tmpfs on /proc/timer_list type tmpfs (rw,mode=755)
tmpfs on /proc/sched_debug type tmpfs (rw,mode=755)
tmpfs on /proc/scsi type tmpfs (ro,relatime)
tmpfs on /sys/firmware type tmpfs (ro,relatime)
username@server-worker01:/test_project/ on /mnt/test type fuse.sshfs (rw,user_id=0,group_id=0)
username@server-worker01:/test_project/ on /tmp/tmp7fig76gx type fuse.sshfs (rw,group_id=0,allow_other)
root@328100dd78be:# cd /tmp/tmp7fig76gx
(hangs forever)
=== 2020-12-04 @colidyre === 的更新
尝试一下...
mount.py
import tempfile
import subprocess
# target_dir = The local full path to which the remote directory will be mounted
target_dir = tempfile.mkdtemp()
user = "<Your username on the remote server >"
server = "<The server name for the remote server>"
remote_dir = "<Full path of the remote directory>"
key = "<Full path of the ssh private key for the remote server>"
options = " -o IdentityFile={IDF},allow_other".format(IDF = key)
command = "sshfs {U}@{S}:{RD} {LD}".format(
U = user,S = server,RD = remote_dir,LD = target_dir)
command += options
cp = subprocess.getoutput(command)
if cp != "":
err = "Something appears to be wrong with the mount. ({})".format(cp)
raise RuntimeError(err)
Dockerfile的代码
注意：这只是来自https://hub.docker.com/r/ensemblorg/ensembl-vep/的工厂VEP Docker容器，中间添加了几行以将mount.py命令复制到/usr/bin，并且入口点已更改为mount.py。参见...
##########################################
# Setup environment for mount.py
##########################################
Dockerfile
###################################################
# Stage 1 - docker container to build ensembl-vep #
###################################################
FROM ubuntu:18.04 as builder
# Update aptitude and install some required packages
# a lot of them are required for Bio::DB::BigFile
RUN apt-get update && apt-get -y install \
build-essential \
apt-utils \
python3 \
python3-pip \
git \
libpng-dev \
zlib1g-dev \
libbz2-dev \
liblzma-dev \
perl \
perl-base \
unzip \
wget
# wget && \
# rm -rf /var/lib/apt/lists/*
# Setup VEP environment
ENV OPT /opt/vep
ENV OPT_SRC $OPT/src
ENV HTSLIB_DIR $OPT_SRC/htslib
ENV BRANCH release/101
# Working directory
WORKDIR $OPT_SRC
# Clone/download repositories/libraries
RUN if [ "$BRANCH" = "master" ]; \
then export BRANCH_OPT=""; \
else export BRANCH_OPT="-b $BRANCH"; \
fi && \
# Get ensembl cpanfile in order to get the list of the required Perl libraries
wget -q "https://raw.githubusercontent.com/Ensembl/ensembl/$BRANCH/cpanfile" -O "ensembl_cpanfile" && \
# Clone ensembl-vep git repository
git clone $BRANCH_OPT --depth 1 https://github.com/Ensembl/ensembl-vep.git && chmod u+x ensembl-vep/*.pl && \
# Clone ensembl-variation git repository and compile C code
git clone $BRANCH_OPT --depth 1 https://github.com/Ensembl/ensembl-variation.git && \
mkdir var_c_code && \
cp ensembl-variation/C_code/*.c ensembl-variation/C_code/Makefile var_c_code/ && \
rm -rf ensembl-variation && \
chmod u+x var_c_code/* && \
# Clone bioperl-ext git repository - used by Haplosaurus
git clone --depth 1 https://github.com/bioperl/bioperl-ext.git && \
# Download ensembl-xs - it contains compiled versions of certain key subroutines used in VEP
wget https://github.com/Ensembl/ensembl-xs/archive/2.3.2.zip -O ensembl-xs.zip && \
unzip -q ensembl-xs.zip && mv ensembl-xs-2.3.2 ensembl-xs && rm -rf ensembl-xs.zip && \
# Clone/Download other repositories: bioperl-live is needed so the cpanm dependencies installation from the ensembl-vep/cpanfile file takes less disk space
ensembl-vep/travisci/get_dependencies.sh && \
# Only keep the bioperl-live "Bio" library
mv bioperl-live bioperl-live_bak && mkdir bioperl-live && mv bioperl-live_bak/Bio bioperl-live/ && rm -rf bioperl-live_bak && \
## A lot of cleanup on the imported libraries,in order to reduce the docker image ##
rm -rf Bio-HTS/.??* Bio-HTS/Changes Bio-HTS/DISCLAIMER Bio-HTS/MANIFEST* Bio-HTS/README Bio-HTS/scripts Bio-HTS/t Bio-HTS/travisci \
bioperl-ext/.??* bioperl-ext/Bio/SeqIO bioperl-ext/Bio/Tools bioperl-ext/Makefile.PL bioperl-ext/README* bioperl-ext/t bioperl-ext/examples \
ensembl-vep/.??* ensembl-vep/docker \
ensembl-xs/.??* ensembl-xs/TODO ensembl-xs/Changes ensembl-xs/INSTALL ensembl-xs/MANIFEST ensembl-xs/README ensembl-xs/t ensembl-xs/travisci \
htslib/.??* htslib/INSTALL htslib/NEWS htslib/README* htslib/test && \
# Only keep needed kent-335_base libraries for VEP - used by Bio::DB::BigFile (bigWig parsing)
mv kent-335_base kent-335_base_bak && mkdir -p kent-335_base/src && \
cp -R kent-335_base_bak/src/lib kent-335_base_bak/src/inc kent-335_base_bak/src/jkOwnLib kent-335_base/src/ && \
cp kent-335_base_bak/src/*.sh kent-335_base/src/ && \
rm -rf kent-335_base_bak
# Setup bioperl-ext
WORKDIR bioperl-ext/Bio/Ext/Align/
RUN perl -pi -e"s|(cd libs.+)CFLAGS=\\\'|\$1CFLAGS=\\\'-fPIC |" Makefile.PL
# Install htslib binaries (for 'bgzip' and 'tabix')
# htslib requires the packages 'zlib1g-dev','libbz2-dev' and 'liblzma-dev'
WORKDIR $HTSLIB_DIR
RUN make install && rm -f Makefile *.c
# Compile Variation LD C scripts
WORKDIR $OPT_SRC/var_c_code
RUN make && rm -f Makefile *.c
###################################################
# Stage 2 - docker container to build ensembl-vep #
###################################################
FROM ubuntu:18.04
# Update aptitude and install some required packages
# a lot of them are required for Bio::DB::BigFile
RUN apt-get update && apt-get -y install \
build-essential \
python3 \
python3-pip \
git \
cpanminus \
curl \
libmysqlclient-dev \
libpng-dev \
libssl-dev \
zlib1g-dev \
libbz2-dev \
liblzma-dev \
locales \
openssl \
perl \
perl-base \
unzip \
sshfs \
vim && \
apt-get -y purge manpages-dev && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
# gitpython must be installed before setup.py is run
RUN pip3 install gitpython
##########################################
# Setup environment for mount.py
##########################################
# Takes the local FUSE conf as the base. This should be properly configured
# with "user_allow_other" enabled
COPY mount.py /usr/bin/mount.py
RUN chown root:root /usr/bin/mount.py
RUN chmod 755 /usr/bin/mount.py
# To hold the GIT downloads prior to install
RUN mkdir /src
RUN chmod 777 /src
##########################################
# Setup VEP environment
##########################################
ENV OPT /opt/vep
ENV OPT_SRC $OPT/src
ENV PERL5LIB_TMP $PERL5LIB:$OPT_SRC/ensembl-vep:$OPT_SRC/ensembl-vep/modules
ENV PERL5LIB $PERL5LIB_TMP:$OPT_SRC/bioperl-live
ENV KENT_SRC $OPT/src/kent-335_base/src
ENV HTSLIB_DIR $OPT_SRC/htslib
ENV MACHTYPE x86_64
ENV DEPS $OPT_SRC
ENV PATH $OPT_SRC/ensembl-vep:$OPT_SRC/var_c_code:$PATH
ENV LANG_VAR en_US.UTF-8
# Create vep user
RUN useradd -r -m -U -d "$OPT" -s /bin/bash -c "VEP User" -p '' vep && usermod -a -G sudo vep && mkdir -p $OPT_SRC
USER vep
# Copy downloaded libraries (stage 1) to this image (stage 2)
COPY --chown=vep:vep --from=builder $OPT_SRC $OPT_SRC
#############################################################
# Change user to root for the following complilations/installations
USER root
# Install bioperl-ext,faster alignments for haplo (XS-based BioPerl extensions to C libraries)
WORKDIR $OPT_SRC/bioperl-ext/Bio/Ext/Align/
RUN perl Makefile.PL && make && make install && rm -f Makefile*
# Install ensembl-xs,faster run using re-implementation in C of some of the Perl subroutines
WORKDIR $OPT_SRC/ensembl-xs
RUN perl Makefile.PL && make && make install && rm -f Makefile* cpanfile
WORKDIR $OPT_SRC
# Install/compile more libraries
RUN ensembl-vep/travisci/build_c.sh && \
# Remove unused Bio-DB-HTS files
rm -rf Bio-HTS/cpanfile Bio-HTS/Build.PL Bio-HTS/Build Bio-HTS/_build Bio-HTS/INSTALL.pl && \
# Install ensembl perl dependencies (cpanm)
cpanm --installdeps --with-recommends --notest --cpanfile ensembl_cpanfile . && \
cpanm --installdeps --with-recommends --notest --cpanfile ensembl-vep/cpanfile . && \
# Delete bioperl and cpanfiles after the cpanm installs as bioperl will be reinstalled by the INSTALL.pl script
rm -rf bioperl-live ensembl_cpanfile ensembl-vep/cpanfile && \
# Configure "locale",see https://github.com/rocker-org/rocker/issues/19
echo "$LANG_VAR UTF-8" >> /etc/locale.gen && locale-gen en_US.utf8 && \
/usr/sbin/update-locale LANG=$LANG_VAR && \
# Copy htslib executables. It also requires the packages 'zlib1g-dev','libbz2-dev' and 'liblzma-dev'
cp $HTSLIB_DIR/bgzip $HTSLIB_DIR/tabix $HTSLIB_DIR/htsfile /usr/local/bin/
ENV LC_ALL $LANG_VAR
ENV LANG $LANG_VAR
# Switch back to vep user
USER vep
ENV PERL5LIB $PERL5LIB_TMP
# Final steps for VEP
WORKDIR $OPT_SRC/ensembl-vep
# Update bash profile
RUN echo >> $OPT/.profile && \
echo PATH=$PATH:\$PATH >> $OPT/.profile && \
echo export PATH >> $OPT/.profile && \
# Run INSTALL.pl and remove the ensemb-vep tests and travis
./INSTALL.pl -a a -l -n && rm -rf t travisci .travis.yml
ENTRYPOINT ["mount.py"]
# CMD tail -f /dev/null
解决方法
我假设您想使用SSHFS将某些服务器的目录挂载到容器的文件系统中。您可以将该指令添加到Dockerfile中：
FROM ubuntu:18.04
RUN apt-get update && apt-get -y install stuff
COPY sshprivkey .ssh/sshprivkey
RUN mkdir /mnt/test
RUN sshfs username@server-worker01:/test_project/ /mnt/test -o IdentityFile=.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks
您可以在此下面的Dockerfile中添加其他内容。您应该将sshprivkey文件添加到当前目录。
现在，如果您构建它：
docker build -t your_desired_image_name .
然后使用该图像运行一个容器，并检查其是否有效：
docker run --privileged -it your_desired_image_name
root@container_id$ ls /mnt/test
请记住添加--privileged标志以使其正常工作。我从values找到了。
服务器目录/目录可以使用SSHFS实用程序直接安装到容器的文件系统中。
Dockerfile
FROM ubuntu:18.04
RUN apt-get update
COPY sshprivkey .ssh/sshprivkey
RUN mkdir /mnt/test
RUN sshfs username@server-worker01:/test_project/ /mnt/test -o IdentityFile=.ssh/sshprivkey,follow_symlinks
使用ubuntu：18.04基本映像创建容器。
FROM ubuntu:18.04
更新软件包列表。 此命令将根据基本图像进行更改。此处命令可以更新以安装必需的软件包
RUN apt-get update
已复制sshprivkey与sshfs一起使用以将服务器直接安装到容器的权限
COPY sshprivkey .ssh/sshprivkey
应在其中为该测试目录创建服务器目录的位置
RUN mkdir /mnt/test
使用sshfs运行sshprivkey命令将服务器目录安装在先前创建的测试目录中
RUN sshfs username@server-worker01:/test_project/ /mnt/test -o IdentityFile=.ssh/sshprivkey,follow_symlinks
使用以下命令从当前目录中的Dockerfile构建映像：
docker build -t your_desired_image_name .
根据先前使用的命令从先前创建的图像运行交互式会话容器：
docker run --cap-add SYS_ADMIN -it your_desired_image_name
在容器交互式会话开始之后，我们可以检查它是否起作用：
root@container_id$ ls /mnt/test
这里我使用--cap-add授予容器安装目录的权限
注意：最好根据需要使用--cap-add而不是使用--privileged添加受限权限，除非出于安全目的没有其他用途，否则也不能提供安全权限该容器有很多权限
您可以了解有关docker权限here
的更多信息版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。