Google Cloud Build：使用Cloud KMS作为secretEnv从JFrog私有存储库进行pip安装

如何解决Google Cloud Build：使用Cloud KMS作为secretEnv从JFrog私有存储库进行pip安装

我正在尝试通过Google Cloud Build在jfrog存储库中pip安装私有软件包。我可以通过https:<USER>:<API_KEY>@<artifactory_url>访问jfrog存储库，也可以进行pip install <package_name_and_version> https:<USER>:<API_KEY>@<artifactory_url>

我想将此步骤集成到使用Cloud KMS的云构建中，以在pip安装期间解密API_KEY。 我已经使用以下命令对API_KEY字符串进行了加密

# Create a local file with the secret
echo "MyAPIKEY" > plain_pwd.txt
# To encrypt a secret using KMS
gcloud kms encrypt \
  --plaintext-file=plain_pwd.txt \
  --ciphertext-file=cipher_pwd.enc.txt \
  --location=global \
  --keyring=<keyring> \
  --key=<key>
# Encode the binary encoded secret as base64 string
base64 cipher_pwd.enc.txt -w 0 > cipher_pw.enc.64.txt

cloudbuild.yaml中指定的机密为：

secrets:
- kmsKeyName: projects/<project_id>/locations/global/keyRings/<keyring>/cryptoKeys/<key>
  secretEnv:
    APIKEY: <base64 encrypted string from cloud kms encrypt command>

这是我的pip安装的cloudbuild.yaml步骤：

-
    args:
      - "-m"
      - pip
      - install
      - "-t"
      - /workspace/lib
      - "schema-registry-client==0.8.14.dev0"
      - "--extra-index-url"
      - 'https://adminuser:$$APIKEY@<artifactory_url>'
    entrypoint: python3
    secretEnv: ['APIKEY']
    id: INSTALL_SCHEMA_REGISTRY
    name: "python:3.7"

在执行此步骤时，我在云构建中收到“ EOFError：读取行时EOF”：

查找索引：https://pypi.org/simple，

https://onpat:****@artifactory.build.****.****.com/artifactory/api/pypi/dpfw-pypi-dev-local/simple
 ERROR: Exception:
 Traceback (most recent call last):
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py",line 216,in _main
     status = self.run(options,args)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/req_command.py",line 182,in wrapper
     return func(self,options,args)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py",line 325,in run
     reqs,check_supported_wheels=not options.target_dir
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py",line 183,in resolve
     discovered_reqs.extend(self._resolve_one(requirement_set,req))
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py",line 388,in _resolve_one
     abstract_dist = self._get_abstract_dist_for(req_to_install)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py",line 339,in _get_abstract_dist_for
     self._populate_link(req)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py",line 305,in _populate_link
     req.link = self._find_requirement_link(req)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py",line 270,in _find_requirement_link
     best_candidate = self.finder.find_requirement(req,upgrade)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/package_finder.py",line 899,in find_requirement
     req.name,specifier=req.specifier,hashes=hashes,File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/package_finder.py",line 881,in find_best_candidate
     candidates = self.find_all_candidates(project_name)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/package_finder.py",line 826,in find_all_candidates
     project_url,link_evaluator=link_evaluator,line 790,in process_project_url
     html_page = self._link_collector.fetch_page(project_url)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/collector.py",line 643,in fetch_page
     return _get_html_page(location,session=self.session)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/collector.py",line 455,in _get_html_page
     resp = _get_html_response(url,session=session)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/collector.py",line 169,in _get_html_response
     "Cache-Control": "max-age=0",File "/usr/local/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py",line 543,in get
     return self.request('GET',url,**kwargs)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/network/session.py",line 421,in request
     return super(PipSession,self).request(method,*args,**kwargs)
   File "/usr/local/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py",line 530,in request
     resp = self.send(prep,**send_kwargs)
   File "/usr/local/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py",line 650,in send
     r = dispatch_hook('response',hooks,r,**kwargs)
   File "/usr/local/lib/python3.7/site-packages/pip/_vendor/requests/hooks.py",line 31,in dispatch_hook
     _hook_data = hook(hook_data,**kwargs)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/network/auth.py",line 256,in handle_401
     username,password,save = self._prompt_for_password(parsed.netloc)
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/network/auth.py",line 226,in _prompt_for_password
     username = ask_input("User for {}: ".format(netloc))
   File "/usr/local/lib/python3.7/site-packages/pip/_internal/utils/misc.py",line 259,in ask_input
     return input(message)
 EOFError: EOF when reading a line
Finished Step #2 - "INSTALL_SCHEMA_REGISTRY"
ERROR
ERROR: build step 2 "python:3.7" failed: step exited with non-zero status: 2

此外，我尝试了在相同密文上进行gcloud kms解密，并获得了原始API KEY。因此，我认为加密/解密不是这里的问题。我还授予了访问Cloud kms的云构建服务代理的必要权限。

有关如何解决此问题的任何建议/帮助？

解决方法

我找到了解决此问题的另一种方法，显然，pip cli在扩展参数和env变量时存在问题。您可以使用Google Cloud Secret管理器来存储凭据（在这种情况下为JFrog API_KEY）。使用gcloud secrets versions access <version_id> --secret=<secret_key>在运行时检索密码。在构建步骤中使用~/.pip/pip.conf在--extra-index-url=https://user:<api_key_secret_value>@<artifactory_url>处创建pip.conf文件，然后在后续构建步骤中执行pip install <package_name>。唯一的缺点是它在构建过程中增加了一个额外的步骤。我创建了bash脚本，以在构建步骤中将密钥读取到文件，并在下一步构建中使用该密钥值创建pip.conf值。使用云构建卷在步骤之间传递文件。

#read_secret.sh

#!/bin/bash
gcloud secrets versions access 1 --secret=artifactory-api-key > "/data/key.txt"

使用以下bash脚本在下一步中安装软件包

#install_module.sh

#!/bin/bash
export API_KEY=$(cat "/data/key.txt")
echo $API_KEY
mkdir -p ~/.pip
echo -e "[global]\n--extra-index-url=https://myuser:$API_KEY@<artifactory_url>" > ~/.pip/pip.conf
pip install -t "/workspace/lib" private-package-lib==0.1.2