Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 75 additions & 0 deletions examples/README-sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,81 @@ python examples/sandbox_injection_rules.py

示例会创建资源并在结束时清理。保存的注入规则还需要 AK/SK;直接设置运行时 HTTP 注入可使用 API key。

## 沙箱列表分页

`client.list_sandboxes(limit=100, nextToken=cursor)` 返回
`{'items': [...], 'nextToken': ...}`,游标来自 `X-Next-Token` 响应头,
Comment thread
miclle marked this conversation as resolved.
Resolved
`nextToken` 为空表示没有下一页。

从 v7.19.0 升级时,把已移除的 `list_sandboxes_v2()`/`listSandboxesV2()` 改为
`list_sandboxes()`。原来直接遍历 `client.list()` 的单页代码改为遍历 `items`:

```python
for item in client.list_sandboxes(limit=100)['items']:
print(item)
```

完整遍历可让 `Sandbox.list()` 分页器自动传递游标。

```python
from qiniu.services.sandbox import Sandbox

pages = Sandbox.list(client=client, limit=100, metadata={'app': 'example'})
while pages.has_next:
for sandbox in pages.next_items():
print(sandbox.sandbox_id)
```

元数据字典会按服务端过滤协议编码,`+`、`&`、`=`、`%` 和中文均保留原值。
直接传入 `metadata` 字符串时,调用方应先分别编码每个键和值,再将这些项编码为查询字符串。

Envd 文件和进程请求不自动跟随 HTTP 重定向;收到 30x 时抛出 `SandboxError`,
避免把访问 token 或签名 URL 带给跳转目标。

## PTY 实时输出与断连

`sandbox.pty.create(on_data=callback)` 和
`sandbox.pty.connect(pid, on_data=callback)` 返回进程句柄。回调接收原始 `bytes`。
首帧含有 PTY 输出时,`create()` 或 `connect()` 会在调用线程执行回调,回调抛错时
会关闭流并直接抛出异常。后续输出由后台读流线程调用回调,无需先调用 `wait()`。
回调应快速返回,可把数据放入队列交给应用处理。

`handle.wait()` 等待进程结束,返回带字符串 `stdout` 的 `CommandResult`,
并抛出后台读流或回调异常。未提供 `on_data` 时,首次调用 `wait()` 会启动后台读流,
传给该次 `wait()` 的 `on_stdout` 和 `on_stderr` 仍在调用线程执行。

`handle.disconnect()` 只关闭本地连接,不终止远端 PTY,可重复调用。
断连后的 `wait()` 返回已收到的结果;未收到退出事件时 `exit_code` 保持 `-1`。
断连会冻结已处理的输出并释放等待者,包括另一个线程正在同步 `wait()` 的情况;
后台任务负责清理连接,之后重复 `wait()` 返回同一份冻结结果。
部分操作系统的读操作或关闭操作会延迟到既有读取时限才退出;`timeout=None`
的语义不变,不保证读线程立即结束。输出回调在状态锁中确认调度后,在锁外执行。
断连不会等待已经调度的回调,它们可能在 `disconnect()` 返回后才开始或结束;
断连后不再调度后续回调,晚到的输出或异常也不再修改冻结结果。
之后可用 `sandbox.pty.connect(handle.pid, on_data=callback)` 重新连接。
需要终止进程时显式调用 `handle.kill()`;任务结束时清理沙箱。

## 私有 Git 克隆

`clone()` 支持 `username` 和 `password`,与 `pull()` / `push()` 一样使用临时
askpass 文件和进程环境,不把凭据写入远程 URL 或持久 credential store:

```python
import os

result = sandbox.git.clone(
'https://github.com/owner/private.git', '/home/user/private',
username=os.environ['GIT_USERNAME'], password=os.environ['GIT_PASSWORD'])
```

两项凭据必须同时提供,且只适用于 HTTP(S) URL。前台执行完成后删除临时文件;
使用 `background=True` 时,在句柄 `wait()` 结束后删除。该方式会把凭据交给
沙箱中的 Git 进程,应仅用于允许沙箱访问这些凭据的任务。

显式用户名在本次 Git 调用中优先于 `credential.username`,已配置的 credential helper
不会保存本次凭据。每次调用使用独立的私有临时目录,准备、上传、
执行及清理均使用本次 `user` 指定的身份;结束后同时删除 askpass 和临时目录。

## 暂停与恢复

`Sandbox.connect()`、实例 `connect()` 和 `SandboxClient.connect_sandbox()` 使用 Connect v2。省略 `timeout` 使用服务端默认的 300 秒;显式值必须至少为 1 秒。连接只延长 TTL,不会缩短已有 TTL;需要主动调整过期时间时使用 `set_timeout()`。
Expand Down
31 changes: 3 additions & 28 deletions examples/sandbox_git.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,6 @@

from sandbox_common import cleanup_sandbox, create_sandbox, run_example

try:
from urllib.parse import quote, urlparse, urlunparse
except ImportError:
from urllib import quote
from urlparse import urlparse, urlunparse


def is_git_ok(result):
message = result.stderr or result.stdout or result.error or ''
Expand Down Expand Up @@ -75,36 +69,17 @@ def remote_git_config():
return repo_url, username, password


def credentialed_repo_url(repo_url, username, password):
parsed = urlparse(repo_url)
if parsed.scheme not in ('http', 'https'):
raise RuntimeError(
'remote git push: only http(s) URLs support credentials')
netloc = parsed.netloc.split('@', 1)[-1]
auth = '{0}:{1}@'.format(
quote(username, safe=''),
quote(password, safe=''),
)
return urlunparse((
parsed.scheme,
auth + netloc,
parsed.path,
parsed.params,
parsed.query,
parsed.fragment,
))


def clone_remote_with_credentials(sandbox, repo_url, repo_path,
username, password, attempts=5):
auth_url = credentialed_repo_url(repo_url, username, password)
result = None
for attempt in range(attempts):
sandbox.commands.run('rm -rf {0}'.format(repo_path))
result = sandbox.git.clone(
auth_url,
repo_url,
repo_path,
depth=1,
username=username,
password=password,
timeout=180,
request_timeout=180,
)
Expand Down
4 changes: 2 additions & 2 deletions examples/sandbox_runtime_configuration.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@
def main():
sandbox = create_sandbox(timeout=300)
try:
listed = sandbox.client.list_sandboxes_v2(
listed = sandbox.client.list_sandboxes(
template=[sandbox.template_id],
)
print('sandboxes using template:', listed)
print('sandboxes using template:', listed['items'])

print('current injections:', sandbox.get_injections())
sandbox.update_injections([{
Expand Down
39 changes: 28 additions & 11 deletions qiniu/services/sandbox/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@

import requests

try:
from urllib.parse import quote
except ImportError:
from urllib import quote

from qiniu.auth import QiniuMacAuth, QiniuMacRequestsAuth
from qiniu.compat import (
basestring,
Expand Down Expand Up @@ -203,14 +208,26 @@ def _require_sandbox_id(sandbox_id):
return sandbox_id


def _encode_metadata_component(value):
if not isinstance(value, (bytes_type, text_type)):
value = text_type(value)
if isinstance(value, text_type):
value = value.encode('utf-8')
return quote(value, safe='')


def _normalize_list_options(opts):
opts = dict(opts or {})
query = opts.pop('query', None) or {}
metadata = query.get('metadata')
if metadata is not None and 'metadata' not in opts:
opts['metadata'] = metadata
if isinstance(opts.get('metadata'), dict):
opts['metadata'] = urlencode(opts.get('metadata'))
# The API unescapes the whole filter, then each key/value separately.
opts['metadata'] = urlencode([
(_encode_metadata_component(key), _encode_metadata_component(value))
for key, value in opts['metadata'].items()
])
for key, value in query.items():
if (
key != 'metadata' and
Expand Down Expand Up @@ -392,8 +409,12 @@ def _request(self, method, path, params=None, body=_UNSET,
return None
data = parse_json_response(response)
if page:
return {'items': data, 'nextToken': response.headers.get(
'X-Next-Token')}
next_token = response.headers.get('X-Next-Token')
if isinstance(data, dict):
if next_token is None:
next_token = data.get('nextToken') or data.get('next_token')
data = data.get('items') or data.get('sandboxes') or []
return {'items': data, 'nextToken': next_token}
return data

def _is_retryable(self, err):
Expand Down Expand Up @@ -438,18 +459,14 @@ def _retry_call(self, fn, retryable=None, retry_after=False,
raise

def list_sandboxes(self, **opts):
return self._request('GET', '/sandboxes', params=opts)

listSandboxes = list_sandboxes

def list_sandboxes_v2(self, **opts):
"""Return {'items': [...], 'nextToken': ...} from the sandbox list."""
return self._request(
'GET',
'/v2/sandboxes',
params=_normalize_list_options(opts))
params=_normalize_list_options(opts), page=True)

listSandboxesV2 = list_sandboxes_v2
list = list_sandboxes_v2
listSandboxes = list_sandboxes
list = list_sandboxes

def create_sandbox(self, template=None, **opts):
body = _normalize_sandbox_create_options(template, **opts)
Expand Down
27 changes: 20 additions & 7 deletions qiniu/services/sandbox/commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ def _decode_bytes(value):


def command_result_from_events(events, on_stdout=None, on_stderr=None,
on_pty=None):
on_pty=None, on_data=None):
pid = 0
stdout = ''
stderr = ''
Expand All @@ -88,6 +88,15 @@ def command_result_from_events(events, on_stdout=None, on_stderr=None,
on_stderr(stderr_chunk)
if pty_chunk and on_pty:
on_pty(pty_chunk)
if data.get('pty') is not None and on_data:
value = data['pty']
if isinstance(value, bytes_type):
pass
elif isinstance(value, basestring):
value = base64.b64decode(value)
else:
value = bytes_type(bytearray(value))
on_data(value)
stdout += stdout_chunk
stderr += stderr_chunk
stdout += pty_chunk
Expand All @@ -109,7 +118,7 @@ def command_result_from_events(events, on_stdout=None, on_stderr=None,

class CommandHandle(object):
def __init__(self, commands, result=None, throw_on_error=False,
events=None, on_stdout=None, on_stderr=None):
events=None, on_stdout=None, on_stderr=None, on_data=None):
self.commands = commands
result = result or CommandResult()
self.result = result
Expand All @@ -122,13 +131,15 @@ def __init__(self, commands, result=None, throw_on_error=False,
self._events = events
self._on_stdout = on_stdout
self._on_stderr = on_stderr
self._on_data = on_data

def wait(self, on_stdout=None, on_stderr=None):
if self._events is not None:
result = command_result_from_events(
self._events,
on_stdout=on_stdout or self._on_stdout,
on_stderr=on_stderr or self._on_stderr,
on_data=self._on_data,
)
if not result.pid:
result.pid = self.pid
Expand All @@ -143,7 +154,9 @@ def wait(self, on_stdout=None, on_stderr=None):
self.stdout = result.stdout
self.stderr = result.stderr
self._events = None
if self.throw_on_error and self.result.exit_code:
if (self.throw_on_error and self.result.exit_code and not (
self.result.exit_code == -1 and
getattr(self, '_disconnected', False))):
raise CommandExitError(self.result)
return self.result

Expand Down Expand Up @@ -253,7 +266,7 @@ def connect(self, pid, tag=None, user=None, timeout=None,
events = connect_stream_rpc(
self.sandbox,
'/process.Process/Connect',
{'process': {'selector': selector}},
{'process': selector},
user=user,
timeout=timeout,
stream=True,
Expand All @@ -280,23 +293,23 @@ def send_stdin(self, pid, data, user=None, timeout=None):
else:
data = str(data).encode('utf-8')
return connect_rpc(self.sandbox, '/process.Process/SendInput', {
'process': {'selector': {'pid': pid}},
'process': {'pid': pid},
'input': {'stdin': base64.b64encode(data).decode('ascii')},
}, user=user, timeout=timeout)

sendStdin = send_stdin

def close_stdin(self, pid, user=None, timeout=None):
return connect_rpc(self.sandbox, '/process.Process/CloseStdin', {
'process': {'selector': {'pid': pid}},
'process': {'pid': pid},
}, user=user, timeout=timeout)

closeStdin = close_stdin

def kill(self, pid, user=None, timeout=None):
try:
connect_rpc(self.sandbox, '/process.Process/SendSignal', {
'process': {'selector': {'pid': pid}},
'process': {'pid': pid},
'signal': 'SIGNAL_SIGKILL',
}, user=user, timeout=timeout)
return True
Expand Down
Loading
Loading