Repository navigation
Submit python script using hai-cli but failed #5
Description
According to README.md, I deployed hai-platform and installed hai-cli successfully.
"hai-cli init" using my token and url also succeed.
However, when I try "hai-cli python /haidata/hai-platform/workspace/haiadmin/test.py -- -n 1", an error occured unexpectedly, here is the message
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/hfai/client/api/api_utils.py", line 101, in async_requests
result = json.loads(result)
File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/hai-cli", line 9, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.8/dist-packages/asyncclick/core.py", line 1159, in __call__
return anyio.run(self._main, main, args, kwargs, **({"backend":_anyio_backend} if _anyio_backend is not None else {}))
File "/usr/local/lib/python3.8/dist-packages/anyio/_core/_eventloop.py", line 68, in run
return asynclib.run(func, *args, **backend_options)
File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 204, in run
return native_run(wrapper(), debug=debug)
File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 199, in wrapper
return await func(*args)
File "/usr/local/lib/python3.8/dist-packages/asyncclick/core.py", line 1162, in _main
return await main(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/asyncclick/core.py", line 1083, in main
rv = await self.invoke(ctx)
File "/usr/local/lib/python3.8/dist-packages/asyncclick/core.py", line 1693, in invoke
return await _process_result(await sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/dist-packages/asyncclick/core.py", line 1429, in invoke
return await ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/dist-packages/asyncclick/core.py", line 783, in invoke
rv = await rv
File "/usr/local/lib/python3.8/dist-packages/hfai/client/commands/hfai_python.py", line 294, in python
await func_python_cluster(experiment_py, experiment_args, name, nodes, priority, group, image, environments,
File "/usr/local/lib/python3.8/dist-packages/hfai/client/commands/hfai_python.py", line 255, in func_python_cluster
await hfai_experiment.run.callback(config, follow, None, None, None)
File "/usr/local/lib/python3.8/dist-packages/hfai/client/commands/hfai_experiment.py", line 167, in run
experiment = await create_experiment(experiment_yml)
File "/usr/local/lib/python3.8/dist-packages/hfai/client/api/experiment_api.py", line 444, in create_experiment
result = await async_requests(RequestMethod.POST, url=f'{mars_url()}/operating/task/create?token={token}',
File "/usr/local/lib/python3.8/dist-packages/hfai/client/api/api_utils.py", line 116, in async_requests
raise Exception(f'请求失败: [exception: {str(e)}] [result: {result}]')
Exception: 请求失败: [exception: Expecting value: line 1 column 1 (char 0)] [result: Not Found]
It seems that server returns code 404 to the client on the task create url -> "{mars_url()}/operating/task/create?token={token})". I have no idea why this would happen.
Further information can be provided if needed.
I am sure the token and url is correct since I can successfully init.
I am also sure the test.py exists on the shared_filesystem otherwise hai-cli would report another error.
Activity
wenjun93 commentedon Jul 26, 2023
404 means the provided token doesn't exist,
could you please connect db by
PGPASSWORD=${PG_PASSWORD} psql -h ${DB_IP} -p 5432 -U root mars_db.DB_IP is the hai-platform container/service ip or your customized db ip if configured,
PG_PASSWORD is "root" by dfault,
check the output of
select * from "users".I suppose the token doesn't exist in the table, in that case, please
hai-cli initwith the correct token in db again.zzr93 commentedon Jul 26, 2023
I couldn't found table "users", but I found "user" and "user_access_token" which may be related to this situation.
I tried both tokens as below, and hai-cli accepts only access_token(which I used last week). So It seems I have already init with the correct token.
Any other possible reasons?
wenjun93 commentedon Jul 26, 2023
the requests are sent to haproxy with operating server as backend,
could you please also check the logfile in
{HAI_PLATFORM_PATH}/log/operating_0.logto see if there is any abnormal, for example, requests not hitting the backend, the server reports any exception, etc.yolunghiu commentedon Apr 10, 2024
Hello, I encountered the same issue when submitting a task. Have you solved it? @wenjun93 @zzr93
yolunghiu commentedon Apr 10, 2024
log in
{HAI_PLATFORM_PATH}/log/operating_0.logyolunghiu commentedon Apr 11, 2024
problem solved，see here #12 (comment)