关于验证场景 & Linux版本问题

[Jungle728]

现有的验证场景主要包含哪些评测指标？具体有哪些？

[734162621]

目前测评指标的部分还在进行调试，主要包括战损比K/D(kill/death ratio) 和胜率 WR(winning rate，也可认为是任务完成率)

在develop分支中我们已经上传了用于实验数据记录的llm_pysc2.lib.data_recorder.DataRecorder、评测指标分析脚本log_analyse.py，他们可以用于分析验证场景的战损比（死亡单位的资源比，并通过1gas=2mineral换算成统一的单位）和胜率，该模块正在进行调试，预计在2024/11/1之前整合到master分支中。

另外，大模型的ave waiting time、ave input tokens、ave output tokens也将被记录，但这些内容暂时仅存放于develop分支中，它们也将与前面所述的内容一同在完成调试后加入master分支。

[Jungle728]

2024/11/1是否会同步公开openai等模型的评测结果？

[734162621]

会的。

计划是11/1左右发布预印本论文，作为实验章节的基准数据出现，代码中提供的各系列的大模型至少选取一个进行测试。（论文的实际发布时间可能会受其他事项干扰，不一定在11/1之前发布，但不会和11/1这个时间节点相差太远）

[Jungle728]

好的，谢谢。

[Jungle728]

我下载了3.16.1版本的星际争霸游戏，但我在测试环境时出现错误。报错如下：
python -m llm_pysc2.bin.experiment_llm_pysc2
pygame 2.6.1 (SDL 2.28.4, Python 3.9.0)
Hello from the pygame community. https://www.pygame.org/contribute.html
pygame 2.6.1 (SDL 2.28.4, Python 3.9.0)
Hello from the pygame community. https://www.pygame.org/contribute.html
I1023 11:36:14.414562 140188840916800 sc_process.py:130] Launching SC2: /home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64 -listen 127.0.0.1 -port 32823 -dataDir /home/ubuntu/StarCraftII/ -tempDir /tmp/sc-19qf_xfi/
I1023 11:36:14.424873 140188840916800 remote_controller.py:162] Connecting to: ws://127.0.0.1:32823/sc2api, attempt: 0, running: True
Version: B55958 (SC2.3.16)
Build: Jul 31 2017 13:19:41
Command Line: '"/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64" -listen 127.0.0.1 -port 32823 -dataDir /home/ubuntu/StarCraftII/ -tempDir /tmp/sc-19qf_xfi/'
Starting up...
Startup Phase 1 complete
I1023 11:36:15.426995 140188840916800 remote_controller.py:162] Connecting to: ws://127.0.0.1:32823/sc2api, attempt: 1, running: True
Startup Phase 2 complete
Creating stub renderer...
Listening on: 127.0.0.1:32823 (32823)
Startup Phase 3 complete. Ready for commands.
I1023 11:36:16.428864 140188840916800 remote_controller.py:162] Connecting to: ws://127.0.0.1:32823/sc2api, attempt: 2, running: True
Requesting to join a single player game
Configuring interface options
Configure: raw interface enabled
Configure: feature layer interface enabled
Configure: score interface enabled
Configure: render interface disabled
Entering load game phase.
Launching next game.
Next launch phase started: 2
Next launch phase started: 3
Next launch phase started: 4
Next launch phase started: 5
Next launch phase started: 6
Next launch phase started: 7
Next launch phase started: 8
Error signal: '11'
Backtrace:
/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64[0x652f73]
/lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7fd3bd6d6090]
/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64[0x1da48c2]
/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64[0x2094299]
/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64[0x2096153]
/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64[0x2096a81]
/lib/x86_64-linux-gnu/libc.so.6(+0x468a7)[0x7fd3bd6d98a7]
/lib/x86_64-linux-gnu/libc.so.6(on_exit+0x0)[0x7fd3bd6d9a60]
/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64[0x674e3e]
/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64[0x6542ec]
/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64[0xe81598]
/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64[0x12d1565]
/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64[0x14c8bfa]
/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64[0x12c0446]
/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64[0x12cc26a]
/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64[0x12c8724]
/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64[0x19338fc]
/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64[0x5ed4d5]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fd3bd6b7083]
/home/ubuntu/StarCraftII/Versions/Base55958/SC2_x64[0x5ed2d9]
Traceback (most recent call last):
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/lib/protocol.py", line 62, in catch_websocket_connection_errors
yield
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/lib/protocol.py", line 179, in _read
response_str = self._sock.recv()
File "/opt/conda/envs/pysc2/lib/python3.9/site-packages/websocket/_core.py", line 388, in recv
opcode, data = self.recv_data()
File "/opt/conda/envs/pysc2/lib/python3.9/site-packages/websocket/_core.py", line 416, in recv_data
opcode, frame = self.recv_data_frame(control_frame)
File "/opt/conda/envs/pysc2/lib/python3.9/site-packages/websocket/_core.py", line 437, in recv_data_frame
frame = self.recv_frame()
File "/opt/conda/envs/pysc2/lib/python3.9/site-packages/websocket/_core.py", line 478, in recv_frame
return self.frame_buffer.recv_frame()
File "/opt/conda/envs/pysc2/lib/python3.9/site-packages/websocket/_abnf.py", line 363, in recv_frame
self.recv_header()
File "/opt/conda/envs/pysc2/lib/python3.9/site-packages/websocket/_abnf.py", line 319, in recv_header
header = self.recv_strict(2)
File "/opt/conda/envs/pysc2/lib/python3.9/site-packages/websocket/abnf.py", line 398, in recv_strict
bytes = self.recv(min(16384, shortage))
File "/opt/conda/envs/pysc2/lib/python3.9/site-packages/websocket/_core.py", line 563, in _recv
return recv(self.sock, bufsize)
File "/opt/conda/envs/pysc2/lib/python3.9/site-packages/websocket/_socket.py", line 132, in recv
raise WebSocketConnectionClosedException("Connection to remote host was lost.")
websocket._exceptions.WebSocketConnectionClosedException: Connection to remote host was lost.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/lib/protocol.py", line 147, in send
res = self.send_req(req)
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/lib/protocol.py", line 127, in send_req
return self.read()
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/lib/stopwatch.py", line 205, in _stopwatch
return func(*args, **kwargs)
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/lib/protocol.py", line 98, in read
response = self._read()
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/lib/protocol.py", line 179, in _read
response_str = self._sock.recv()
File "/opt/conda/envs/pysc2/lib/python3.9/contextlib.py", line 135, in exit
self.gen.throw(type, value, traceback)
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/lib/protocol.py", line 64, in catch_websocket_connection_errors
raise ConnectionError("Connection already closed. SC2 probably crashed. "
pysc2.lib.protocol.ConnectionError: Connection already closed. SC2 probably crashed. Check the error log.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/conda/envs/pysc2/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/pysc2/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/bin/agent.py", line 164, in
app.run(main)
File "/opt/conda/envs/pysc2/lib/python3.9/site-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/opt/conda/envs/pysc2/lib/python3.9/site-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/bin/agent.py", line 150, in main
run_thread(agent_classes, players, FLAGS.map, FLAGS.render)
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/bin/agent.py", line 88, in run_thread
with sc2_env.SC2Env(
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/env/sc2_env.py", line 262, in init
self._create_join()
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/env/sc2_env.py", line 417, in _create_join
self._parallel.run((c.join_game, join)
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/lib/run_parallel.py", line 51, in run
return [funcs0]
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/lib/remote_controller.py", line 95, in _valid_status
return func(self, *args, **kwargs)
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/lib/remote_controller.py", line 70, in _check_error
return check_error(func(*args, **kwargs), error_enum)
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/lib/stopwatch.py", line 205, in _stopwatch
return func(*args, **kwargs)
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/lib/remote_controller.py", line 205, in join_game
return self._client.send(join_game=req_join_game)
File "/opt/luhailun/Agent/Agent_eval/LLM-PySC2/pysc2/lib/protocol.py", line 149, in send
raise ConnectionError("Error during %s: %s" % (name, e)) from e
pysc2.lib.protocol.ConnectionError: Error during join_game: Connection already closed. SC2 probably crashed. Check the error log.
I1023 11:36:22.680248 140188840916800 sc2_env.py:741] Environment Close
I1023 11:36:22.680866 140188840916800 sc_process.py:227] Shutdown gracefully.
I1023 11:36:22.680967 140188840916800 sc_process.py:205] Shutdown with return code: -11
我完全按照readme中的要求配置，但在测试环境的过程中一直出现Error signal: '11'的报错。

[734162621]

这似乎是一个星际争霸程序启动和SC2-pysc2通讯的报错（并且未涉及到我们做过修改的pysc2部分，我们只启用了pysc2.lib.feature的raw_unit的tag，并加入了一个动作函数到pysc2.lib.actions，未修改过sc2client、protocol相关的东西）

这个问题我们目前暂未遇到过，可能需要查找pysc2库相关文章确定具体问题，或者按照报错提示查看SC2的error log。
（pysc2.lib.protocol.ConnectionError: Error during join_game: Connection already closed. SC2 probably crashed. Check the error log.）

[Jungle728]

我在三台不同的ubuntu机器上分别部署了LLM-pysc2，尝试了3.16.1、4.10、4.6.2三个不同版本的StarCraftII包，均出现上述Error signal: '11'的报错。或者是否能出一份更详细的复现说明？
目前我的复现操作如下：

1. git clone https://github.com/NKAI-Decision-Team/LLM-PySC2.git；
2. 从 https://github.com/Blizzard/s2client-proto 下载3.16.1版本的linux package并解压保存到~/StarCraftII路径；
3. 复制./llm_pysc/maps中的文件至~/StarCraftII/Maps/；
4. pip install -e .；
5. 运行测试脚本 llm_pysc2/bin/experiment_llm_pysc2.py 出现Error signal: '11'报错；
6. 运行测试脚本llm_pysc2/bin/experiment_llm_smac.py 成功。

[734162621]

尝试一下下载5.0.13(92440)版本的游戏（地图编辑器记录的版本为该版本），可能是地图版本高于游戏版本引发了兼容性问题，最终导致SC2进程报错。

您可以考虑在Windows系统上运行星际争霸程序，在Windows上与远程大模型/部署在linux的局域网大模型交互。

README中关于获取SC2游戏的部分来源于PySC2项目的README，这一部分确实存在疏忽，暂时没有对其他系统的游戏-地图版本进行测试。我们将在基准测试完成后立即开展程序版本的兼容性测试。

[Jungle728]

好的，感谢你的回复！ 希望后续能出一个兼容linux/ubuntu的版本。

[734162621]

好的，感谢理解。我们后续会尝试确定是否是地图和游戏版本导致了上述问题，并尝试解决这个问题。如果您对于这一问题的处理有新的进展，也欢迎在此处分享，感谢您的贡献 [鞠躬]。

[Jungle728]

似乎暂时不会有新的linux包发布。 Blizzard/s2client-proto#185

[734162621]

如果我们后续确认是地图版本的问题，可能会尝试一下使用低版本的地图编辑器重制一遍地图。技术上的阻碍不大，主要是需要一些时间来完成这个工作。