Features / Cleanup for MP Frontend #387
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge). To run full CI, you can do one of these:
🚀 |
| # task.add_done_callback(_running_tasks.remove) | ||
| if not engine_args.disable_log_stats: | ||
| task = asyncio.create_task(_force_log()) | ||
| _running_tasks.add(task) |
There was a problem hiding this comment.
if this is here just to be a hard ref to the task so it doesn't get GC'ed, then I think it would work fine as a local variable inside lifespan instead of a module-scope variable
| @@ -82,6 +82,7 @@ async def _force_log(): | |||
| async def build_backend(args) -> AsyncIterator[VLLMBackend]: | |||
| # Context manager to handle backend lifecycle | |||
| # Ensures everything is shutdown and cleaned up on error/exit | |||
| global engine_args | |||
| # Wait for server process to join | ||
| rpc_server_process.join() | ||
| # Wait for server process to join | ||
| rpc_server_process.join() |
There was a problem hiding this comment.
I actually meant this to be in the finally statement so that it will run on exit when there's an unhandled exception, which can happen if there's an exception before we get to the guarded
try:
await server_task
There was a problem hiding this comment.
this was a typo, will fix
| self.get_data_socket.send(pickle.dumps(GetDataRequest.MODEL_CONFIG)) | ||
| model_config = await self.get_data_socket.recv() | ||
| return pickle.loads(model_config) | ||
| # Await acknowledgement from RPCServer that it aborted. |
There was a problem hiding this comment.
copy/paste comment error?
robertgshaw2-redhat
merged commit 1f33286
into
isolate-oai-server-process
SUMMARY:
do_log_statsabort