Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why my json response back from local host is always empty? #12

Open
66deft123456 opened this issue Jan 24, 2025 · 10 comments
Open

Why my json response back from local host is always empty? #12

66deft123456 opened this issue Jan 24, 2025 · 10 comments

Comments

@66deft123456
Copy link

root@v02e09045:/workspace/parrotserve/ParrotServe# python3 examples/write_recommendation_letter.py
2025-01-24 02:57:01,530 - Interface - WARNING - Use a conversation template ConversationTemplate to transform the function. This only works well for requests which are dispatched to engines with the corresponding models.
2025-01-24 02:57:01,531 - PFunc VM - INFO - VM (session_id=NONE) registers function: write_recommendation_letter
2025-01-24 02:57:01,531 - PFunc VM - INFO - VM (session_id=NONE) runs program: main
2025-01-24 02:57:01,534 - PFunc VM - INFO - VM registered a Session (session_id=0).
2025-01-24 02:57:01,535 - PFunc VM - INFO - VM (session_id=0) submits SemanticCall: write_recommendation_letter
2025-01-24 02:57:01,537 - Public API - ERROR - Submit semantic call (session_id=0) error in http://localhost:9000. Error: Expecting value: line 1 column 1 (char 0)
Error happens when executing Parrot program: <class 'requests.exceptions.JSONDecodeError'> JSONDecodeError('Expecting value: line 1 column 1 (char 0)')
Traceback: Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.10/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/vm.py", line 402, in running_scope
yield
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/vm.py", line 454, in run
program(*args)
File "/workspace/parrotserve/ParrotServe/examples/write_recommendation_letter.py", line 21, in main
letter = letter_generator(
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/function.py", line 328, in call
return self._call_func(*args, **kwargs)
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/function.py", line 355, in _call_func
param_info = self._submit_semantic_call(call)
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/function.py", line 277, in _submit_semantic_call
return BasicFunction._virtual_machine_env.submit_semantic_call_handler(call)
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/vm.py", line 197, in submit_semantic_call_handler
resp = submit_semantic_call(
File "/workspace/parrotserve/ParrotServe/parrot/protocol/public/apis.py", line 141, in submit_semantic_call
raise e
File "/workspace/parrotserve/ParrotServe/parrot/protocol/public/apis.py", line 129, in submit_semantic_call
return send_http_request(
File "/workspace/parrotserve/ParrotServe/parrot/protocol/http_utils.py", line 48, in send_http_request
resp_data = error_resp.json()
File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

2025-01-24 02:57:01,540 - PFunc VM - INFO - VM unregistered its Session (session_id=0).

@66deft123456
Copy link
Author

My local host return 404, but I successfully launch it.

Image

Image

@SiriusNEO
Copy link
Collaborator

@66deft123456 Hi! Could you please try some other examples like write_recommendation_letter?I'm worried that tell_me_joke uses multiple outputs which may not be a stable feature in main branch. By the way, for the 404 issue you mentioned, the reason is that there is no route entry for http://localhost:9000. You can check https://github.com/microsoft/ParrotServe/blob/main/parrot/serve/http_server.py for details.

@66deft123456
Copy link
Author

66deft123456 commented Feb 5, 2025

@66deft123456 Hi! Could you please try some other examples like write_recommendation_letter?I'm worried that tell_me_joke uses multiple outputs which may not be a stable feature in main branch. By the way, for the 404 issue you mentioned, the reason is that there is no route entry for http://localhost:9000. You can check https://github.com/microsoft/ParrotServe/blob/main/parrot/serve/http_server.py for details.

Actually most of these examples show the same error that shows requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0) and others include problems about some missing packages. The 404 issue is solved through adding the route entry. Thank you! So I am confused that whether the examples should include ‘/v1/session’ but it still fails when I add ‘/v1/session’.

Image

Image

@SiriusNEO
Copy link
Collaborator

No, the example should not include v1/session since it's added in the server side (You can check the implementation in https://github.com/microsoft/ParrotServe/blob/main/parrot/serve/http_server.py). Could you please check the log whether the server is launched at localhost:9000 (May be it's launched at another port number)? And for debugging, can you try to run a minimal server-client communication example using FastAPI, to ensure the localhost HTTP communication is OK?

@66deft123456
Copy link
Author

No, the example should not include v1/session since it's added in the server side (You can check the implementation in https://github.com/microsoft/ParrotServe/blob/main/parrot/serve/http_server.py). Could you please check the log whether the server is launched at localhost:9000 (May be it's launched at another port number)? And for debugging, can you try to run a minimal server-client communication example using FastAPI, to ensure the localhost HTTP communication is OK?

Image
I verified that the service on port 9000 was successfully started, and I also tried port 9002, which also successfully started the service. The issue is that while the session routing is functioning, the semantic_call is not. I attempted to add a time.sleep(10) code at the semantic_call definition in the HTTP server, but the results showed that the route was not being called, as there was no 10-second wait observed.
Image

@66deft123456
Copy link
Author

No, the example should not include v1/session since it's added in the server side (You can check the implementation in https://github.com/microsoft/ParrotServe/blob/main/parrot/serve/http_server.py). Could you please check the log whether the server is launched at localhost:9000 (May be it's launched at another port number)? And for debugging, can you try to run a minimal server-client communication example using FastAPI, to ensure the localhost HTTP communication is OK?

Image
I have tried to run a minimal server-client communication example using FastAPI,and the localhost HTTP communication seems to be OK.

@SiriusNEO
Copy link
Collaborator

Get it. I guess you may not set the environment variables (source .env). The semantic_call simulates a client-server latency using SIMULATE_NETWORK_LATENCY_PRT environment variable.

@66deft123456
Copy link
Author

Get it. I guess you may not set the environment variables (source .env). The semantic_call simulates a client-server latency using SIMULATE_NETWORK_LATENCY_PRT environment variable.

After executing source .env and restarting the service, I noticed that the local service automatically disconnects after a period of time. For example, shortly after I run python3 examples/write_recommendation_letter.py, the output is as shown in the image below. However, after a long period of time, it throws an error.

Image

However, after a long period of time, it throws an error.

**Task was destroyed but it is pending!
task: <Task pending name='Task-9' coro=<GraphExecutor._execute_coroutine() running at /workspace/parrotserve/ParrotServe/parrot/serve/session/graph_executor.py:81> wait_for= cb=[_task_error_callback_fail_fast() at /workspace/parrotserve/ParrotServe/parrot/utils/async_utils.py:11]>
Task was destroyed but it is pending!
task: <Task pending name='Task-11' coro=<RequestResponseCycle.run_asgi() running at /usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py:403> wait_for= cb=[set.discard()]>
--- Logging error ---
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
GeneratorExit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.10/logging/init.py", line 1100, in emit
msg = self.format(record)
File "/usr/lib/python3.10/logging/init.py", line 943, in format
return fmt.format(record)
File "/usr/lib/python3.10/logging/init.py", line 681, in format
s = self.formatMessage(record)
File "/usr/local/lib/python3.10/dist-packages/uvicorn/logging.py", line 56, in formatMessage
recordcopy = copy(record)
File "/usr/lib/python3.10/copy.py", line 92, in copy
rv = reductor(4)
ImportError: sys.meta_path is None, Python is likely shutting down
Call stack:
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
self.logger.error(msg, exc_info=exc)
Message: 'Exception in ASGI application\n'
Arguments: ()
--- Logging error ---
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
GeneratorExit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.10/logging/init.py", line 1100, in emit
msg = self.format(record)
File "/usr/lib/python3.10/logging/init.py", line 943, in format
return fmt.format(record)
File "/usr/lib/python3.10/logging/init.py", line 681, in format
s = self.formatMessage(record)
File "/usr/local/lib/python3.10/dist-packages/uvicorn/logging.py", line 98, in formatMessage
recordcopy = copy(record)
File "/usr/lib/python3.10/copy.py", line 92, in copy
rv = reductor(4)
ImportError: sys.meta_path is None, Python is likely shutting down
Call stack:
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 410, in run_asgi
await self.send_500_response()
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 438, in send_500_response
await self.send(response_start_event)
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 473, in send
self.access_logger.info(
Message: '%s - "%s %s HTTP/%s" %d'
Arguments: ('::1:59888', 'GET', '/v1/semantic_var/1ed57add-2748-37d6-816b-22bca23fd011', '1.1', 500)
Exception ignored in: <coroutine object RequestResponseCycle.run_asgi at 0x7fb07805fd10>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 410, in run_asgi
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 444, in send_500_response
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 518, in send
File "/usr/lib/python3.10/asyncio/selector_events.py", line 706, in close
File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
RuntimeError: Event loop is closed
Task was destroyed but it is pending!
task: <Task pending name='Task-3' coro=<LifespanOn.main() running at /usr/local/lib/python3.10/dist-packages/uvicorn/lifespan/on.py:86> wait_for=>
2025-02-07 03:37:34,309 - Public API - ERROR - Get semantic variable 1ed57add-2748-37d6-816b-22bca23fd011 (session_id=0) error in http://localhost:9000. Error: Expecting value: line 1 column 1 (char 0)
Error happens when executing Parrot program: <class 'requests.exceptions.JSONDecodeError'> JSONDecodeError('Expecting value: line 1 column 1 (char 0)')
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/queues.py", line 159, in get
await getter
GeneratorExit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 700, in lifespan
await receive()
File "/usr/local/lib/python3.10/dist-packages/uvicorn/lifespan/on.py", line 137, in receive
return await self.receive_queue.get()
File "/usr/lib/python3.10/asyncio/queues.py", line 161, in get
getter.cancel() # Just in case getter is not done yet.
File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
self._check_closed()
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.10/logging/init.py", line 1100, in emit
msg = self.format(record)
File "/usr/lib/python3.10/logging/init.py", line 943, in format
return fmt.format(record)
File "/usr/lib/python3.10/logging/init.py", line 681, in format
s = self.formatMessage(record)
File "/usr/local/lib/python3.10/dist-packages/uvicorn/logging.py", line 56, in formatMessage
recordcopy = copy(record)
File "/usr/lib/python3.10/copy.py", line 92, in copy
rv = reductor(4)
ImportError: sys.meta_path is None, Python is likely shutting down
Call stack:
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 704, in lifespan
await send({"type": "lifespan.shutdown.failed", "message": exc_text})
File "/usr/local/lib/python3.10/dist-packages/uvicorn/lifespan/on.py", line 134, in send
self.logger.error(message["message"])
Message: 'Traceback (most recent call last):\n File "/usr/lib/python3.10/asyncio/queues.py", line 159, in get\n await getter\nGeneratorExit\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 700, in lifespan\n await receive()\n File "/usr/local/lib/python3.10/dist-packages/uvicorn/lifespan/on.py", line 137, in receive\n return await self.receive_queue.get()\n File "/usr/lib/python3.10/asyncio/queues.py", line 161, in get\n getter.cancel() # Just in case getter is not done yet.\n File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon\n self._check_closed()\n File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed\n raise RuntimeError('Event loop is closed')\nRuntimeError: Event loop is closed\n'
Arguments: ()
Traceback: Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.10/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/vm.py", line 402, in running_scope
yield
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/vm.py", line 454, in run
program(*args)
File "/workspace/parrotserve/ParrotServe/examples/write_recommendation_letter.py", line 29, in main
letter_str = letter.get(P.PerformanceCriteria.LATENCY)
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/semantic_variable.py", line 129, in get
self.content = self._get_semantic_variable(criteria)
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/semantic_variable.py", line 75, in _get_semantic_variable
return self._virtual_machine_env.get_semantic_variable_handler(
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/vm.py", line 139, in get_semantic_variable_handler
resp = get_semantic_variable(
File "/workspace/parrotserve/ParrotServe/parrot/protocol/public/apis.py", line 266, in get_semantic_variable
raise e
File "/workspace/parrotserve/ParrotServe/parrot/protocol/public/apis.py", line 252, in get_semantic_variable
return send_http_request(
File "/workspace/parrotserve/ParrotServe/parrot/protocol/http_utils.py", line 48, in send_http_request
resp_data = error_resp.json()
File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

2025-02-07 03:37:34,312 - Public API - ERROR - Remove session error in http://localhost:9000. Error: HTTPConnectionPool(host='localhost', port=9000): Max retries exceeded with url: /v1/session/0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9f02eee5f0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.10/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/vm.py", line 402, in running_scope
yield
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/vm.py", line 454, in run
program(*args)
File "/workspace/parrotserve/ParrotServe/examples/write_recommendation_letter.py", line 29, in main
letter_str = letter.get(P.PerformanceCriteria.LATENCY)
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/semantic_variable.py", line 129, in get
self.content = self._get_semantic_variable(criteria)
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/semantic_variable.py", line 75, in _get_semantic_variable
return self._virtual_machine_env.get_semantic_variable_handler(
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/vm.py", line 139, in get_semantic_variable_handler
resp = get_semantic_variable(
File "/workspace/parrotserve/ParrotServe/parrot/protocol/public/apis.py", line 266, in get_semantic_variable
raise e
File "/workspace/parrotserve/ParrotServe/parrot/protocol/public/apis.py", line 252, in get_semantic_variable
return send_http_request(
File "/workspace/parrotserve/ParrotServe/parrot/protocol/http_utils.py", line 48, in send_http_request
resp_data = error_resp.json()
File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/workspace/parrotserve/ParrotServe/examples/write_recommendation_letter.py", line 34, in
vm.run(main)
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/vm.py", line 445, in run
with self.running_scope(timeit):
File "/usr/lib/python3.10/contextlib.py", line 153, in exit
self.gen.throw(typ, value, traceback)
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/vm.py", line 410, in running_scope
self.unset_global_env()
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/vm.py", line 385, in unset_global_env
self.unregister_session()
File "/workspace/parrotserve/ParrotServe/parrot/frontend/pfunc/vm.py", line 298, in unregister_session
remove_session(
File "/workspace/parrotserve/ParrotServe/parrot/protocol/public/apis.py", line 122, in remove_session
raise e
File "/workspace/parrotserve/ParrotServe/parrot/protocol/public/apis.py", line 112, in remove_session
send_http_request(
File "/workspace/parrotserve/ParrotServe/parrot/protocol/http_utils.py", line 55, in send_http_request
raise error
File "/workspace/parrotserve/ParrotServe/parrot/protocol/http_utils.py", line 36, in send_http_request
resp = requests.delete(url, json=kwargs, timeout=timeout)
File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 157, in delete
return request("delete", url, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, kwargs)
File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=9000): Max retries exceeded with url: /v1/session/0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9f02eee5f0>: Failed to establish a new connection: [Errno 111] Connection refused'))

@66deft123456
Copy link
Author

Get it. I guess you may not set the environment variables (source .env). The semantic_call simulates a client-server latency using SIMULATE_NETWORK_LATENCY_PRT environment variable.

I reconfigured the environment completely and strictly according to the README, started the service, and tried to run the write_recommendation_letter.py script. However, the result is still the same as yesterday.

@SiriusNEO
Copy link
Collaborator

SiriusNEO commented Feb 10, 2025

@66deft123456 Hi, I reconfigured the environment from scratch following the documentation in my local environment and don't meet the problem you encountered.

Image

Can you example the log/ files more carefully? Normally, Parrot create a log/ directory containing 4 files:

Image

The real cause of the error may be in the stdout file rather than in the log. Also, you can run the benchmark/bench_kernel.py to check whether the kernel is compiled successfully and run tests/engine/test_engine.py to check whether you can run a single engine (without ServeCore).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants