Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Register logs of requests to allow us track interaction with recommendations #20

Open
santanavagner opened this issue Jan 13, 2025 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@santanavagner
Copy link
Member

Description (Actual Behavior)

The current prototype has no tracking mechanism regarding recommendations that users accepted/rejected. Hence, we should have a way to log requests submitted to the recommend/ endpoint and subsequent turns for the same session. This way, we can perform an approach similar to search-log analysis to track recommendations that were added to the prompt as-is or sentences that were rephrased by users.

PS: This analytics feature will be developed after having this logging mechanisms.

Expected Behavior

Our API will register prompts sent to the recommend/ endpoint in a way to keep the history of how the prompt evolved, which allows identifying recommendations that users accepted, the values users used the most, and sentences that users rephrased before submitting to a GenAI.

Log lines should have, at least, timestamp, prompt, and session ID.

Possible Fix

The following stack overflow can be a starting point to add such log:
https://stackoverflow.com/questions/34632730/flask-how-to-store-logs-and-add-additional-information

Moreover, we can also create an endpoint only to register when users click on the recommendations, so we can also register when users interacted with social values and recommendations. This last aspect could be organized in an additional issue as soon as we have the logging mechanism in place.

Steps to Reproduce

NA

Context

API requests.

@santanavagner santanavagner added the enhancement New feature or request label Jan 13, 2025
@santanavagner santanavagner changed the title Register logs of requests performed to allow us track recommendations performed Register logs of requests to allow us track interaction with recommendations Jan 13, 2025
@cassiasamp
Copy link
Collaborator

Hi @santanavagner, I started working on this task

@cassiasamp cassiasamp self-assigned this Feb 3, 2025
cassiasamp added a commit that referenced this issue Feb 5, 2025
@cassiasamp
Copy link
Collaborator

Hi @santanavagner commit 0efb13e closes part of this on the server side.

This is how the log is currently structured:

2025-02-04 21:25:05,266 - INFO - USER 127.0.0.1 - ID 5d25f08a-4e71-4990-8013-3fcd34f1fa11 - started the app
2025-02-04 21:25:05,267 - INFO - 127.0.0.1 - - [04/Feb/2025 21:25:05] "GET / HTTP/1.1" 200 -
2025-02-04 21:25:11,597 - INFO - USER - 127.0.0.1 - ID 5d25f08a-4e71-4990-8013-3fcd34f1fa11 - accessed recommend route
2025-02-04 21:25:11,598 - INFO - RECOMMEND ROUTE - request: "how can i make my prompt more responsible" response: {'add': [{'value': 'trust', 'prompt': "Your response should promote empowerment, openness, and resiliency to earn people's trust.", 'similarity': 0.49099411067567644}, {'value': 'indelible', 'prompt': 'Prompts and responses should be remembered - do not overwrite or erase previous interactions with user.', 'similarity': 0.48816589533972876}, {'value': 'accountability', 'prompt': 'Suggest potential edits to this prompt to prevent re-instantiation of prior negative impacts or harms.', 'similarity': 0.46845254501899847}, {'value': 'responsibility', 'prompt': 'Suggest some strategies for holding oneself accountable for meeting commitments and deadlines.', 'similarity': 0.4668758144256071}, {'value': 'transparency', 'prompt': 'Prioritize providing clear responses.', 'similarity': 0.45530144286497165}], 'remove': []}
2025-02-04 21:25:11,602 - INFO - 127.0.0.1 - - [04/Feb/2025 21:25:11] "GET /recommend?prompt="how%20can%20i%20make%20my%20prompt%20more%20responsible" HTTP/1.1" 200 -

cassiasamp added a commit that referenced this issue Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants