[go: nahoru, domu]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ENHANCEMENT] Full trace from spanid, incl evaluations, from REST api/python client #3558

Open
stdweird opened this issue Jun 18, 2024 · 8 comments
Labels
c/evals c/traces enhancement New feature or request triage issues that need triage

Comments

@stdweird
Copy link

Is your feature request related to a problem? Please describe.
We want to provided detailed feedback to "expert" users, and thus want to give them access to the full trace of a eg a query/response flow from a chatbot; but not to phoenix itself. We currently only gather a single spanid from the whole flow, and want to find the trace this spanid belongs too, and retrieve all other spans and some hierarchy info. This ideally also contains the evaluations.

Describe the solution you'd like
A rest api where we can post the spanid, and get back as answer all data a trace detail/overlay would be able to show. Hierarchical data can be in nested "dict" structure, or with references between parent/child.

Also an extra method to the python client to retrieve this data.

Describe alternatives you've considered
Eg to retrieve the evaluations belong to a span, we have no other way bit to retrieve all evaluations, and filter out everything except the one we need.

@stdweird stdweird added enhancement New feature or request triage issues that need triage labels Jun 18, 2024
@axiomofjoy
Copy link
Contributor

Hey @stdweird, we'll be building out auth in Phoenix. Maybe what you're asking for is the ability to share traces and spans? Or is it important for you to be able to pull this information programmatically?

@stdweird
Copy link
Author

@axiomofjoy hmm, ideally we have both ;)

programmatic access to the evaluations associated with a single spanid is something we will actually need. pulling all evaluation to get single set associated with known spanid, not sure how long this will last.

on the authenticated traces, what are you planning to provide: group based access mapped to projects? some user metatadata in the span, and authenticated user can see all its spans, and nothing else? our use case with the full traces view is that (certain classes of) users can see the traces, but we want to give them a url of some sort to click on and point it to the trace of the answer. now we have the spanid per generated answer (in a chatbot scenario), so we would need a url like "/traces/?spanid=abcdef" to resolve to the trace. we are not going to let users search for the trace.

@axiomofjoy
Copy link
Contributor

on the authenticated traces, what are you planning to provide: group based access mapped to projects? some user metatadata in the span, and authenticated user can see all its spans, and nothing else? our use case with the full traces view is that (certain classes of) users can see the traces, but we want to give them a url of some sort to click on and point it to the trace of the answer. now we have the spanid per generated answer (in a chatbot scenario), so we would need a url like "/traces/?spanid=abcdef" to resolve to the trace. we are not going to let users search for the trace.

We haven't scoped authentication yet so I am not sure, but thanks for the feedback!

To unblock for now, you can query the GraphQL API directly.

from httpx import Client

query = """
  query ($spanId: GlobalID!) {
    span: node(id: $spanId) {
      ... on Span {
        id
      	spanEvaluations {
          name
          score
          label
          explanation
        }
      }
    }
  }
"""
span_id = "U3BhbjoxMA=="  # replace with your span id
test_client = Client(base_url="http://localhost:6006")
response = test_client.post(
    "/graphql",
    json={
        "query": query,
        "variables": {
            "spanId": span_id,
        },
    },
)
assert response.status_code == 200
response_json = response.json()
assert response_json.get("errors") is None
print(response_json["data"])

@stdweird
Copy link
Author
stdweird commented Jul 5, 2024

@axiomofjoy sorry for getting back so late to this. i get an error

graphql query='\n  query ($spanId: GlobalID!) {\n    span: node(id: $spanId) {\n      ... on Span {\n        id\n      \tspanEvaluations {\n          name\n          score\n          label\n          explanation\n        }\n      }\n    }\n  }\n' variables={'spanId': '2437f393a545acfc'} returned errors=[{'message': "Variable '$spanId' got invalid value '2437f393a545acfc'; Expected type 'GlobalID'. 'utf-8' codec can't decode byte 0xfb in position 2: invalid start byte", 'locations': [{'line': 2, 'column': 10}]}]: response=<Response [200]>

any idea what this could be about? also, the spanid you give, what format is this? i am used to either int or the hex version of the int (what is eg used when adding an evaluation via the python client)

@stdweird
Copy link
Author
stdweird commented Jul 5, 2024

ok, after playing around with GlobalID (and the spanId a str of the int of the span id), i get another error

ERROR:root:graphql query='\n  query ($spanId: GlobalID!) {\n    span: node(id: $spanId) {\n      ... on Span {\n        id\n      \tspanEvaluations {\n          name\n          score\n          label\n          explanation\n        }\n      }\n    }\n  }\n' variables={'spanId': 'U3BhbjoyNjA5ODIyMzI0NTQzMDQwNzY0'} returned errors=[{'message': "(sqlalchemy.dialects.postgresql.asyncpg.Error) <class 'asyncpg.exceptions.DataError'>: invalid input for query argument $1: 2609822324543040764 (value out of int32 range)\n[SQL: SELECT traces.id, traces.project_rowid, traces.trace_id, traces.start_time, traces.end_time, spans.id AS id_1, spans.trace_rowid, spans.span_id, spans.parent_id, spans.name, spans.span_kind, spans.start_time AS start_time_1, spans.end_time AS end_time_1, spans.attributes, spans.events, spans.status_code, spans.status_message, spans.cumulative_error_count, spans.cumulative_llm_token_count_prompt, spans.cumulative_llm_token_count_completion \nFROM spans JOIN traces ON traces.id = spans.trace_rowid \nWHERE spans.id = $1::INTEGER]\n[parameters: (2609822324543040764,)]\n(Background on this error at: https://sqlalche.me/e/20/dbapi)", 'locations': [{'line': 3, 'column': 5}], 'path': ['span']}]: response=<Response [200]>

@stdweird
Copy link
Author
stdweird commented Jul 5, 2024

so the int value of the spanid i use is way too big.
this is the spanid i get from eg a ReadableSpan context.span_id (if i convert this to a hex string and use it to create an evaluation, this works)

@axiomofjoy
Copy link
Contributor

@stdweird I may have lead you astray, it looks like the span ID you need to use this script is not available in the UI.

@axiomofjoy
Copy link
Contributor
axiomofjoy commented Jul 8, 2024

We'll likely need to build out a REST route to satisfy this need.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c/evals c/traces enhancement New feature or request triage issues that need triage
Projects
Status: 📘 Todo
Development

No branches or pull requests

2 participants