[AE-1003] Add new ad-provider-request-stats ping to observe server-side fill rate#7
Conversation
mashalifshin
changed the title
telemetry/glean/metrics.yaml
Outdated
| The type of ads requested from the provider. Possible values are from Shepherd's Canonical Mozilla Ad Products: | ||
| Tile, Native, Billboard, Rectangle, Skyscraper, Leaderboard. |
There was a problem hiding this comment.
this stands out since mars has no awareness of shepherd's list of ad products (I didn't even know shepherd had one). if we're going to include this, we should have a clearer idea of how we would have this ad product concept supported for any placement or provider.
with this being a metric tied to requests to partners, i'd kind of expect any attributes we send to be the params we sent to the provider. we have a subset of those pulled out explicitly, should we include them all?
this also makes me wonder if there is a way we could specify the request information in glean without having to disassemble it to individual fields. I wouldn't be surprised if not since that'd probably be a generic or open-ended object that we could put any key into, but then we wouldn't need to model provider requests in glean too.
not related to this pr, Native as an ad product name is confusing since that's more a method of delivery than a product; we could have a native billboard/native rectangle/etc.
There was a problem hiding this comment.
this stands out since mars has no awareness of shepherd's list of ad products (I didn't even know shepherd had one). if we're going to include this, we should have a clearer idea of how we would have this ad product concept supported for any placement or provider.
Hmm good callout... yep you're right this info doesn't surface in mars, but from back in my shepherd days I'm aware that they are trying to standardize how we refer to the products across all ad systems, so seemed like a good starting point.
And over on my branch for this in MARS, I've been wondering how to represent that for now. With AMP it's easy/simple since it only ever serves Tile, but not sure what would be best moving forward. Was thinking of just having a simple list of constants right in the repo for now, since these don't change often. And we could move towards a map or function to derive product from placement names.
Would love your thoughts/ideas on that.
There was a problem hiding this comment.
with this being a metric tied to requests to partners, i'd kind of expect any attributes we send to be the params we sent to the provider. we have a subset of those pulled out explicitly, should we include them all?
this also makes me wonder if there is a way we could specify the request information in glean without having to disassemble it to individual fields. I wouldn't be surprised if not since that'd probably be a generic or open-ended object that we could put any key into, but then we wouldn't need to model provider requests in glean too.
Ahh yeah great discussion point, this is at the heart of some of the data design conversations we've had about this. When Glenda and I started designing this, the goal was to generalize as much as we could, so that every field here would apply across all providers, and would not be specific to partner params.
We were also looking for a way to have something like a jsonb column in the metrics, so we could have a general field like provider_params and it would vary based on the partner.
As far as I could tell as a glean newbie, this isn't possible. The object type can't be an arbitrary object structure that varies, it needs to have specific, pre-defined fields.
So I think in the future as we instrument more partner integrations, we can add amp-params, equativ-params, etc etc, each as their own specific object type with their own specific structure, and meant to be nullable, so only one would be populated for each partner request.
I think that isn't as elegant but it will work. And that's not part of this PR since the focus here is really observing the server-side fill rate for tiles, and we can add specific params later if/when we need that data. Lmk what you think, or if I'm missing something about how glean works with objects...
There was a problem hiding this comment.
not related to this pr, Native as an ad product name is confusing since that's more a method of delivery than a product; we could have a native billboard/native rectangle/etc.
Yep this is tough. I think when Sales and Ad Ops say "Native", they are using a standard industry term to mean what we call "Spoc". (and I think they are really keen to move away from calling them Spocs...).
And you're saying "Native" being a method of delivery is in the programmatic ads context? As opposed to the "bundle of js/css/html creatives"? Maybe we should call the ad unit type "Native" and also call this "native delivery"? Not sure...
There was a problem hiding this comment.
I missed your replies until yesterday and mulled on this overnight, what do you think about this:
on shepherd's list of ad products -- this is implied by some combination of provider+endpoint or provider+params and isn't actually sent as a parameter in the request, so maybe it's not worth trying to include in glean here. mars also has a proto enum of the ad formats https://github.com/mozilla-services/mars/blob/723cbec9ca8962144fdbd860ba47e48c76277d79/internal/ad/enum/format.proto#L7-L15 which is what populates the format field so that callers know what type of ad was returned.
on "native" -- I agree that it's a standard industry term (for an ad that is presented to look similar to the content) in addition to a method in programmatic delivery (receive fields individually to enable presenting the ad like content). neither of those have any meaning with what that ad layout is. "spocs" is also kind of meaningless (I think it is a portmanteau of "sponsored contents"?) but is the historic norm we have. in some conversations I refer to them as "sponsored stories" but there isn't clear acceptance or agreement on their name so we'd probably be creating n+1 names for them if we tried defining a new one
on glean fields -- My question was motivated in thinking this effort was for AMP specifically, and if we wanted that data for discussion with them on success/fill rate then they would probably ask what set of parameters had low success/fill rate. In that conversation if we only had a subset of the parameters in our calls to them that seems like it wouldn't be great for the discussion.
I came to the same conclusion on glean types that none of them would allow for open ended field names. I don't know if I like these ideas, but...
- what if we went with a separate ping for each provider we want stats on. That would trade off between more ping definitions, but might be easier to reason around since there wouldn't be columns that only existed for one provider and the field names in the ping could match the request parameter names more directly.
- what if we used the text type and had two fields like queryString or postBody and put the whole object in there? this would make the table generic but would also make it much harder to query since all consumers would need to parse those first. I think there's also extra scrutiny on the text type in glean so it may not be keen on getting past data review.
I haven't been in the modeling discussions on this at all and wouldn't be surprised if you progressed across those options already too.
There was a problem hiding this comment.
All righty. Thank you for this and for all the in-person discussions earlier today.
Re. ad products, I ripped that field out for now to avoid dealing with these questions, since for the first instrumentation of AMP the Tile will be implied. I'll ask about this in the ads data and eng channel re. what, if anything, they might like to see in BQ for this.
And summarizing our discussion re. the rest, we are punting on future questions of instrumenting other providers and if we'll create separate pings, or separate fields for params, or continue with this design. All those options will be open later, and for now this captures all the data we need for server-side fill rate.
|
DATA REVIEW REQUEST
We are adding a new server-side ping to Unified Ads API (MARS), for logging telemetry about our requests to and responses from 3rd party ad providers, such as how many ads were requested/returned, which geos and ad products were requested, and which specific ad ids were returned by the provider. The first goal of this instrumentation will be to determine our Server-Side Fill Rate for Tiles. However we'd like this to be general enough to instrument instrument other ad provider integrations. More details available in the Data Design Doc for this work, as well as the Engineering Design Doc.
For 3rd party contract negotiations, we need to be able to understand whether those parties are meeting their contract obligations. More generally, this ping will allow us to instrument our partner integrations to understand if the partners are working correctly, performant, and according to contract. It will also give us more observability into the life cycle of an ad by capturing the ids returned from the provider, allowing us to trace an ad impression not just from MARS, but from its source with the third party, which will be important for understanding and debugging our ad serving behavior.
I think for consistency with our other metrics, and for Data team to leverage the metrics, continuing to instrument MARS with Glean is the best choice.
Not currently. We instrument the request from the client to MARS, but we currently have no instrumentation and little observability into our third party provider interactions beyond Prometheus metrics.
This collection is Glean so is documented in the Glean Dictionary.
This collection will be collected permanently.
All channels, countries, and locales. No filters. That said, we don't instrument much about the client in this ping, as this is the proxied request from MARS to the ad providers.
These collections are Glean. The opt-out can be found in the product's preferences. That said, this is a server-side ping. It is not initiated by a user or client action, doesn't happen on the user's machine, so no need to provide them opt out for sending data. It's fired when MARS makes a request for Tiles every 10minutes to refresh its cache. So this is backend instrumentation, there isn't a user or client in this part of the flow (though the ads will potentially be served from the cache to a client down the line, that would be a separate request).
The first analysis will focus on server-side fill rate, and if our providers are meeting the contractually required rate of returning to us as many tiles as we request. See the Data Design Doc for more details. Later we can instrument all our providers to make sure they are meeting SLAs and are performant and correct.
Data team will share this data with Business Development to help negotiate contracts with 3rd party ad providers. More generally the MozAds team will use this data for observability into partner ad provider behavior.
No. |
data-review+
Yes, the usual Glean dictionary at https://dictionary.telemetry.mozilla.org.
This is a server-side collection, not tied to any specific client or user, so it is not subject to user-specific opt-out.
Yes, mlifshin@mozilla.com will monitor.
Category 1, technical data.
Default-on.
No.
Yes.
No. |
| lifetime: application | ||
| send_in_pings: | ||
| - interaction | ||
| - provider-request-stats |
There was a problem hiding this comment.
thanks for the discussion in standup helping me to realize this and the line on 271 were for the request params I thought we were omitting
There was a problem hiding this comment.
Thanks so much for confirming I did indeed get 'em all, had a moment of horror that I missed something after all that design doc-ing 🫠
mashalifshin
deleted the
ae-1003-new-ping-for-ad-provider-request-stats
branch
4 participants
Problem Statement:
AE-1003
We'd like to observe the "server side fill rate", to measure how many ads MARS requested from AMP and how often AMP returned that number of tiles, so that we can measure how AMP is meeting its SLAs to us.
Instrumenting the AMP integration is the immediate goal here, but we want this ping to be generally applicable to any ad provider requests, to measure fill rate as well as add more general observability into the MARS <-> provider part of the ad serving chain.
More details in the design doc.
Proposed Solution
This change adds a new ping called
provider-request-stats, to capture the server-side requested and received ad counts and timestamps alongside key provider params like geo, os, form factor.To Verify
Pull down the MARS branch called
ae-1004-instrument-amp-request-with-provider-request-stats-ping, spin up the backend withdocker compose up, and hit the server with a curl for tiles:You can then see the
AdProviderRequestStatsping go out in the MARS logs.