Adaptive response sampling for surveys

[annikaschmid]

Feature request

We currently allow %-based rollouts for surveys, as well as the option to set a response limit.

For some users with high volumes, that's not enough. Here is what we've heard:

• Being able to specify a response limit per time period, e.g. "250 responses per month", or "7 responses per day"
• Ideally, PostHog would adapt the rollout of surveys across the time period specified by the user, e.g. when a user wants 250 responses over a one-month period, we make sure they receive approximately 8 responses per day

One proposal how this could be achieved from @neilkakkar: "I think the naive targeting all users and adjusting the rollout % based on how many responses have been received so far would work pretty well - and better than trying to extra optimise it based on several other factors / guestimating cohort sizes who are active.. This is also reasonably straightforward to implement - we have all the pieces in place already."

More details in this Slack thread here: https://posthog.slack.com/archives/C07QD3LT8U9/p1728406374642269

Debug info

No response

[Phanatic]

ok, I spent some time thinking about this today and this is an approach that would work.

Changes to Survey data model

1. We will introduce yet another custom targeting flag : survey_response_rate_throttling_feature_flag to a Survey model which is either at a 100% or 0%.
2. We will introduce a property called survey_response_rate_throttling which is an array of dates and their allowed survey response counts. example : [{'10/29/24': 17, '10/30/24': 34}]

posthog-js changes

1. We need to account for the new feature flag survey_response_rate_throttling_feature_flag here : https://github.com/PostHog/posthog-js/blob/192e6c2af2b07f4605a940050447c6c709020a98/src/posthog-surveys.ts#L203-L205

Distributing survey response limits over a time period

S = <total number of responses accepted> 
D = <duration>
n = <number of days in duration>
u = <per day response count>

If a customer wants to collect 500 responses over 30 days :

S = 500
n = 30 days
u = Math.ceil(500/30)  = 17

We would evenly distribute the number of responses we collect over the time period, giving us 17 responses per day.
We would then turn this into an arithmetic progression where element in the array is the total number of responses we hope to get bythat day.

For example, on day 1 , we hope to get 17 responses, and on day 2, we hope to have a total count of 34 responses to the survey.
This way, we can catch up if survey response rates are down on a specific day.

17,34,51,68,85,102,119,136,153,170,187,204,221,238,255,272,289,306,323,340,357,374,391,408,425,442,459,476,493,510

Yes, the last number is higher than 500, but we have a guard clause in the section below.

Turning off survey response collection if over daily limit

We will run a periodic task, every 30 minutes, which will enable the flag as long as :

1. Total number of survey responses collected today is less than total expected count ( 500 )
2. Total number of survey responses collected today is less than daily expected count ( element in the sequence above )

Once we either of these conditions are false, we will turn off the survey_response_rate_throttling_feature_flag by setting rollout to 0%, which will disable the survey from being shown to the user.

Caveats

1. Surveys are targeted by a global switch, based on total response count. We don't adjust the sampling to account for user behavior or any specific user targeting metric.
2. Customers cannot adjust the rollout percentage to be anything other than monotonically increasing in an arithmetic sequence.
3. Since the task is periodic, we might get more than the expected total count of responses. but not fewer. We can, alternatively, put this code in the ingestion pipeline to run on every survey response, need to investigate that some more.

[neilkakkar]

Nice!

one caveat, related to (1): I'm not sure if toggling between 0 and 100% would work well here. The problem is, this effectively means we're hiding / showing the survey to all users at any time. This doesn't really help with the catchup mechanism because we don't know in advance how many users will respond to the survey. For example, if a customer wants 500 responses, buuut there are only 20 people who will respond when they see the survey (this is unknown to all of us at time of creation, but probable in the real world), then showing the survey to everyone would mean we probably get [17-20] responses on day one, rollout to 0, and on day two get [0-3] responses when we roll out to 100% again, and on day 3 onwards get none, because all the people who would've responded have responded. This problem is much less stark when we're dealing with lots of new incoming users though.

I'd propose instead adjusting the throttling rollout based on the n (=30 days) as well. I.e. everyday by default we increase rollout % by (100/n). This ensures new users start seeing the survey everyday, even if no new users are coming from the client. And now, if expected no. of responses is lower than what it should be today, we increase it further, based on how close we are to the target of 500. If the expected no. of responses is higher, we don't increase the rollout %, and keep it where it is. The advantage is clients don't randomly see and then not see surveys based on other people's behaviours. It's sort-of how I've seen @annikaschmid run the product feedback surveys internally too, and seems to work well.

[Phanatic]

Thanks for the comments @neilkakkar! I like the idea of adjusting the rollout percentage of the feature flag. Given this, does the below section make sense?

Turning off survey response collection if over daily limit

Every day, the rollout percentage of the feature flag is calculated by this forumla

S  = <total number of responses expected>  = 500
T = <total number of responses sent> 
D = <duration> = 10 days
n = <number of days in duration> = 10
u = <per day response count> = 500/10 = 50

if (T <= (d*n) ) {
 rollout_percentage_on_day_d = 100 * n/D
}

example: 
D = 10
n = 1
rollout_percentage_on_day_1 = 100 * 1/10 = 10
rollout_percentage_on_day_2= 100 * 2/10 = 20
and so on.

[marandaneto]

Figure out #27115 and #27311 as well