-
Notifications
You must be signed in to change notification settings - Fork 2.8k
feat(slack-bot): bill slack-initiated posthog code tasks under AI credits #59040
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
0cb7501
feat(llm-gateway): register slack_app product as billable
VojtechBartos b685a8f
feat(slack-bot): gate task creation and follow-ups on AI credits quota
VojtechBartos c008271
feat(llm-gateway): rename slack-posthog-code product to slack_app_rou…
VojtechBartos 87c5c91
feat(llm-gateway): gate billable products on AI credits quota at the …
VojtechBartos 1b686e6
chore(slackbot): llm gtw using quata limits django endpoint for usage…
VojtechBartos fc31b55
feat(slack-bot): gate AI credits at webhook and workflow entry
VojtechBartos 5fafefb
chore(ee): drop trailing newline in test_quota_limits.py
VojtechBartos 8b9eb78
chore: update OpenAPI generated types
tests-posthog[bot] 9a125bb
feat(llm-gateway): report quota state for every resource
VojtechBartos c445434
refactor(llm-gateway): extract resolve_plan_and_quota helper
VojtechBartos 9159625
feat(llm-gateway): bump quota cache TTL to 5 minutes
VojtechBartos 0baee26
refactor(slack-bot): keep AI credits gate inside the workflow only
VojtechBartos 6cb735f
feat(llm-gateway): retry transient quota fetches with exponential bac…
VojtechBartos 9d08641
fix(slack-bot): move quota check to temporal layer to satisfy tach
VojtechBartos File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| """Expose a team's quota-limit state. | ||
|
|
||
| Backs the LLM gateway's `QuotaResolver`, which forwards the caller's auth | ||
| header here to learn whether a given team is currently over its AI credits | ||
| quota. Project-nested so org membership and token `scoped_teams`/ | ||
| `scoped_organizations` enforcement come from the standard | ||
| `TeamAndOrgViewSetMixin` permission chain — see | ||
| `posthog.permissions.APIScopePermission.check_team_and_org_permissions`. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| from typing import Any | ||
|
|
||
| from drf_spectacular.utils import extend_schema | ||
| from rest_framework import serializers, viewsets | ||
| from rest_framework.request import Request | ||
| from rest_framework.response import Response | ||
|
|
||
| from posthog.api.routing import TeamAndOrgViewSetMixin | ||
|
|
||
| from ee.billing.quota_limiting import QuotaLimitingCaches, QuotaResource, is_team_limited | ||
|
|
||
|
|
||
| class QuotaResourceLimitSerializer(serializers.Serializer): | ||
| limited = serializers.BooleanField( | ||
| help_text="True when the team is currently over its quota for this resource and limits are in effect.", | ||
| ) | ||
|
|
||
|
|
||
| class QuotaLimitsResponseSerializer(serializers.Serializer): | ||
| limited = serializers.DictField( | ||
| child=QuotaResourceLimitSerializer(), | ||
| help_text=( | ||
| "Per-resource limit state keyed by `QuotaResource` value. " | ||
| "Currently only `ai_credits` is reported; additional resources may be added." | ||
| ), | ||
| ) | ||
|
|
||
|
|
||
| @extend_schema(tags=["quota_limits"]) | ||
| class QuotaLimitsViewSet(TeamAndOrgViewSetMixin, viewsets.ViewSet): | ||
| """Read-only view of a team's quota-limit state.""" | ||
|
|
||
| scope_object = "project" | ||
| required_scopes = ["project:read"] | ||
| http_method_names = ["get", "head", "options"] | ||
|
|
||
| @extend_schema( | ||
| summary="Get a team's quota-limit state", | ||
| description=( | ||
| "Return the current quota-limit state for the team identified in the URL, " | ||
| "keyed by `QuotaResource` value. Used by the LLM gateway to gate billable " | ||
| "products on AI credits exhaustion." | ||
| ), | ||
| responses={200: QuotaLimitsResponseSerializer}, | ||
| ) | ||
| def list(self, request: Request, *args: Any, **kwargs: Any) -> Response: | ||
| limited = { | ||
| resource.value: { | ||
| "limited": is_team_limited( | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what does this function call? is it a clickhouse query, or hitting redis? (I think redis in which case great)
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yup, redis 🚀 |
||
| self.team.api_token, | ||
| resource, | ||
| QuotaLimitingCaches.QUOTA_LIMITER_CACHE_KEY, | ||
| ), | ||
| } | ||
| for resource in QuotaResource | ||
| } | ||
| return Response(QuotaLimitsResponseSerializer({"limited": limited}).data) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,164 @@ | ||
| from __future__ import annotations | ||
|
|
||
| from posthog.test.base import APIBaseTest | ||
|
|
||
| from rest_framework import status | ||
|
|
||
| from posthog.models.organization import Organization | ||
| from posthog.models.personal_api_key import PersonalAPIKey | ||
| from posthog.models.team import Team | ||
| from posthog.models.utils import generate_random_token_personal, hash_key_value | ||
|
|
||
| from ee.billing.quota_limiting import ( | ||
| QuotaLimitingCaches, | ||
| QuotaResource, | ||
| add_limited_team_tokens, | ||
| replace_limited_team_tokens, | ||
| ) | ||
|
|
||
|
|
||
| def _clear_ai_credits_limits() -> None: | ||
| replace_limited_team_tokens(QuotaResource.AI_CREDITS, {}, QuotaLimitingCaches.QUOTA_LIMITER_CACHE_KEY) | ||
|
|
||
|
|
||
| class TestQuotaLimitsAPI(APIBaseTest): | ||
| def setUp(self) -> None: | ||
| super().setUp() | ||
| _clear_ai_credits_limits() | ||
|
|
||
| def tearDown(self) -> None: | ||
| _clear_ai_credits_limits() | ||
| super().tearDown() | ||
|
|
||
| def _url(self, team_id: int | None = None) -> str: | ||
| return f"/api/projects/{team_id if team_id is not None else self.team.pk}/quota_limits/" | ||
|
|
||
| def _set_ai_credits_limit(self, team_api_token: str, expires_at: int) -> None: | ||
| add_limited_team_tokens( | ||
| QuotaResource.AI_CREDITS, | ||
| {team_api_token: expires_at}, | ||
| QuotaLimitingCaches.QUOTA_LIMITER_CACHE_KEY, | ||
| ) | ||
|
|
||
| def test_unauthenticated_returns_401_or_403(self) -> None: | ||
| self.client.logout() | ||
| response = self.client.get(self._url()) | ||
| # DRF returns 401 when no creds are presented and an authenticator that supports | ||
| # a WWW-Authenticate challenge is configured; otherwise it returns 403. Either is | ||
| # an auth failure — we only care that the endpoint refuses unauthenticated reads. | ||
| self.assertIn(response.status_code, (status.HTTP_401_UNAUTHORIZED, status.HTTP_403_FORBIDDEN)) | ||
|
|
||
| def test_session_auth_returns_under_quota_when_team_not_limited(self) -> None: | ||
| response = self.client.get(self._url()) | ||
| self.assertEqual(response.status_code, status.HTTP_200_OK) | ||
| data = response.json() | ||
| self.assertEqual(data["limited"]["ai_credits"], {"limited": False}) | ||
|
|
||
| def test_returns_limited_when_team_is_over_quota(self) -> None: | ||
| self._set_ai_credits_limit(self.team.api_token, 9_999_999_999) | ||
|
|
||
| response = self.client.get(self._url()) | ||
| self.assertEqual(response.status_code, status.HTTP_200_OK) | ||
| self.assertEqual(response.json()["limited"]["ai_credits"], {"limited": True}) | ||
|
|
||
| def test_returns_unlimited_when_limit_has_already_expired(self) -> None: | ||
| self._set_ai_credits_limit(self.team.api_token, 1) # epoch 1970 | ||
|
|
||
| response = self.client.get(self._url()) | ||
| self.assertEqual(response.status_code, status.HTTP_200_OK) | ||
| self.assertEqual(response.json()["limited"]["ai_credits"], {"limited": False}) | ||
|
|
||
| def test_personal_api_key_auth_works(self) -> None: | ||
| self.client.logout() | ||
| raw_key = generate_random_token_personal() | ||
| PersonalAPIKey.objects.create( | ||
| label="quota_limits-test", | ||
| user=self.user, | ||
| secure_value=hash_key_value(raw_key), | ||
| scopes=["project:read"], | ||
| ) | ||
|
|
||
| self._set_ai_credits_limit(self.team.api_token, 9_999_999_999) | ||
|
|
||
| response = self.client.get( | ||
| self._url(), | ||
| headers={"authorization": f"Bearer {raw_key}"}, | ||
| ) | ||
| self.assertEqual(response.status_code, status.HTTP_200_OK) | ||
| self.assertEqual(response.json()["limited"]["ai_credits"], {"limited": True}) | ||
|
|
||
| def test_user_not_in_teams_org_is_forbidden(self) -> None: | ||
| other_org = Organization.objects.create(name="other-org") | ||
| other_team = Team.objects.create(organization=other_org, name="other-team") | ||
|
|
||
| response = self.client.get(self._url(other_team.pk)) | ||
| # The caller is logged in to a team in a different org — TeamMemberAccessPermission | ||
| # rejects with 403 (or 404 if the queryset can't see the team at all). | ||
| self.assertIn(response.status_code, (status.HTTP_403_FORBIDDEN, status.HTTP_404_NOT_FOUND)) | ||
|
|
||
| def test_personal_api_key_scoped_to_a_different_team_is_forbidden(self) -> None: | ||
| # Caller has access to both teams via membership, but the token is scoped to | ||
| # `other_team` only — the standalone-endpoint design would have missed this and | ||
| # leaked the other team's state. | ||
| other_team = Team.objects.create(organization=self.organization, name="other-team") | ||
| self.client.logout() | ||
| raw_key = generate_random_token_personal() | ||
| PersonalAPIKey.objects.create( | ||
| label="quota_limits-test", | ||
| user=self.user, | ||
| secure_value=hash_key_value(raw_key), | ||
| scopes=["project:read"], | ||
| scoped_teams=[other_team.pk], | ||
| ) | ||
|
|
||
| response = self.client.get( | ||
| self._url(), | ||
| headers={"authorization": f"Bearer {raw_key}"}, | ||
| ) | ||
| self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN) | ||
|
|
||
| def test_personal_api_key_missing_required_scope_is_forbidden(self) -> None: | ||
| # A token with only `feature_flag:read` shouldn't be able to read quota state. | ||
| self.client.logout() | ||
| raw_key = generate_random_token_personal() | ||
| PersonalAPIKey.objects.create( | ||
| label="quota_limits-test", | ||
| user=self.user, | ||
| secure_value=hash_key_value(raw_key), | ||
| scopes=["feature_flag:read"], | ||
| ) | ||
|
|
||
| response = self.client.get( | ||
| self._url(), | ||
| headers={"authorization": f"Bearer {raw_key}"}, | ||
| ) | ||
| self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN) | ||
|
|
||
| def test_response_includes_every_quota_resource(self) -> None: | ||
| # Limiting one resource must not hide the unlimited state of the rest. | ||
| self._set_ai_credits_limit(self.team.api_token, 9_999_999_999) | ||
|
|
||
| response = self.client.get(self._url()) | ||
| self.assertEqual(response.status_code, status.HTTP_200_OK) | ||
| limited = response.json()["limited"] | ||
| expected_keys = {resource.value for resource in QuotaResource} | ||
| self.assertEqual(set(limited.keys()), expected_keys) | ||
| self.assertTrue(limited["ai_credits"]["limited"]) | ||
| for resource in QuotaResource: | ||
| if resource is QuotaResource.AI_CREDITS: | ||
| continue | ||
| self.assertFalse(limited[resource.value]["limited"], resource.value) | ||
|
|
||
| def test_multi_team_user_gets_per_team_answers(self) -> None: | ||
| # Same user belongs to two teams in their org; each team's quota is independent. | ||
| # This is the regression that "me" couldn't model — `user.team` (current team) | ||
| # picked one arbitrary answer for users in multiple teams. | ||
| other_team = Team.objects.create(organization=self.organization, name="other-team") | ||
| self._set_ai_credits_limit(self.team.api_token, 9_999_999_999) | ||
| # other_team's token deliberately not limited | ||
|
|
||
| resp_self = self.client.get(self._url()) | ||
| resp_other = self.client.get(self._url(other_team.pk)) | ||
|
|
||
| self.assertEqual(resp_self.json()["limited"]["ai_credits"], {"limited": True}) | ||
| self.assertEqual(resp_other.json()["limited"]["ai_credits"], {"limited": False}) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should be general given the endpoints registration, lets show them quota limiting status for all products?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is technically only data that should be available to admins since it's directed off of billing data, but given we're just exposing is limited or not this seems fine to be available to anyone with project:read imo
cc @PostHog/team-billing in case one of them disagrees, but you can assume not
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated