the count_token feature in python gives different value than what the gemini input tokens

[anshulkr04]

Description of the bug:

When I upload a PDF file (~33k tokens per count_tokens()), generate_content() fails with an INVALID_ARGUMENT error claiming the input token count exceeds the model limit (~1.2M tokens).
It seems the SDK is serializing or expanding the uploaded file differently between count_tokens() and generate_content().

2025-10-25 00:09:21,699 - httpx - INFO - HTTP Request: POST https://generativelanguage.googleapis.com/upload/v1beta/files "HTTP/1.1 200 OK"
2025-10-25 00:09:24,932 - httpx - INFO - HTTP Request: POST https://generativelanguage.googleapis.com/upload/v1beta/files?upload_id=AAwnv3LE0Tqc2iJbl8YuN8TXK6SUYhYJY4dMaVrLWd_WkLcDpvohyxjHCvtA1pHyN-vf9lLgVuoGtkjZ4VgZjPMxFWk_JSLo0GudQidxicPQ2y0&upload_protocol=resumable "HTTP/1.1 200 OK"
2025-10-25 00:09:27,655 - httpx - INFO - HTTP Request: POST https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-lite:countTokens "HTTP/1.1 200 OK"
Uploaded file tokens: total_tokens=33541 cached_content_token_count=None
2025-10-25 00:09:27,658 - google_genai.models - INFO - AFC is enabled with max remote calls: 10.
2025-10-25 00:10:03,628 - httpx - INFO - HTTP Request: POST https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-lite:generateContent "HTTP/1.1 400 Bad Request"
Traceback (most recent call last):
File "/Users/anshulkumar/backfin/tet.py", line 22, in
response = client.models.generate_content(model="gemini-2.5-flash-lite",
contents=[uploaded_files,prompt])
File "/Users/anshulkumar/backfin/.venv/lib/python3.13/site-packages/google/genai/models.py", line 5202, in generate_content
response = self._generate_content(
model=model, contents=contents, config=config
)
File "/Users/anshulkumar/backfin/.venv/lib/python3.13/site-packages/google/genai/models.py", line 4178, in _generate_content
response_dict = self._api_client.request(
'post', path, request_dict, http_options
)
File "/Users/anshulkumar/backfin/.venv/lib/python3.13/site-packages/google/genai/_api_client.py", line 755, in request
response = self._request(http_request, stream=False)
File "/Users/anshulkumar/backfin/.venv/lib/python3.13/site-packages/google/genai/_api_client.py", line 684, in _request
errors.APIError.raise_for_response(response)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/Users/anshulkumar/backfin/.venv/lib/python3.13/site-packages/google/genai/errors.py", line 101, in raise_for_response
raise ClientError(status_code, response_json, response)
google.genai.errors.ClientError: 400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'The input token count exceeds the maximum number of tokens allowed 1237083.', 'status': 'INVALID_ARGUMENT'}}

Actual vs expected behavior:

No response

Any other information you'd like to share?

No response