How to handle LLM rate limits in the pipeline?", Body "As we add more agents, rate limits could be an issue. Suggestions?

[sorcerer2073]

Scaling issues?

[kunwar-vikrant]

we can implement a caching layer in llm_providers.py using a lightweight database like SQLite or a key-value store like Redis for frequently repeated prompts. Here’s a claude generated SQLite-based approach, but i am leaning more towards using postGres:
pythonimport sqlite3
import hashlib
import os
import json

def get_cached_response(prompt, provider):
# Generate a unique key for the prompt
cache_key = hashlib.md5(f'{provider}:{prompt}'.encode()).hexdigest()
conn = sqlite3.connect('cache.db')
cursor = conn.cursor()
cursor.execute('CREATE TABLE IF NOT EXISTS cache (key TEXT PRIMARY KEY, response TEXT)')
cursor.execute('SELECT response FROM cache WHERE key = ?', (cache_key,))
result = cursor.fetchone()
if result:
conn.close()
return json.loads(result[0])
return None

def cache_response(prompt, provider, response):
cache_key = hashlib.md5(f'{provider}:{prompt}'.encode()).hexdigest()
conn = sqlite3.connect('cache.db')
cursor = conn.cursor()
cursor.execute('INSERT OR REPLACE INTO cache (key, response) VALUES (?, ?)',
(cache_key, json.dumps(response)))
conn.commit()
conn.close()

def get_llm_response(provider, prompt):
# Check cache first
cached = get_cached_response(prompt, provider)
if cached:
return cached
# Existing API call logic
if provider == 'openai':
api_key = os.getenv('OPENAI_API_KEY')
# ... (rest of the API call code)
response = requests.post(url, headers=headers, json=data).json()['choices'][0]['message']['content']
cache_response(prompt, provider, response)
return response
# ... (other providers)
This caches responses based on a hash of the prompt and provider, reducing API calls for repeated requests (e.g., similar use cases). We should also add a cache expiration mechanism (e.g., 24 hours) to keep data fresh.”