Skip to content

Leaderboard migration #1907

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
SamXop123 wants to merge 5 commits into
base: main
Choose a base branch
from
+33 −427
Open

Leaderboard migration #1907

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1e88777
Update docusaurus.config.ts
SamXop123 Jun 17, 2026
dc2b118
Update statsProvider.tsx
SamXop123 Jun 17, 2026
bff6d8f
Update githubService.ts
SamXop123 Jun 17, 2026
a2f5b2d
Potential fix for pull request finding
sanjay-kv Jun 18, 2026
2bac08e
removed dead code
SamXop123 Jun 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docusaurus.config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -281,8 +281,9 @@ const config: Config = {
],
],

// ✅ Add this customFields object to expose the token to the client-side
// Custom fields exposed to client-side code (browser-visible at build time)
customFields: {
backendApiUrl: process.env.BACKEND_API_URL || "http://localhost:5000",
gitToken: process.env.DOCUSAURUS_GIT_TOKEN,
clerkPublishableKey: process.env.VITE_CLERK_PUBLISHABLE_KEY || "",
Comment on lines +286 to 288

@Adez017 Adez017 Jun 18, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think now we don't need to mount the git token , let me know if it is required or we can move forward and remove this one @SamXop123

@SamXop123 SamXop123 Jun 19, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is still there due to the discussions section in the dashboard

image

if we remove it, the discussions page will break.

but if we really want to remove the PAT from the client side.... we need to create a seperate route for this discussions section (like /api/discussions) in the leaderboard backend itself, but i guess it would become a whole seperate issue. lmk what you prefer?

@Adez017 Adez017 Jun 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets proceed with this one @SamXop123 , we can create a separate issue later on for that one .

// Shopify credentials for merch store
Expand Down
225 changes: 31 additions & 194 deletions src/lib/statsProvider.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ export function CommunityStatsProvider({
const {
siteConfig: { customFields },
} = useDocusaurusContext();
const token = customFields?.gitToken || "";
const backendApiUrl = (customFields?.backendApiUrl as string) || "http://localhost:5000";

const [loading, setLoading] = useState(false); // Start with false to avoid hourglass
const [error, setError] = useState<string | null>(null);
Expand Down Expand Up @@ -251,164 +251,6 @@ export function CommunityStatsProvider({
setCurrentTimeFilter(filter);
}, []);

const fetchAllOrgRepos = useCallback(
async (headers: Record<string, string>) => {
const repos: any[] = [];
let page = 1;
while (true) {
const resp = await fetch(
`https://api.github.com/orgs/${GITHUB_ORG}/repos?type=public&per_page=100&page=${page}`,
{
headers,
},
);
if (!resp.ok) {
throw new Error(
`Failed to fetch org repos: ${resp.status} ${resp.statusText}`,
);
}
const data = await resp.json();
repos.push(...data);
if (!Array.isArray(data) || data.length < 100) break;
page++;
}
return repos;
},
[],
);

const fetchMergedPRsForRepo = useCallback(
async (repoName: string, headers: Record<string, string>) => {
const mergedPRs: PullRequestItem[] = [];

// First, get the first page to estimate total pages
const firstResp = await fetch(
`https://api.github.com/repos/${GITHUB_ORG}/${repoName}/pulls?state=closed&per_page=100&page=1`,
{ headers },
);

if (!firstResp.ok) {
console.warn(
`Failed to fetch PRs for ${repoName}: ${firstResp.status} ${firstResp.statusText}`,
);
return [];
}

const firstPRs: PullRequestItem[] = await firstResp.json();
if (!Array.isArray(firstPRs) || firstPRs.length === 0) return [];

const firstPageMerged = firstPRs.filter((pr) => Boolean(pr.merged_at));
mergedPRs.push(...firstPageMerged);

// If we got less than 100, that's all there is
if (firstPRs.length < 100) return mergedPRs;

// Create parallel requests for remaining pages
const pagePromises: Promise<PullRequestItem[]>[] = [];
const maxPages = Math.min(MAX_PAGES_PER_REPO, 10);

for (let i = 2; i <= maxPages; i++) {
pagePromises.push(
fetch(
`https://api.github.com/repos/${GITHUB_ORG}/${repoName}/pulls?state=closed&per_page=100&page=${i}`,
{ headers },
)
.then(async (resp) => {
if (!resp.ok) return [];
const prs: PullRequestItem[] = await resp.json();
if (!Array.isArray(prs)) return [];
return prs.filter((pr) => Boolean(pr.merged_at));
})
.catch(() => []),
);
}

// Wait for all pages in parallel
const remainingPages = await Promise.all(pagePromises);
remainingPages.forEach((pagePRs) => {
if (pagePRs.length > 0) mergedPRs.push(...pagePRs);
});

return mergedPRs;
},
[],
);

// Enhanced processing function that stores only valid PRs with points
const processBatch = useCallback(
async (
repos: any[],
headers: Record<string, string>,
): Promise<{
contributorMap: Map<string, FullContributor>;
totalMergedPRs: number;
}> => {
const contributorMap = new Map<string, FullContributor>();
let totalMergedPRs = 0;

// Process repos in batches to control concurrency
for (let i = 0; i < repos.length; i += MAX_CONCURRENT_REQUESTS) {
const batch = repos.slice(i, i + MAX_CONCURRENT_REQUESTS);

const promises = batch.map(async (repo) => {
if (repo.archived) return { mergedPRs: [], repoName: repo.name };

try {
const mergedPRs = await fetchMergedPRsForRepo(repo.name, headers);
return { mergedPRs, repoName: repo.name };
} catch (error) {
console.warn(`Skipping repo ${repo.name} due to error:`, error);
return { mergedPRs: [], repoName: repo.name };
}
});

// Wait for current batch to complete
const results = await Promise.all(promises);

// Process results from this batch
results.forEach(({ mergedPRs, repoName }) => {
mergedPRs.forEach((pr) => {
// Calculate points for this PR based on labels
const prPoints = calculatePointsForPR(pr.labels);

// ONLY store PRs that have points (i.e., have "recode" label and a level label)
if (prPoints > 0) {
totalMergedPRs++;

const username = pr.user.login;
if (!contributorMap.has(username)) {
contributorMap.set(username, {
username,
avatar: pr.user.avatar_url,
profile: pr.user.html_url,
points: 0, // Will be calculated later based on filter
prs: 0, // Will be calculated later based on filter
allPRDetails: [], // Store only valid PRs here
});
}
const contributor = contributorMap.get(username)!;

// Add detailed PR information only if it has all required fields
if (pr.title && pr.html_url && pr.merged_at && pr.number) {
contributor.allPRDetails.push({
title: pr.title,
url: pr.html_url,
mergedAt: pr.merged_at,
repoName,
number: pr.number,
points: prPoints,
});
}
}
});
});
}

return { contributorMap, totalMergedPRs };
},
[fetchMergedPRsForRepo],
);

const fetchAllStats = useCallback(
async (signal: AbortSignal) => {
// Check cache first and load it immediately without showing loading state
Expand All @@ -433,57 +275,52 @@ export function CommunityStatsProvider({

setError(null);

if (!token) {
setError(
"GitHub token not found. Please set customFields.gitToken in docusaurus.config.js.",
);
setLoading(false);
return;
}

try {
const headers: Record<string, string> = {
Authorization: `token ${token}`,
Accept: "application/vnd.github.v3+json",
};

// Fetch both org stats and repos in parallel
const [orgStats, repos] = await Promise.all([
githubService.fetchOrganizationStats(signal),
fetchAllOrgRepos(headers),
const [leaderboardResp, statsResp] = await Promise.all([
fetch(`${backendApiUrl}/api/leaderboard`, { signal }),
fetch(`${backendApiUrl}/api/stats`, { signal })
]);

// Set org stats immediately
setGithubStarCount(orgStats.totalStars);
setGithubContributorsCount(orgStats.totalContributors);
setGithubForksCount(orgStats.totalForks);
setGithubReposCount(orgStats.publicRepositories);
setGithubDiscussionsCount(orgStats.discussionsCount);
setLastUpdated(new Date(orgStats.lastUpdated));

// Process leaderboard data with concurrent processing
const { contributorMap, totalMergedPRs } = await processBatch(
repos,
headers,
);
if (!leaderboardResp.ok || !statsResp.ok) {
throw new Error("Failed to fetch leaderboard data from backend server");
}

const contributorsArray = Array.from(contributorMap.values());
const leaderboardData = await leaderboardResp.json();
const statsData = await statsResp.json();

// Set org stats immediately
setGithubStarCount(statsData.totalStars);
setGithubContributorsCount(statsData.totalContributors);
setGithubForksCount(statsData.totalForks);
setGithubReposCount(statsData.publicRepositories);
setGithubDiscussionsCount(statsData.discussionsCount);
setLastUpdated(new Date(statsData.lastUpdated));

// Format to FullContributor (which matches contributor mapping)
const contributorsArray: FullContributor[] = (leaderboardData.contributors || []).map((c: any) => ({
username: c.username,
avatar: c.avatar,
profile: c.profile,
points: c.points,
prs: c.prs,
allPRDetails: c.prDetails || [] // Backend stores complete list
}));

setAllContributors(contributorsArray);

// Cache the results (raw data without filtering)
setCache({
data: {
contributors: contributorsArray,
rawStats: { totalPRs: totalMergedPRs },
rawStats: { totalPRs: statsData.totalContributors },
},
Comment on lines 313 to 316

@sanjay-kv sanjay-kv Jun 18, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Adez017 I need your review on merged PR
https://github.com/recodehive/leaderboard/pull/1/changes

This is the backend code.

@SamXop123 SamXop123 Jun 19, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should i fix this thing @Adez017 @sanjay-kv ?

timestamp: now,
});
} catch (err: any) {
if (err.name !== "AbortError") {
console.error("Error fetching GitHub organization stats:", err);
console.error("Error fetching stats from backend:", err);
setError(
err instanceof Error ? err.message : "Failed to fetch GitHub stats",
err instanceof Error ? err.message : "Failed to fetch stats from backend",
);

// Set fallback values on error
Expand All @@ -497,7 +334,7 @@ export function CommunityStatsProvider({
setLoading(false);
}
},
[token, fetchAllOrgRepos, processBatch, cache],
[backendApiUrl, cache],
);

const clearCache = useCallback(() => {
Expand Down
Loading
Loading