Implement AI image generation, management, and settings enhancements#19
Open
wernerbihl wants to merge 10 commits into
Open
Implement AI image generation, management, and settings enhancements#19wernerbihl wants to merge 10 commits into
wernerbihl wants to merge 10 commits into
Conversation
- Added persistence functions for AI-generated images, including saving, retrieving, and listing records. - Introduced new types for AI-generated images and their context in the application. - Implemented provider utilities for handling image data and fetching images from various AI providers. - Enhanced the quota management system to track API usage limits. - Created tests for persistence, provider utilities, and quota handling to ensure reliability. - Updated the image editor to support AI-generated images and integrate with existing tileset functionality.
…onality - Implement tests for OpenAI image generation with JSON and multipart requests. - Add tests for Gemini image generation and error handling. - Introduce tests for Together and xAI image generation. - Enhance Hugging Face provider route tests for error payloads. - Create tests for Unity map import handling manifest-only bundles and missing resources. - Validate input and unsupported tile metadata in Unity map import tests. - Add GameMaker import helper tests for resource reading and JSON entry parsing. - Introduce Godot import helper tests for document parsing and resource resolution. - Create Tiled Lua format tests for document normalization and JSON entry creation. - Ensure coverage for edge cases and error handling in all new tests.
…port - Add functions to save and load Lospec palette sync state and enabled status in local storage. - Create a sync controller for managing background synchronization of Lospec palettes. - Introduce hooks for using the Lospec palette sync state in React components. - Normalize and decode HTML entities in palette descriptions. - Enhance sync functionality to handle rate limiting and resume from checkpoints. - Update tests to cover new sync features and ensure proper functionality.
…add settings section types
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
This PR adds end-to-end AI image generation + persistence (multi-provider), enhances settings navigation/layout, and expands import/export + Lospec palette sync capabilities while tightening the app’s CSP to explicitly allow AI provider network calls.
Changes:
- Add AI image generation provider layer (OpenAI/Gemini/Together/xAI/Hugging Face), quota parsing, persistence to IndexedDB, and actions to download/save/open/edit/add-to-tileset.
- Implement Lospec palette background sync with persisted checkpoints, richer Lospec normalization, and an updated Lospec import dialog (pagination, tag search, copyable swatches).
- Update CSP/connect-src configuration (Vite dev headers + Cloudflare
_headers) and add broad test coverage for the new capabilities.
Reviewed changes
Copilot reviewed 46 out of 46 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| vite.config.ts | Uses shared CSP connect-src sources; updates Vitest coverage exclusions/thresholds. |
| TODO.txt | Updates internal TODO list. |
| tests/services/db.test.ts | Adds localStorage helper coverage for Lospec sync prefs/checkpoint. |
| tests/features/import-export/lib/unity-map-import.test.ts | Adds Unity manifest-only and validation scenarios. |
| tests/features/import-export/lib/tiled-lua-format.test.ts | New tests for Tiled Lua document formatting/normalization. |
| tests/features/import-export/lib/import-export-godot-complex.test.ts | New complex Godot export/import preservation tests. |
| tests/features/import-export/lib/godot-import-helpers.test.ts | New unit tests for Godot parsing + helper utilities. |
| tests/features/import-export/lib/gamemaker-import-helpers.test.ts | New unit tests for GameMaker import helper utilities. |
| tests/features/image-editor/lib/lospec-sync-controller.test.ts | New tests for Lospec background sync controller + retry behavior. |
| tests/features/image-editor/lib/lospec-palettes.test.ts | Expands Lospec normalization/sync behavior tests (HTML, paging, status codes). |
| tests/features/ai-assets/lib/tileset-actions.test.ts | New tests for adding AI images as tilesets in editor state. |
| tests/features/ai-assets/lib/quota.test.ts | New tests for quota header parsing. |
| tests/features/ai-assets/lib/providers.test.ts | New tests for provider-specific request/response handling. |
| tests/features/ai-assets/lib/provider-utils.test.ts | New tests for image source normalization + scaling. |
| tests/features/ai-assets/lib/persistence.test.ts | New tests for AI image history/gallery persistence in IndexedDB. |
| tests/features/ai-assets/lib/dimensions.test.ts | New tests for structured target-dimension calculation + ratio selection. |
| tests/config/content-security-policy.test.ts | Validates provider origins are covered by CSP definitions + _headers. |
| src/types/integrations/ai-assets.ts | Extends AI integration types (providers, quota, persistence records, etc.). |
| src/services/db.ts | Adds aiImages Dexie table + Lospec sync localStorage helpers. |
| src/features/image-editor/types/lospec.ts | Extends Lospec sync types (progress, status codes, page counts). |
| src/features/image-editor/types/lospec-sync.ts | New persisted background sync checkpoint/snapshot types. |
| src/features/image-editor/types/index.ts | Re-exports lospec-sync types. |
| src/features/image-editor/lib/lospec-sync-controller.ts | New background sync controller with persisted checkpoints + retry scheduling. |
| src/features/image-editor/lib/lospec-palettes.ts | Adds richer Lospec normalization + sync progress/status handling. |
| src/features/image-editor/hooks/use-lospec-palette-sync.ts | New hook bridging controller snapshot via useSyncExternalStore. |
| src/features/image-editor/hooks/use-image-editor-request-loader.ts | Adds loading path for standalone AI images into the image editor. |
| src/features/image-editor/components/LospecPaletteDialog.tsx | Refreshes dialog UX (pagination, tag click, color tooltips/copy, background sync status). |
| src/features/app-shell/types/settings-dialog.ts | New settings section model for tabbed navigation. |
| src/features/app-shell/types/index.ts | Re-exports settings dialog types. |
| src/features/ai-assets/types/index.ts | New UI-facing types for AI image grids/actions + editor context. |
| src/features/ai-assets/lib/tileset-actions.ts | Adds utilities to save AI images as assets and append as tilesets. |
| src/features/ai-assets/lib/standalone-editor-context.ts | Adds global pending-context bridge for opening AI images in editor. |
| src/features/ai-assets/lib/quota.ts | Adds quota header parsing utilities + UNKNOWN_QUOTA constant. |
| src/features/ai-assets/lib/providers.ts | Adds multi-provider request logic + image normalization/scaling. |
| src/features/ai-assets/lib/provider-utils.ts | Adds image source decoding + optional scaling to target dimensions. |
| src/features/ai-assets/lib/persistence.ts | Adds AI image record CRUD + history/gallery queries. |
| src/features/ai-assets/lib/dimensions.ts | Adds target-dimension derivation + closest aspect-ratio selection. |
| src/features/ai-assets/lib/constants.ts | Adds Hugging Face model defs + provider labels updates. |
| src/features/ai-assets/components/ImageCell.tsx | Adds per-image action toolbar (download/save/add/edit/delete). |
| src/features/ai-assets/components/Generator.tsx | Refactors generator to use provider layer + persistence + scheduler + tabs. |
| src/config/content-security-policy.ts | Centralizes CSP connect-src sources (app + AI providers). |
| src/config/api-keys.ts | Adds Hugging Face API key provider configuration. |
| src/components/dialogs/SettingsDialog.tsx | Replaces accordion with tabbed/section-based settings UI. |
| src/App.tsx | Auto-starts Lospec background sync when enabled in localStorage. |
| public/_headers | Updates CSP connect-src to include AI providers. |
| AGENTS.md | Updates contributor guidance around running lint/build commands. |
Comment on lines
+41
to
+77
| if (initImageB64 && initImageMime) { | ||
| await Promise.all( | ||
| Array.from({ length: count }, async () => { | ||
| const form = new FormData(); | ||
| const byteStr = atob(initImageB64); | ||
| const bytes = new Uint8Array(byteStr.length); | ||
| for (let index = 0; index < byteStr.length; index += 1) { | ||
| bytes[index] = byteStr.charCodeAt(index); | ||
| } | ||
| form.append( | ||
| "image", | ||
| new Blob([bytes], { type: initImageMime }), | ||
| "reference.png", | ||
| ); | ||
| form.append("prompt", prompt); | ||
| form.append("model", model); | ||
| form.append("n", "1"); | ||
| form.append("size", `${width}x${height}`); | ||
| form.append("response_format", "b64_json"); | ||
| const response = await fetch("https://api.openai.com/v1/images/edits", { | ||
| method: "POST", | ||
| headers: { Authorization: `Bearer ${apiKey}` }, | ||
| body: form, | ||
| }); | ||
| if (!response.ok) { | ||
| await readJsonError(response, `OpenAI error ${response.status}`); | ||
| } | ||
| const data = (await response.json()) as { | ||
| data: { b64_json: string }[]; | ||
| }; | ||
| dataUrls.push( | ||
| ...data.data.map( | ||
| (image) => `data:image/png;base64,${image.b64_json}`, | ||
| ), | ||
| ); | ||
| }), | ||
| ); |
Comment on lines
+311
to
+319
| quota = parseQuotaHeaders(response.headers); | ||
| if (!response.ok) { | ||
| const error = await response.json().catch(async () => ({ | ||
| message: await response.text().catch(() => ""), | ||
| })); | ||
| throw new Error( | ||
| getApiErrorMessage(`Hugging Face error ${response.status}`, error), | ||
| ); | ||
| } |
Comment on lines
+165
to
+278
| const start = async () => { | ||
| if (disposed) { | ||
| return; | ||
| } | ||
|
|
||
| await ensureInitialized(); | ||
|
|
||
| if (disposed || runPromise || snapshot.status === "complete") { | ||
| return; | ||
| } | ||
|
|
||
| if (snapshot.retryAtMs !== null && snapshot.retryAtMs > dependencies.now()) { | ||
| scheduleRetry(snapshot.retryAtMs); | ||
| return; | ||
| } | ||
|
|
||
| clearRetry(); | ||
|
|
||
| const runStartPage = snapshot.nextPage; | ||
| const baseFetchedPageCount = snapshot.fetchedPageCount; | ||
| const baseAddedCount = snapshot.addedCount; | ||
|
|
||
| commit({ | ||
| ...snapshot, | ||
| status: "syncing", | ||
| retryAtMs: null, | ||
| errorStatus: undefined, | ||
| errorMessage: undefined, | ||
| updatedAt: dependencies.now(), | ||
| }); | ||
|
|
||
| runPromise = (async () => { | ||
| const result = await dependencies.syncCatalog({ | ||
| startPage: runStartPage, | ||
| stopAtKnownPalette: false, | ||
| onProgress: (progress) => { | ||
| if (disposed) { | ||
| return; | ||
| } | ||
|
|
||
| commit({ | ||
| ...snapshot, | ||
| palettes: progress.palettes, | ||
| status: "syncing", | ||
| nextPage: progress.page === null ? runStartPage : progress.page + 1, | ||
| retryAtMs: null, | ||
| fetchedPageCount: baseFetchedPageCount + progress.fetchedPageCount, | ||
| addedCount: baseAddedCount + progress.addedCount, | ||
| updatedAt: dependencies.now(), | ||
| errorStatus: undefined, | ||
| errorMessage: undefined, | ||
| }); | ||
| }, | ||
| }); | ||
|
|
||
| if (disposed) { | ||
| return; | ||
| } | ||
|
|
||
| const nextFetchedPageCount = | ||
| baseFetchedPageCount + result.fetchedPageCount; | ||
| const nextAddedCount = baseAddedCount + result.addedCount; | ||
|
|
||
| if (result.status === "synced") { | ||
| commit({ | ||
| ...snapshot, | ||
| palettes: result.palettes, | ||
| status: result.reachedEnd ? "complete" : "idle", | ||
| nextPage: runStartPage + result.fetchedPageCount, | ||
| retryAtMs: null, | ||
| fetchedPageCount: nextFetchedPageCount, | ||
| addedCount: nextAddedCount, | ||
| updatedAt: dependencies.now(), | ||
| errorStatus: undefined, | ||
| errorMessage: undefined, | ||
| }); | ||
| return; | ||
| } | ||
|
|
||
| if (result.status === "cache-only" && result.errorStatus === 429) { | ||
| const retryAtMs = dependencies.now() + LOSPEC_RATE_LIMIT_RETRY_MS; | ||
| commit({ | ||
| ...snapshot, | ||
| palettes: result.palettes, | ||
| status: "rate-limited", | ||
| nextPage: result.retryPage ?? runStartPage, | ||
| retryAtMs, | ||
| fetchedPageCount: nextFetchedPageCount, | ||
| addedCount: nextAddedCount, | ||
| updatedAt: dependencies.now(), | ||
| errorStatus: result.errorStatus, | ||
| errorMessage: result.errorMessage, | ||
| }); | ||
| scheduleRetry(retryAtMs); | ||
| return; | ||
| } | ||
|
|
||
| commit({ | ||
| ...snapshot, | ||
| palettes: result.palettes, | ||
| status: "error", | ||
| retryAtMs: null, | ||
| fetchedPageCount: nextFetchedPageCount, | ||
| addedCount: nextAddedCount, | ||
| updatedAt: dependencies.now(), | ||
| errorStatus: result.errorStatus, | ||
| errorMessage: result.errorMessage, | ||
| }); | ||
| })().finally(() => { | ||
| runPromise = null; | ||
| }); | ||
|
|
||
| await runPromise; | ||
| }; |
Comment on lines
+194
to
+254
| const loadPendingStandaloneAiImageRequest = useCallback(async () => { | ||
| const pendingRequest = getPendingStandaloneAiImageEditorRequest(); | ||
| if (!pendingRequest) { | ||
| return; | ||
| } | ||
|
|
||
| const { requestId, context: ctx } = pendingRequest; | ||
| const runId = ++loadRunIdRef.current; | ||
| const isCurrentRun = () => | ||
| isMountedRef.current && loadRunIdRef.current === runId; | ||
|
|
||
| const objectUrl = URL.createObjectURL( | ||
| new Blob([ctx.data], { type: ctx.mimeType }), | ||
| ); | ||
|
|
||
| try { | ||
| const image = new Image(); | ||
| image.src = objectUrl; | ||
| if (typeof image.decode === "function") { | ||
| await image.decode(); | ||
| } else { | ||
| await new Promise<void>((resolve, reject) => { | ||
| image.onload = () => resolve(); | ||
| image.onerror = () => reject(new Error("Failed to decode image.")); | ||
| }); | ||
| } | ||
| if (!isCurrentRun()) return; | ||
|
|
||
| const width = ctx.width || image.naturalWidth || image.width; | ||
| const height = ctx.height || image.naturalHeight || image.height; | ||
| const canvas = document.createElement("canvas"); | ||
| canvas.width = width; | ||
| canvas.height = height; | ||
| const context = canvas.getContext("2d"); | ||
| if (!context || !isCurrentRun()) return; | ||
|
|
||
| context.imageSmoothingEnabled = false; | ||
| context.drawImage(image, 0, 0, width, height); | ||
| const imageData = context.getImageData(0, 0, width, height); | ||
| if (!isCurrentRun()) return; | ||
|
|
||
| const savedPalettes = projectId ? loadPaletteLibrary(projectId) : null; | ||
| editor.initProject(width, height, savedPalettes ?? undefined); | ||
| if (!isCurrentRun()) return; | ||
|
|
||
| if (isImageEditorStoreReady()) { | ||
| const state = getImageEditorStore().getState(); | ||
| if (state.frames.length > 0) { | ||
| editor.setFrameData(state.frames[0].id, imageData); | ||
| editor.markSavePoint(); | ||
| } | ||
| } | ||
|
|
||
| clearStandaloneAiImageEditorContext(requestId); | ||
| onRequestLoaded(); | ||
| setActiveTileCtx(null); | ||
| setActiveImageLayerCtx(null); | ||
| } finally { | ||
| URL.revokeObjectURL(objectUrl); | ||
| } | ||
| }, [editor, onRequestLoaded, projectId]); |
| - Do not inline typescript types and interfaces - move them to types folder | ||
| - When working on a file, and it is more than 1000 lines of code then refactor it into smaller, easier to test and maintain files. There are no exceptions to this. Always do this even if it is outside the scope of what was asked | ||
| - Always run `bun run lint` and `bun run build` after making code changes to verify that the build succeeds and there are no linting issues | ||
| - Always run `bun run lint` after making code changes to verify that the build succeeds and there are no linting issues |
| > | ||
| <span className="flex min-w-0 flex-col items-start"> | ||
| <span>{section.label}</span> | ||
| <span className="text-[10px] leading-relaxed text-muted-foreground whitespace-normal wrap-break-word"> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.