Feat/applywithllm by kavitha186 · Pull Request #1067 · shepherd-tools/shepherd

kavitha186 · 2026-01-04T02:53:08Z

This pull request introduces comprehensive support for the new applywithllm command in Shepherd, enabling automated code changes using Large Language Models (LLMs) such as OpenAI's GPT and Groq. It adds documentation, usage examples, configuration guidelines, and updates the CLI and package dependencies to support this feature. The changes make it easy for users to leverage LLMs for both single-file edits and multi-repository migrations, with robust error handling and best practices guidance.

Major features and documentation:

Added a detailed quick start guide (APPLYWITHLLM_QUICKSTART.md) covering installation, configuration, usage examples, troubleshooting, and best practices for the applywithllm command.
Introduced a comprehensive reference documentation file (docs/applywithllm.md) describing command modes, environment variables, prompt formats, error handling, implementation details,
Codebase and configuration updates:
Registered the applywithllm command in the CLI by importing it in src/cli.ts.
Added the groq-sdk package as a dependency in package.json to enable Groq LLM integration.

Examples and test data:

Added an example migration configuration in examples/apply-code-with-llm/shepherd.yml to demonstrate enabling and customizing applywithllm.
Included sample LLM and OpenAI API responses (llm_response.json, openai_response.json) for demonstration and testing.

Screen.Recording.2026-01-03.at.9.37.34.PM.mov

Copilot

Pull request overview

This pull request introduces the applywithllm command to Shepherd, enabling automated code changes using Large Language Models (OpenAI GPT and Groq). The feature supports both single-file modifications and multi-repository migrations with LLM-powered code generation.

Key Changes:

New LLM service integration supporting OpenAI and Groq providers
New applywithllm command with single-file and repo-based modes
Configuration schema updates to support LLM settings in migration specs
Comprehensive documentation including quickstart guide and reference docs

Reviewed changes

Copilot reviewed 14 out of 16 changed files in this pull request and generated 28 comments.

Show a summary per file

File	Description
src/services/llm.ts	Core LLM service with OpenAI and Groq provider implementations
src/services/llm.test.ts	Unit tests for LLM service functionality
src/commands/applywithllm.ts	Main command implementation for LLM-based code modifications
src/commands/applywithllm.test.ts	Test suite for applywithllm command
src/cli.ts	CLI registration for the new applywithllm command
src/util/migration-spec.ts	Extended migration spec interface to support LLM configuration
src/migration-context.ts	Added LLM config to migration context interface
package.json	Added groq-sdk dependency
package-lock.json	Dependency lockfile updates for groq-sdk and related packages
docs/applywithllm.md	Comprehensive reference documentation for the command
APPLYWITHLLM_QUICKSTART.md	Quick start guide with examples and best practices
README.md	Updated with applywithllm command overview
examples/apply-code-with-llm/shepherd.yml	Example migration configuration using LLM
openai_response.json	Sample OpenAI API response for testing/demonstration
llm_response.json	Sample LLM response structure for testing/demonstration

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/commands/applywithllm.ts

Copilot · 2026-01-04T03:00:55Z

APPLYWITHLLM_QUICKSTART.md

+
+```bash
+# Export the API key (add to .bashrc or .zshrc for persistence)
+export GROQ_API_KEY="sk-your-openai-api-key-here"


The documentation states to set GROQ_API_KEY but then says "sk-your-openai-api-key-here" in the comment. Groq API keys start with "gsk_" not "sk-". This is misleading and should be corrected to use a proper Groq API key format like "gsk-...".

Suggested change

export GROQ_API_KEY="sk-your-openai-api-key-here"

export GROQ_API_KEY="gsk-your-groq-api-key-here"

Copilot · 2026-01-04T03:00:55Z

APPLYWITHLLM_QUICKSTART.md

+## Troubleshooting
+
+### "GROQ_API_KEY environment variable is not set"
+
+```bash
+# Solution: Export your API key
+export GROQ_API_KEY="sk-your-key"
+```
+
+### "Diff validation failed"
+
+The LLM may have generated an invalid diff. Try:
+
+1. Use `--dry-run` first to see the error
+2. Refine your prompt to be more specific
+3. Use a simpler, more targeted prompt
+4. Check if the prompt format is correct (with `@files`)
+
+### "File not found: src/example.ts"
+
+Ensure:
+
+- File paths are relative to repository root
+- Spell file names correctly
+- Files are actually committed (not untracked)
+
+### "LLM API error"
+


The documentation section describes Groq configuration but references OpenAI API keys and GPT models throughout. This entire section appears to have copy-paste errors where GROQ was substituted for OPENAI without updating the key format ("sk-" prefix is for OpenAI, not Groq which uses "gsk_") or model names (GPT models are OpenAI, not Groq).

@copilot open a new pull request to apply changes based on this feedback

Copilot · 2026-01-04T03:00:55Z

src/services/llm.ts

+    // Save response to file for debugging
+    const responseFile = path.join(process.cwd(), 'openai_response.json');
+    await fs.writeFile(responseFile, JSON.stringify(data, null, 2), 'utf-8');
+    console.log(`OpenAI response saved to ${responseFile}`);


The OpenAIProvider unconditionally writes debug files to the current working directory. This should only happen when DEBUG_LLM is enabled, or should be removed entirely as it can clutter the filesystem and may expose sensitive data in production environments.

Suggested change

// Save response to file for debugging

const responseFile = path.join(process.cwd(), 'openai_response.json');

await fs.writeFile(responseFile, JSON.stringify(data, null, 2), 'utf-8');

console.log(`OpenAI response saved to ${responseFile}`);

// Save response to file for debugging (only when DEBUG_LLM is enabled)

if (process.env.DEBUG_LLM === 'true') {

const responseFile = path.join(process.cwd(), 'openai_response.json');

await fs.writeFile(responseFile, JSON.stringify(data, null, 2), 'utf-8');

console.log(`OpenAI response saved to ${responseFile}`);

}

Copilot · 2026-01-04T03:00:55Z

src/services/llm.ts

+
+    // Handle plain text response - remove markdown code fences if present
+    let modifiedContent = content.trim();
+    // Remove markdown code fences if present


The comment on line 83 is redundant as it duplicates the comment on line 81. This redundancy should be removed for clarity.

Suggested change

// Remove markdown code fences if present

Copilot · 2026-01-04T03:01:01Z

src/services/llm.ts

+
+    // Handle plain text response - remove markdown code fences if present
+    let modifiedContent = content.trim();
+    // Remove markdown code fences if present


The comment on line 205 is redundant as it duplicates the comment on line 203. This redundancy should be removed for clarity.

Suggested change

// Remove markdown code fences if present

Copilot · 2026-01-04T03:01:01Z

src/services/llm.test.ts

+    });
+
+    it('should return provider with provided API key', () => {
+      const provider = getLLMProvider('test-key');


The test calls getLLMProvider with an API key, but the implementation now checks environment variables (OPENAI_API_KEY, GROQ_API_KEY) and ignores the apiKey parameter unless it's used as OPENAI_API_KEY. This test may not behave as expected since neither environment variable is set, and will likely throw an error rather than returning a provider.

Copilot · 2026-01-04T03:01:01Z

src/commands/applywithllm.ts

+    // Save LLM response to shepherd directory
+    const shepherdResponsePath = path.join(process.cwd(), 'llm_response.json');
+    await fs.writeFile(shepherdResponsePath, JSON.stringify(llmResponse, null, 2), 'utf-8');
+    repoLogs.push(`LLM response saved to ${shepherdResponsePath}`);


The applywithllm command unconditionally saves LLM responses to the current working directory. This can clutter the filesystem and potentially expose sensitive data. This should be conditional on DEBUG_LLM environment variable or removed entirely.

Suggested change

// Save LLM response to shepherd directory

const shepherdResponsePath = path.join(process.cwd(), 'llm_response.json');

await fs.writeFile(shepherdResponsePath, JSON.stringify(llmResponse, null, 2), 'utf-8');

repoLogs.push(`LLM response saved to ${shepherdResponsePath}`);

if (process.env.DEBUG_LLM === 'true') {

// Save LLM response to shepherd directory (debug only)

const shepherdResponsePath = path.join(process.cwd(), 'llm_response.json');

await fs.writeFile(shepherdResponsePath, JSON.stringify(llmResponse, null, 2), 'utf-8');

repoLogs.push(`LLM response saved to ${shepherdResponsePath}`);

}

Copilot · 2026-01-04T03:01:01Z

src/services/llm.test.ts

@@ -0,0 +1,101 @@
+import { getLLMProvider, readFilesForContext, GroqProvider } from './llm';
+import fs from 'fs-extra';
+import path from 'path';


Unused import path.

Suggested change

import path from 'path';

Copilot · 2026-01-04T03:01:02Z

src/services/llm.test.ts

+  const mockFsPathExists = fs.pathExists as jest.MockedFunction<typeof fs.pathExists>;
+


Unused variable mockFsPathExists.

Suggested change

const mockFsPathExists = fs.pathExists as jest.MockedFunction<typeof fs.pathExists>;

Copilot · 2026-01-04T03:02:58Z

@kavitha186 I've opened a new pull request, #1068, to work on those changes. Once the pull request is ready, I'll request review from you.

Kavitha Kesavalu added 2 commits January 3, 2026 20:22

feat: added a command to apply code changes with llm

4561677

feat: apply code changes with prompt

275fad4

kavitha186 requested review from aorinevo and Copilot January 4, 2026 02:53

Copilot started reviewing on behalf of kavitha186 January 4, 2026 02:53 View session

style: fix code formatting with prettier

489e18a

Copilot AI reviewed Jan 4, 2026

View reviewed changes

Copilot AI mentioned this pull request Jan 4, 2026

Fix Groq API key format and model references in documentation #1068

Closed

fix: unit tests

6f79941

kavitha186 force-pushed the feat/applywithllm branch from cba6174 to 6f79941 Compare January 4, 2026 03:16

style: fix code formatting with prettier

fb30475

kavitha186 mentioned this pull request Jan 4, 2026

Apply Code changes with LLM #1069

Open

	export GROQ_API_KEY="sk-your-openai-api-key-here"
	export GROQ_API_KEY="gsk-your-groq-api-key-here"

		const mockFsPathExists = fs.pathExists as jest.MockedFunction<typeof fs.pathExists>;

Uh oh!

Conversation

kavitha186 commented Jan 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

kavitha186 Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Jan 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kavitha186 commented Jan 4, 2026 •

edited

Loading