Skip to content

Update README to address missing entry script The README.md previously referenced a script 'generate-llmstxt-crawl4ai.py' for all command examples. This script is not present in the repository. This commit updates the README to: - Add a prominent note in 'Quick Start', 'Command Line Options', and 'Example Output' sections explaining the missing script. - Suggest potential reasons and guide the user on how to proceed, such as pointing to the 'llmsgen.generator' module for the core logic. - Preserve existing command examples as they are likely correct if the script were present or recreated by the user. - Correct minor formatting issues arising from the insertions. #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Hrishikeshgupta2002 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update README to address missing entry script The README.md previously referenced a script 'generate-llmstxt-crawl4ai.py' for all command examples. This script is not present in the repository. This commit updates the README to: - Add a prominent note in 'Quick Start', 'Command Line Options', and 'Example Output' sections explaining the missing script. - Suggest potential reasons and guide the user on how to proceed, such as pointing to the 'llmsgen.generator' module for the core logic. - Preserve existing command examples as they are likely correct if the script were present or recreated by the user. - Correct minor formatting issues arising from the insertions. #2

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,23 @@ ollama pull phi:2.7b

## 🚀 Quick Start

> **Important Note on Running the Tool:**
>
> The command examples in this README (e.g., `python generate-llmstxt-crawl4ai.py ...`) assume the presence of a script named `generate-llmstxt-crawl4ai.py` in the project's root directory. **This script does not appear to be included in the current repository.**
>
> This could be due to a few reasons:
> * It might be a helper script that you need to create locally.
> * It may have been inadvertently excluded from the repository.
> * The primary execution method for this tool might have changed, and this README has not yet been fully updated.
>
> If you have this script, the examples below should work as described.
>
> If the script is missing, the core logic for generating `llms.txt` files likely resides within the `llmsgen` package, particularly in the `llmsgen.generator` module (potentially in a file that would have been named `llms_generator.py`, though its source also appears to be missing from the repository). You may need to either:
> 1. Create a Python wrapper script that imports and calls the main function from `llmsgen.generator`.
> 2. Investigate how to directly invoke the intended functionality from the `llmsgen` package modules.
>
> The command-line options listed further down are likely still relevant for the tool's underlying functionality.

### Basic Usage
```bash
# Generate llms.txt for a website
Expand Down Expand Up @@ -114,6 +131,8 @@ python generate-llmstxt-crawl4ai.py https://example.com \

## 📖 Command Line Options

> **Note:** The commands shown below (and throughout this document) assume you are using the `generate-llmstxt-crawl4ai.py` script. As mentioned in the "Quick Start" section, this script is not found in the repository. If you are invoking the tool's functionality directly from the `llmsgen` package, you will need to adapt how these options are passed. The options themselves are likely still applicable to the core logic.

| Option | Default | Description |
|--------|---------|-------------|
| `url` | Required | Target website URL |
Expand Down Expand Up @@ -259,6 +278,8 @@ Raw HTML → Content Extraction → Text Cleaning → AI Summarization → Categ

## 📊 Example Output

> **Note:** The example input command below assumes the `generate-llmstxt-crawl4ai.py` script is available. Please see the note in the "Quick Start" section if this script is missing.

### Input
```bash
python generate-llmstxt-crawl4ai.py https://alternates.ai --max-pages 10
Expand Down