Request support to exclude indexing of specified folders or files

[BryceWG]

I noticed that some folders can currently be automatically excluded, but according to the README, it should not be possible to set custom exclusions, right?

[badmonster0]

thanks for the question @BryceWG
@georgeh0 george would you like to take a look?

[georgeh0]

Yes, currently we maintain an exclude list internally for simplicity:

We'll allow customizing later. Stay tuned!

Feel free to send PR to add common exclude patterns to the file.

[Mreddi1]

Hello @georgeh0, I am also waiting for entering custom excluding paths feature, because I have small size project, but it has a lot of big sql files with data and sql migrations, and md files, but they are all in .gitignore.

I have started to use your tool today and used COCOINDEX_CODE_EMBEDDING_MODEL=openai, and I did approximatly 7 claude code cli requests and platform.openai shows me 2.247K requests, 68.869M input tokens and .cocoindex_code/target_sqlite.db was 678MB, yes, it spends only $1.30, before I switched to default model and did force cocoindex-code index command, I will provide some data about project.

Full project, includes md, sql files, repomix.config:

import { defineConfig } from 'repomix';

export default defineConfig({
  $schema: 'https://repomix.com/schemas/latest/schema.json',
  input: {
    maxFileSize: 52428800,
  },
  output: {
    filePath: 'repomix-output.md',
    style: 'markdown',
    parsableStyle: false,
    fileSummary: true,
    directoryStructure: true,
    files: true,
    removeComments: false,
    removeEmptyLines: true,
    compress: true,
    topFilesLength: 5,
    showLineNumbers: false,
    truncateBase64: true,
    copyToClipboard: false,
    includeFullDirectoryStructure: true,
    tokenCountTree: true,
    git: {
      sortByChanges: true,
      sortByChangesMaxCommits: 30,
      includeDiffs: false,
      includeLogs: false,
      includeLogsCount: 20,
    },
  },
  include: [],
  ignore: {
    useGitignore: true,
    useDotIgnore: true,
    useDefaultPatterns: true,
    customPatterns: [
      '**/node_modules/**',
      '**/dist/**',
      '**/*.log',
      '**/public/**',
      '**/.yarn/**',
      '**/.vscode/**',
      '**/.idea/**',
      '**/.cursor/**',
      'package-lock.json',
      'AGENTS.md',
      'CLAUDE.md',
      'GEMINI.md',
    ],
  },
  security: {
    enableSecurityCheck: true,
  },
  tokenCount: {
    encoding: 'o200k_base',
  },
});

Shows output:

Total Files: 1,121 files
Total Tokens: 1,533,436 tokens
Total Chars: 6,443,492 chars

Without sql, md files, repomix config:

import { defineConfig } from 'repomix';

export default defineConfig({
  $schema: 'https://repomix.com/schemas/latest/schema.json',
  input: {
    maxFileSize: 52428800,
  },
  output: {
    filePath: 'repomix-output.md',
    style: 'markdown',
    parsableStyle: false,
    fileSummary: true,
    directoryStructure: true,
    files: true,
    removeComments: false,
    removeEmptyLines: true,
    compress: true,
    topFilesLength: 5,
    showLineNumbers: false,
    truncateBase64: true,
    copyToClipboard: false,
    includeFullDirectoryStructure: true,
    tokenCountTree: true,
    git: {
      sortByChanges: true,
      sortByChangesMaxCommits: 30,
      includeDiffs: false,
      includeLogs: false,
      includeLogsCount: 20,
    },
  },
  include: [],
  ignore: {
    useGitignore: true,
    useDotIgnore: true,
    useDefaultPatterns: true,
    customPatterns: [
      '**/node_modules/**',
      '**/dist/**',
      '**/*.log',
      '**/public/**',
      '**/.yarn/**',
      '**/.vscode/**',
      '**/.idea/**',
      '**/.cursor/**',
      'package-lock.json',
      'AGENTS.md',
      'CLAUDE.md',
      'GEMINI.md',
      '**/*.md',
      '**/*.sql',
    ],
  },
  security: {
    enableSecurityCheck: true,
  },
  tokenCount: {
    encoding: 'o200k_base',
  },
});

Shows output:

 Total Files: 613 files
 Total Tokens: 503,454 tokens
 Total Chars: 2,097,877 chars

Force reindexing by cocoindex-code index output with sql and md files:

cocoindex-code index                INT ✘  27m 59s  23:33:53 
⠒⠒ app_main: ▕                                        ▏0/1
Loading weights: 100%|███████████████████████████████████████████████████████████| 103/103 [00:00<00:00, 6655.47it/s]
BertModel LOAD REPORT from: sentence-transformers/all-MiniLM-L6-v2
Key                     | Status     |  | 
------------------------+------------+--+-
embeddings.position_ids | UNEXPECTED |  | 

Notes:
⠚⠚ app_main: ▕                                        ▏0/1
✅ declare_target_state_with_child: ▕████████████████████████████████████████▏1/1: 1 reprocessed
⠒⠒ process_file: ▕███████████████████████████████████████ ▏51201/52207: 129 added, 23 reprocessed, 51049 unchanged
                                         

It had duration approximetly 20 minutes, I stopped a few times just to gave my laptop a little rest :) But, I did not finish reindexing, because I did not undertend where it found 52207 files..
And .cocoindex_code/target_sqlite.db was 1.7G

Whole reindexing time my MacBook Pro M1 14-inch, 2021, Chip Apple M1 Pro, RAM 16GB, had 90C processor temperature.

Then, I turned off mcp temporarily for this project..

[overbit]

it would also make sense to exclude anything matching .gitignore @badmonster0 @georgeh0

[georgeh0]

Yes, ignoring everything matching .gitignore makes perfect sense. Created #62 for this.

(BTW, I thought that before but sometimes users adds .gitignore later for a new project so large amount of generated files can be included before we have that, but at least it can be a good list complementary to the predefined exclude list)

Closing this issue since it's fixed by #54