#1 Created notebook that visualized simple monitor logs by jathavaan · Pull Request #2 · kartAI/doppa-analytics

jathavaan · 2026-02-18T13:18:06Z

This pull request introduces a new helper function for initializing a DuckDB context with required extensions and configuration, and makes it available for import throughout the codebase. The main focus is on enabling spatial and Azure support in DuckDB, along with platform-specific configuration for Linux environments.

New DuckDB context initialization:

Added a new function create_duckdb_context in duckdb.py to create a DuckDB connection, install and load the "spatial" and "azure" extensions, set up an Azure secret, and apply a Linux-specific transport option.
Exposed create_duckdb_context from the persistence package by importing it in __init__.py.

Copilot

Pull request overview

Adds a reusable DuckDB connection initializer under src/persistence to support loading DuckDB spatial + Azure features (and related configuration), and wires it for package-level import.

Changes:

Introduces create_duckdb_context() to connect to DuckDB, install/load spatial + azure, create an Azure secret, and apply a Linux-specific Azure transport setting.
Re-exports create_duckdb_context from src/persistence/__init__.py.
Adds a requirements.txt lock-style dependency list and updates .gitignore to ignore JetBrains IDE files.

Reviewed changes

Copilot reviewed 2 out of 6 changed files in this pull request and generated 6 comments.

File	Description
`src/persistence/duckdb.py`	New DuckDB context helper with extension loading, Azure secret creation, and Linux transport configuration.
`src/persistence/__init__.py`	Exposes `create_duckdb_context` for `from src.persistence import ...` usage.
`requirements.txt`	Adds pinned Python dependencies for the project environment.
`.gitignore`	Adds ignoring of `.idea` and related formatting changes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-18T13:22:00Z

src/persistence/__init__.py

@@ -0,0 +1 @@
+from .duckdb import create_duckdb_context


This file also appears to start with a UTF-8 BOM character before the first from import. Please remove the BOM and save as UTF-8 without BOM to avoid tooling issues.

Suggested change

from .duckdb import create_duckdb_context

from .duckdb import create_duckdb_context

Copilot · 2026-02-18T13:22:01Z

src/persistence/duckdb.py

@@ -0,0 +1,24 @@
+import platform


The file appears to include a UTF-8 BOM character before the first import (the invisible character shown before import platform). This can cause linting/formatting issues and noisy diffs; please remove the BOM and ensure the file is saved as UTF-8 without BOM.

Suggested change

import platform

import platform

Copilot · 2026-02-18T13:22:01Z

src/persistence/duckdb.py

+    """)
+
+    if platform.system() == "Linux":
+        db_context.execute("SET azure_transport_option_type = curl")


SET azure_transport_option_type = curl is likely invalid SQL because curl will be parsed as an identifier rather than a string/enum value. Use the proper DuckDB syntax for setting a string option (typically quoting the value) so this statement succeeds on Linux.

Suggested change

db_context.execute("SET azure_transport_option_type = curl")

db_context.execute("SET azure_transport_option_type = 'curl'")

Copilot · 2026-02-18T13:22:01Z

src/persistence/duckdb.py

+
+import duckdb
+
+
+def create_duckdb_context() -> duckdb.DuckDBPyConnection:
+    db_context: duckdb.DuckDBPyConnection = duckdb.connect()
+    db_context.install_extension("spatial")
+    db_context.load_extension("spatial")
+    db_context.install_extension("azure")
+    db_context.load_extension("azure")
+
+    db_context.execute("""
+    CREATE OR REPLACE SECRET azure_secret(
+        TYPE azure,
+        PROVIDER config,
+        ACCOUNT_NAME 'doppablobstorage'
+    );
+    """)


create_duckdb_context() hardcodes the Azure storage account name and secret name (azure_secret). This makes the helper non-reusable and couples library code to one environment; consider accepting these as parameters or reading from configuration/env vars, and allow opting out of secret creation when not needed.

Suggested change

import duckdb

def create_duckdb_context() -> duckdb.DuckDBPyConnection:

db_context: duckdb.DuckDBPyConnection = duckdb.connect()

db_context.install_extension("spatial")

db_context.load_extension("spatial")

db_context.install_extension("azure")

db_context.load_extension("azure")

db_context.execute("""

CREATE OR REPLACE SECRET azure_secret(

TYPE azure,

PROVIDER config,

ACCOUNT_NAME 'doppablobstorage'

);

""")

import os

import duckdb

def create_duckdb_context(

account_name: str | None = None,

secret_name: str = "azure_secret",

create_secret: bool = True,

) -> duckdb.DuckDBPyConnection:

db_context: duckdb.DuckDBPyConnection = duckdb.connect()

db_context.install_extension("spatial")

db_context.load_extension("spatial")

db_context.install_extension("azure")

db_context.load_extension("azure")

if account_name is None:

account_name = os.getenv("AZURE_STORAGE_ACCOUNT_NAME", "doppablobstorage")

if create_secret and account_name:

# Minimal escaping of single quotes in the account name to keep the SQL valid

escaped_account_name = account_name.replace("'", "''")

db_context.execute(

f"""

CREATE OR REPLACE SECRET {secret_name}(

TYPE azure,

PROVIDER config,

ACCOUNT_NAME '{escaped_account_name}'

);

"""

)

Copilot · 2026-02-18T13:22:02Z

src/persistence/duckdb.py

+def create_duckdb_context() -> duckdb.DuckDBPyConnection:
+    db_context: duckdb.DuckDBPyConnection = duckdb.connect()
+    db_context.install_extension("spatial")
+    db_context.load_extension("spatial")
+    db_context.install_extension("azure")
+    db_context.load_extension("azure")
+


Installing DuckDB extensions on every call (install_extension) has runtime/network side effects and can slow down imports or fail in locked-down/offline environments. Consider attempting load_extension first and only installing on failure, or moving installation to a one-time setup step and keeping this helper to configuration/loading.

Suggested change

def create_duckdb_context() -> duckdb.DuckDBPyConnection:

db_context: duckdb.DuckDBPyConnection = duckdb.connect()

db_context.install_extension("spatial")

db_context.load_extension("spatial")

db_context.install_extension("azure")

db_context.load_extension("azure")

_SPATIAL_EXTENSION_INSTALLED = False

_AZURE_EXTENSION_INSTALLED = False

def create_duckdb_context() -> duckdb.DuckDBPyConnection:

global _SPATIAL_EXTENSION_INSTALLED, _AZURE_EXTENSION_INSTALLED

db_context: duckdb.DuckDBPyConnection = duckdb.connect()

# Ensure the spatial extension is available and loaded

if not _SPATIAL_EXTENSION_INSTALLED:

try:

db_context.load_extension("spatial")

except duckdb.Error:

db_context.install_extension("spatial")

db_context.load_extension("spatial")

_SPATIAL_EXTENSION_INSTALLED = True

else:

db_context.load_extension("spatial")

# Ensure the azure extension is available and loaded

if not _AZURE_EXTENSION_INSTALLED:

try:

db_context.load_extension("azure")

except duckdb.Error:

db_context.install_extension("azure")

db_context.load_extension("azure")

_AZURE_EXTENSION_INSTALLED = True

else:

db_context.load_extension("azure")

Copilot · 2026-02-18T13:22:02Z

src/persistence/duckdb.py

+import duckdb
+
+
+def create_duckdb_context() -> duckdb.DuckDBPyConnection:


PR title/description mismatch: the title mentions creating a notebook, but the diff adds a DuckDB persistence helper, requirements, and gitignore updates (no notebook changes shown). Please align the PR title/description with the actual changes (or include the missing notebook diff if intended).

#1 Created notebook that visualized simple monitor logs

4bc3c8e

jathavaan self-assigned this Feb 18, 2026

Copilot AI review requested due to automatic review settings February 18, 2026 13:18

jathavaan linked an issue Feb 18, 2026 that may be closed by this pull request

Plot Blob Storage DB Scan runs #1

Open

Copilot started reviewing on behalf of jathavaan February 18, 2026 13:18 View session

Copilot AI reviewed Feb 18, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#1 Created notebook that visualized simple monitor logs#2

#1 Created notebook that visualized simple monitor logs#2
jathavaan wants to merge 1 commit intomainfrom
feature/1-plot-blob-storage-db-scan-runs

jathavaan commented Feb 18, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 18, 2026

Uh oh!

Copilot AI Feb 18, 2026

Uh oh!

Copilot AI Feb 18, 2026

Uh oh!

Copilot AI Feb 18, 2026

Uh oh!

Copilot AI Feb 18, 2026

Uh oh!

Copilot AI Feb 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Comments

	from .duckdb import create_duckdb_context
	from .duckdb import create_duckdb_context

	db_context.execute("SET azure_transport_option_type = curl")
	db_context.execute("SET azure_transport_option_type = 'curl'")

		import duckdb


		def create_duckdb_context() -> duckdb.DuckDBPyConnection:

Conversation

jathavaan commented Feb 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Comments