fix: Adding Python shutdown check in _close to fix sys.meta_path is None error

[Bidek56]

What type of PR is this?

• Refactor
• Feature
• Bug Fix
• Other

Description

This PR adds Python shutdown check in _close function to avoid this error:
ERROR:Attempt to close session raised a local exception: sys.meta_path is None, Python is likely shutting down

How is this tested?

• [X ] Unit tests
• E2E Tests
• Manually
• N/A

Related Tickets & Documents

[Copilot]

Pull request overview

This PR fixes a bug where closing a Databricks SQL connection during Python shutdown would raise an error due to sys.meta_path being None. The fix adds a shutdown detection mechanism and suppresses error logging when Python is shutting down.

Changes:

• Added Python shutdown detection using sys.meta_path is None check
• Wrapped close operations in try-except blocks to prevent errors during shutdown
• Suppressed error logging when shutdown is detected to avoid spurious error messages

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The shutdown check variable is computed once at the beginning of _close, but Python shutdown state could change during the execution of this method. Consider checking sys.meta_path is None directly in each exception handler for more accurate shutdown detection.

[Copilot]

Catching bare Exception is too broad and could mask unexpected errors during normal operation. Consider catching specific exceptions or at minimum using except Exception as e: to log the exception details when not shutting down.

[Copilot]

Catching bare Exception is too broad and could mask unexpected errors during normal operation. Consider catching specific exceptions or at minimum using except Exception as e: to log the exception details when not shutting down.

[Copilot]

Catching bare Exception is too broad and could mask unexpected errors during normal operation. Consider catching specific exceptions or at minimum using except Exception as e: to log the exception details when not shutting down.

[Bidek56]

@msrathore-db Please review this PR, it resolves the following error that we are seeing. Thx
Attempt to close session raised a local exception: sys.meta_path is None, Python is likely shutting down

[msrathore-db]

@msrathore-db Please review this PR, it resolves the following error that we are seeing. Thx Attempt to close session raised a local exception: sys.meta_path is None, Python is likely shutting down

When are you observing this error. Can you provide repro steps?

[Bidek56]

@msrathore-db Please review this PR, it resolves the following error that we are seeing. Thx Attempt to close session raised a local exception: sys.meta_path is None, Python is likely shutting down

When are you observing this error. Can you provide repro steps?

I am observing this issue when setting: connect_args={"enable_telemetry": False} in create_engine

from sqlalchemy import create_engine
import pandas as pd
import os

host = os.environ.get('DATABRICKS_HOST').replace("https://", "") 
http_path = os.environ.get('HTTP_PATH')
token = os.environ.get('DATABRICKS_TOKEN')

if not host or not http_path or not token:
    raise ValueError("DATABRICKS_HOST, HTTP_PATH, and DATABRICKS_TOKEN environment variables must be set.")

engine = create_engine(f"databricks://token:{token}@{host}?http_path={http_path}", connect_args={"enable_telemetry": False} )

with engine.connect() as conn:
    pd.read_sql("SELECT VERSION() AS version", conn)