Feature Request: Add `header` parameter to `BData.save()` method

[micchu]

Feature Request: Add header parameter to BData.save() method

Summary

Add an optional header parameter to the BData.save() method to allow users to control whether header information (creation time, call stack, code) is saved. This is particularly important when preparing data for public release.

Motivation

Currently, BData.save() automatically saves header information that may contain sensitive details such as:

• Absolute file paths from the call stack
• Source code contents
• User-specific directory structures

When preparing datasets for public release or sharing with collaborators, it is often necessary to exclude such metadata for:

• Public data repositories: Clean data for platforms like Figshare, OpenNeuro
• Privacy and security: Prevent leakage of internal paths and code
• Collaboration: Share data without exposing implementation details
• File size optimization: Reduce unnecessary metadata

Current Workaround

The only way to avoid saving header information is to use the private method:

bdata._BData__save_h5(filepath, header=None)

This violates Python conventions and is neither documented nor officially supported.

Proposed Solution

Add an optional header parameter to the BData.save() method:

def save(self, file_name: str, file_type: Optional[str] = None, header: bool = True) -> None:
    """
    Save 'dataset' and 'metadata' to a file.
    
    Parameters
    ----------
    file_name : str
        File name
    file_type : str, optional
        File type (currently only 'HDF5' is supported)
    header : bool, optional (default: True)
        If True (default), save header information (creation time, call stack, code).
        If False, do not save header information.
    
    Returns
    -------
    None
    """

Implementation Details

• Default behavior: header=True (preserve backward compatibility)
• When header=True: Current behavior - update self.__header with timestamp, call stack, and code, then pass to __save_h5()
• When header=False: Skip header update and pass None to __save_h5()
• The conditional logic should be added before the header update section

Suggested Implementation

def save(self, file_name: str, file_type: Optional[str] = None, header: bool = True) -> None:
    """Save 'dataset' and 'metadata' to a file."""
    
    # Store data creation information (only if header=True)
    if header:
        t_now = time.time()
        t_now_str = datetime.datetime.fromtimestamp(t_now).strftime('%Y-%m-%d %H:%M:%S')

        callstack = []
        callstack_code = []
        f = inspect.currentframe()
        while True:
            f = f.f_back
            if f is None:
                break
            fname = os.path.abspath(f.f_code.co_filename)
            fline = f.f_lineno
            callstack.append('%s:%d' % (fname, fline))
            if os.path.exists(fname):
                with open(fname, 'r') as fl:
                    fcode = fl.read()
            else:
                fcode = ''
            callstack_code.append(fcode)

        self.__header.update({'ctime': t_now_str,
                              'ctime_epoch': t_now,
                              'callstack': callstack,
                              'callstack_code': callstack_code})
        header_to_save = self.__header
    else:
        header_to_save = None

    if file_type is None:
        file_type = self.__get_filetype(file_name)

    if file_type == "Matlab":
        raise RuntimeError('Saving BData as a mat file is no longer supported. Please save the data as HDF5 (.h5).')
    elif file_type == "HDF5":
        self.__save_h5(file_name, header=header_to_save)
    else:
        raise ValueError("Unknown file type: %s" % (file_type))

Example Usage

# Save with header (current default behavior)
bdata.save('output.h5')
bdata.save('output.h5', header=True)

# Save without header (new feature)
bdata.save('output.h5', header=False)

[micchu]

Additional note after discussing this with @ganow :

We may want to change the direction of this issue slightly. Rather than simply adding an option to disable the header, the current behavior of header=True may itself need to be reconsidered. Automatically saving the call stack and source code contents can be risky, especially when BData files are shared outside the lab or prepared for public release.

This is even more problematic because this header-saving behavior is likely not widely known to users. Users may save and share BData files without realizing that source code contents, file paths, and other execution-context information can be embedded in the output file.

Keeping creation time information such as ctime / ctime_epoch seems reasonable, but other header-saving features, particularly callstack and callstack_code, should probably be disabled by default or removed from the default save behavior.

[ganow]

Thanks @micchu — I agree with the motivation, and your follow-up captures the key point: this behavior is not widely known, so the risk is largely invisible to users.

That's exactly why simply adding an opt-out flag (header=False) wouldn't be enough. An opt-out only protects users who already know the header exists and remember to disable it. The people most at risk — those who save and share a BData file without realizing that source code and absolute paths are embedded — are exactly the ones who would never set the flag. So adding header=False would leave the harmful default unchanged for the very users it's meant to protect.

So let's change the default rather than add an opt-out:

1. Stop collecting callstack / callstack_code entirely — remove them from save(). They embed absolute paths and the full contents of every source file on the call stack (potentially unpublished code or secrets), which goes well beyond what provenance tracking needs.
2. Keep ctime / ctime_epoch, as you suggested — they're harmless, low-information, and genuinely useful.
3. Sanitize existing files on re-save. Just deleting the collection code is not sufficient: on load, a file's /header is restored into memory, so a legacy file's callstack_code would survive load and be re-written on the next save. So save() should also strip these legacy keys before writing. Net effect: opening an old (contaminated) file with the new version and saving it produces a clean file automatically.
4. Warn on drop. When legacy callstack / callstack_code are stripped during save, emit a warning so the change is visible rather than silent.

With this, the leakage motivation behind the original header=False request disappears: the remaining header is just ctime / ctime_epoch plus whatever the user intentionally added via update_header(). So I don't think the header=False flag is needed anymore.

This is a behavior change to saved-file contents, so it should be called out in the release notes.