A high-performance, type-safe, and secure Python library for processing ONIX 3.0 XML feeds. Built on top of xsdata and lxml, pyonix-core provides a robust foundation for bibliographic data exchange in the publishing industry.
ONIX for Books is the international standard for representing and communicating book industry product information in electronic form. pyonix-core simplifies the complexity of the ONIX 3.0 standard by providing:
- Strict Type Safety: Fully typed Python dataclasses generated directly from EDItEUR's official XSD schemas.
- Memory Efficiency: Iterative, streaming parsing capabilities designed to handle multi-gigabyte ONIX files with minimal memory footprint.
- Security First: Hardened XML parsing configuration to prevent XXE (XML External Entity) attacks and other common XML vulnerabilities.
- Developer Friendly: A high-level "Facade" pattern to abstract away the deeply nested structure of raw ONIX messages, providing simple access to common fields like ISBNs, titles, and prices.
Requires Python 3.11 or higher.
pip install pyonix-coreThe core entry point is parse_onix_stream, which yields product records one by one. This allows you to process massive files without loading the entire document into memory.
from pyonix_core.parsing.parser import parse_onix_stream
from pyonix_core.facade.product import ProductFacade
file_path = "path/to/onix_feed.xml"
# parse_onix_stream automatically detects Reference vs Short tags
for product in parse_onix_stream(file_path):
# Wrap the raw data model in a Facade for easier access
facade = ProductFacade(product)
print(f"Title: {facade.title}")
print(f"ISBN-13: {facade.isbn13}")
print(f"Price: {facade.price_amount} {facade.price_currency}")
print("-" * 20)The ProductFacade simplifies data extraction. Instead of navigating complex nested objects, you can access properties directly.
# Example of accessing data via Facade
print(f"Record Reference: {facade.record_reference}")
print(f"Contributors: {', '.join(facade.contributors)}")The data models in pyonix_core.models are auto-generated using xsdata from the official ONIX 3.0 schemas. This ensures 100% compliance with the standard and provides excellent IDE support (autocompletion, type checking).
XML parsing is handled by lxml with strict security settings:
resolve_entities=False: Prevents XXE attacks.no_network=True: Blocks remote resource fetching.load_dtd=False: Disables DTD processing.
python -m unittest discover testsIf the schemas change, you can regenerate the data models:
python scripts/generate_models.pyThis project is licensed under the MIT License.
Note: This project is currently in active development.
The following high-value features were recently added to pyonix-core. Each one is implemented to integrate cleanly with the existing facade and parsing architecture.
- Short Tag to Reference Tag Normalizer
- Module:
pyonix_core.parsing.normalization - Description: Uses an XSLT transform (bundleable EDItEUR XSLT) to convert ONIX Short Tags into Reference Tags before parsing. This lets the parser and generated dataclasses work consistently regardless of input tag style. For very large files, a streaming/SAX-based fallback is planned.
- Quick usage:
from pyonix_core.parsing.normalization import TagNormalizer
from pyonix_core.parsing.parser import parse_onix_stream
norm = TagNormalizer()
with open('short_tag_onix.xml', 'rb') as fh:
normalized_stream = norm.normalize_stream(fh)
for product in parse_onix_stream(normalized_stream):
...- Data Flattener (Pandas/CSV ready)
- Module:
pyonix_core.utils.flatten - Description:
ProductFlattenerconverts aProductFacadeinto a flat dictionary using a configurableSerializationProfile. Useful for creating CSV rows or constructing Pandas DataFrames without manually traversing the ONIX model. - Quick usage:
from pyonix_core.utils.flatten import ProductFlattener
from pyonix_core.facade.product import ProductFacade
f = ProductFlattener()
row = f.flatten(ProductFacade(product_model))- HTML Sanitizer & Extractor
- Module:
pyonix_core.utils.text - Description: Safe extraction and sanitization of HTML-like content found in ONIX
TextContentcomposites. Usesbleachto sanitize andhtml2textto produce Markdown. These are optional dependencies under thetextextra inpyproject.toml. - Quick usage:
from pyonix_core.utils.text import clean_html, to_markdown
safe_html = clean_html(raw_html)
md = to_markdown(raw_html)- ISBN Tools (10/13 converter)
- Module:
pyonix_core.utils.identifiers - Description: Pure-Python
ISBNutility for cleaning, validating, and converting ISBN-10 to ISBN-13.ProductFacade.isbn13now prefers an explicit ISBN-13 but will auto-convert valid ISBN-10 values when necessary. - Quick usage:
from pyonix_core.utils.identifiers import ISBN
ISBN.clean('0-306-40615-2')
ISBN.validate('9780306406157')
ISBN.to_13('0-306-40615-2')- Media Asset Helper
- Module:
pyonix_core.facade.assets - Description:
AssetHelpersimplifies locating front cover images and other collateral assets within theCollateralDetailcomposite and exposes ahelperproperty onProductFacade. - Quick usage:
from pyonix_core.facade.product import ProductFacade
facade = ProductFacade(product_model)
cover_url = facade.helper.get_cover_image()All of these features are exercised in the test-suite (excluding extremely large-file performance tests). Optional dependencies are declared under [project.optional-dependencies] in pyproject.toml (see the text extra for HTML utilities).
To support memory-safe, resilient ETL pipelines and to bridge the impedance mismatch between generic client libraries and strict pipeline contracts, pyonix-core now includes small, focused wrappers to handle streaming, retries, and adaptation to external Source interfaces.
- Streaming / Iterator Wrapper —
OnyxStreamWrapper
- Module:
pyonix_core.integration.stream_wrapper - Purpose: Convert paginated client list calls into a seamless, memory-efficient generator that yields items one-by-one. Supports several pagination shapes (
items/resultsattributes,has_next,next_page_token, or simple page/limit parameters). - Key options:
page_param,items_attr,page_size, andmax_items(handy for tests or bounded runs). - Example:
from pyonix_core.integration.stream_wrapper import OnyxStreamWrapper
def client_list(page=1):
# returns an object with .items and .has_next
...
stream = OnyxStreamWrapper(client_list, page_param='page')
for record in stream:
process(record)- Resilience / Retry Decorator —
with_onyx_retries
- Module:
pyonix_core.integration.resilience - Purpose: Lightweight exponential-backoff retry decorator for wrapping flaky API calls without modifying library code. Configurable
max_attempts,initial_delay, andbackoff_factor. Supportsfail_fast_exceptionsto immediately abort on unrecoverable errors (e.g., auth failures). - Example:
from pyonix_core.integration.resilience import with_onyx_retries
@with_onyx_retries(max_attempts=4, initial_delay=1.0)
def fetch_page(page):
return client.list(page=page)- Protocol Adapter / Source Plugin —
OnyxSourceAdapter
- Module:
pyonix_core.integration.adapter - Purpose: Adapter that wraps a client and one of its list methods to provide a minimal Source-like interface for pipelines (methods:
configure(),extract() -> Iterable,cleanup()). Theextract()method returns anOnyxStreamWrapperof the underlying list method and automatically applies the retry decorator. - Example:
from pyonix_core.integration.adapter import OnyxSourceAdapter
adapter = OnyxSourceAdapter(client, client.list)
adapter.configure(api_key='...')
for record in adapter.extract(page_size=100):
mapper(record) # yields one record at a time
adapter.cleanup()Notes
- These wrappers are intentionally small and dependency-free (tests use mocks). For production, you may want to extend the adapter to accept an HTTP session, custom logging hooks, or pluggable error mappings.
max_itemson the stream wrapper can be used to bound streaming during tests or dry runs.