picosparql

About

Picosparql is a small lightweight SPARQL server. Its primary goal is to be resource efficient. It should handle large datasets with minimal memory usage. I wanted something I could deploy on a server with less than 1GB RAM. Picosparql has been implemented in C99 and uses libmicrohttpd for the server and librdf (redland) for handling RDF and SPARQL. Officially, only Linux platforms are supported. The server supports multiple concurrent clients and offers a basic REST API and web interface. SPARQL queries are executed in a separate process. The process is monitored by a watchdog in order to not exceed configurable time or memory constraints. As underlying storage for triples, Berkeley DB Hashes are used, as these performed best and allowed for fast buildup of databases from RDF data.

Support

Picosparql supports SPARQL 1.0 and an incomplete subset of SPARQL 1.1. Because picosparql uses the unmaintained redland library, recent additions to the SPARQL standard are not implemented. Refer to the notice by the redland library below for further information.

Unimplemented parts of SPARQL 1.1:

    BIND scope
    VALUES: part of federated query
    Decimal and double canonical format details
    EXISTS / NOT EXISTS
    JSON result format reading
    MINUS
    Property Paths: These are likely never be supported since it is a lot of work (estimate: 3 months full time) and might need multiple new APIs to talk to the storage layer.
    SERVICE: Part of federated query

Out of scope parts of SPARQL 1.1:

    Entailment and inference: Rasqal is not an inference engine
    SPARQL Federated Query
    SPARQL Update (also called SPARQL Protocol): The Update syntax is parsed with parser name 'sparql-update' but nothing is executed inside Rasqal. Supporting SPARQL Update requires a protocol server that responds to requests to perform operations on a graph store. Rasqal is a query library and does not have an event loop or triple store. Redstore is a system supporting SPARQL Update and Service Description built with Rasqal by using Redland librdf for graph storage and query.

Quickstart

The following instructions expect a Debian Linux system.

Installation

git clone https://github.com/palomena/picosparql.git
cd picosparql
apt install build-essential librdf-dev libraptor2-dev librasqal3-dev libxml2-dev zlib1g-dev libmicrohttpd-dev
make

Usage

Creating a new database at ./path/to/database/name. This will also clear an existing database at that location.

./picosparql new -d ./path/to/database/name

Feeding data into the database

# Load data from file ./my-triples.ttl into the database
./picosparql add -d ./path/to/database/name -i ./my-triples.ttl -f turtle
# Load data from a gzip compressed file via pipe into the database
zcat ./my-triples.ttl.gz | ./picosparql add -d ./path/to/database/name -i /dev/stdin -f turtle
# Load data from multiple files via pipe into the database
zcat `ls ./triples/*.ttl.gz | head -n 20` | ./picosparql add -d ./path/to/database/name -i /dev/stdin -f turtle

Querying the database locally via the command line

# Interactive: write query on command line, Press Ctrl+D to Submit
./picosparql query -d ./path/to/database/name
# Pipe the query from a file
cat ./my-query.txt | ./picosparql query -d ./path/to/database/name

Launching the SPARQL webserver on port 3030 (default)

./picosparql serve -d ./path/to/database/name -p 3030

Serializing a database to stdout or a file

# Dump database to stdout
./picosparql serialize -d ./path/to/database/name
# Dump database to file
./picosparql serialize -d ./path/to/database/name -f turtle > my-triples.txt

Query running picosparql server via requests module from Python

import requests
querystring = """
SELECT * WHERE {
	?s ?p ?o .
}
LIMIT 10
"""
response = requests.post("http://localhost:3030/sparql", data=querystring)
print(response.json())

Reference

Usage: ./picosparql action [options]

Actions:
  serve       Start HTTP SPARQL endpoint
  new         Create new database
  add         Add data into database
  query       Read SPARQL query from stdin and execute it
  serialize   Serialize database to stdout

Options:
  -h          Show help
  -v          Show version
  -p PORT     HTTP port, default 3030
  -i INPUT    Input file/path, use /dev/stdin for stdin/pipe
  -o OUTPUT   Output file/path, use /dev/stdout for stdout
  -f FORMAT   RDF syntax format, default turtle
  -d DATABASE Database name/path
  -t TIMEOUT  SPARQL Query timeout
  -m MEMORY   SPARQL Query memory limit

Web Interface

Future Work

With the advent of Large Language Models as coding assistants, it could possibly be feasible to "vibe code" SPARQL 1.1 support into redland. I may investigate this in the near future. If it works, picosparql could provide a feature-complete SPARQL server.

License

This software is licensed under the MIT License. It uses third-party components licensed under the GNU LESSER GENERAL PUBLIC LICENSE Version 2.1 - see LICENSE.txt for more information.