Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ evalio can be used both as a python library and as a CLI for both datasets and p
Once evalio is installed, datasets can be listed and downloaded via the CLI interface. For example, to list all datasets and then download a sequence from the hilti-2022 dataset,
```bash
evalio ls datasets
evalio download hilti_2022/basement_2
evalio dl hilti_2022/basement_2
```

Once downloaded, a trajectory can then be easily used in python,
Expand Down Expand Up @@ -108,4 +108,4 @@ If you use evalio in your research, please cite the following paper,
primaryClass={cs.RO},
url={https://arxiv.org/abs/2507.16000},
}
```
```
4 changes: 2 additions & 2 deletions docs/examples/data_loading.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,12 +81,12 @@ from evalio.rerun import convert

# Initialize rerun
rr.init("evalio")
rr.connect_tcp()
rr.connect_grpc()

# Stream lidar scans to rerun
for scan in Hilti2022.basement_2.lidar():
rr.set_time("timeline", timestamp=scan.stamp.to_sec())
rr.log("lidar", convert(scan, color="z"))
```

If anything is unclear, please open an issue on the evalio repository. We are always looking to improve the documentation and make it easier to use.
If anything is unclear, please open an issue on the evalio repository. We are always looking to improve the documentation and make it easier to use.
4 changes: 2 additions & 2 deletions docs/examples/dataset.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ Additionally, there is a number of optional methods that you can implement to ad

def vehicle(self) -> str: ...

def download(self) -> str: ...
def download(self) -> None: ...

def quick_len(self) -> Optional[int]: ...
```
Expand All @@ -82,4 +82,4 @@ The next three are again self-explanatory, all of which provide information for

`quick_len` returns a hardcoded number of scans in a dataset, used for `evalio ls` and for computing time estimates in `evalio run`. If not set, evalio will load the data to compute the length.

That's all there is to it! Datasets are fairly simple - mostly just parameter setting and easy-to-use iterator wrappers. If you have an dataset implementation of an open-source dataset, feel free to make a PR to add it to evalio so others can use it as well.
That's all there is to it! Datasets are fairly simple - mostly just parameter setting and easy-to-use iterator wrappers. If you have an dataset implementation of an open-source dataset, feel free to make a PR to add it to evalio so others can use it as well.
4 changes: 2 additions & 2 deletions docs/examples/evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,6 @@ Results will be saved to the `output_dir` specified in the config file, with nes

Once trajectories have been run, statistics can be calculated,
```bash
evalio stats -d results -m mean -w 200 -s RTEt
evalio stats results --metric mean --w-meters 200 --sort RTEt_200.0m
```
With `-m/--metric` specifying the metric to calculate with options including mean, median, and sse and `-w/--window` specifying the window size for RTE, with a default of 100 scans. Only first part of all trajectories can also be done using the `-l/--length` option. Sorting of the results can be done with the `-s/--sort` option, with any column heading being an allowed option.
With `--metric` specifying the metric to calculate with options including mean, median, and sse, and `--w-meters` / `--w-seconds` specifying RTE windows (defaults to `--w-meters 30` and can be repeated). Only first part of all trajectories can also be done using the `-l/--length` option. Sorting of the results can be done with `-S/--sort`, with any column heading being an allowed option.
4 changes: 2 additions & 2 deletions docs/examples/odometry.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,12 @@ from evalio.types import Trajectory
from evalio.datasets import NewerCollege2020
from evalio import stats

traj = Trajectory.from_experiment("odometry.csv")
traj = Trajectory.from_file("odometry.csv")
gt = NewerCollege2020.short_experiment.ground_truth()

# Align the odometry to the ground truth
traj_aligned, gt_aligned = stats.align(traj, gt)

# Compute metrics as desired (will align if not already aligned)
error = stats.rte(traj, gt).mean()
```
```
26 changes: 16 additions & 10 deletions docs/examples/pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ To create a pipeline, simply inherit from the `Pipeline` class,

# Getters
def pose(self) -> SE3: ...
def map(self) -> list[Point]: ...
def map(self) -> dict[str, list[Point]]: ...

# Setters
def set_imu_params(self, params: ImuParams): ...
Expand All @@ -44,7 +44,7 @@ To create a pipeline, simply inherit from the `Pipeline` class,
# Doers
def initialize(self): ...
def add_imu(self, mm: ImuMeasurement): ...
def add_lidar(self, mm: LidarMeasurement) -> list[Point]: ...
def add_lidar(self, mm: LidarMeasurement) -> None: ...
```

=== "C++"
Expand Down Expand Up @@ -72,7 +72,7 @@ To create a pipeline, simply inherit from the `Pipeline` class,

// Getters
const SE3 pose() { ... };
const std::vector<Point> map() { ... };
const evalio::Map<> map() { ... };

// Setters
void set_imu_params(ImuParams params) { ... };
Expand All @@ -83,7 +83,7 @@ To create a pipeline, simply inherit from the `Pipeline` class,
// Doers
void initialize() { ... };
void add_imu(ImuMeasurement mm) { ... };
std::vector<Point> add_lidar(LidarMeasurement mm) { ... };
void add_lidar(LidarMeasurement mm) { ... };
}
```

Expand All @@ -93,6 +93,8 @@ We'll cover each section of methods in turn.

The first four methods are all static methods that provide information about the pipeline. `version`, `url`, and `name` are all self-explanatory. `default_params` is a static method that returns a dictionary of the default parameters for the pipeline. This is used to verify parameters before they are passed in, as well as ensure a consistent output for each run.

In C++ there is additionally a number of helper type conversion functions that can make converting between iterators, point types, and geometry types simpler. A good example of this can be found in the `lio_sam.h` binding where it is used to convert pose and point types. These converters can additionally be leveraged by the `save` methods described below.

## Getters

The next two methods are getters for the pose and map. The pose is the most up-to-date estimate for the IMU and is polled after each lidar measurement is passed in.
Expand All @@ -112,7 +114,11 @@ Arguably the most important part.

`add_imu` is called for each IMU measurement. This is where the IMU data is processed and used to update the pose.

`add_lidar` is called for each lidar measurement. This is where the lidar data is processed and used to update the map. It returns a list of features were extracted from the scan and are used for visualization.
`add_lidar` is called for each lidar measurement. This is where the lidar data is processed and used to update the map.

Saving poses can be done asynchronously, using `save(stamp, pose)`. In C++, the type of `pose` can be anything that has the method `evalio::convert<evalio::SE3>(const MyPose& pose)` implemented.

For visualization, features can be save similarly with either `save(stamp, {"key": features})` in python, or `save(stamp, "key1", feat1, "key2", feat2)` in C++. Again, `feat1` and `feat2` can be any type that are iterators with their internal point types convertible to `evalio::Point`.

## C++ Building

Expand All @@ -125,11 +131,11 @@ NB_MODULE(_core, m) {

// Only have to override the static methods here
// All the others will be automatically inherited from the base class
nb::class_<MyCppPipeline, evalio::Pipeline>(m, "MyCppPipeline")
nb::class_<MyPipeline, evalio::Pipeline>(m, "MyPipeline")
.def(nb::init<>())
.def_static("name", &MyCppPipeline::name)
.def_static("url", &MyCppPipeline::url)
.def_static("default_params", &MyCppPipeline::default_params);
.def_static("name", &MyPipeline::name)
.def_static("url", &MyPipeline::url)
.def_static("default_params", &MyPipeline::default_params);
}
```

Expand All @@ -145,4 +151,4 @@ We recommend then setting everything up to be built with [`scikit-build-core`](h
m.def("abi_tag", []() { return nb::detail::abi_tag(); });
```

That's all there is to it! Pipelines should be fairly easy to implement and are usually just a simple wrapper around your existing code to provide a common interface. Once your pipeline is open-source/published/etc, feel free to make a PR to add it to evalio. This both improves the visibility of your work and of evalio.
That's all there is to it! Pipelines should be fairly easy to implement and are usually just a simple wrapper around your existing code to provide a common interface. Once your pipeline is open-source/published/etc, feel free to make a PR to add it to evalio. This both improves the visibility of your work and of evalio.
2 changes: 1 addition & 1 deletion docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ make
```

### Pipelines
By default, pipelines are not included due to their large dependencies. We use vpckg to handle a reliable build of these dependencies and pipelines. vcpkg and the pipelines can be setup via running
By default, pipelines are not included due to their large dependencies. We use vcpkg to handle a reliable build of these dependencies and pipelines. vcpkg and the pipelines can be setup via running
```bash
./cpp/setup_pipelines.sh
```
Expand Down
8 changes: 4 additions & 4 deletions docs/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ evalio can be used both as a python library and as a [CLI](ref/cli.md) for both
Once evalio is installed, datasets can be listed and downloaded via the [CLI](ref/cli.md) interface. For example, to list all datasets and then download a sequence from the hilti-2022 dataset,
```bash
evalio ls datasets
evalio download hilti_2022/basement_2
evalio dl hilti_2022/basement_2
```
evalio downloads data to the path given by `-D`, `EVALIO_DATA` environment variable, or if both are unset to the local folder `./evalio_data`. All the trajectories in a dataset can also be downloaded by using the wildcard `hilti_2022/*`, making sure to escape the asterisk as needed.

Expand Down Expand Up @@ -67,10 +67,10 @@ import rerun as rr
from evalio.rerun import convert

rr.init("evalio")
rr.connect_tcp()
rr.connect_grpc()
for scan in ds.Hilti2022.basement_2.lidar():
rr.set_time("timeline", timestamp=scan.stamp.to_sec())
rr.log("lidar", convert(scan, color=[255, 0, 255]))
rr.log("lidar", convert(scan, color=(255, 0, 255)))
```

!!! note
Expand Down Expand Up @@ -143,4 +143,4 @@ where m -> map, s -> scan, i -> intensity image, and f -> extracted features can

That's about the gist of it! Try playing around the [CLI](ref/cli.md) interface to see what else is possible. Feel free to open an issue if you have any questions, suggestions, or problems.

Additionally, we recommend checking out the examples section for specific use cases for evalio.
Additionally, we recommend checking out the examples section for specific use cases for evalio.
5 changes: 3 additions & 2 deletions docs/ref/pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,13 @@ For more information about the pipelines included in evalio, see the [included p
members:
- Pipeline
- CTICP
- KissICP
- DLIO
- FORM
- GenZICP
- KissICP
- LOAM
- LioSAM
- MadICP
- FORM
- PipelineNotFound
- UnusedPipelineParam
- InvalidPipelineParamType
Expand Down
3 changes: 0 additions & 3 deletions python/evalio/cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,6 @@ def module_callback(value: Optional[list[str]]) -> list[Any]:

@app.callback()
def global_options(
# Marking this as a str for now to get autocomplete to work,
# Once this fix is released (hasn't been as of 0.15.2), we can change it to a Path
# https://github.com/fastapi/typer/pull/1138
data_dir: Annotated[
Optional[Path],
typer.Option(
Expand Down
2 changes: 1 addition & 1 deletion python/evalio/cli/stats.py
Original file line number Diff line number Diff line change
Expand Up @@ -286,7 +286,7 @@ def evaluate_typer(
# Parse some of the options
if only_complete and only_failed:
raise typer.BadParameter(
"Can only use one of --only-complete, --only-incomplete, or --only-failed."
"Can only use one of --only-complete or --only-failed."
)

# Parse the filtering options
Expand Down
2 changes: 1 addition & 1 deletion python/evalio/datasets/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ def _fail_not_downloaded(self):
if not self.is_downloaded():
# TODO: Make this print with rich?
raise ValueError(
f"Data for {self} not found, please use `evalio download {self}` to download"
f"Data for {self} not found, please use `evalio dl {self}` to download"
)

@classmethod
Expand Down
Loading