You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _conferences/eurosys2026/badges.md
+62-53Lines changed: 62 additions & 53 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,64 +4,73 @@ order: 10
4
4
---
5
5
6
6
<style>
7
-
img { width: 10em; }
7
+
img { width: 4em; }
8
8
</style>
9
9
10
-
Submitted artifacts can select to be evaluated against the following badges,
11
-
which are defined in the [ACM Artifact Review and Badging policy v1.1](https://www.acm.org/publications/policies/artifact-review-and-badging-current):
10
+
EuroSys is an ACM conference and thus uses [ACM's badges](https://www.acm.org/publications/policies/artifact-review-and-badging-current).
12
11
13
-
||**Artifacts Available**<br>Author-created artifacts relevant to this paper have been placed on a publicly accessible archival repository. A DOI or link to this repository along with a unique identifier for the object is provided. |
14
-
||**Artifacts Evaluated - Functional**<br>The artifacts associated with the research are found to be documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation. |
15
-
||**Results Reproduced**<br>The main results of the paper have been obtained in a subsequent study by a person or team other than the authors, using, in part, artifacts provided by the author. |
12
+
Authors can apply for, and be awarded, one of three combinations for their artifacts and associated papers:
16
13
14
+
|<br>**Available, Functional, and Reproduced**| For the vast majority of software artifacts,<br>and for hardware artifacts whenever possible. |
15
+
|<br>**Available and Functional**| For data sets,<br>as well as artifacts that require custom environments authors can't give access to. |
16
+
|<br>**Functional and Reproduced**| For software and hardware artifacts that the authors cannot make public. |
17
17
18
-
## Checklists
19
-
20
-
Unfortunately, artifacts sometimes miss badges because they were not tested on a clean setup, or not documented enough, or because running experiments is too error-prone due to complex manual steps.
21
-
Below we provide checklists for authors to minimize the risk of an artifact unnecessarily missing a badge.
22
-
23
-
24
-
25
-
### Artifact Available
26
-
27
-
- The artifact is available on a public website with a specific version such as a git commit
28
-
- The artifact has a "read me" file with a reference to the paper
29
-
- The artifact has an associated license and ideally one that at least allows use for comparison purposes
30
-
18
+
Artifacts submitted for the first target may be awarded one of the other two if availability or reproducibility evaluations fail, respectively.
19
+
Authors cannot apply for other combinations of badges as these make little sense, such as "an artifact that is not public, does not appear functional, but outputs the right numbers".
31
20
32
-
Artifacts must meet these criteria _at the time of evaluation_.
33
-
Promises of future availability, such as artifacts "temporarily" gated behind credentials given to evaluators, do not qualify for the badge.
34
21
22
+
## Checklists
35
23
36
-
### Artifact Functional
37
-
38
-
- The artifact has a "read me" file with high-level documentation:
39
-
- A description, such as which folders correspond to code, benchmarks, data, ...
40
-
- A list of supported environments, including OS, specific hardware if necessary, ...
41
-
- Compilation and running instructions, including dependencies and pre-installation steps,
42
-
with a reasonable degree of automation such as scripts to download and build exotic dependencies
43
-
- Configuration instructions, such as selecting IP addresses or disks
44
-
- Usage instructions, such as analyzing a new data set
45
-
- Instructions for a "minimal working example"
46
-
- The artifact has documentation explaining the high-level organization of modules, and code comments explaining non-obvious code,
47
-
such that other researchers can fully understand it
48
-
- The artifact contains all components the paper describes using the same terminology as the paper, and no obsolete code/data
49
-
- If the artifact includes a container/VM, it must also contain a script to create it from scratch
50
-
51
-
Artifacts must be usable on other machines than the authors', though they may require hardware such as specific network cards.
52
-
Information such as IP addresses must not be hardcoded.
53
-
54
-
55
-
### Results Reproduced
56
-
57
-
- The artifact has a "read me" file that documents:
58
-
- The exact environment the authors used, including OS version and any special hardware
59
-
- The exact commands to run to reproduce each claim from the paper
60
-
- The approximate resources used per claim, such as "5 minutes, 1 GB of disk space"
61
-
- The scripts to reproduce claims are documented, allowing researchers to ensure they correspond to the claims;
62
-
merely producing the right output is not enough
63
-
- The artifact's external dependencies are fetched from well-known sources such as official websites or GitHub repositories
64
-
- Changes to such dependencies should be clearly separated, such as using a patch file or a repository fork with a clear commit history
65
-
66
-
The amount of manual work, such as writing configuration files, should be reasonably minimized.
67
-
In particular, there should be no redundant manual steps such as writing the same configuration values in multiple places, as this inevitably leads to human error.
24
+
To provide fair evaluation across artifacts, to help authors prepare, and to help evaluators work efficiently, each badge has an associated checklist.
25
+
26
+
### "Available" checklist
27
+
28
+
- The artifact is available on a **public archive with irrevocable versioning and long-term storage**, such as Zenodo but not GitHub
29
+
- The artifact has a **license that allows comparison and extension**, such as the [CC-BY](https://creativecommons.org/licenses/by/4.0/) or [MIT](https://opensource.org/license/mit/) licenses
30
+
- The artifact has a **"read me" file referencing the paper**
31
+
32
+
These criteria must be met *at the time artifact evaluation finishes*.
33
+
Authors only need to put the data in long-term storage once evaluators are otherwise satisfied. Development may take place on a platform like GitHub.
34
+
Promises of future availability are *not* acceptable, such as uploading the artifact to a private repository with the goal of "eventually" making it public.
35
+
36
+
### "Functional" checklist
37
+
38
+
- The artifact has a **"read me" file** with:
39
+
- A description of each artifact component and how it relates to the paper
40
+
- A description of the exact environment the authors used, such as OS version and hardware
41
+
- If the artifact includes code that deliberately performs malicious or destructive operations, appropriate warnings and context
42
+
- The artifact includes **all code and data relevant to the paper**, and only those
43
+
- The artifact must not include obsolete or unrelated code nor data
44
+
- If existing code or data has been modified, the artifact should clearly separate the modifications from the original
45
+
- If the paper makes soundness claims, such as proofs, there should be simple scripts to verify these, such as listing proof assumptions
46
+
- If the paper makes quantifiable claims, such as code size per module, there should be simple scripts to output these
47
+
- For data, **modifications made to the raw data are documented**
48
+
- For instance, whether parts of the raw data were anonymized or discarded
49
+
- For executable artifacts, the "read me" file also contains **documentation** to:
50
+
- Run and extend a "minimal working example"
51
+
- Compile and execute the artifact, including pre-installation steps
52
+
- Configure the artifact, such as selecting IP addresses or disks
53
+
- Know the expected resource use per kind of experiment, such as "5 minutes, 10 GB of disk space"
54
+
- Know what unusual behavior to expect, such as warning messages emitted by another system used as baseline for experiments
55
+
- For executable artifacts, the artifact includes a **precise list of dependencies**:
56
+
- Whenever possible, it should be usable by a package manager
57
+
- Exotic dependencies must have associated automation to download and build them
58
+
- OS-level dependencies must involve a VM/container, accompanied by a script to generate the VM/container
59
+
- Proprietary dependencies must have associated instructions to obtain them along with "mock" versions to demonstrate their use
60
+
- The artifact includes an **example input and configuration for each kind of experiment** in the paper
61
+
- Authors are encouraged, but not required, to provide inputs, configurations, and outputs for all experiments described in the paper
62
+
63
+
Artifacts must be usable on other environments than the authors', though software may require specific hardware such as one model of network card.
64
+
Manual work such as writing configuration files must be minimized. There must be no redundant manual steps such as writing the same configuration values in multiple places, as this inevitably leads to human error.
65
+
66
+
### "Reproduced" checklist
67
+
68
+
- The artifact includes a **single script to run each experiment** and output results, given the necessary input and configuration
69
+
- The scripts must be documented, allowing researchers to ensure they correspond to the claims, merely producing the right output is not enough
70
+
- The scripts must handle common edge cases in a reasonable fashion, such as forgetting arguments or running the same script twice
71
+
- The artifact includes a **script to convert each experiment's results into human-readable ones** as close to the paper presentation as possible
72
+
- For simple results presentation such as tables, this and the previous script can be merged into one
73
+
- The artifact may contain separate installation steps for the dependencies of plotting scripts, subject to the same criteria
74
+
75
+
The expected workflow for an evaluator or a researcher looking to reuse the artifact is to install the artifact using a handful of commands, run experiments with one command each, and plot data as necessary.
76
+
In the absence of problems requiring debugging, active time must not exceed a few minutes.
Copy file name to clipboardExpand all lines: _conferences/eurosys2026/packaging.md
+3-82Lines changed: 3 additions & 82 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,18 +6,14 @@ order: 30
6
6
Artifacts must be packaged to ease evaluation and use, including [instructions](#instructions) for the artifact and an [artifact appendix](#artifact-appendix).
7
7
Packaging is not only about evaluation but about future use of the artifact by other researchers who may want to build on top of it or use it as a baseline.
8
8
9
-
We provide a few guidelines about packaging the artifact below.
10
9
If you have further questions about how best to package your artifact, contact the AEC chairs at [aec-2026@eurosys.org](mailto:aec-2026@eurosys.org).
11
10
12
-
*Note*: Some artifacts may attempt to perform malicious or destructive operations by design.
13
-
These cases should be boldly and explicitly flagged in detail at submission time (in the artifact instructions and appendix) so the AEC can take appropriate precautions before installing and running these artifacts. Please contact the AEC chairs if you believe that your artifacts fall into this category.
14
-
15
11
16
12
## Instructions
17
13
18
-
Your artifact package must include an obvious "read me" document containing suitable instructions and documentation.
19
-
A tool without a quick tutorial is generally very difficult to use. Similarly, a dataset is useless without some explanation on how to browse the data.
20
-
Please see the [badges](badges) page for more details on what the instructions should contain.
14
+
Follow the packaging guide for artifact authors [here](/packaging-guide).
15
+
16
+
Also see the [badges](badges) page for more details on what the instructions should contain.
21
17
22
18
23
19
## Artifact Appendix
@@ -39,78 +35,3 @@ The intention for the artifact appendix is to be published in conjunction with y
39
35
A template for the artifact appendix can be found here: [LaTeX Template](appendix/EuroSys26_ArtifactAppendix_template.tex) (to be used in conjuction with the EuroSys'26 template for research papers).
40
36
41
37
**Artifact Appendices are limited to 2 pages.**
42
-
43
-
## Formats
44
-
45
-
Authors should consider one of the following methods to package the software components of their artifacts
46
-
(although the AEC is open to other reasonable formats as well):
47
-
48
-
-*Container/virtual machine:* This is the preferred method. We recommend using a format that is easy for evaluators to work with, such as Docker images or an OVF virtual machine (e.g., to run in VirtualBox).
49
-
An AWS EC2 instance is also possible. In any case, the Dockerfile or script used to initialize the virtual machine should be available.
50
-
A Docker image or virtual machine should already be set up with the right toolchain and runtime environment. For example:
51
-
- For raw data, the container/VM would contain the data and the analysis scripts.
52
-
- For mechanized proofs, the container/VM would contain the right version of the relevant theorem prover
53
-
- For a mobile phone application, the VM would have a phone emulator installed
54
-
55
-
-*Source code:* If your artifact has few dependencies and can be installed easily on several operating systems,
56
-
you may submit source code and build scripts. However, if your artifact has a long list of dependencies, please use one of the other formats below.
57
-
58
-
-*Live instance on the web:* Ensure that it is available during the artifact evaluation process.
59
-
60
-
-*Internet-accessible hardware:* If your artifact requires special hardware (e.g., SGX or another trusted execution environment), or if your artifact is actually a piece of hardware, please make sure that evaluators can access the device. SSH or VPN-based access to the device might be an option. Authors must ensure anonymity of the reviewers while evaluating the artifacts. No web-forms or access requests that require the reviewers personal details are an acceptable way for giving access to remote infrastructure. An example approach for SSH access could be that of creating a single user (e.g., `eurosys-aec-review`) on the target machine and then collecting reviewers' SSH keys for granting access. No one technique would serve all purposes, and we leave the final choice to the authors. Should you have issues in granting access to your remote infrastructure, please contact the AEC chairs at [aec-2026@eurosys.org](mailto:aec-2026@eurosys.org) as soon as possible.
61
-
62
-
-*Screencast:* A detailed screencast of the tool along with the results can be an option if one of the following special cases applies:
63
-
- The artifact needs proprietary/commercial software or proprietary data that is not easily available or cannot be distributed to the committee.
64
-
- The artifact requires significant computation resources (e.g., more than 24 hours of execution time to produce the results) or requires huge data sets.
65
-
- The artifact requires specific hardware or software that is not generally available in a typical lab and where no access can be provided in a reasonable way.
66
-
- If your artifact falls in this category please contact the AEC chairs at [aec-2026@eurosys.org](mailto:aec-2026@eurosys.org) as soon as possible.
67
-
68
-
69
-
70
-
## Tools
71
-
72
-
[GitHub](https://github.com) and [GitLab](https://gitlab.com) are good options to host a Git repository for your artifact.
73
-
[Zenodo](https://zenodo.org) provides long-term storage and DOI for a specific version, which is useful once your artifact has been evaluated.
74
-
[Docker](https://docs.docker.com/get-started/overview/) allows you to create a lightweight container with all of your artifact's dependencies, and even write scripts to manage multiple containers locally instead of using a cloud provider.
75
-
76
-
Please see the [tips](#tips) section for specific tips.
77
-
78
-
There are also a growing number of tools and mechanisms that are designed specifically to meet the needs of research reproducibility; authors may want to consider using such tools when appropriate. A partial list includes:
A framework for packaging and sharing research artifacts on the [Chameleon Cloud](https://www.chameleoncloud.org/) facility. Chameleon Cloud supports a wide range of [hardware and software environments](https://chameleoncloud.org/hardware/) (please **contact the AEC Chairs beforehand** if you plan to use Chameleon Cloud). Check out a demo video [here](https://www.youtube.com/watch?v=grUOrKkiYuQ).
81
-
-*[CloudLab Profiles](https://docs.cloudlab.us/repeatable-research.html):* A mechanism for encapsulating and sharing research environments on the [CloudLab](https://cloudlab.us) facility (please **contact the AEC Chairs beforehand** if you plan to use CloudLab)
82
-
-*[Popper](https://getpopper.io/):* A container-native system for automating workflow
83
-
84
-
## Tips
85
-
86
-
*We thank the EuroSys'22 AEC chairs for the materials below.*
87
-
88
-
The following guides will be useful when preparing your artifact:
89
-
-[HOWTO for AEC Submitters](https://docs.google.com/document/d/1pqzPtLVIvwLwJsZwCb2r7yzWMaifudHe1Xvn42T4CcA/edit),
90
-
by Dan Barowy, Charlie Curtsinger, Emma Tosch, John Vilk, and Emery Berger
91
-
-[Artifact Evaluation: Tips for Authors](https://blog.padhye.org/Artifact-Evaluation-Tips-for-Authors/),
92
-
by Rohan Padhye
93
-
94
-
You may also find these examples of past artifacts useful:
95
-
-[Bundler](https://github.com/bundler-project/evaluation), a middlebox and its multi-machine benchmarks (EuroSys'21)
96
-
-[Serval](https://unsat.cs.washington.edu/projects/serval/sosp19-artifact.html), a verification tool with correct and buggy code to test it (SOSP'19)
97
-
-[TinyNF](https://github.com/dslab-epfl/tinynf), a network driver with low-level multi-machine benchmarks (OSDI'20)
98
-
99
-
100
-
Here are some general tips to make life easier for both artifact authors and evaluators:
101
-
102
-
-**Automate as much as possible**, you will save a lot of time in the end from not having to re-run experiments that suffered from human error.
103
-
This is feasible even for artifacts that need multiple nodes or to replicate configuration in multiple places.
104
-
See [this repository](https://github.com/SolalPirelli/docker-artifact-eval) for an example of artifact automation with Docker.
105
-
106
-
-**Try out your own artifact on a blank environment**, following the steps you documented.
107
-
One lightweight way to do this is to create a Docker container from a base OS image, such as `ubuntu:latest`.
108
-
You can also use a virtual machine or even provision a real machine if you have the infrastructure to do so.
109
-
110
-
-**Log both successes and failures**, so that users know that something happened.
111
-
Avoid logging unnecessary or confusing information, such as verbose output or failures that are actually expected.
112
-
Log potential issues, such as an optional but recommended library not being present.
113
-
114
-
-**Measure resource use** using tools such as `mpstat`, `iostat`, `vmstat`, and `ifstat` to measure CPU, I/O, memory, and network use respectively on Linux,
115
-
or `/usr/bin/time -v` to measures the time and memory used by a command also on Linux.
0 commit comments