Skip to content

Commit d300070

Browse files
SolalPirelliSolalPirelli
authored
Add EuroSys 2027 (#179)
1 parent a3cdcc9 commit d300070

4 files changed

Lines changed: 187 additions & 0 deletions

File tree

‎_conferences/eurosys2027/aec-call.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
---
2+
title: Call for Evaluators
3+
order: 20
4+
---
5+
6+
Join the Artifact Evaluation committee, learn about the latest research before everyone else, and help make science more reusable and reproducible!
7+
8+
**Apply now**: [fill this self-nomination form](https://forms.gle/ovEbCPgViT1s9UCW8) (deadline: **August 21st, 2026 (AoE)**)
9+
10+
You can be located anywhere in the world as all committee discussions will happen online.
11+
AEC membership is especially suitable if you are early in your career, such as the first or second year of your PhD.
12+
You are welcome to join the AEC if you are working in a topic area covered by the conference such as
13+
operating systems, file and storage systems, distributed systems, cloud computing, and other areas of computer systems research.
14+
15+
As an artifact evaluator, you will:
16+
- Read the papers whose artifacts you are assigned to review
17+
- Install, audit, and run artifacts
18+
- Discuss with other evaluators to award [badges](./badges)
19+
20+
Check out the [evaluator guide](/evaluator-guide) for an overview of what artifact evaluation involves.
21+
You must carry out evaluations independently, without sharing data nor information with people outside the committee.
22+
23+
Please ensure you have available time during the evaluation period stated in the [home page](./index)'s important dates section.
24+
We hope to assign 2 artifacts per evaluator, each taking around 5h to review.
25+
26+
This work can usually be done on your own computer, any moderately recent desktop or laptop will do.
27+
In specific cases and to the extent possible, authors will arrange their artifacts so as to run in community research testbeds or will provide remote access to their systems.
28+
If you have access to special hardware, that might come in handy, but it is by no means required.
29+
30+
Contact the [chairs](./committee) if you have any questions.

‎_conferences/eurosys2027/badges.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
---
2+
title: Badges
3+
order: 10
4+
---
5+
6+
<style>
7+
img { width: 4em; }
8+
</style>
9+
10+
EuroSys is an ACM conference and thus uses [ACM's badges](https://www.acm.org/publications/policies/artifact-review-and-badging-current).
11+
12+
Authors can apply for, and be awarded, one of three combinations for their artifacts and associated papers:
13+
14+
| ![Available](/images/acm_available_1.1.png) ![Functional](/images/acm_functional_1.1.png) ![Reproduced](/images/acm_reproduced_1.1.png)<br>**Available, Functional, and Reproduced** | For the vast majority of software artifacts,<br>and for hardware artifacts whenever possible. |
15+
| ![Available](/images/acm_available_1.1.png) ![Functional](/images/acm_functional_1.1.png)<br>**Available and Functional** | For data sets,<br>as well as artifacts that require custom environments authors can't give access to. |
16+
| ![Functional](/images/acm_functional_1.1.png) ![Reproduced](/images/acm_reproduced_1.1.png)<br>**Functional and Reproduced** | For software and hardware artifacts that the authors cannot make public. |
17+
18+
Artifacts submitted for the first target may be awarded one of the other two if availability or reproducibility evaluations fail, respectively.
19+
Authors cannot apply for other combinations of badges as these make little sense, such as "an artifact that is not public, does not appear functional, but outputs the right numbers".
20+
21+
22+
## Checklists
23+
24+
To provide fair evaluation across artifacts, to help authors prepare, and to help evaluators work efficiently, each badge has an associated checklist.
25+
26+
### "Available" checklist
27+
28+
- The artifact is available on a **public archive with irrevocable versioning and long-term storage**, such as Zenodo but not GitHub
29+
- The artifact has a **license that allows comparison and extension**, such as the [CC-BY](https://creativecommons.org/licenses/by/4.0/) or [MIT](https://opensource.org/license/mit/) licenses
30+
- The artifact has a **"read me" file referencing the paper**
31+
32+
These criteria must be met *at the time artifact evaluation finishes*.
33+
Authors only need to put the data in long-term storage once evaluators are otherwise satisfied. Development may take place on a platform like GitHub.
34+
Promises of future availability are *not* acceptable, such as uploading the artifact to a private repository with the goal of "eventually" making it public.
35+
36+
### "Functional" checklist
37+
38+
- The artifact has a **"read me" file** with:
39+
- A description of each artifact component and how it relates to the paper
40+
- A description of the exact environment the authors used, such as OS version and hardware
41+
- If the artifact includes code that deliberately performs malicious or destructive operations, appropriate warnings and context
42+
- The artifact includes **all code and data relevant to the paper**, and only those
43+
- The artifact must not include obsolete or unrelated code nor data
44+
- If existing code or data has been modified, the artifact should clearly separate the modifications from the original
45+
- If the paper makes soundness claims, such as proofs, there should be simple scripts to verify these, such as listing proof assumptions
46+
- If the paper makes quantifiable claims, such as code size per module, there should be simple scripts to output these
47+
- For data, **modifications made to the raw data are documented**
48+
- For instance, whether parts of the raw data were anonymized or discarded
49+
- For executable artifacts, the "read me" file also contains **documentation** to:
50+
- Run and extend a "minimal working example"
51+
- Compile and execute the artifact, including pre-installation steps
52+
- Configure the artifact, such as selecting IP addresses or disks
53+
- Know the expected resource use per kind of experiment, such as "5 minutes, 10 GB of disk space"
54+
- Know what unusual behavior to expect, such as warning messages emitted by another system used as baseline for experiments
55+
- For executable artifacts, the artifact includes a **precise list of dependencies**:
56+
- Whenever possible, it should be usable by a package manager
57+
- Exotic dependencies must have associated automation to download and build them
58+
- OS-level dependencies must involve a VM/container, accompanied by a script to generate the VM/container
59+
- Proprietary dependencies must have associated instructions to obtain them along with "mock" versions to demonstrate their use
60+
- The artifact includes an **example input and configuration for each kind of experiment** in the paper
61+
- Authors are encouraged, but not required, to provide inputs, configurations, and outputs for all experiments described in the paper
62+
63+
Artifacts must be usable on other environments than the authors', though software may require specific hardware such as one model of network card.
64+
Manual work such as writing configuration files must be minimized. There must be no redundant manual steps such as writing the same configuration values in multiple places, as this inevitably leads to human error.
65+
66+
### "Reproduced" checklist
67+
68+
- The artifact includes a **single script to run each experiment** and output results, given the necessary input and configuration
69+
- The scripts must be documented, allowing researchers to ensure they correspond to the claims, merely producing the right output is not enough
70+
- The scripts must handle common edge cases in a reasonable fashion, such as forgetting arguments or running the same script twice
71+
- The artifact includes a **script to convert each experiment's results into human-readable ones** as close to the paper presentation as possible
72+
- For simple results presentation such as tables, this and the previous script can be merged into one
73+
- The artifact may contain separate installation steps for the dependencies of plotting scripts, subject to the same criteria
74+
75+
The expected workflow for an evaluator or a researcher looking to reuse the artifact is to install the artifact using a handful of commands, run experiments with one command each, and plot data as necessary.
76+
In the absence of problems requiring debugging, active time must not exceed a few minutes.

‎_conferences/eurosys2027/committee.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
---
2+
title: Committee
3+
order: 30
4+
---
5+
6+
**Chairs**, reachable at [aec-2027@eurosys.org](mailto:aec-2027@eurosys.org):
7+
- Redha Gouicem, RWTH Aachen
8+
- Solal Pirelli, EPFL
9+
10+
_Please do not contact the chairs to self-nominate for the committee; use the form instead._
11+
12+
**Members**:
13+
- TBA

‎_conferences/eurosys2027/index.md‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
---
2+
title: Artifact Evaluation
3+
order: 0
4+
---
5+
6+
A scientific paper consists of a constellation of artifacts that extend beyond the document itself:
7+
software, hardware, evaluation data and documentation, raw survey results, mechanized proofs, models, test suites, benchmarks, and so on.
8+
Based on the success of artifact evaluation for past editions, EuroSys 2027 will have an artifact evaluation process.
9+
10+
The artifact evaluation process considers the reusability and reproducibility of artifacts associated with papers.
11+
Artifact evaluation is _single blind_: authors do not know evaluators' identity, but evaluators know who the authors are.
12+
13+
**Call for evaluators**: Apply [here](./aec-call) to join the artifact evaluation committee!
14+
15+
**Artifact registration and submission**: TBA
16+
17+
18+
## Process
19+
20+
Artifact evaluation is a cooperative process, unlike paper reviewing, meaning that authors have the opportunity to fix problems found during evaluation if they can do so in reasonable time.
21+
22+
1. Authors decide which [badges and associated checklists](./badges) they target.
23+
2. Authors _register_ their artifact indicating which badges they target, along with the paper PDF and some metadata.
24+
3. Authors then _submit_ their artifact, packaged according to [the artifact packaging guide](/packaging-guide) using a service such as GitHub.
25+
4. Evaluators begin with a short "kick the tires" phase to double-check that the artifacts are complete and appear usable and contact authors if that is not the case.
26+
5. Evaluators proceed with the main evaluation period, operating mostly independently but occasionally contacting authors if an issue arises.
27+
6. Authors upload the final version of their artifact to permanent storage such as Zenodo and submit their artifact DOI.
28+
7. Authors are awarded badges which they add to their camera-ready paper submission.
29+
30+
31+
## Important Dates
32+
33+
### Spring Deadline
34+
35+
- Paper acceptance notification: **August 21, 2026**
36+
- Artifact submission deadline: **August 31, 2026 (AoE)**
37+
- Artifact "kick the tires" evaluation period ends: **September 8, 2026**
38+
- Artifact decisions: **September 21, 2026**
39+
- Paper camera-ready deadline: **September 25, 2026**
40+
41+
### Fall Deadline
42+
43+
- Paper acceptance notification: **January 29, 2027**
44+
- Artifact submission deadline: **TBA**
45+
- Artifact "kick the tires" evaluation period ends: **TBA**
46+
- Artifact decisions: **TBA**
47+
- Paper camera-ready deadline: **March 5, 2027**
48+
49+
50+
At least one author for each artifact submission must be reachable via email and respond to questions in a timely manner during the kick-the-tires period.
51+
Ideally, authors should also be available to respond to reasonable requests during the main evaluation period.
52+
53+
54+
## Awards
55+
56+
Artifacts that obtain all available badges are eligible to receive a Distinguished Artifact award.
57+
This award will be given to the artifacts that best exemplify reusability and reproducibility practices, including clear documentation, simple installation, ease of use, and extensibility.
58+
59+
The chairs may, at their discretion, award Distinguished Evaluator awards to evaluators who are particularly effective, friendly, and efficient.
60+
61+
62+
## Committee
63+
64+
Chairs:
65+
- Redha Gouicem, RWTH Aachen
66+
- Solal Pirelli, EPFL
67+
68+
Full membership list [here](./committee).

0 commit comments

Comments
 (0)