Skip to content

Commit f8abb70

Browse files
bastoicabastoica
authored
feat: adding a new systems+AI conferece, ACM CAIS (https://www.caisconf.org/), which provides artifact evaluation (#152)
1 parent e8eaee2 commit f8abb70

9 files changed

Lines changed: 229 additions & 0 deletions

File tree

‎_conferences/cais.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
title: CAIS
3+
---
4+
5+
The [ACM Conference on AI and Agentic Systems](https://www.caisconf.org/) is organized by the
6+
is the premier venue for rigorous, reproducible research on compound
7+
AI architectures, optimization, and deployment.
8+
Research artifacts were first evaluated in 2026.

‎_conferences/cais2026/aec-call.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
title: Call for AEC Members
3+
order: 60
4+
---
5+
6+
We are looking for reviewers who can provide detailed, constructive evaluations of 1-2 artifacts during the ACM CAIS'26 AE [review period](call). AEC members evaluate artifact quality, reproducibility, and relevance; provide clear feedback to help authors improve their work; and participate in committee discussions on artifact badge decisions. In return, they gain early insight into emerging research, opportunities to connect with peers in the community, and recognition for their service on the ACM CAIS [website](https://www.caisconf.org/) and, potentially, in the conference proceedings.
7+
8+
How to Apply
9+
------------
10+
11+
If you are interested in taking part in the AEC, please nominate yourself using [this form](https://forms.gle/tc7xca6PaYFqyYqW7) by **Sunday, April 12, 2026**.
12+
13+
You can reach us at [aec-chairs@caisconf.org](mailto:aec-chairs@caisconf.org) with any questions.
14+
15+
AEC responsabilities
16+
--------------------
17+
18+
As an AEC member, you will contribute to the reproducibility of experimental results in systems research by evaluating artifacts associated with papers accepted for publication at ACM CAIS'26. For each artifact, you will be asked to evaluate its public availability, functionality, and ability to reproduce the results reported in the paper. You will be able to discuss artifacts with other AEC members and interact anonymously with the authors when necessary. Finally, you will submit a review with constructive feedback, discuss the artifact with fellow reviewers, and help determine artifact evaluation badge outcomes.
19+
20+
We expect each AEC member to evaluate 1-2 artifacts, and we estimate that each evaluation will take around 10-20 hours. AEC members are expected to allocate time to indicate their artifact preferences and, once assigned, read the corresponding papers, evaluate the artifacts, and participate in online discussions through each notification deadline. Please ensure that you have sufficient time and availability during the AEC [reviewing period](call) and can carry out the evaluation confidentially and independently, without sharing artifacts or related information outside the AEC.
21+
22+
We expect evaluators to be able to use conference-provided cloud resources when available, particularly for artifacts that require substantial compute capacity or specialized hardware. Evaluators may also use their own machines when artifacts support local execution (including via containers or virtual machines, where appropriate). If neither option is suitable, authors may provide remote access to their own systems (e.g., via SSH) with proper anonymization. Commercial cloud services should be used only as a last resort and only with prior coordination with the authors and conference organizers to minimize unnecessary costs.

‎_conferences/cais2026/badges.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
---
2+
title: Badges
3+
order: 10
4+
---
5+
6+
<style>
7+
img { width: 4em; }
8+
</style>
9+
10+
CAIS is an ACM conference and thus uses [ACM's badges](https://www.acm.org/publications/policies/artifact-review-and-badging-current). Authors can apply for, and be awarded, one of following three badge combinations:
11+
12+
| ![Available](/images/acm_available_1.1.png) ![Functional](/images/acm_functional_1.1.png) ![Reproduced](/images/acm_reproduced_1.1.png)<br>**Available, Functional, and Reproduced** | For the vast majority of software artifacts,<br>and for hardware artifacts whenever possible. |
13+
| ![Available](/images/acm_available_1.1.png) ![Functional](/images/acm_functional_1.1.png)<br>**Available and Functional** | For data sets,<br>as well as artifacts that require custom environments authors can't give access to. |
14+
| ![Functional](/images/acm_functional_1.1.png) ![Reproduced](/images/acm_reproduced_1.1.png)<br>**Functional and Reproduced** | For software and hardware artifacts that the authors cannot make public. |
15+
16+
Artifacts submitted for the first target may be awarded one of the other two if availability or reproducibility evaluations fail, respectively. Authors cannot apply for other combinations of badges as these make little sense (e.g., an artifact that is not public, does not appear functional, but outputs the expected measurements).
17+
18+
19+
## Checklists
20+
21+
To provide a fair evaluation across artifacts, help authors prepare, and enable evaluators to work efficiently, each badge has an associated checklist.
22+
23+
### "Available" checklist
24+
25+
- The artifact is available on a **public archive with irrevocable versioning and long-term storage**, such as Zenodo but not GitHub
26+
- The artifact has a **license that allows comparison and extension**, such as the [CC-BY](https://creativecommons.org/licenses/by/4.0/) or [MIT](https://opensource.org/license/mit/) licenses
27+
- The artifact has a **"read me" file referencing the paper**
28+
29+
Note that these criteria must be met *at the time artifact evaluation finishes*, but development may take place on a platform like GitHub. Authors only need to save their artifact to long-term storage once evaluators are otherwise satisfied. However, promises of future availability are *not* acceptable, such as uploading the artifact to a private repository with the goal of "eventually" making it public.
30+
31+
### "Functional" checklist
32+
33+
- The artifact has a **"read me" file** with:
34+
- A description of each artifact component and how it relates to the paper
35+
- A description of the exact environment the authors used, such as OS version and hardware
36+
- If the artifact includes code that deliberately performs malicious or destructive operations, appropriate warnings and context
37+
- The artifact includes **all code and data relevant to the paper**, and only those
38+
- The artifact must not include obsolete or unrelated code nor data
39+
- If existing code or data has been modified, the artifact should clearly separate the modifications from the original
40+
- If the paper makes soundness claims, such as proofs, there should be simple scripts to verify these, such as listing proof assumptions
41+
- If the paper makes quantifiable claims, such as code size per module, there should be simple scripts to output these
42+
- For data, **modifications made to the raw data are documented**
43+
- For instance, whether parts of the raw data were anonymized or discarded
44+
- For executable artifacts, the "read me" file also contains **documentation** to:
45+
- Run and extend a "minimal working example"
46+
- Compile and execute the artifact, including pre-installation steps
47+
- Configure the artifact, such as selecting IP addresses or disks
48+
- Know the expected resource use per kind of experiment, such as "5 minutes, 10 GB of disk space"
49+
- Know what unusual behavior to expect, such as warning messages emitted by another system used as baseline for experiments
50+
- For executable artifacts, the artifact includes a **precise list of dependencies**:
51+
- Whenever possible, it should be usable by a package manager
52+
- Exotic dependencies must have associated automation to download and build them
53+
- OS-level dependencies must involve a VM/container, accompanied by a script to generate the VM/container
54+
- Proprietary dependencies must have associated instructions to obtain them along with "mock" versions to demonstrate their use
55+
- The artifact includes an **example input and configuration for each kind of experiment** in the paper
56+
- Authors are encouraged, but not required, to provide inputs, configurations, and outputs for all experiments described in the paper
57+
58+
Manual work such as writing configuration files must be minimized. There must be no redundant manual steps such as writing the same configuration values in multiple places, as this inevitably leads to human error. Also, artifacts must be usable on other environments than the authors', though software may require specific hardware (e.g., a particular network card, a specific GPU, etc.).
59+
60+
### "Reproduced" checklist
61+
62+
- The artifact includes a **single script to run each experiment** and output results, given the necessary input and configuration
63+
- The scripts must be documented, allowing researchers to ensure they correspond to the claims, merely producing the right output is not enough
64+
- The scripts must handle common edge cases in a reasonable fashion, such as forgetting arguments or running the same script twice
65+
- The artifact includes a **script to convert each experiment's results into human-readable ones** as close to the paper presentation as possible
66+
- For simple results presentation such as tables, this and the previous script can be merged into one
67+
- The artifact may contain separate installation steps for the dependencies of plotting scripts, subject to the same criteria
68+
69+
The expected workflow for an evaluator or a researcher looking to reuse the artifact is to install the artifact using a handful of commands, run experiments with minimal effort (ideally running a single script per experiment or group of experiments), and display experimental data as necessary. In absence of issues requiring in-depth debugging, active time must not exceed a few minutes.

‎_conferences/cais2026/call.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
---
2+
title: Call for Artifacts
3+
order: 20
4+
---
5+
6+
All papers accepted at ACM CAIS'26 are *encouraged to participate in the artifact evaluation* process.
7+
8+
Artifacts must be consistent with the paper, as complete as possible, reasonably well documented, and easy to reuse. These goals are reflected in the three [badges](badges) that can be awarded to each paper: Available, Functional, and Results Reproduced (or Reproduced, for short). The purpose of the AEC is to help authors meet these goals and to award badges to artifacts that satisfy the criteria. Note that for ACM CAIS'26, the AE is *single-blind* (see below).
9+
10+
Questions about artifact evaluation can be directed to [aec-chairs@caisconf.org](mailto:aec-chairs@caisconf.org).
11+
12+
To be considered in the AE process, at least one contact author for the submission must be reachable and respond to questions in a timely manner during the evaluation period, allowing sufficient back-and-forth between the AEC and the authors. Please check the AEC timeline and important dates [here](dates).
13+
14+
15+
## Registration and Submission
16+
17+
**Link to HotCRP portal:** [https://acm-cais26-ae.hotcrp.com/](https://acm-cais26-ae.hotcrp.com/)
18+
19+
Please submit your artifacts to the [AE HotCRP portal](https://acm-cais26-ae.hotcrp.com/) by providing an URL or a packaged artifact, selecting which artifact badges you apply for, and provide an artifact appendix that describes the artifact.
20+
21+
The effort that you put into packaging your artifacts has a direct impact on the committee's ability to make well-informed decisions.
22+
Please package your artifacts with care to make it as straightforward and easy as possible for the AEC to understand and evaluate their quality.
23+
24+
*Note*: If you need permission from your organization's legal or IT department to publish your artifact or give evaluators access to custom hardware, submit that request as soon as possible, otherwise evaluators may not have sufficient time to audit your artifact.
25+
26+
27+
## Process
28+
29+
Authors are invited to submit artifacts shortly after their papers are accepted. Because the time between paper acceptance and artifact submission is short, the AEC Chairs encourage authors to begin preparing their artifacts while their papers are still under review.
30+
31+
At the time of artifact submission, authors choose which badges they want to pursue. Please read instructions and criteria, [here](badges).
32+
33+
After the artifact submission deadline, evaluators will review each artifact using the corresponding paper and artifact appendix as guides. Evaluators may communicate with authors *exclusively through HotCRP* (to preserve anonymity) to resolve minor issues and ask clarifying questions throughout the evaluation process. Evaluation begins with a "kick-the-tires" period, during which evaluators confirm that they can access their assigned artifacts and perform basic operations, such as compiling and running a minimal working example. Artifact evaluations include feedback, allowing authors to improve both their artifacts and their papers based on that feedback.
34+
35+
36+
## Artifact Details
37+
38+
Artifacts can include software, datasets, survey results, test suites, mechanized proofs, and similar materials. Pen-and-paper proofs are not accepted, as evaluators often lack the time and expertise to review them carefully. Physical objects, such as computer hardware, also cannot be accepted because they are difficult to make available to evaluators. To the extent possible, artifacts should be able to run on commodity hardware (e.g., laptop or desktop systems). If this requirement cannot be met, please contact the AEC Chairs in advance so that arrangements can be made for evaluators to access special hardware (e.g., your own). More detailed artifact packaging instructions are available [here](packaging).
39+
40+
When submitting your artifact, please specify which combination of the [three badges](badges) you are applying for. For the Functional and Reproduced badges, AEC members will attempt to use your artifact to run the experiments described in your paper.
41+
42+
Submitting an artifact for evaluation does **not** give the AEC permission to make its contents public or to retain any part of it after the evaluation. Thus, authors are free to include proprietary models, data files, or code in their artifacts. Participating in artifact evaluation does not require public release of the artifact, though public release is highly encouraged.
43+
44+
AEC members may contact authors during the evaluation period, for example, to ask for help if they are unable to get the artifact to work, and authors are expected to respond to such requests. However, your goal as an author should be to present and document your artifact so that AEC members can use it and complete the evaluation successfully on their own (ideally without needing to interact with the authors). To ensure that your instructions are complete, we recommend testing them on a fresh setup before submission, following exactly the instructions you provide.
45+
46+
47+
## Reviewing and Anonymity
48+
49+
Artifact evaluation is "***single-blind***", meaning that the identities of authors will be known to reviewers, but authors will not know which Artifact Evaluation Committee (AEC) members reviewed their artifacts.
50+
51+
To maintain reviewer anonymity, authors should not embed analytics or other tracking tools in the websites hosting their artifacts for the duration of the artifact evaluation period. This helps maintain reviewer confidentiality. In cases where tracking is unavoidable, authors should notify the AEC Chairs in advance so that AEC members can take adequate precautions

‎_conferences/cais2026/dates.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
title: Important Dates
3+
order: 25
4+
---
5+
6+
#### Important dates
7+
- Acceptance notification to paper authors: **Tue, April 21, 2026**
8+
- Artifact submission deadline: **Thu, April 23, 2026 (AoE)**
9+
- Kick-the-tires response period: **Thu, April 23 - Mon, April 27, 2026**
10+
- Camera-ready deadline: **Mon, April 27, 2026 (AoE)**
11+
- Artifact decisions announced: **Fri, May 8, 2026**

‎_conferences/cais2026/index.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
title: Artifact Evaluation
3+
order: 0
4+
---
5+
6+
7+
The Artifact Evaluation (AE) process is a community service intended to strengthen the long-term value of accepted papers by encouraging authors to provide substantial supplementary materials for review. These artifacts enable future researchers to reproduce, compare, and extend prior work more effectively. ACM CAIS invites authors of accepted papers to submit their artifacts for evaluation. Research artifacts are digital objects developed by the authors for use in the reported study or generated during the experimental process.
8+
9+
Artifact evaluation is optional at [ACM CAIS'26](https://www.caisconf.org/). Authors of accepted papers may submit their artifacts shortly after receiving the acceptance notification, but they are strongly encouraged to prepare these materials in advance. If an artifact passes evaluation, the paper will receive one or more official ACM AE badges displayed on its first page, and the final camera-ready version must include an appendix describing the artifact. For more information about the AE timeline, see the [important dates](dates) page.
10+
11+
The AE process for ACM CAIS'26 is *single-blind*, meaning that reviewers will know the identities of the authors, but authors will not know which Artifact Evaluation Committee (AEC) members reviewed their artifacts. To preserve reviewer anonymity, authors should not embed analytics or other tracking tools in websites hosting their artifacts during the artifact evaluation period. In cases where tracking is unavoidable, authors should notify the AEC Chairs in advance so that AEC members can take appropriate precautions.

‎_conferences/cais2026/organizers.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
---
2+
title: Organizers
3+
order: 50
4+
---
5+
6+
## Artifact Evaluation Chairs
7+
8+
* [Bogdan "Bo" Stoica](https://bastoica.github.io/) (University of Illinois Urbana-Champaign, USA)
9+
10+
You can reach the AEC chairs at [aec-chairs@caisconf.org](mailto:aec-chairs@caisconf.org).
11+
12+
## Artifact Evaluation Committee
13+
14+
*TBD*

‎_conferences/cais2026/packaging.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
---
2+
title: Artifact Packaging
3+
order: 30
4+
---
5+
6+
Artifacts must be packaged to ease evaluation and use, including [instructions](#instructions) for the artifact and an [artifact appendix](#artifact-appendix).
7+
Packaging is not only about evaluation but about future use of the artifact by other researchers who may want to build on top of it or use it as a baseline.
8+
9+
If you have further questions about how best to package your artifact, contact the AEC chairs at [aec-chairs@caisconf.org](mailto:aec-chairs@caisconf.org).
10+
11+
12+
## Instructions
13+
14+
Follow the packaging guide for artifact authors [here](/packaging-guide).
15+
16+
Also see the [badges](badges) page for more details on what the instructions should contain.
17+
18+
19+
## Artifact Appendix
20+
21+
22+
The artifact appendix is a self-contained document up to **2 pages** that provides a roadmap for evaluators. In particular, it should include a description of the hardware, software, and configuration requirements, as well as a **list of the major claims** made in the paper that can be reproduced using your artifact.
23+
24+
Linking the paper's claims to the artifact is a necessary step that enables evaluators to reproduce your results. It is especially important to state your paper's key results and claims clearly. You should also make your claims about the artifact concrete. This is particularly important if those claims differ from the expectations set by your paper.
25+
26+
The AEC members will still evaluate your artifact relative to your paper, but your explanation can help set expectations up front, especially in cases that might otherwise frustrate evaluators. For example, inform the AEC about difficulties they might encounter when using the artifact or about the artifact's maturity relative to the content of the paper.
27+
28+
An artifact appendix must provide details on the time and hardware resources (e.g., storage) required to run the experiments in your paper. If possible, the appendix should also describe how to compare the results of a reproduced experiment with those reported in the paper (e.g., by providing access to the underlying dataset).
29+
30+
The intention for the artifact appendix is to be published in conjunction with your artifact, using an ACM official [LaTeX template](https://www.acm.org/publications/proceedings-template). For further information, please check ACM CAIS'26 [formatting requirements](https://www.caisconf.org/pages/cfp/).

0 commit comments

Comments
 (0)