Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 42 additions & 31 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,52 +6,64 @@
[![Codecov][cov-badge]][cov-link] 
[![Downloads](https://static.pepy.tech/personalized-badge/wfcommons?period=total&units=international_system&left_color=grey&right_color=yellowgreen&left_text=Downloads)](https://pepy.tech/project/wfcommons)

<a href="https://wfcommons.org" target="_blank"><img src="https://wfcommons.org/images/wfcommons-horizontal.png" width="350" /></a>
<br/>_A Framework for Enabling Scientific Workflow Research and Development_
<a href="https://wfcommons.org" target="_blank"><img src="https://wfcommons.org/images/wfcommons-logo-white.png" width="350" /></a>
<br/>An open-source ecosystem of workflow execution instances, synthetic workflow generators, and benchmark specifications. It helps the community study scheduling, performance, resilience, and emerging AI-driven workflow automation on modern distributed and HPC platforms.

This Python package provides a collection of tools for:
- **Real instances:** Workflow executions curated in a common JSON format (WfFormat).
- **Synthetic realism:** Generate realistic workflows from real traces.
- **Benchmarks:** Produce executable specs for repeatable experiments and fair comparisons.

- Analyzing instances of actual workflow executions;
- Producing recipes structures for creating workflow recipes for workflow generation;
- Generating synthetic realistic workflow instances; and
- Generating realistic workflow benchmark specifications.
Quick links: [Documentation](https://wfcommons.readthedocs.io/en/latest/) · [Website](https://wfcommons.org) · [GitHub Issues](https://github.com/wfcommons/wfcommons/issues)

[![Open In Gitpod](https://gitpod.io/button/open-in-gitpod.svg)](https://gitpod.io/#https://github.com/wfcommons/wfcommons/tree/main)
## Quickstart

## Installation
WfCommons requires Python 3.11+ and has been tested on Linux and macOS.

WfCommons is available on [PyPI](https://pypi.org/project/wfcommons).
WfCommons requires Python3.11+ and has been tested on Linux and MacOS.
```
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install wfcommons
```

### Installation using pip
Generate a synthetic workflow in a few lines:

While `pip` can be used to install WfCommons, we suggest the following
approach for reliable installation when many Python environments are available:
```python
import pathlib
from wfcommons.wfchef.recipes import SeismologyRecipe
from wfcommons import WorkflowGenerator

```
$ python3 -m pip install wfcommons
generator = WorkflowGenerator(SeismologyRecipe.from_num_tasks(250))
workflow = generator.build_workflow()
workflow.write_json(pathlib.Path("seismology-workflow.json"))
```

### Retrieving the latest unstable version
Next steps:

If you want to use the latest WfCommons unstable version, that will contain
brand new features (but also contain bugs as the stabilization work is still
underway), you may consider retrieving the latest unstable version.
- Learn how to build recipes in the [WfChef guide](https://wfcommons.readthedocs.io/en/latest/generating_workflows_recipe.html).
- Generate larger workflow families in the [WfGen guide](https://wfcommons.readthedocs.io/en/latest/generating_workflows.html).
- Produce benchmark specs in the [WfBench guide](https://wfcommons.readthedocs.io/en/latest/generating_workflow_benchmarks.html).

Cloning from [WfCommons's GitHub](https://github.com/wfcommons/wfcommons)
repository:
## Installation

WfCommons is available on [PyPI](https://pypi.org/project/wfcommons).

```
python3 -m pip install wfcommons
```

### Installing from source (latest)

```
$ git clone https://github.com/wfcommons/wfcommons
$ cd wfcommons
$ pip install .
git clone https://github.com/wfcommons/wfcommons
cd wfcommons
python3 -m pip install .
```

### Optional Requirements

#### Graphviz
WfCommons uses _pygraphviz_ for generating visualizations for the workflow task graph.
If you want to enable this feature, you will have to install the
WfCommons uses _pygraphviz_ for generating visualizations for the workflow task graph.
If you want to enable this feature, you will have to install the
[graphviz](https://www.graphviz.org/) package (version 2.16 or later).
You can install graphviz easily on Linux with your favorite package manager,
for example for Debian-based distributions:
Expand Down Expand Up @@ -82,15 +94,14 @@ python3 -m pip install pydot

## Get in Touch

The main channel to reach the WfCommons team is via the support email:
The main channel to reach the WfCommons team is via the support email:
[support@wfcommons.org](mailto:support@wfcommons.org).

**Bug Report / Feature Request:** our preferred channel to report a bug or request a feature is via
**Bug Report / Feature Request:** our preferred channel to report a bug or request a feature is via
WfCommons's [Github Issues Track](https://github.com/wfcommons/wfcommons/issues).


## Citing WfCommons
When citing WfCommons, please use the following paper. You should also actually read
When citing WfCommons, please use the following paper. You should also actually read
that paper, as it provides a recent and general overview on the framework.

```
Expand Down
3 changes: 2 additions & 1 deletion docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
sphinx>=5.3.0
furo>=2024.5.6
sphinx_rtd_theme>=1.2.0
recommonmark>=0.7.1
jsonschema~=3.2.0
Expand All @@ -12,4 +13,4 @@ setuptools>=49.3.1
pyyaml>=5.3.1
pandas>=1.2.4
stringcase>=1.2.0
filelock>=3.6.0
filelock>=3.6.0
248 changes: 248 additions & 0 deletions docs/source/_static/css/custom.css
Original file line number Diff line number Diff line change
@@ -0,0 +1,248 @@
:root {
--content-max-width: 72rem;
--sidebar-width: 18rem;
--color-background-primary: #f7f6f2;
--color-background-secondary: #f0efe9;
--color-sidebar-background: #f0efe9;
--color-background-hover: #e9ecf2;
--color-background-border: #e1e4ee;
--color-brand-primary: #1f5eff;
--color-brand-content: #1a55e6;
--color-foreground-primary: #16181d;
--color-foreground-secondary: #4b5563;
--accent-warm: #ff7a59;
--accent-mint: #22c997;
--accent-sky: #4cc9f0;
--accent-violet: #9b5de5;
color-scheme: light;
}

article h1 {
letter-spacing: -0.01em;
}

article h2 {
letter-spacing: -0.005em;
}

body {
background:
radial-gradient(1200px 600px at 10% -10%, rgba(155, 93, 229, 0.12), transparent 60%),
radial-gradient(900px 500px at 90% -15%, rgba(76, 201, 240, 0.15), transparent 55%),
radial-gradient(700px 400px at 50% 110%, rgba(34, 201, 151, 0.12), transparent 60%),
var(--color-background-primary);
}

.sidebar-drawer,
.sidebar-container,
.toc-drawer,
.page-sidebar,
.sidebar-tree {
background: var(--color-sidebar-background);
}

.content {
background: transparent;
}

/* Ensure any theme variant still uses the light palette. */
html[data-theme="dark"],
html[data-theme="auto"],
body[data-theme="dark"],
body[data-theme="auto"] {
color-scheme: light;
}

html[data-theme="dark"],
html[data-theme="auto"],
body[data-theme="dark"],
body[data-theme="auto"] {
--color-background-primary: #f7f6f2;
--color-background-secondary: #f0efe9;
--color-sidebar-background: #f0efe9;
--color-background-hover: #e9ecf2;
--color-background-border: #e1e4ee;
--color-brand-primary: #1f5eff;
--color-brand-content: #1a55e6;
--color-foreground-primary: #16181d;
--color-foreground-secondary: #4b5563;
--accent-warm: #ff7a59;
--accent-mint: #22c997;
--accent-sky: #4cc9f0;
--accent-violet: #9b5de5;
}

body[data-theme="dark"],
body[data-theme="auto"] {
background:
radial-gradient(1200px 600px at 10% -10%, rgba(155, 93, 229, 0.12), transparent 60%),
radial-gradient(900px 500px at 90% -15%, rgba(76, 201, 240, 0.15), transparent 55%),
radial-gradient(700px 400px at 50% 110%, rgba(34, 201, 151, 0.12), transparent 60%),
var(--color-background-primary);
}

body[data-theme="dark"] .sidebar-drawer,
body[data-theme="auto"] .sidebar-drawer,
body[data-theme="dark"] .sidebar-container,
body[data-theme="auto"] .sidebar-container,
body[data-theme="dark"] .toc-drawer,
body[data-theme="auto"] .toc-drawer,
body[data-theme="dark"] .page-sidebar,
body[data-theme="auto"] .page-sidebar,
body[data-theme="dark"] .sidebar-tree,
body[data-theme="auto"] .sidebar-tree,
body[data-theme="dark"] .mobile-header,
body[data-theme="auto"] .mobile-header {
background: var(--color-sidebar-background);
}

/* Never show dark-only elements. */
.only-dark {
display: none !important;
}

article {
background:
linear-gradient(180deg, rgba(255, 255, 255, 0.85), rgba(255, 255, 255, 0.92));
border: 1px solid var(--color-background-border);
border-radius: 20px;
box-shadow: 0 24px 48px rgba(19, 25, 39, 0.08);
padding: 2.25rem 2.5rem;
}

article h1 {
color: #0f172a;
}

article h1::after {
content: "";
display: block;
width: 72px;
height: 6px;
margin-top: 0.5rem;
border-radius: 999px;
background: linear-gradient(90deg, var(--accent-warm), var(--accent-sky), var(--accent-mint));
}

article h2 {
color: #1e293b;
}

article h3 {
color: #1f2937;
}

article a {
color: var(--color-brand-primary);
font-weight: 600;
}

article a:hover {
color: var(--accent-violet);
}

code,
pre {
background: #eef2ff;
border-color: #d8def2;
}

pre {
box-shadow: inset 0 0 0 1px rgba(26, 85, 230, 0.08);
}

.sidebar-tree .current > a {
background: rgba(26, 85, 230, 0.08);
border-radius: 8px;
}

.sidebar-tree a:hover {
background: rgba(34, 201, 151, 0.12);
border-radius: 8px;
}

.toc-drawer .toc-tree li > a,
.toc-tree li > a {
border-radius: 8px;
}

.toc-tree li > a:hover {
background: rgba(76, 201, 240, 0.14);
}

table {
border-radius: 12px;
overflow: hidden;
box-shadow: 0 12px 24px rgba(15, 23, 42, 0.08);
}

th {
background: rgba(31, 94, 255, 0.08);
}

hr {
border: none;
height: 1px;
background: linear-gradient(90deg, transparent, rgba(31, 94, 255, 0.3), transparent);
}

/* Hide heading link glyphs while keeping the anchor element. */
.headerlink {
color: transparent !important;
text-decoration: none !important;
font-size: 0 !important;
width: 0;
margin-left: 0;
}

.headerlink::before {
content: "";
}

/* Force light palette even if the theme requests dark mode. */
html[data-theme="dark"],
html[data-theme="dark"] body,
html[data-theme="auto"],
html[data-theme="auto"] body {
--color-background-primary: #f7f6f2;
--color-background-secondary: #f0efe9;
--color-sidebar-background: #f0efe9;
--color-background-hover: #e9ecf2;
--color-background-border: #e1e4ee;
--color-brand-primary: #1f5eff;
--color-brand-content: #1a55e6;
--color-foreground-primary: #16181d;
--color-foreground-secondary: #4b5563;
--accent-warm: #ff7a59;
--accent-mint: #22c997;
--accent-sky: #4cc9f0;
--accent-violet: #9b5de5;
color-scheme: light;
}

html[data-theme="dark"] body,
html[data-theme="auto"] body {
background:
radial-gradient(1200px 600px at 10% -10%, rgba(155, 93, 229, 0.12), transparent 60%),
radial-gradient(900px 500px at 90% -15%, rgba(76, 201, 240, 0.15), transparent 55%),
radial-gradient(700px 400px at 50% 110%, rgba(34, 201, 151, 0.12), transparent 60%),
var(--color-background-primary);
}

/* Hide the theme toggle UI entirely. */
.theme-toggle,
.theme-toggle-container,
.theme-toggle-container button {
display: none !important;
}

/* Hide project title text in the sidebar, keep the logo visible. */
.sidebar-brand-text,
.sidebar-brand .brand-text,
.sidebar-brand .title {
display: none !important;
}

article .align-left {
float: none !important
}
Loading
Loading