chore: Use Dune package management by sabine · Pull Request #3132 · ocaml/ocaml.org

sabine · 2025-05-26T14:04:58Z

This patch changes the build configuration to use Dune Package Management.

Done / To do:

To test this

$ make start

$ make playground

$ make scrape

Current status

I believe this is ready to merge.

gridbugs · 2025-06-05T01:23:24Z

I've been using dune package management to build this project whenever I've needed to contribute to it over the past year and it's worked fine until recently. Here are some PRs/issues relating to fixing the problems I've encountered recently:

gridbugs · 2025-06-05T01:26:24Z

There's also currently an issue with the ocamlnet package source being missing. I've added it to the opam-source-archives repo and I have a PR to point the ocamlnet package at the archived source. Once the latter PR is merged, updating the opam-repository revision in this project's dune-workspace to the current tip of opam-repository will fix the problem with ocamlnet.

Sudha247 · 2025-06-05T13:20:46Z

This is great! Thanks for the work, @sabine and @gridbugs. Upon offline discussion with Sabine, we wanted to test this on MacOS and Windows to ensure the workflow remains unaffected for contributors on those platforms. I tested with WSL2 (which is the recommended way to develop ocaml.org on Windows), and the ocaml.org project (with make start), builds and runs locally on WSL2, in this branch with dune pkg. Moreover, the testsuite passes too.

The ocaml.org playground doesn't build, as expected - as it seems to have some dependency issues. I'll try to investigate what's happening there!

sabine · 2025-06-06T07:35:46Z

Sounds great! Looks like we'll be migrating fully to dune pkg as soon as the playground builds.

sabine · 2025-06-16T07:41:19Z

Thanks @Sudha247 for sabine#53. This makes the playground dune build succeed.

One thing I observe on my machine:

Both on the main branch, and on this branch, running the playground build just deletes the .cmi files in the asset/ folder of the playground. It should be copying the .cmi files from somewhere. The script gen_stdlib.sh performs this part by looking up via opam the location of these .cmi files, so we need to either find where they reside with a dune build, or find another way to regenerate these files. 🤔

cuihtlauac · 2025-06-20T13:27:56Z

HACKING.md

Replace all ````` by ````bash` in this file

cuihtlauac · 2025-06-20T13:32:03Z

dune-project

Could the removal of the dependency on Anton's river fork be merged ahead in a separate PR?

Leonidas-from-XIV · 2025-06-20T14:45:03Z

Dockerfile

-RUN opam install . --deps-only
+# Install Dune Developer Preview
+RUN curl -fsSL https://get.dune.build/install | sh
+RUN /bin/bash -c 'source "/root/.local/share/dune/env/env.bash"'


Does this have any effect? The environment disappears after this call.

We must not download a dev preview on each build, that's brittle and wasteful.

Leonidas-from-XIV · 2025-06-20T14:46:50Z

Dockerfile

-RUN sudo apk -U upgrade --no-cache && sudo apk add --no-cache \
+RUN apk -U upgrade --no-cache && apk add --no-cache \
+    # to download and install Dune Developer Preview with alpine:3.21
+    build-base patch tar ca-certificates git rsync curl sudo bash \


Is there a need for sudo, rsync, bash or nano?

You also probably don't need curl. Docker can itself download remote files. See https://docs.docker.com/reference/dockerfile/#adding-files-from-a-url

Indeed, but dune uses curl internally for (url ..) fields, so we still need it.

playground/dune-project

v-gb · 2025-06-20T23:08:19Z

Makefile

 watch: ## Watch for the filesystem and rebuild on every change
-	opam exec -- dune build @run -w --force --no-buffer
+	dune build @run -w --force --no-buffer


I read the HACKING doc a bit quickly, and ran make watch instead of make start first:

$ make watch dune build @run -w --force --no-buffer File "src/ocamlorg_web/lib/dune", line 13, characters 2-15: 13 | mirage-kv-mem)) ^^^^^^^^^^^^^ Error: Library "mirage-kv-mem" not found. -> required by library "ocamlorg_web" in _build/default/src/ocamlorg_web/lib -> required by executable main in src/ocamlorg_web/bin/dune:2 -> required by _build/default/src/ocamlorg_web/bin/main.exe -> required by alias src/ocamlorg_web/bin/run in src/ocamlorg_web/bin/dune:8 ######################################################################## 100.0% Done: 95% (95/100, 5 left, 1 failed) (jobs: 1)Browserslist: caniuse-lite is outdated. Please run: npx update-browserslist-db@latest Why you should do it regularly: https://github.com/browserslist/update-db#readme Rebuilding... Done in 1911ms.

Shouldn't every rule that does a build depend on dune.lock, instead of just a couple of them? (or taking a step back, I thought the usual thing to do is commit the dune.lock, so I'm a bit confused about why it's being generated on the fly).

thanks, you're right. need to update the Makefile

dev-tools.locks/ocamlformat/ocamlformat.pkg

shonfeder · 2025-07-31T21:45:02Z

The install script is ready here, thanks to @gridbugs' work: https://github.com/ocaml-dune/dune-bin-install

@Sudha247 is planning to add documentation here about how to use it.

It also sounds like we should take Steve's advice about ignoring the lock dir, and IIUC @cuihtlauac's review has been addressed and just needs to be re-requested to clear the change request?

cuihtlauac

Small remarks:

File .github/workflows/scrape_changelog.yml should be updated.
There are a few remaining references to olinkcheck that should be removed, it is no longer used

Big question

Is this PR proposing to build ocaml.org with a version of Dune that changes daily? If that is the case, I must object. This is unwise. I support building ocaml.org as a regression test in Dune developer preview CI. However, using an unstable build tool in a production delivery chain is reckless.

cuihtlauac · 2025-08-01T06:14:15Z

dune-workspace

+(pin
+(name olinkcheck)
+ (url "git+https://github.com/tarides/olinkcheck")
+ (package
+  (name olinkcheck)))


PR #3144 Removed olinkcheck. I dont think this is needed any longer

cuihtlauac · 2025-08-01T06:14:51Z

dune-workspace

+  (pins olinkcheck)
+   (version_preference newest)
+  )


Same remark w.r.t. PR#3144

cuihtlauac · 2025-08-01T06:31:00Z

.github/workflows/ci.yml

+      - name: Use Dune Developer Preview
+        uses: ocaml-dune/setup-dune@v0


I'm not sure I fully understand this.

Are we attempting to build ocaml.org with a build tool that changes daily?

Sudha247 · 2025-08-01T08:47:14Z

Is this PR proposing to build ocaml.org with a version of Dune that changes daily? If that is the case, I must object.

The PR in its current state proposes this approach. However, as @shonfeder noted above, we can use @gridbugs' installer to install the released Dune binary (currently version 3.19.1). I'll update the PR and documentation accordingly. The idea is to use dune pkg for building and deploying the website. Since dune pkg is available in the released version of Dune, we don’t need the developer preview as the experimental features it offers aren’t essential for this PR.

sabine · 2025-08-19T16:36:47Z

Merged @Sudha247's changes and rebased on main.

cuihtlauac

The following changes are required:

Update .github/workflows/scrape_changelog.yml
Update README.md, section “Getting Started”
Do not use developer preview in ci.yml the same build configuration must be used there, in Dockerfile and locally
Fix make docker build under Ubuntu 25.04
Remove file ocamlorg.opam

shonfeder · 2025-08-21T23:11:07Z

Remove file ocamlorg.opam

IIUC, this is not desirable, as we still want to support opam-based workflows.

shonfeder · 2025-08-26T14:17:16Z

For the record, at one point @cuihtlauac was getting this failure on the docker build

ocaml.org:dune_pkg>make docker                                                                                                                    ocaml.org
docker build -f Dockerfile . -t ocamlorg:latest
[+] Building 115.5s (13/23)                                                                                                                  docker:default
 => [internal] load build definition from Dockerfile                                                                                                   0.0s
 => => transferring dockerfile: 1.79kB                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/alpine:3.21                                                                                         0.4s
 => [internal] load .dockerignore                                                                                                                      0.0s
 => => transferring context: 163B                                                                                                                      0.0s
 => [internal] load build context                                                                                                                      0.1s
 => => transferring context: 278.91kB                                                                                                                  0.1s
 => [build  1/11] FROM docker.io/library/alpine:3.21@sha256:b6a6be0ff92ab6db8acd94f5d1b7a6c2f0f5d10ce3c24af348d333ac6da80685                           0.0s
 => CACHED [build  2/11] RUN apk -U upgrade --no-cache && apk add --no-cache     build-base patch tar ca-certificates git        libx11-dev coreutils  0.0s
 => CACHED [build  3/11] RUN curl -sSL https://github.com/ocaml-dune/dune-bin-install/releases/download/v1/install.sh | sh -s 3.19.1 --install-root /  0.0s
 => CACHED [build  4/11] RUN dune --version                                                                                                            0.0s
 => CACHED [build  5/11] WORKDIR /root/ocaml.org                                                                                                       0.0s
 => CACHED [build  6/11] COPY --chown=root . .                                                                                                         0.0s
 => CACHED [build  7/11] RUN ls                                                                                                                        0.0s
 => CACHED [build  8/11] RUN dune pkg lock --verbose                                                                                                   0.0s
 => ERROR [build  9/11] RUN dune build @pkg-install                                                                                                  115.0s
------                                                                                                                                                      
 > [build  9/11] RUN dune build @pkg-install:                                                                                                               
104.7 File "dune.lock/stdune.pkg", line 17, characters 3-73:                                                                                                
104.7 17 |    https://github.com/ocaml/dune/releases/download/3.19.0/dune-3.19.0.tbz)                                                                       
104.7         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
104.7 Error: curl returned an invalid error code 6
104.7        
104.7        
106.5 File "dune.lock/result.pkg", line 11, characters 3-76:
106.5 11 |    https://github.com/janestreet/result/releases/download/1.5/result-1.5.tbz)
106.5         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
106.5 Error: curl returned an invalid error code 6
106.5        
106.5        
------
Dockerfile:25
--------------------
  23 |     
  24 |     RUN dune pkg lock --verbose
  25 | >>> RUN dune build @pkg-install
  26 |     RUN dune build @install --profile=release #--verbose -j1 --no-buffer
  27 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c dune build @pkg-install" did not complete successfully: exit code: 1
make: *** [Makefile:62: docker] Error 1

This seemed to be an issue with the latest Ubuntu that was addressed by

DOCKER_BUILDKIT=1 docker build --network=host .

cuihtlauac · 2025-08-26T14:28:08Z

Unfortunately using --network=host isn't a fix. It's a workaround. Something is wrong in the way dune triggers networking activity. That's what needs to be fixed.

shonfeder · 2025-08-26T15:03:46Z

Something is wrong in the way dune triggers networking activity.

Oh! That's an interesting claim. This is a good finding if so. But afaik, we have only seen problems on a particular OS version when running specifically on docker. What about this leads us to the conclusions that is a general problem with dune's network requests?

sabine · 2025-08-28T09:22:49Z

Closing and reopening this, in order to not bottleneck progress on my availability to respond to pull requests on my fork. Here's the new PR: #3281

sabine mentioned this pull request May 26, 2025

Add GitHub Actions workflow for dune pkg #3127

Closed

sabine force-pushed the dune_pkg branch from 8786902 to 0b27bb5 Compare June 3, 2025 10:54

sabine force-pushed the dune_pkg branch from 0b27bb5 to 35c9778 Compare June 5, 2025 10:12

sabine force-pushed the dune_pkg branch 3 times, most recently from a279318 to d48543c Compare June 16, 2025 07:37

sabine mentioned this pull request Jun 17, 2025

Dune pkg sabine/ocaml.org#53

Closed

sabine marked this pull request as ready for review June 17, 2025 09:49

sabine changed the title ~~chore: migrate to using dune package management~~ chore: Use Dune package management Jun 20, 2025

cuihtlauac requested changes Jun 20, 2025

View reviewed changes

Leonidas-from-XIV reviewed Jun 20, 2025

View reviewed changes

This was referenced Jun 20, 2025

Do not use Anton's river fork #3178

Merged

No code blocks w/o language #3181

Merged

Only use Alpine 3.21 #3180

Merged

Call make instead of dune build #3179

Merged

v-gb reviewed Jun 20, 2025

View reviewed changes

sabine force-pushed the dune_pkg branch from 936ba3f to 7d4f405 Compare June 26, 2025 12:58

Sudha247 mentioned this pull request Jul 17, 2025

Updates sabine/ocaml.org#58

Merged

4 tasks

sabine force-pushed the dune_pkg branch from 7d4f405 to 5674aad Compare July 23, 2025 10:03

sabine added the stalled Blocked waiting. Will not progress w/o external progress label Jul 23, 2025

shonfeder mentioned this pull request Jul 29, 2025

changelog: add a post about OCaml.org migration to Dune Package Management #3152

Closed

gridbugs reviewed Jul 30, 2025

View reviewed changes

dev-tools.locks/ocamlformat/ocamlformat.pkg Outdated Show resolved Hide resolved

cuihtlauac requested changes Aug 1, 2025

View reviewed changes

sabine and others added 18 commits August 19, 2025 18:36

attempt to make Playground build with Dune Developer Preview (fails)

620267a

Build playground with dune pkg

017f2a5

update CI to use Dune Developer Preview

59dd5ef

update scraper workflow to use dev preview

e5af6bd

copy playground cmi files

6e622e2

update playground gitignore

f1e40ec

remove opam.locked file

95bfd9c

rely on dune.lock

22a546a

remove obsolete config

12ed046

build scraper in makefile

5d6f6ca

rely on dune.lock

a247128

Pin dune pkg to a specific version in the Dockerfile

3a1d3e6

update Dockerfile and HACKING.md

343d032

revert accidental playground changes

dffbaa2

Update to use Dune binary

b0b8c50

Address review comments

fd507da

add ocamlformat as a test dependency

3a0eef0

remove river pin

2501b76

sabine force-pushed the dune_pkg branch from cc5395e to 2501b76 Compare August 19, 2025 16:36

reduce changes to playground dune-project

b693e90

cuihtlauac requested changes Aug 21, 2025

View reviewed changes

sabine mentioned this pull request Aug 28, 2025

chore: Adopt Dune package management #3280

Closed

9 tasks

sabine closed this Aug 28, 2025

sabine mentioned this pull request Aug 28, 2025

chore: Adopt Dune Package Management for the OCaml.org Build #3281

Open

11 tasks

		- name: Use Dune Developer Preview
		uses: ocaml-dune/setup-dune@v0

Conversation

sabine commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

To test this

Current status

Uh oh!

gridbugs commented Jun 5, 2025

Uh oh!

gridbugs commented Jun 5, 2025

Uh oh!

Sudha247 commented Jun 5, 2025

Uh oh!

sabine commented Jun 6, 2025

Uh oh!

sabine commented Jun 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

v-gb Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shonfeder commented Jul 31, 2025

Uh oh!

cuihtlauac left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cuihtlauac Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sudha247 commented Aug 1, 2025

Uh oh!

sabine commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cuihtlauac left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shonfeder commented Aug 21, 2025

Uh oh!

shonfeder commented Aug 26, 2025

Uh oh!

cuihtlauac commented Aug 26, 2025

Uh oh!

shonfeder commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sabine commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

sabine commented May 26, 2025 •

edited

Loading

v-gb Jun 20, 2025 •

edited

Loading

cuihtlauac left a comment •

edited

Loading

cuihtlauac Aug 1, 2025 •

edited

Loading

sabine commented Aug 19, 2025 •

edited

Loading

cuihtlauac left a comment •

edited

Loading

shonfeder commented Aug 26, 2025 •

edited

Loading

sabine commented Aug 28, 2025 •

edited

Loading