Release docker image

[homersimpsons]

As a user I want to be able to run the crawler as a docker container without needing to pull the broject and build the image.

Also document it as requestest in #10

[thiswillbeyourgithub]

After looking at the docker compose file I believe this is done no? There is an image to download without building

[homersimpsons]

After looking at the docker compose file I believe this is done no?

I do not think so, the docker-compose.yml file references an image without namespace, that would mean that this is an official docker image (on the docker side), I highly doubt that mwmbl would have an official docker image.

There is an image to download without building

I didn't try with the one in the compose file yet, checking quickly there is https://hub.docker.com/layers/mwmbl/crawler

But the steps differ from what I read in the Dockerfile:
https://hub.docker.com/layers/mwmbl/crawler/latest/images/sha256-14d430c8828314305062a54a6ba861cd31a5c569f460b1184da96902335289a8

And the low download count does not help to believe in it.

[thiswillbeyourgithub]

Thanks I just was about to tell you!

That's really too bad because I was making a PR to the good karma kit to add the mwmbl crawler but can't!

I gave it a go to the cralwer on docker (latest release was 13h ago) but it crashes asking for an MWMBL_API_KEY and I can't find anywhere in the documentation how to add it.

Similarly, the mwmbl stats mention users by some kind of ID but I don't know how to know who I am!

[thiswillbeyourgithub]

Actually I was able to make a standalone docker compose using inline Dockerfile that clones the repo:

  mwmbl-crawler:
    # Help crawl for mwmbl.org, the open source, non profit web search engine
    # More info: https://book.mwmbl.org/
    # Crawler source: https://github.com/mwmbl/crawler-script
    environment:
      - THREADS=1
    restart: always
    build:
      context: .
      dockerfile_inline: |
          FROM alpine:latest AS repo
          RUN apk add --no-cache git
          WORKDIR /app
          RUN git clone https://github.com/mwmbl/crawler-script.git .

          FROM python:3.9-slim-bullseye
          COPY --from=repo /app/entrypoint.sh /
          RUN chmod +x /entrypoint.sh
          RUN mkdir -p /srv/mwmbl/crawler-script
          RUN useradd mwmbl -r -d /srv/mwmbl && \
              chown mwmbl:mwmbl -R /srv/mwmbl
          USER mwmbl
          WORKDIR /srv/mwmbl/crawler-script
          COPY --from=repo /app/justext /srv/mwmbl/crawler-script/justext
          COPY --from=repo /app/LICENSE /app/README.md /app/pyproject.toml /app/poetry.lock /app/main.py /srv/mwmbl/crawler-script/
          RUN python -m venv venv && \
              . venv/bin/activate && \
              pip install . && \
              pip cache purge
          ENTRYPOINT ["/entrypoint.sh"]

[MatthK]

I just found out about mwmbl and would love to contribute to this. While it should be possible to install it direct, I would also prefer to just fire up a simple docker container.

Could you please elaborate a bit more on how you got that docker compose up and running? Do I simply use that file, or do I have to clone the repository first?

[MatthK]

Ok, so it works just using this file.

1. On a linux computer, create a directory (call it mwmbl)
2. run nano docker-compose.yml
3. Paste the above file into it and save it. (I had to add services: to the top of the file for it to work though)
4. Alternatively change the number of threads you can spare
5. Run docker compose up -d

[thiswillbeyourgithub]

Yes! I recommend checking out that thread mwmbl/mwmbl#45

I don't find the author's answer but basically the new crawler is still in the works

[homersimpsons]

For the record there are steps to build on publish the image to the GitHub registry on https://docs.github.com/en/actions/use-cases-and-examples/publishing-packages/publishing-docker-images#publishing-images-to-github-packages