Is it possible to use `wikitextprocessor` with Wikisource template?

[LeMoussel]

Wikisource has specific templates, for example {{T}}, {{c}}, ....
An example can be found on this page: Chapotot - L’Estomac et le Corset.djvu/3

{{T2|{{t|L’ESTOMAC et LE CORSET|140}}}}

Is it possible to process this Wikisource template with wikitextprocessor?

Is it necessary to generate a specific SQLite Database? If so, which dump should be used?

wtp = Wtp(
    db_path="fr_wikisource.db", lang_code="fr", project="wikisource"
)

# extract dump file then save pages to SQLite DB
process_dump(
    wtp,
   => Which dump should be used?,
    {0, 10, 110, 828},  # namespace id, can be found at the start of dump file
)

[kristian-clausal]

It should be possible, you'd just need to create the right configs, unless there are some bigger changes to how the wiki works, but Wikisource is Wikimedia and most probably doesn't do anything weird.

Each template and module is a page in the project, and gets extracted into the database file together with all the other articles. This is where Wikitextprocessor has access to templates and modules, so you need to create the .db (or extract everything again and again if you don't create a .db).

Take a look at how individual extractors are configured in Wiktextract to get a good idea of what to do. There's the stuff in data/langcode/ in Wiktextract that shows some things to look out for when parsing things and data/langcode/ in Wikitextprocessor that shows how namespace ids are usually done in these Wikis. This config data depends on the source wiki, so it can be different between different projects, unfortunately, but there's a ton of overlap and usually you have the same id codes.

Although looking at how the hierarchy is constructed in data/langcode, there seems to be only one directory for each language code, which is not ideal if there are differences between it.wiktionary.org and it.wikisource.org... @xxyzz how would handle this?

[LeMoussel]

I created a .db file with the dump frwiki-20260101-pages-articles.xml.bz2
I'm wondering if this is the correct dump to use.

With the .db file generated from this dump, for this wikitext:

{{T2|{{t|L’ESTOMAC et LE CORSET|140}}}}

{{c|DÉVIATIONS, DISLOCATIONS, TROUBLES FONCTIONNELS}}

I get this result:

T2Catégorie:DÉVIATIONS, DISLOCATIONS, TROUBLES FONCTIONNELS

Which is incorrect.

Code to parse :

        wtp = Wtp(
            db_path="fr_wikisource.db",
            lang_code="fr",
            project="wikipedia",
        )
        wtp.start_page("")
        root = wtp.parse(
            text=html.unescape(wikitext or ""),
            expand_all=True,
        )
        text = wtp.node_to_text(node=root)

[xxyzz]

Wrong dump file, download this: https://dumps.wikimedia.org/frwikisource/20260101/frwikisource-20260101-pages-articles.xml.bz2

Templates should be expanded without problem.

I guess you are gathering data for training LLM? Can't believe you still haven't tried the free HTML dump file: https://enterprise.wikimedia.com

[LeMoussel]

Yes, I'm collecting data for training LLM to french.

Which free HTML export file are you referring to?

I'm using Wikimedia Enterprise to download a chunk of a specific segment from a snapshot (https://enterprise.wikimedia.com/docs/snapshot/#download-snapshot-chunk), from which I retrieve Wikitext code for the articles.

[xxyzz]

I downloaded this chunk https://api.enterprise.wikimedia.com/v2/snapshots/frwikisource_namespace_0/chunks/frwikisource_namespace_0_chunk_9/download and the data in the "article_body.html" JSON field is HTML data.

[xxyzz]

To answer Kristian's question: if the namespace data is incompatible with other wiki, the "tools/get_namespaces.py" script can be used to update namespace data, and the generated file need to be examined manually. For now I don't see the need to create and maintain namespace files for other wikis.

Also the unformatted single line namespace files in "data" folder are not created by the above mentioned script and need to be updated and examined when a new extractor is added.