Presence of spurious text.

[LeMoussel]

In some analyzed texts we have the presence of characters such as {{!-}}, }}|, {{{blbla bla|}}} etc.

For example in the following articles:

• Alpes-de-Haute-Provence
• Akhenaton
• Anubis
• Algèbre de Boole (logique)

[xxyzz]

Probably same as tatuylonen/wiktextract#533, we will check this bug next week.

[xxyzz]

#250 should fix this bug if it's the same bug as the wiktextract 533 issue.

[LeMoussel]

Alpes-de-Haute-Provence
The output is OK. No errors, warnings and debugs messages.

Akhenaton
The output is KO.
In the output presence of

|
|align="center" valign="middle"|
|}
| style="text-align:center;padding:2px;" |
| style="text-align:left;padding:2px;" |
|}

|
|align="center" valign="middle"|
|}
| style="text-align:center;padding:2px;" |
| style="text-align:left;padding:2px;" |
|}

debugs message:

Akhenaton: DEBUG: HTML tag not properly closed at ['Akhenaton'] parsing Akhenaton/Règne/Révolution religieuse/Période noire ?
started on line 136, detected on line 470
Akhenaton: DEBUG: HTML tag not properly closed at ['Akhenaton'] parsing Akhenaton/Règne/Révolution religieuse
started on line 102, detected on line 470
Akhenaton: DEBUG: HTML tag not properly closed at ['Akhenaton'] parsing Akhenaton/Règne/Révolution religieuse
started on line 100, detected on line 470

Anubis
The output is OK. No errors, warnings and debugs messages.

Algèbre de Boole (logique)
The output is OK. No errors, warnings and debugs messages.

Almost good ☺️, but for "Akhenaton" maybe it's another anomaly?

[kristian-clausal]

Currently downloading the French Wikipedia dump to make a new .db file with the updated data structure, so I'll be taking a look a look at this maybe by tomorrow.

[kristian-clausal]

I'm getting mostly correct output for Akhenaton. I've found issues, but not the ones that you have here.

Some of the image links, like in the homonym template at the start of the page, don't have alt texts at all; they take the last argument, class=noviewer.

I also found a few broken table ends, |}, pairs after another, which is probably what is left of the broken tables in your post. I'll take a deeper look tomorrow.

[kristian-clausal]

There's a PR for wiktextract that should take care of the last of the fixes I've attempted here.

clean_value() is supposed to remove wikitext tables {| ... |}, which we will continue to do. However, HTML tag tables <table>...</table> will be left, and will just render its contents on cell after another linearly. There hasn't been any need for wiktextract to handle this any better, and we'll keep it this way (at least for a while); if you do not want to see HTML tables or wikitext tables in the output, or do want to see both, they need to be handled with a node handling function. This might change in the future, if we change how clean_value is implemented, or create a separate thing (a library or a wikitextprocessor-specific cleaning function).

Other issues addressed: image links are rare in a Wiktionary context, so they weren't handled well, but now several issues should be fixed. There were other things but I can't actually remember what they were, so they couldn't have been that important!

Please pull the newest commits of wikitextprocessor and wiktextract sometime tomorrow and check if everything looks better now. The PR in wiktextract needs to be looked at by someone other than me because it involves negative lookahead in regex and I am already starting to see into the sixth dimension.

[LeMoussel]

• Alpes-de-Haute-Provence KO
  Presence of [Alt: Page d’aide sur l’homonymie] and this [Alt: icône décorative]

[Alt: icône décorative] Portail des Alpes-de-Haute-Provence

[Alt: icône décorative] Portail des Alpes

• Akhenaton KO
  Presence of [Alt: Page d’aide sur l’homonymie] and this (many times)

|
|align="center" valign="middle"|
|}
| style="text-align:center;padding:2px;" |
| style="text-align:left;padding:2px;" |
|}

• Anubis OK
• Algèbre de Boole (logique) KO
  Presence of [Alt: Page d’aide sur l’homonymie]

wikitext_parse-Akhenaton.txt
wikitext_parse-Algèbre de Boole (logique).txt
wikitext_parse-Alpes-de-Haute-Provence.txt

Python Test code:

import re
import requests
from typing import Optional

# https://github.com/tatuylonen/wikitextprocessor/
from wikitextprocessor import (
    Wtp,
    NodeKind,
    WikiNode,
)
# https://github.com/tatuylonen/wiktextract
from wiktextract.wxr_context import WiktextractContext
from wiktextract.config import WiktionaryConfig
from wiktextract.page import clean_node


def clean_node_handler(node)-> Optional[str]:
    """Process nodes when encountering them.
    For example by filtering them or changing them if needed."""
    assert isinstance(node, WikiNode)

    if node.kind == NodeKind.TEMPLATE:
        if node.largs[0][0] in ['Semi-protection', 'Semi-protection longue', 'Confusion', 'coord']:
            return ""
        if re.match('Infobox', node.largs[0][0], re.I):
            return ""
        if re.match('Article', node.largs[0][0], re.I):
            return ""
        if re.match('Référence', node.largs[0][0], re.I):
            return ""

    if node.kind == NodeKind.LEVEL2:
        if node.largs[0][0] in  ['Annexes', 'Notes et références', 'Voir aussi']:
            return ""

    if node.kind == NodeKind.LINK:
        if re.match('Fichier:', node.largs[0][0], re.I):
            return ""

    #if node.kind == NodeKind.HTML:
        #print(node.sarg)

    #if hasattr(node, 'largs') and len(node.largs) > 0:
    #    if node.largs[0][0] in  ['=== Langues ===']:

    return None

def template_handler(name, args_ht):
    if len(args_ht) == 0:
        return ""
    return None

if __name__ == '__main__':
    extension_tags = {
        "maplink": {"parents": ["phrasing"], "content": ["phrasing"]},
        "poem": {"parents": ["phrasing"], "content": ["phrasing"]},
        "gallery": {"parents": ["phrasing"], "content": ["phrasing"]},
        "graph": {"parents": ["phrasing"], "content": ["phrasing"]},
        "mapframe": {"parents": ["phrasing"], "content": ["phrasing"]},
        "timeline": {"parents": ["phrasing"], "content": ["phrasing"]},
    }
    wxr = WiktextractContext(
        wtp = Wtp(
            db_path="fr-wiki-latest.db",
            lang_code="fr",
            project="wikipedia",
            extension_tags=extension_tags,
        ),
        config=WiktionaryConfig(),
    )

    wiki_page_title = 'Alpes-de-Haute-Provence'

    wiki_page = wxr.wtp.get_page(wiki_page_title)

    wxr.wtp.start_page(wiki_page.title)
    wxr.wtp.invoke_aliases = wxr.wtp.invoke_aliases | {"#invoque"}

    info_log = f"Analyse: '{wiki_page_title}'\n"

    wiki_nodes = wxr.wtp.parse(text=wiki_page.body)
    text = clean_node(
            wxr=wxr,
            sense_data={},
            wikinode=wiki_nodes,
            collect_links=False,
            node_handler_fn=clean_node_handler,
            template_fn=template_handler,
        )

    if len(wxr.wtp.errors) > 0:
        info_log += f"# Erreurs: {len(wxr.wtp.errors)}\n"
    if len(wxr.wtp.warnings) > 0:
            info_log += f"# Warnings: {len(wxr.wtp.warnings)}"

    print(info_log)

    with open(f'wikitext-{wiki_page_title}.txt', 'w', encoding='utf-8') as f:
        f.write(wiki_page.body)
    with open(f'wikitext_parse-{wiki_page_title}.txt', 'w', encoding='utf-8') as f:
        f.write(text)

[xxyzz]

[Alt: something] is the "alt" text of an image, this was added from a recent pr: tatuylonen/wiktextract#539

I guess you only need the entire page text but don't need the wikitext node type or structure data, have you tried the HTML dump file or the zim dump file?

[kristian-clausal]

I've used Akhenaton as the text to test these changes on, so it should be fine. I can't recreate the specific error you have there. I will try using your specific code.

Also a tip with regexes: using r-strings r"like this" allows you to type strings with escape characters (like \n or \) and other things so that the escape strings are not escaped like in a normal Python string; it's a 'raw' string literal. The regex for 'Fichier:' would be r'Fichier:' and then r'\s*Fichier\s*: because it turns out there can be whitespace in those places.

[kristian-clausal]

Yeah, this is the table removal stuff that hadn't been yet merged when you tried to merge it; there was still a pull request waiting, which is why I said to wait until tomorrow (it's now tomorrow morning in Europe). Please pull the newest commits and try again.

[LeMoussel]

Yes, I only need the entire text of the page, but not the wikitext node type or structure data.

Just like the result with the Wikipedia API action=query&prop=extracts|revisions&explaintext
Example: Alpes-de-Haute-Provence

HTML format is complicated to parse. I tried different tools like Trafilatura, but the results are not relevant.

I don't know the ZIM format. But from what I quickly saw, it's also in HTML format.

[LeMoussel]

Pull:

dev@dev-B550M-DS3H:~/Python/WikiExtractor$ cd wiktextract
dev@dev-B550M-DS3H:~/Python/WikiExtractor/wiktextract$ git pull
Updating 5d6bb0e2..e992e954
Fast-forward
 src/wiktextract/clean.py | 23 +++++++++++++++++++----
 tests/test_clean.py      | 12 ++++++++++++
 2 files changed, 31 insertions(+), 4 deletions(-)
dev@dev-B550M-DS3H:~/Python/WikiExtractor/wiktextract$ cd ../wikitextprocessor
dev@dev-B550M-DS3H:~/Python/WikiExtractor/wikitextprocessor$ git pull
Already up to date.

Test on "Alpes-de-Haute-Provence". Same errors.
Presence of [Alt: Page d’aide sur l’homonymie] and this [Alt: icône décorative]

Rem: Following your advice I modified all the regex's
NB: I'm away until Monday.

[xxyzz]

HTML is complicated then wikitext, seriously? You could try HTML/XML parsers like lxml or Beautiful Soup to find the body element and use the element method or attribute to get the text.