Skip to content

update default for the filter_both_ac parameter in IAA score calculation #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ypislon wants to merge 6 commits into
base: main
Choose a base branch
from
Open

update default for the filter_both_ac parameter in IAA score calculation #18

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e21123d
update default for the filter_both_ac parameter to be True in IAA sco…
ypislon Jun 10, 2026
0a11de3
update demo notebook to show default usage of filter_both_ac paramete…
ypislon Jun 18, 2026
5e3a889
fix wrong change in verbose default parameter
ypislon Jun 18, 2026
89b41f2
Merge branch 'main' into fix-filter-both-ac-parameter-defaults
ypislon Jun 18, 2026
15b65e1
add more verbose warning in the demo notebooks about the filter_both_…
ypislon Jul 7, 2026
0f82fcd
make comments and docs more explicit about filter_both_ac parameter
ypislon Jul 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions demo/notebooks/inter_annotator_agreement.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,10 +219,8 @@
"metadata": {},
"cell_type": "markdown",
"source": [
"As the confusion matrix shows for the calculation of Scott's pi, only the annotations from annotation collection 1\n",
"with the tag 'process_event' have been taken into account.\n",
"From annotation collection 2 there are still two annotations considered, with the tags 'stative_event' and 'change_of_state' respectively.\n",
"But we can filter both annotation collections, too:"
"As the confusion matrix shows for the calculation of Scott's Pi, only the annotations with the tag 'process_event' have been taken into account.\n",
"If we would want to filter only the first collection and compare it with a second, unfiltered collection, we can use the parameter `filter_both_ac`:"
]
},
{
Expand All @@ -233,12 +231,17 @@
" ac1_name_or_inst='ac_1',\n",
" ac2_name_or_inst='ac_2',\n",
" tag_filter=['process_event'],\n",
" filter_both_ac=True\n",
" filter_both_ac=False\n",
")"
],
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"cell_type": "markdown",
"source": "This can be helpful when exploring the matches and mismatches of different annotation collections. But be careful: since the calculation of IAA scores is always a based on a comparison of two or more annotation collections, filtering just one annotation collection and comparing it to a \"full\" annotation collection can influence the IAA score significantly."
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down
21 changes: 12 additions & 9 deletions gitma/_metrics.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,11 @@ def filter_ac_by_tag(
ac1 (AnnotationCollection): First annotation collection.
ac2 (AnnotationCollection): Second annotation collection.
tag_filter (list, optional): The list of tags to be included. Defaults to None.
filter_both_ac (bool, optional): If `True` both collections get filtered . Defaults to True.
filter_both_ac (bool, optional): If `True` both collections get filtered. If `False`, only the first collection's annotations get filtered.\
Defaults to True.

Returns:
Tuple[List[Annotation]]: Two filtered list of annotations.
Tuple[List[Annotation]]: Two filtered lists of annotations.
"""
if tag_filter:
ac1_annotations = [
Expand Down Expand Up @@ -173,21 +174,21 @@ def get_confusion_matrix(pair_list: List[Tuple[Annotation]], level: str = 'tag')


def get_cooccurence_matrix(annotationdata: set) -> pd.DataFrame:
"""Generates cooccurence matrix for annotation data
"""Generates co-occurence matrix for annotation data

Args:
annotationdata (Set[List[coderid,itemid,tag]]): List of overlapping annotations as a set.

Returns:
pd.DataFrame: Cooccurence matrix as pandas data frame.
pd.DataFramefilter_ac_by_tag: Co-occurence matrix as pandas data frame.
"""

# group tags by item
items_tags_grouped = defaultdict(list)
for coderid, itemid, tag in annotationdata:
items_tags_grouped[itemid].append(tag)

# count co-coccurences of tags
# count co-occurences of tags
cooccurence_counter = Counter()
for tags in items_tags_grouped.values():
for pair in combinations(sorted(tags), 2): # all possible combinations of two elements
Expand Down Expand Up @@ -231,7 +232,7 @@ def get_annotation_pairs(
ac1: AnnotationCollection,
ac2: AnnotationCollection,
tag_filter: list = None,
filter_both_ac: bool = False,
filter_both_ac: bool = True,
property_filter: str = None,
verbose: bool = True) -> List[Union[Tuple[Annotation, EmptyAnnotation], Tuple[Annotation, Annotation]]]:
"""
Expand All @@ -245,7 +246,8 @@ def get_annotation_pairs(
ac1 (AnnotationCollection): First annotation collection.
ac2 (AnnotationCollection): Second annotation collection.
tag_filter (list, optional): The list of tags to be included. Defaults to `None` (no filter / all tags included).
filter_both_ac (bool, optional): If `True` the `tag_filter` is applied to both collections. Defaults to `False`.
filter_both_ac (bool, optional): If `True` the `tag_filter` is applied to both collections, if `False`, the `tag_filter` is applied\
only to the first collection. Defaults to `True`.
property_filter (str, optional): If not `None`, only annotations with this property are included. Defaults to\
`None` (no filter / all annotations included).
verbose (bool, optional): Whether to print results to stdout. Defaults to `True`.
Expand Down Expand Up @@ -336,9 +338,10 @@ def get_annotation_pairs_for_multiple_annotators(
Get annotation data in the NLTK format for IAA calculation for two or more annotators without duplicate pairs.
Args:
ac_dict (dict): Dictionary of all annotation collections.
ac_names (list): List of annotation collection names to include in the IAA calculation. If empty, all ACs in the project will be used.
ac_names (list): List of annotation collection names to include in the IAA calculation. If empty, all annotation collections in the project will be used.
tag_filter (list): List of tags that should be included for iaa calculation. If empty, all tags will be used.
filter_both_ac (bool): Whether to apply tag_filter on both ACs in the pair or just on the first AC. Default is True.
filter_both_ac (bool): Whether to apply tag_filter on both annotation collections in the pair or just on the first annotation collection.\
Default is True.
include_empty_annotations (bool): Whether to include empty annotations in the IAA data. If `False`, only annotations with a matching annotation\
in the second collection are included. Default is True.
property_filter (str, optional): Property to filter by as a string with the property name. If None, all properties will be used.\
Expand Down
27 changes: 14 additions & 13 deletions gitma/project.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -726,7 +726,7 @@ def get_iaa(
ac1_name_or_inst: Union[str, AnnotationCollection],
ac2_name_or_inst: Union[str, AnnotationCollection],
tag_filter: list = None,
filter_both_ac: bool = False,
filter_both_ac: bool = True,
level: str = 'tag',
include_empty_annotations: bool = True,
distance: str = 'binary',
Expand All @@ -744,8 +744,8 @@ def get_iaa(
ac2_name_or_inst (str): The name or instance of the second annotation collection, whose annotations will be searched for\
matches to those in the first.
tag_filter (list, optional): Which tags should be included. Defaults to `None` (all tags).
filter_both_ac (bool, optional): Whether the tag filter should be applied to both annotation collections. Defaults to `False`\
(only applied to the first collection).
filter_both_ac (bool, optional): Whether the tag filter should be applied to both annotation collections. Defaults to `True`\
(applies to both collections).
level (str, optional): Whether the annotations' tags or a specified property (prefixed with 'prop:') should be compared. Defaults\
to 'tag'.
include_empty_annotations (bool, optional): If `False`, only annotations with a matching annotation in the second collection are\
Expand Down Expand Up @@ -894,7 +894,7 @@ def calculate_scotts_pi(
ac1_name_or_inst: Union[str, AnnotationCollection],
ac2_name_or_inst: Union[str, AnnotationCollection],
tag_filter: list = None,
filter_both_ac: bool = False,
filter_both_ac: bool = True,
level: str = 'tag',
include_empty_annotations: bool = True,
distance: str = 'binary',
Expand All @@ -910,8 +910,8 @@ def calculate_scotts_pi(
ac2_name_or_inst (str): The name or instance of the second annotation collection, whose annotations will be searched for\
matches to those in the first.
tag_filter (list, optional): Which tags should be included. Defaults to `None` (all tags).
filter_both_ac (bool, optional): Whether the tag filter should be applied to both annotation collections. Defaults to `False`\
(only applied to the first collection).
filter_both_ac (bool, optional): Whether the tag filter should be applied to both annotation collections. Defaults to `True`\
(applies to both collections).
level (str, optional): Whether the annotations' tags or a specified property (prefixed with 'prop:') should be compared. Defaults\
to 'tag'.
include_empty_annotations (bool, optional): If `False`, only annotations with a matching annotation in the second collection are\
Expand All @@ -936,7 +936,7 @@ def calculate_cohens_kappa(
ac1_name_or_inst: Union[str, AnnotationCollection],
ac2_name_or_inst: Union[str, AnnotationCollection],
tag_filter: list = None,
filter_both_ac: bool = False,
filter_both_ac: bool = True,
level: str = 'tag',
include_empty_annotations: bool = True,
distance: str = 'binary',
Expand All @@ -952,8 +952,8 @@ def calculate_cohens_kappa(
ac2_name_or_inst (str): The name or instance of the second annotation collection, whose annotations will be searched for\
matches to those in the first.
tag_filter (list, optional): Which tags should be included. Defaults to `None` (all tags).
filter_both_ac (bool, optional): Whether the tag filter should be applied to both annotation collections. Defaults to `False`\
(only applied to the first collection).
filter_both_ac (bool, optional): Whether the tag filter should be applied to both annotation collections. Defaults to `True`\
(applies `tag_filter` to both collections).
level (str, optional): Whether the annotations' tags or a specified property (prefixed with 'prop:') should be compared. Defaults\
to 'tag'.
include_empty_annotations (bool, optional): If `False`, only annotations with a matching annotation in the second collection are\
Expand Down Expand Up @@ -990,10 +990,11 @@ def calculate_krippendorffs_alpha(
See the [demo notebook](https://github.com/forTEXT/gitma/blob/main/demo/notebooks/inter_annotator_agreement.ipynb) for details.

Args:
ac_names (list): List of annotation collection names for IAA calculation. If empty, all ACs in the project will be used.
tag_filter (list): List of tags that should be included for iaa calculation. If empty, all tags will be used.
filter_both_ac (bool): Whether to apply tag_filter on both ACs in the pair or just on the first AC. Default is True.
include_empty_annotations (bool): Whether to include empty annotations in the IAA data. If `False`, only annotations with a matching annotation in the second collection are\
ac_names (list): List of annotation collection names for IAA calculation. If empty, all annotation collections in the project will be used.
tag_filter (list, optional): List of tags that should be included for iaa calculation. If empty, all tags will be used.
filter_both_ac (bool, optional): Whether the tag filter should be applied to both annotation collections. Defaults to `True`\
(applies `tag_filter` to both collections).
include_empty_annotations (bool, optional): Whether to include empty annotations in the IAA data. If `False`, only annotations with a matching annotation in the second collection are\
included. Default is True.
property_filter (str, optional): Property to filter by. If None, all properties will be used.
distance (str, optional): The IAA distance function. Either 'binary' or 'interval'. See the\
Expand Down