Supplementary Material for Credibility Score Comparison: Expert vs. DeepSeek

This repository accompanies the notebook extract.ipynb, which evaluates and compares credibility scores of political statements using expert-derived and LLM-inferred approaches. The analysis supports our methodology in the IEEE SMC submission.

---

📁 Included Files

• extract.ipynb
  Contains the full experimental workflow for:

  • Loading expert and LLM-inferred credibility scores
  • Comparing score distributions
  • Computing Pearson correlation between the two methods
  • Evaluating impact on classification accuracy
  • Visualizing and analyzing disagreements

• sample_test.csv
  Expert-annotated credibility scores for a sample of political statements derived from the LIAR dataset. Each row includes:

  • Credibility Score: Derived from fact-check count features
  • Metadata about the statement and speaker

• deep_seek.csv
  Contains scores: Credibility values inferred by a Large Language Model (LLM), such as DeepSeek, based on contextual and external reasoning about the source.

• FinalEvaluatedSample.csv
  Unified file combining:

  • Credibility Score (expert)
  • Deep Seek score (LLM-inferred)
  • True Label: Ground-truth label from LIAR
  • Predicted Class: Prediction based on expert score
  • Predicted Class2: Prediction based on DeepSeek score

---

🧠 Purpose

To explore how LLM-based reasoning compares to structured, fact-count-based credibility scoring in:

• Score distribution
• Statistical correlation
• Classification performance (based on downstream label prediction)
• Discrepancy analysis

---

📊 Key Evaluation Steps

1. Descriptive Statistics & Distributions

   • Use seaborn histograms to compare score distributions between expert and DeepSeek evaluations.

2. Pearson Correlation

   • Quantify statistical similarity between the two scoring methods.

3. Confusion Matrix & Classification Report

   • Compare classification performance using:
     • Expert-based Credibility Score
     • LLM-inferred Deep Seek score

4. Error Analysis

   • Highlight statements where the two approaches lead to different predictions.

---

📌 Example Output

• Pearson correlation coefficient between expert and DeepSeek scores
• Histograms of both score distributions
• Side-by-side confusion matrices and F1-scores for both scoring approaches
• Table of discrepancies where Predicted Class ≠ Predicted Class2

---

📬 Contact

For questions or collaboration inquiries, please reach out to:

• Dimitrios P. Panagoulias
  LinkedIn

---

✅ Requirements

• Python 3.7+
• pandas, seaborn, matplotlib, scikit-learn

---

📦 Run Instructions

To reproduce the results:

1. Ensure all three .csv files are in the same directory as extract.ipynb.
2. Launch the notebook and run each cell sequentially.