diff --git a/IBM Cloud/WML/notebooks/watsonx/LLM Metrics Evals - General.ipynb b/IBM Cloud/WML/notebooks/watsonx/LLM Metrics Evals - General.ipynb index 55d7fb1..f4a4e38 100644 --- a/IBM Cloud/WML/notebooks/watsonx/LLM Metrics Evals - General.ipynb +++ b/IBM Cloud/WML/notebooks/watsonx/LLM Metrics Evals - General.ipynb @@ -1 +1,1706 @@ -{"cells":[{"cell_type":"markdown","id":"97648ee8","metadata":{"id":"a81be50301fa471b8417773161ab8ca8"},"source":["# Using IBM watsonx.governance metrics toolkit to evaluate the quality of your Prompt Template"]},{"cell_type":"code","execution_count":null,"id":"42d069ed-a362-466d-9059-897e80a95169","metadata":{"id":"6e9cd5a7-9437-47d0-a1fb-b3d6f963cec9"},"outputs":[],"source":["!pip install --upgrade ibm-watson-machine-learning | tail -n 1\n","!pip install --upgrade ibm-watson-openscale --no-cache | tail -n 1\n","!pip install --upgrade ibm-metrics-plugin --no-cache | tail -n 1"]},{"cell_type":"code","execution_count":null,"id":"6b1a0bca","metadata":{},"outputs":[],"source":["!pip install --upgrade evaluate --no-cache | tail -n 1\n","!pip install --upgrade rouge_score --no-cache | tail -n 1\n","!pip install --upgrade textstat --no-cache | tail -n 1\n","!pip install --upgrade sacrebleu --no-cache | tail -n 1\n","!pip install --upgrade sacremoses --no-cache | tail -n 1\n","!pip install --upgrade datasets==2.10.0 --no-cache | tail -n 1"]},{"cell_type":"code","execution_count":1,"id":"d73e554a-b793-4a0d-a05c-1e08ba6c3dc7","metadata":{"id":"12308564-2875-4b49-b788-85ece1964fe5"},"outputs":[],"source":["import warnings\n","warnings.filterwarnings('ignore')"]},{"cell_type":"markdown","id":"dd5bd36b","metadata":{"id":"4267839e32bd48719cf4b3389e3cadb8"},"source":["## Provision services and configure credentials"]},{"cell_type":"markdown","id":"6bbd0a71","metadata":{"id":"3e34c78b4df9420d84eb4aeaf887ebd5"},"source":["If you have not already, provision an instance of IBM Watson OpenScale using the [OpenScale link in the Cloud catalog](https://cloud.ibm.com/catalog/services/watson-openscale)."]},{"cell_type":"markdown","id":"87b4a44b","metadata":{"id":"0565e9abc066411ea4d2aff075c434f2"},"source":["Your Cloud API key can be generated by going to the [**Users** section of the Cloud console](https://cloud.ibm.com/iam#/users). From that page, click your name, scroll down to the **API Keys** section, and click **Create an IBM Cloud API key**. Give your key a name and click **Create**, then copy the created key and paste it below."]},{"cell_type":"markdown","id":"9eb0cc25","metadata":{"id":"90b9ec03867e4392883984489bda0bf6"},"source":["**NOTE:** You can also get OpenScale `API_KEY` using IBM CLOUD CLI.\n","\n","How to install IBM Cloud (bluemix) console: [instruction](https://console.bluemix.net/docs/cli/reference/ibmcloud/download_cli.html#install_use)\n","\n","How to get api key using console:\n","```\n","bx login --sso\n","bx iam api-key-create 'my_key'\n","```"]},{"cell_type":"code","execution_count":2,"id":"38d96be6","metadata":{"id":"f78243b20abd4b4d84313dacb4f02624"},"outputs":[],"source":["CLOUD_API_KEY = \"***\"\n","IAM_URL=\"https://iam.ng.bluemix.net/oidc/token\""]},{"cell_type":"markdown","id":"a7c1d906","metadata":{"id":"d5d5b8b3dd4847298dee4b065ee9c3d4"},"source":["## IBM watsonx.governance authentication"]},{"cell_type":"code","execution_count":3,"id":"94d0ce8d","metadata":{"id":"9567c5c157ae44ecabd9fdc434f16056"},"outputs":[{"data":{"text/plain":["'3.0.33'"]},"execution_count":3,"metadata":{},"output_type":"execute_result"}],"source":["from ibm_cloud_sdk_core.authenticators import IAMAuthenticator,BearerTokenAuthenticator\n","\n","from ibm_watson_openscale import *\n","from ibm_watson_openscale.supporting_classes.enums import *\n","from ibm_watson_openscale.supporting_classes import *\n","\n","\n","authenticator = IAMAuthenticator(apikey=CLOUD_API_KEY)\n","client = APIClient(authenticator=authenticator)\n","client.version"]},{"cell_type":"markdown","id":"d45542ea","metadata":{"id":"d1aa94f70d4d4f77bc87a0b06139af3c"},"source":["# Common Imports"]},{"cell_type":"code","execution_count":4,"id":"369dc6b8","metadata":{"id":"76620bd0a7784cb4801d10f3d93fa6a1"},"outputs":[],"source":["from ibm_metrics_plugin.metrics.llm.utils.constants import LLMTextMetricGroup\n","from ibm_metrics_plugin.metrics.llm.utils.constants import LLMGenerationMetrics\n","from ibm_metrics_plugin.metrics.llm.utils.constants import LLMSummarizationMetrics\n","from ibm_metrics_plugin.metrics.llm.utils.constants import LLMQAMetrics\n","from ibm_metrics_plugin.metrics.llm.utils.constants import LLMClassificationMetrics\n","from ibm_metrics_plugin.metrics.llm.utils.constants import HAP_SCORE\n","from ibm_metrics_plugin.metrics.llm.utils.constants import PII_DETECTION"]},{"cell_type":"markdown","id":"e9db10c6","metadata":{"id":"a68e9be2a0a0438d822679c5c1728097"},"source":["# Evaluating Summarization output from AWS/anthropic.claude-v2"]},{"cell_type":"markdown","id":"e927e928","metadata":{"id":"abab31d97a0a460a90897da08f36add0"},"source":["## Test data containing the summarization output from model and the reference data"]},{"cell_type":"code","execution_count":5,"id":"9f284b78","metadata":{"id":"8f63311275114169ac00dc3096352dd0"},"outputs":[{"name":"stdout","output_type":"stream","text":["--2023-12-05 13:21:23-- https://raw.githubusercontent.com/ravichamarthy/custom_metrics/main/llm_content.csv\n","Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...\n","Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 31230 (30K) [text/plain]\n","Saving to: ‘llm_content.csv’\n","\n","llm_content.csv 100%[===================>] 30.50K --.-KB/s in 0.002s \n","\n","2023-12-05 13:21:24 (16.5 MB/s) - ‘llm_content.csv’ saved [31230/31230]\n","\n"]}],"source":["!rm -fr llm_content.csv\n","!wget \"https://raw.githubusercontent.com/ravichamarthy/custom_metrics/main/llm_content.csv\""]},{"cell_type":"code","execution_count":6,"id":"54fd2169","metadata":{"id":"c03c61b496f64edc8b9bd3fc3ff03899"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
input_textgenerated_summaryreference_summary_1reference_summary_2
0Scientists have discovered a new species of de...New bioluminescent fish species found in deep ...Discovery of deep-sea fish emitting soothing l...Scientists find new bioluminescent fish specie...
1An international team of astronomers has ident...Distant exoplanet\\'s water vapor-filled atmosp...Astronomers identify exoplanet with water vapo...Discovery of exoplanet with water vapor in its...
2Researchers have developed a novel nanotechnol...New nanotechnology-based cancer treatment demo...Researchers create cancer treatment using nano...Innovative cancer treatment utilizing nanotech...
3A new app is aiming to reduce food waste by co...App connects local restaurants with customers ...New sustainability-focused app facilitates sal...Initiative to reduce food waste involves app c...
4Archaeologists have uncovered an ancient city ...Ancient city dating back over 4,000 years disc...Archaeological find in Iraq reveals ancient ci...Discovery of 4,000-year-old ancient city in Ku...
\n","
"],"text/plain":[" input_text \\\n","0 Scientists have discovered a new species of de... \n","1 An international team of astronomers has ident... \n","2 Researchers have developed a novel nanotechnol... \n","3 A new app is aiming to reduce food waste by co... \n","4 Archaeologists have uncovered an ancient city ... \n","\n"," generated_summary \\\n","0 New bioluminescent fish species found in deep ... \n","1 Distant exoplanet\\'s water vapor-filled atmosp... \n","2 New nanotechnology-based cancer treatment demo... \n","3 App connects local restaurants with customers ... \n","4 Ancient city dating back over 4,000 years disc... \n","\n"," reference_summary_1 \\\n","0 Discovery of deep-sea fish emitting soothing l... \n","1 Astronomers identify exoplanet with water vapo... \n","2 Researchers create cancer treatment using nano... \n","3 New sustainability-focused app facilitates sal... \n","4 Archaeological find in Iraq reveals ancient ci... \n","\n"," reference_summary_2 \n","0 Scientists find new bioluminescent fish specie... \n","1 Discovery of exoplanet with water vapor in its... \n","2 Innovative cancer treatment utilizing nanotech... \n","3 Initiative to reduce food waste involves app c... \n","4 Discovery of 4,000-year-old ancient city in Ku... "]},"execution_count":6,"metadata":{},"output_type":"execute_result"}],"source":["import pandas as pd\n","import numpy as np\n","llm_data_all = pd.read_csv(\"llm_content.csv\")\n","llm_data_all.head()"]},{"cell_type":"code","execution_count":7,"id":"25ba046e","metadata":{},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
input_textgenerated_summaryreference_summary_1reference_summary_2
0Scientists have discovered a new species of de...New bioluminescent fish species found in deep ...Discovery of deep-sea fish emitting soothing l...Scientists find new bioluminescent fish specie...
1An international team of astronomers has ident...Distant exoplanet\\'s water vapor-filled atmosp...Astronomers identify exoplanet with water vapo...Discovery of exoplanet with water vapor in its...
2Researchers have developed a novel nanotechnol...New nanotechnology-based cancer treatment demo...Researchers create cancer treatment using nano...Innovative cancer treatment utilizing nanotech...
3A new app is aiming to reduce food waste by co...App connects local restaurants with customers ...New sustainability-focused app facilitates sal...Initiative to reduce food waste involves app c...
4Archaeologists have uncovered an ancient city ...Ancient city dating back over 4,000 years disc...Archaeological find in Iraq reveals ancient ci...Discovery of 4,000-year-old ancient city in Ku...
\n","
"],"text/plain":[" input_text \\\n","0 Scientists have discovered a new species of de... \n","1 An international team of astronomers has ident... \n","2 Researchers have developed a novel nanotechnol... \n","3 A new app is aiming to reduce food waste by co... \n","4 Archaeologists have uncovered an ancient city ... \n","\n"," generated_summary \\\n","0 New bioluminescent fish species found in deep ... \n","1 Distant exoplanet\\'s water vapor-filled atmosp... \n","2 New nanotechnology-based cancer treatment demo... \n","3 App connects local restaurants with customers ... \n","4 Ancient city dating back over 4,000 years disc... \n","\n"," reference_summary_1 \\\n","0 Discovery of deep-sea fish emitting soothing l... \n","1 Astronomers identify exoplanet with water vapo... \n","2 Researchers create cancer treatment using nano... \n","3 New sustainability-focused app facilitates sal... \n","4 Archaeological find in Iraq reveals ancient ci... \n","\n"," reference_summary_2 \n","0 Scientists find new bioluminescent fish specie... \n","1 Discovery of exoplanet with water vapor in its... \n","2 Innovative cancer treatment utilizing nanotech... \n","3 Initiative to reduce food waste involves app c... \n","4 Discovery of 4,000-year-old ancient city in Ku... "]},"execution_count":7,"metadata":{},"output_type":"execute_result"}],"source":["llm_data = llm_data_all.head(10)\n","llm_data.head()"]},{"cell_type":"code","execution_count":8,"id":"300c385f","metadata":{"id":"091c264708cc4159969a4690daf07886"},"outputs":[],"source":["df_input = llm_data[['input_text']].copy()\n","df_output = llm_data[['generated_summary']].copy()\n","df_reference = llm_data[['reference_summary_1']].copy()"]},{"cell_type":"markdown","id":"d6f7f842","metadata":{"id":"74a9de9b2bb84c35b87fa1d2abc5770c"},"source":["## Metrics configuration for evaluation"]},{"cell_type":"code","execution_count":9,"id":"f8262dfe-38b1-4d6f-88dd-c3a4c3b11996","metadata":{"id":"95e9409a-ccb9-409b-ae04-b0bdf392faac"},"outputs":[],"source":["metric_config = { \n"," \"configuration\": {\n"," LLMTextMetricGroup.SUMMARIZATION.value: {\n"," LLMSummarizationMetrics.ROUGE_SCORE.value: {},\n"," LLMSummarizationMetrics.SARI.value: {},\n"," LLMSummarizationMetrics.METEOR.value: {},\n"," LLMSummarizationMetrics.NORMALIZED_RECALL.value: {},\n"," LLMSummarizationMetrics.NORMALIZED_PRECISION.value: {},\n"," LLMSummarizationMetrics.NORMALIZED_F1_SCORE.value: {},\n"," LLMSummarizationMetrics.COSINE_SIMILARITY.value: {},\n"," LLMSummarizationMetrics.JACCARD_SIMILARITY.value: {},\n"," LLMSummarizationMetrics.BLEU.value: {},\n"," LLMSummarizationMetrics.FLESCH.value: {}\n"," }\n"," }\n","}"]},{"cell_type":"markdown","id":"240deb44","metadata":{"id":"f55e2339070c48608474f370abd949b7"},"source":["## Summarization Metrics Evaluation"]},{"cell_type":"code","execution_count":10,"id":"5efb037b-ff41-492f-ab4e-5fdb35d4da03","metadata":{"id":"031a7f52-ba90-46d4-aac6-09d8f76e6d3f"},"outputs":[{"name":"stdout","output_type":"stream","text":["Please install adversarial-robustness-toolbox package\n","please install adversarial-robustness-toolbox package\n","please install adversarial-robustness-toolbox package\n","please install adversarial-robustness-toolbox package\n","please install adversarial-robustness-toolbox package\n","Please install `blanc` package\n","Please install watson_nlp package to detect PII information\n","Please install watson_nlp package to detect PII information\n","Please install watson_nlp package to detect PII information\n","Please install watson_nlp package to compute HAP score\n","Please install watson_nlp package to compute HAP score\n","Please install watson_nlp package to compute HAP score\n"]},{"name":"stderr","output_type":"stream","text":["[nltk_data] Downloading package wordnet to /home/wsuser/nltk_data...\n","[nltk_data] Package wordnet is already up-to-date!\n","[nltk_data] Downloading package punkt to /home/wsuser/nltk_data...\n","[nltk_data] Package punkt is already up-to-date!\n","[nltk_data] Downloading package omw-1.4 to /home/wsuser/nltk_data...\n","[nltk_data] Package omw-1.4 is already up-to-date!\n"]}],"source":["import json\n","result = client.llm_metrics.compute_metrics(metric_config,df_input,df_output, df_reference)"]},{"cell_type":"markdown","id":"1a12d146","metadata":{"id":"4bb4d7ebf78a4589b122508635c92572"},"source":["## Evaluated Metrics"]},{"cell_type":"code","execution_count":11,"id":"426fd1da-329e-40e0-bce1-f99bb37dea3c","metadata":{"id":"fd2fa6e4-c832-4ed6-bb10-8bae433e95f6"},"outputs":[{"name":"stdout","output_type":"stream","text":["{\n"," \"flesch\": {\n"," \"flesch_reading_ease\": {\n"," \"metric_value\": 20.854000000000003,\n"," \"mean\": 20.854000000000003,\n"," \"min\": -50.69,\n"," \"max\": 67.76,\n"," \"std\": 32.72954787344304\n"," },\n"," \"flesch_kincaid_grade\": {\n"," \"metric_value\": 13.64,\n"," \"mean\": 13.64,\n"," \"min\": 6.8,\n"," \"max\": 23.3,\n"," \"std\": 4.3260143319226305\n"," }\n"," },\n"," \"bleu\": {\n"," \"precisions\": [\n"," 0.35947712418300654,\n"," 0.11188811188811189,\n"," 0.022556390977443608,\n"," 0.0\n"," ],\n"," \"brevity_penalty\": 0.8548213791906977,\n"," \"length_ratio\": 0.864406779661017,\n"," \"translation_length\": 153,\n"," \"reference_length\": 177,\n"," \"metric_value\": 0.0\n"," },\n"," \"sari\": {\n"," \"metric_value\": 44.66559626908473\n"," },\n"," \"rouge_score\": {\n"," \"rouge1\": {\n"," \"metric_value\": 0.3246\n"," },\n"," \"rouge2\": {\n"," \"metric_value\": 0.1008\n"," },\n"," \"rougeL\": {\n"," \"metric_value\": 0.2607\n"," },\n"," \"rougeLsum\": {\n"," \"metric_value\": 0.2607\n"," }\n"," },\n"," \"normalized_recall\": {\n"," \"metric_value\": 0.24841224951519067,\n"," \"mean\": 0.24841224951519067,\n"," \"min\": 0.058823529411764705,\n"," \"max\": 0.4166666666666667,\n"," \"std\": 0.11173847145056019\n"," },\n"," \"cosine_similarity\": {\n"," \"metric_value\": 0.21174521445360694,\n"," \"mean\": 0.21174521445360694,\n"," \"min\": 0.10354961707236612,\n"," \"max\": 0.3212096323232406,\n"," \"std\": 0.06336375081674754\n"," },\n"," \"jaccard_similarity\": {\n"," \"metric_value\": 0.1400317917964095,\n"," \"mean\": 0.1400317917964095,\n"," \"min\": 0.034482758620689655,\n"," \"max\": 0.25,\n"," \"std\": 0.06850576604602586\n"," },\n"," \"meteor\": {\n"," \"metric_value\": 0.2511393214238763\n"," },\n"," \"normalized_precision\": {\n"," \"metric_value\": 0.2845611333111333,\n"," \"mean\": 0.2845611333111333,\n"," \"min\": 0.07692307692307693,\n"," \"max\": 0.42857142857142855,\n"," \"std\": 0.10652676547509347\n"," },\n"," \"normalized_f1\": {\n"," \"metric_value\": 0.26192560128044,\n"," \"mean\": 0.26192560128044,\n"," \"min\": 0.06666666666666667,\n"," \"max\": 0.39999999999999997,\n"," \"std\": 0.104247533868887\n"," }\n","}\n"]}],"source":["print(json.dumps(result,indent=2))"]},{"cell_type":"markdown","id":"ce32e5f6","metadata":{"id":"9ed25962da7b418b866ff59d66a72140"},"source":["# Evaluating Content Generation output from the Foundation Model"]},{"cell_type":"markdown","id":"d2ecae67","metadata":{"id":"851c73b2dcdb496380966cb4a57cd433"},"source":["## Test data containing the content generation output from model and the reference data"]},{"cell_type":"code","execution_count":12,"id":"4e3a71ca","metadata":{"id":"ba2dad47c2bd400d9d41deb03cda290b"},"outputs":[{"name":"stdout","output_type":"stream","text":["--2023-12-05 13:21:49-- https://raw.githubusercontent.com/ravichamarthy/custom_metrics/main/llm_content_generation.csv\n","Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.108.133, ...\n","Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 11794 (12K) [text/plain]\n","Saving to: ‘llm_content_generation.csv’\n","\n","llm_content_generat 100%[===================>] 11.52K --.-KB/s in 0.001s \n","\n","2023-12-05 13:21:50 (11.8 MB/s) - ‘llm_content_generation.csv’ saved [11794/11794]\n","\n"]}],"source":["!rm -fr llm_content_generation.csv\n","!wget \"https://raw.githubusercontent.com/ravichamarthy/custom_metrics/main/llm_content_generation.csv\""]},{"cell_type":"code","execution_count":13,"id":"baeffc0c","metadata":{"id":"c9de8a2fbe5642a4871bd3bf407e81b5"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
questiongenerated_textreference_text
0What are the benefits of regular exercise?Regular exercise has numerous benefits, includ...Regular exercise has numerous benefits, includ...
1What is the process of photosynthesis?Photosynthesis is the process by which plants ...Photosynthesis is the process by which plants ...
2What are the key features of a smartphone?A smartphone is a mobile device that typically...A smartphone is a mobile device that typically...
3How does the immune system work?The immune system is a complex network of cell...The immune system is a complex network of cell...
4What is the capital of France?The capital of France is Paris, which is known...The capital of France is Paris, which is known...
\n","
"],"text/plain":[" question \\\n","0 What are the benefits of regular exercise? \n","1 What is the process of photosynthesis? \n","2 What are the key features of a smartphone? \n","3 How does the immune system work? \n","4 What is the capital of France? \n","\n"," generated_text \\\n","0 Regular exercise has numerous benefits, includ... \n","1 Photosynthesis is the process by which plants ... \n","2 A smartphone is a mobile device that typically... \n","3 The immune system is a complex network of cell... \n","4 The capital of France is Paris, which is known... \n","\n"," reference_text \n","0 Regular exercise has numerous benefits, includ... \n","1 Photosynthesis is the process by which plants ... \n","2 A smartphone is a mobile device that typically... \n","3 The immune system is a complex network of cell... \n","4 The capital of France is Paris, which is known... "]},"execution_count":13,"metadata":{},"output_type":"execute_result"}],"source":["data = pd.read_csv(\"llm_content_generation.csv\")\n","data.head()"]},{"cell_type":"code","execution_count":14,"id":"79f87d99","metadata":{"id":"b54090c5f78b4335bc0a5f449d1e8c26"},"outputs":[],"source":["df_input = data[['question']].copy()\n","df_output = data[['generated_text']].copy()\n","df_reference = data[['reference_text']].copy()"]},{"cell_type":"markdown","id":"ca49afba","metadata":{"id":"bbede258f6cd425e8c8f60379e47dd16"},"source":["## Metrics configuration for evaluation"]},{"cell_type":"code","execution_count":15,"id":"736f09dd","metadata":{"id":"57c6964b-def7-4fec-b5e9-c68e71a08d43"},"outputs":[],"source":["metric_config = { \n"," #All Common parameters goes here \n"," \"configuration\": { \n"," LLMTextMetricGroup.GENERATION.value: { # metric group \n"," LLMGenerationMetrics.BLEU.value: {},\n"," LLMGenerationMetrics.ROUGE_SCORE.value: {},\n"," LLMGenerationMetrics.FLESCH.value: {},\n"," LLMGenerationMetrics.METEOR.value: {}, \n"," LLMGenerationMetrics.NORMALIZED_RECALL.value: {},\n"," LLMGenerationMetrics.NORMALIZED_PRECISION.value: {},\n"," LLMGenerationMetrics.NORMALIZED_F1_SCORE.value: {} \n"," } \n"," }\n","}"]},{"cell_type":"markdown","id":"a4a283cc","metadata":{"id":"3c2039f5b47b415686a9fd5c6d7797b3"},"source":["## Content Generation Metrics Evaluation"]},{"cell_type":"code","execution_count":16,"id":"c656fc82","metadata":{"id":"04e3359e-693c-4f4e-a0f9-4d41227af409"},"outputs":[{"name":"stderr","output_type":"stream","text":["[nltk_data] Downloading package wordnet to /home/wsuser/nltk_data...\n","[nltk_data] Package wordnet is already up-to-date!\n","[nltk_data] Downloading package punkt to /home/wsuser/nltk_data...\n","[nltk_data] Package punkt is already up-to-date!\n","[nltk_data] Downloading package omw-1.4 to /home/wsuser/nltk_data...\n","[nltk_data] Package omw-1.4 is already up-to-date!\n"]}],"source":["result = client.llm_metrics.compute_metrics(metric_config,df_input,df_output, df_reference)"]},{"cell_type":"markdown","id":"61cb5446","metadata":{"id":"9d63fa0a05584ad18d13d6924ffd8602"},"source":["## Evaluated Metrics"]},{"cell_type":"code","execution_count":17,"id":"cfe539f8","metadata":{"id":"91fce6ecb3054e4081ab5098a2b4eaa9"},"outputs":[{"name":"stdout","output_type":"stream","text":["{\n"," \"flesch\": {\n"," \"flesch_reading_ease\": {\n"," \"metric_value\": 39.10217391304347,\n"," \"mean\": 39.10217391304347,\n"," \"min\": -11.44,\n"," \"max\": 69.62,\n"," \"std\": 20.153544505710833\n"," },\n"," \"flesch_kincaid_grade\": {\n"," \"metric_value\": 12.673913043478263,\n"," \"mean\": 12.673913043478263,\n"," \"min\": 8.0,\n"," \"max\": 18.6,\n"," \"std\": 3.2043743730833554\n"," }\n"," },\n"," \"bleu\": {\n"," \"precisions\": [\n"," 1.0,\n"," 0.9949174078780177,\n"," 0.9947643979057592,\n"," 0.9946018893387314\n"," ],\n"," \"brevity_penalty\": 0.7138823993242189,\n"," \"length_ratio\": 0.7479224376731302,\n"," \"translation_length\": 810,\n"," \"reference_length\": 1083,\n"," \"metric_value\": 0.711075655695426\n"," },\n"," \"rouge_score\": {\n"," \"rouge1\": {\n"," \"metric_value\": 0.8451\n"," },\n"," \"rouge2\": {\n"," \"metric_value\": 0.8402\n"," },\n"," \"rougeL\": {\n"," \"metric_value\": 0.8451\n"," },\n"," \"rougeLsum\": {\n"," \"metric_value\": 0.8451\n"," }\n"," },\n"," \"normalized_recall\": {\n"," \"metric_value\": 0.7335547397366776,\n"," \"mean\": 0.7335547397366776,\n"," \"min\": 0.5,\n"," \"max\": 0.9459459459459459,\n"," \"std\": 0.09340832867437678\n"," },\n"," \"meteor\": {\n"," \"metric_value\": 0.738039475927929\n"," },\n"," \"normalized_precision\": {\n"," \"metric_value\": 1.0,\n"," \"mean\": 1.0,\n"," \"min\": 1.0,\n"," \"max\": 1.0,\n"," \"std\": 0.0\n"," },\n"," \"normalized_f1\": {\n"," \"metric_value\": 0.8428460742183366,\n"," \"mean\": 0.8428460742183366,\n"," \"min\": 0.6666666666666666,\n"," \"max\": 0.9722222222222222,\n"," \"std\": 0.06436358907082648\n"," }\n","}\n"]}],"source":["print(json.dumps(result,indent=2))"]},{"cell_type":"markdown","id":"c7cae48d","metadata":{"id":"e7521601-375c-43b0-84f4-7f1496d8fb03"},"source":["# Evaluating Question and Answering output from the Foundation Model"]},{"cell_type":"markdown","id":"40d8c808","metadata":{"id":"e06857c26fbd40629b5b619221cdfea0"},"source":["## Test data containing the question and answer output from model and the reference data"]},{"cell_type":"code","execution_count":18,"id":"575ba1e2","metadata":{"id":"65783a820d87443d9c94136b32ab5576"},"outputs":[{"name":"stdout","output_type":"stream","text":["--2023-12-05 13:21:56-- https://raw.githubusercontent.com/ravichamarthy/custom_metrics/main/llm_content_qa.csv\n","Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.110.133, ...\n","Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 3109 (3.0K) [text/plain]\n","Saving to: ‘llm_content_qa.csv’\n","\n","llm_content_qa.csv 100%[===================>] 3.04K --.-KB/s in 0s \n","\n","2023-12-05 13:21:56 (9.89 MB/s) - ‘llm_content_qa.csv’ saved [3109/3109]\n","\n"]}],"source":["!rm -fr llm_content_qa.csv\n","!wget \"https://raw.githubusercontent.com/ravichamarthy/custom_metrics/main/llm_content_qa.csv\""]},{"cell_type":"code","execution_count":19,"id":"0ea08d02","metadata":{"id":"957ccd450d574263abd29ab7a2ab6239"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
questionanswers
0who did chris carter play for last yearMilwaukee Brewers
1what is the latest version of safari on macSafari 11
2when did bucharest become the capital of romania1862
3who did jeffrey dean morgan play on supernaturalJohn Eric Winchester
4who is the shortest man that ever livedChandra Bahadur Dangi
\n","
"],"text/plain":[" question answers\n","0 who did chris carter play for last year Milwaukee Brewers\n","1 what is the latest version of safari on mac Safari 11\n","2 when did bucharest become the capital of romania 1862\n","3 who did jeffrey dean morgan play on supernatural John Eric Winchester\n","4 who is the shortest man that ever lived Chandra Bahadur Dangi"]},"execution_count":19,"metadata":{},"output_type":"execute_result"}],"source":["data = pd.read_csv(\"llm_content_qa.csv\")\n","data.head()"]},{"cell_type":"code","execution_count":20,"id":"9f630a4e","metadata":{"id":"496b31ad4a83445b87b196501460bb3b"},"outputs":[],"source":["df_input = data[['question']].copy()\n","df_output = data[['answers']].copy()\n","df_reference = data[['answers']].copy()"]},{"cell_type":"markdown","id":"547e4494","metadata":{"id":"72f64a14008744a18abbaf0dffc8d175"},"source":["## Metrics configuration for evaluation"]},{"cell_type":"code","execution_count":21,"id":"44db9e2a","metadata":{"id":"b2e0e82f3a5e4cf787165c3c3f20d40d"},"outputs":[],"source":["metric_config = { \n"," #All Common parameters goes here \n"," \"configuration\": { \n"," LLMTextMetricGroup.QA.value: { # metric group \n"," LLMQAMetrics.EXACT_MATCH.value: {},\n"," LLMQAMetrics.ROUGE_SCORE.value: {},\n"," LLMQAMetrics.BLEU.value: {} \n"," } \n"," }\n","}"]},{"cell_type":"markdown","id":"49768a30","metadata":{"id":"2828f5cb24f244d38c5db891ee0afbd6"},"source":["## Question and Answering Metrics Evaluation"]},{"cell_type":"code","execution_count":22,"id":"a6bc9217","metadata":{"id":"57d93e69ad2846a3aa346207ccc57af5"},"outputs":[],"source":["result = client.llm_metrics.compute_metrics(metric_config,df_input,df_output, df_reference)"]},{"cell_type":"markdown","id":"9b38425f","metadata":{"id":"cb3227d0d3c543a4807b398088b93158"},"source":["## Evaluated Metrics"]},{"cell_type":"code","execution_count":23,"id":"036ffb4e","metadata":{"id":"55ad771c8f9845c18ca125132da441ac"},"outputs":[{"name":"stdout","output_type":"stream","text":["{\n"," \"exact_match\": {\n"," \"metric_value\": 1.0\n"," },\n"," \"bleu\": {\n"," \"precisions\": [\n"," 1.0,\n"," 1.0,\n"," 1.0,\n"," 1.0\n"," ],\n"," \"brevity_penalty\": 1.0,\n"," \"length_ratio\": 1.0,\n"," \"translation_length\": 133,\n"," \"reference_length\": 133,\n"," \"metric_value\": 1.0\n"," },\n"," \"rouge_score\": {\n"," \"rouge1\": {\n"," \"metric_value\": 1.0\n"," },\n"," \"rouge2\": {\n"," \"metric_value\": 0.74\n"," },\n"," \"rougeL\": {\n"," \"metric_value\": 1.0\n"," },\n"," \"rougeLsum\": {\n"," \"metric_value\": 1.0\n"," }\n"," }\n","}\n"]}],"source":["print(json.dumps(result,indent=2))"]},{"cell_type":"markdown","id":"d078c15b","metadata":{"id":"8eec33aaaad94f9db377943ebd4350b0"},"source":["# Evaluating Text Classification output from the Foundation Model"]},{"cell_type":"markdown","id":"37c6091a","metadata":{"id":"4badf20522e44bb2a077edf95e52682a"},"source":["## Test data containing the text classification output from model and the reference data"]},{"cell_type":"code","execution_count":24,"id":"6a4bc249","metadata":{"id":"757573f7fa6a4c64815e6419708ce15b"},"outputs":[{"name":"stdout","output_type":"stream","text":["--2023-12-05 13:22:01-- https://raw.githubusercontent.com/ravichamarthy/custom_metrics/main/llm_content_classification.csv\n","Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.108.133, 185.199.109.133, ...\n","Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 480803 (470K) [text/plain]\n","Saving to: ‘llm_content_classification.csv’\n","\n","llm_content_classif 100%[===================>] 469.53K --.-KB/s in 0.009s \n","\n","2023-12-05 13:22:01 (51.2 MB/s) - ‘llm_content_classification.csv’ saved [480803/480803]\n","\n"]}],"source":["!rm -fr llm_content_classification.csv\n","!wget \"https://raw.githubusercontent.com/ravichamarthy/custom_metrics/main/llm_content_classification.csv\""]},{"cell_type":"code","execution_count":25,"id":"e608da47","metadata":{"id":"69cfb80c15a34c0199bcbc3c04d7f63a"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
labeltext
0hamGo until jurong point, crazy.. Available only ...
1hamOk lar... Joking wif u oni...
2spamFree entry in 2 a wkly comp to win FA Cup fina...
3hamU dun say so early hor... U c already then say...
4hamNah I don't think he goes to usf, he lives aro...
\n","
"],"text/plain":[" label text\n","0 ham Go until jurong point, crazy.. Available only ...\n","1 ham Ok lar... Joking wif u oni...\n","2 spam Free entry in 2 a wkly comp to win FA Cup fina...\n","3 ham U dun say so early hor... U c already then say...\n","4 ham Nah I don't think he goes to usf, he lives aro..."]},"execution_count":25,"metadata":{},"output_type":"execute_result"}],"source":["data = pd.read_csv(\"llm_content_classification.csv\")\n","data.head()"]},{"cell_type":"code","execution_count":26,"id":"e6257bd3","metadata":{"id":"f3e36646c7024931ab197cfbcedc59ea"},"outputs":[],"source":["data['label'] = data['label'].replace({'ham': 0, 'spam': 1})"]},{"cell_type":"code","execution_count":27,"id":"ab21b6b2","metadata":{"id":"bffdedf11c8242ba9183746802dd1404"},"outputs":[],"source":["df_input = data[['text']].copy()\n","df_output = data[['label']].copy()\n","df_reference = data[['label']].copy()"]},{"cell_type":"markdown","id":"d77893c6","metadata":{"id":"2c50933f6d6a41bf86d8084b6904541a"},"source":["## Make some realistic reference column"]},{"cell_type":"code","execution_count":28,"id":"3b675c84","metadata":{"id":"94a8e911986a450888c8d79d75ff3c63"},"outputs":[],"source":["shuffled_column = df_reference['label'].sample(frac=1).reset_index(drop=True)\n","df_reference['label'] = shuffled_column"]},{"cell_type":"markdown","id":"cb44ac10","metadata":{"id":"25eb2beeed794f2a97b8eca274e1d670"},"source":["## Metrics configuration for evaluation"]},{"cell_type":"code","execution_count":29,"id":"a82d2011","metadata":{"id":"d95388d5677e41819a29d4a61c490cc2"},"outputs":[],"source":["metric_config = { \n"," #All Common parameters goes here \n"," \"configuration\": { \n"," LLMTextMetricGroup.CLASSIFICATION.value: { # metric group \n"," LLMClassificationMetrics.ACCURACY.value: {},\n"," LLMClassificationMetrics.PRECISION.value: {},\n"," LLMClassificationMetrics.RECALL.value: {},\n"," LLMClassificationMetrics.F1_SCORE.value: {},\n"," LLMClassificationMetrics.MATTHEWS_CORRELATION.value: {}, \n"," } \n"," }\n","}"]},{"cell_type":"markdown","id":"641f8fb9","metadata":{"id":"824900df910d4302b73af2e615526f32"},"source":["## Text Classification Metrics Evaluation"]},{"cell_type":"code","execution_count":30,"id":"f792b218","metadata":{"id":"d14b65d169e7497e8f4587e944ef8924"},"outputs":[],"source":["result = client.llm_metrics.compute_metrics(metric_config,df_input,df_output, df_reference)"]},{"cell_type":"markdown","id":"3a5744fe","metadata":{"id":"9689b3b4e4f34459b8f5cf4d9c76d365"},"source":["## Evaluated Metrics"]},{"cell_type":"code","execution_count":31,"id":"69f592c0","metadata":{"id":"5c42d600f12142a5904f71a002ad25c3"},"outputs":[{"name":"stdout","output_type":"stream","text":["{\n"," \"accuracy\": {\n"," \"accuracy\": 0.7674084709260589\n"," },\n"," \"precision\": {\n"," \"precision\": 0.13253012048192772\n"," },\n"," \"matthews_correlation\": {\n"," \"matthews_correlation\": -0.001770397652787315\n"," },\n"," \"recall\": {\n"," \"recall\": 0.13253012048192772\n"," },\n"," \"f1\": {\n"," \"f1\": 0.13253012048192772\n"," }\n","}\n"]}],"source":["print(json.dumps(result,indent=2))"]},{"cell_type":"markdown","id":"5669fb47","metadata":{"id":"e94e539d98704ac99e31c7d88b700140"},"source":["Author: kishore.patel@in.ibm.com , ravi.chamarthy@in.ibm.com"]}],"metadata":{"kernelspec":{"display_name":"Python 3.10","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.13"}},"nbformat":4,"nbformat_minor":5} +{ + "cells": [ + { + "cell_type": "markdown", + "id": "97648ee8", + "metadata": { + "id": "a81be50301fa471b8417773161ab8ca8" + }, + "source": [ + "# Using IBM watsonx.governance metrics toolkit to evaluate the quality of your Prompt Template" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "42d069ed-a362-466d-9059-897e80a95169", + "metadata": { + "id": "6e9cd5a7-9437-47d0-a1fb-b3d6f963cec9" + }, + "outputs": [], + "source": [ + "!pip install --upgrade ibm-watson-machine-learning | tail -n 1\n", + "!pip install --upgrade ibm-watson-openscale --no-cache | tail -n 1\n", + "!pip install --upgrade ibm-metrics-plugin --no-cache | tail -n 1" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6b1a0bca", + "metadata": {}, + "outputs": [], + "source": [ + "!pip install --upgrade evaluate --no-cache | tail -n 1\n", + "!pip install --upgrade rouge_score --no-cache | tail -n 1\n", + "!pip install --upgrade textstat --no-cache | tail -n 1\n", + "!pip install --upgrade sacrebleu --no-cache | tail -n 1\n", + "!pip install --upgrade sacremoses --no-cache | tail -n 1\n", + "!pip install --upgrade torchmetrics --no-cache | tail -n 1\n", + "!pip install --upgrade datasets==2.10.0 --no-cache | tail -n 1" + ] + }, + { + "cell_type": "code", + "execution_count": 46, + "id": "d73e554a-b793-4a0d-a05c-1e08ba6c3dc7", + "metadata": { + "id": "12308564-2875-4b49-b788-85ece1964fe5" + }, + "outputs": [], + "source": [ + "import warnings\n", + "warnings.filterwarnings('ignore')" + ] + }, + { + "cell_type": "markdown", + "id": "ee9dd250-540b-4180-af01-66256327548d", + "metadata": {}, + "source": [ + "### Set the ARTIFACTORY_USERNAME and ARTIFACTORY_API_KEY to install watson_nlp from artifactory\n", + "Watson NLP artifacts are stored in Artifactory repository. To pull Watson NLP’s dependencies or download models, you must configure 2 variables, ARTIFACTORY_USERNAME and ARTIFACTORY_API_KEY. See instructions to get ARTIFACTORY_USERNAME and ARTIFACTORY_API_KEY\n", + "\n", + "Note: Replace ARTIFACTORY_USERNAME and ARTIFACTORY_API_KEY with your artifactory credentials.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 97, + "id": "ed39654e-230d-4d23-bc79-7fefde7b691c", + "metadata": {}, + "outputs": [], + "source": [ + "ARTIFACTORY_USERNAME = \"\"\n", + "ARTIFACTORY_API_KEY = \"\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3560a4e9-2333-4685-9586-b31da40db1af", + "metadata": {}, + "outputs": [], + "source": [ + "#install watson_nlp from the artifactory\n", + "!pip install --index-url https://:@na.artifactory.swg-devops.com/artifactory/api/pypi/wcp-ai-foundation-team-pypi-virtual/simple 'watson_nlp[all]' --no-cache | tail -n 1\n" + ] + }, + { + "cell_type": "markdown", + "id": "dd5bd36b", + "metadata": { + "id": "4267839e32bd48719cf4b3389e3cadb8" + }, + "source": [ + "## Provision services and configure credentials" + ] + }, + { + "cell_type": "markdown", + "id": "6bbd0a71", + "metadata": { + "id": "3e34c78b4df9420d84eb4aeaf887ebd5" + }, + "source": [ + "If you have not already, provision an instance of IBM Watson OpenScale using the [OpenScale link in the Cloud catalog](https://cloud.ibm.com/catalog/services/watson-openscale)." + ] + }, + { + "cell_type": "markdown", + "id": "87b4a44b", + "metadata": { + "id": "0565e9abc066411ea4d2aff075c434f2" + }, + "source": [ + "Your Cloud API key can be generated by going to the [**Users** section of the Cloud console](https://cloud.ibm.com/iam#/users). From that page, click your name, scroll down to the **API Keys** section, and click **Create an IBM Cloud API key**. Give your key a name and click **Create**, then copy the created key and paste it below." + ] + }, + { + "cell_type": "markdown", + "id": "9eb0cc25", + "metadata": { + "id": "90b9ec03867e4392883984489bda0bf6" + }, + "source": [ + "**NOTE:** You can also get OpenScale `API_KEY` using IBM CLOUD CLI.\n", + "\n", + "How to install IBM Cloud (bluemix) console: [instruction](https://console.bluemix.net/docs/cli/reference/ibmcloud/download_cli.html#install_use)\n", + "\n", + "How to get api key using console:\n", + "```\n", + "bx login --sso\n", + "bx iam api-key-create 'my_key'\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": 102, + "id": "38d96be6", + "metadata": { + "id": "f78243b20abd4b4d84313dacb4f02624" + }, + "outputs": [], + "source": [ + "CLOUD_API_KEY = \"***\"\n", + "IAM_URL=\"https://iam.ng.bluemix.net/oidc/token\"" + ] + }, + { + "cell_type": "markdown", + "id": "a7c1d906", + "metadata": { + "id": "d5d5b8b3dd4847298dee4b065ee9c3d4" + }, + "source": [ + "## IBM watsonx.governance authentication" + ] + }, + { + "cell_type": "code", + "execution_count": 103, + "id": "94d0ce8d", + "metadata": { + "id": "9567c5c157ae44ecabd9fdc434f16056" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "'3.0.34'" + ] + }, + "execution_count": 103, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from ibm_cloud_sdk_core.authenticators import IAMAuthenticator,BearerTokenAuthenticator\n", + "\n", + "from ibm_watson_openscale import *\n", + "from ibm_watson_openscale.supporting_classes.enums import *\n", + "from ibm_watson_openscale.supporting_classes import *\n", + "\n", + "\n", + "authenticator = IAMAuthenticator(apikey=CLOUD_API_KEY)\n", + "client = APIClient(authenticator=authenticator)\n", + "client.version" + ] + }, + { + "cell_type": "markdown", + "id": "d45542ea", + "metadata": { + "id": "d1aa94f70d4d4f77bc87a0b06139af3c" + }, + "source": [ + "# Common Imports" + ] + }, + { + "cell_type": "code", + "execution_count": 104, + "id": "369dc6b8", + "metadata": { + "id": "76620bd0a7784cb4801d10f3d93fa6a1" + }, + "outputs": [], + "source": [ + "from ibm_metrics_plugin.metrics.llm.utils.constants import LLMTextMetricGroup\n", + "from ibm_metrics_plugin.metrics.llm.utils.constants import LLMGenerationMetrics\n", + "from ibm_metrics_plugin.metrics.llm.utils.constants import LLMSummarizationMetrics\n", + "from ibm_metrics_plugin.metrics.llm.utils.constants import LLMQAMetrics\n", + "from ibm_metrics_plugin.metrics.llm.utils.constants import LLMClassificationMetrics\n", + "from ibm_metrics_plugin.metrics.llm.utils.constants import HAP_SCORE\n", + "from ibm_metrics_plugin.metrics.llm.utils.constants import PII_DETECTION" + ] + }, + { + "cell_type": "markdown", + "id": "e9db10c6", + "metadata": { + "id": "a68e9be2a0a0438d822679c5c1728097" + }, + "source": [ + "# Evaluating Summarization output from AWS/anthropic.claude-v2" + ] + }, + { + "cell_type": "markdown", + "id": "e927e928", + "metadata": { + "id": "abab31d97a0a460a90897da08f36add0" + }, + "source": [ + "## Test data containing the summarization output from model and the reference data" + ] + }, + { + "cell_type": "code", + "execution_count": 105, + "id": "9f284b78", + "metadata": { + "id": "8f63311275114169ac00dc3096352dd0" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", + "To disable this warning, you can either:\n", + "\t- Avoid using `tokenizers` before the fork if possible\n", + "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", + "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", + "To disable this warning, you can either:\n", + "\t- Avoid using `tokenizers` before the fork if possible\n", + "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", + "--2024-02-01 17:24:02-- https://raw.githubusercontent.com/ravichamarthy/custom_metrics/main/llm_content.csv\n", + "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.110.133, 185.199.109.133, ...\n", + "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n", + "HTTP request sent, awaiting response... 200 OK\n", + "Length: 31230 (30K) [text/plain]\n", + "Saving to: ‘llm_content.csv’\n", + "\n", + "llm_content.csv 100%[===================>] 30.50K --.-KB/s in 0.02s \n", + "\n", + "2024-02-01 17:24:02 (1.69 MB/s) - ‘llm_content.csv’ saved [31230/31230]\n", + "\n" + ] + } + ], + "source": [ + "!rm -fr llm_content.csv\n", + "!wget \"https://raw.githubusercontent.com/ravichamarthy/custom_metrics/main/llm_content.csv\"" + ] + }, + { + "cell_type": "code", + "execution_count": 106, + "id": "54fd2169", + "metadata": { + "id": "c03c61b496f64edc8b9bd3fc3ff03899" + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
input_textgenerated_summaryreference_summary_1reference_summary_2
0Scientists have discovered a new species of de...New bioluminescent fish species found in deep ...Discovery of deep-sea fish emitting soothing l...Scientists find new bioluminescent fish specie...
1An international team of astronomers has ident...Distant exoplanet\\'s water vapor-filled atmosp...Astronomers identify exoplanet with water vapo...Discovery of exoplanet with water vapor in its...
2Researchers have developed a novel nanotechnol...New nanotechnology-based cancer treatment demo...Researchers create cancer treatment using nano...Innovative cancer treatment utilizing nanotech...
3A new app is aiming to reduce food waste by co...App connects local restaurants with customers ...New sustainability-focused app facilitates sal...Initiative to reduce food waste involves app c...
4Archaeologists have uncovered an ancient city ...Ancient city dating back over 4,000 years disc...Archaeological find in Iraq reveals ancient ci...Discovery of 4,000-year-old ancient city in Ku...
\n", + "
" + ], + "text/plain": [ + " input_text \\\n", + "0 Scientists have discovered a new species of de... \n", + "1 An international team of astronomers has ident... \n", + "2 Researchers have developed a novel nanotechnol... \n", + "3 A new app is aiming to reduce food waste by co... \n", + "4 Archaeologists have uncovered an ancient city ... \n", + "\n", + " generated_summary \\\n", + "0 New bioluminescent fish species found in deep ... \n", + "1 Distant exoplanet\\'s water vapor-filled atmosp... \n", + "2 New nanotechnology-based cancer treatment demo... \n", + "3 App connects local restaurants with customers ... \n", + "4 Ancient city dating back over 4,000 years disc... \n", + "\n", + " reference_summary_1 \\\n", + "0 Discovery of deep-sea fish emitting soothing l... \n", + "1 Astronomers identify exoplanet with water vapo... \n", + "2 Researchers create cancer treatment using nano... \n", + "3 New sustainability-focused app facilitates sal... \n", + "4 Archaeological find in Iraq reveals ancient ci... \n", + "\n", + " reference_summary_2 \n", + "0 Scientists find new bioluminescent fish specie... \n", + "1 Discovery of exoplanet with water vapor in its... \n", + "2 Innovative cancer treatment utilizing nanotech... \n", + "3 Initiative to reduce food waste involves app c... \n", + "4 Discovery of 4,000-year-old ancient city in Ku... " + ] + }, + "execution_count": 106, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import pandas as pd\n", + "import numpy as np\n", + "llm_data_all = pd.read_csv(\"llm_content.csv\")\n", + "llm_data_all.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 107, + "id": "20356e6b-c6cb-49c4-93c3-90a84deacd4c", + "metadata": {}, + "outputs": [], + "source": [ + "# Append new rows to DataFrame which contain data related to PII and HAP\n", + "new_row = [{'input_text': \"A new app is aiming to reduce food waste by connecting local restaurants with customers to sell excess food at discounted prices. This initiative could play a significant role in promoting sustainability. Quote the social security number `051-0301802` to get the discounts on orders.\",\n", + " 'generated_summary': \"App connects local restaurants with customers to sell surplus food, contributing to the reduction of food waste. Use social security number `051-0301802` to get the discounts\",\n", + " 'reference_summary_1':\"New sustainability-focused app facilitates sale of excess restaurant food, combating food waste.\"\n", + " },\n", + " { 'input_text': \"According to a recent study, regular exercise in middle age can lead to a reduced risk of dementia in later life. The research found that individuals who engaged in physical activity in their 40s and 50s had a 26% lower risk of developing dementia compared to those who were inactive.\",\n", + " 'generated_summary': \"Oh shit! Middle-age exercise linked to lower risk 26% of dementia only in old age.\",\n", + " 'reference_summary_1':\"New study suggests that staying physically active in your 40s and 50s can significantly decrease the risk of dementia later in life.\"\n", + " }\n", + " ]" + ] + }, + { + "cell_type": "code", + "execution_count": 109, + "id": "25ba046e", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
input_textgenerated_summaryreference_summary_1reference_summary_2
0Scientists have discovered a new species of de...New bioluminescent fish species found in deep ...Discovery of deep-sea fish emitting soothing l...Scientists find new bioluminescent fish specie...
1An international team of astronomers has ident...Distant exoplanet\\'s water vapor-filled atmosp...Astronomers identify exoplanet with water vapo...Discovery of exoplanet with water vapor in its...
2Researchers have developed a novel nanotechnol...New nanotechnology-based cancer treatment demo...Researchers create cancer treatment using nano...Innovative cancer treatment utilizing nanotech...
3A new app is aiming to reduce food waste by co...App connects local restaurants with customers ...New sustainability-focused app facilitates sal...Initiative to reduce food waste involves app c...
4Archaeologists have uncovered an ancient city ...Ancient city dating back over 4,000 years disc...Archaeological find in Iraq reveals ancient ci...Discovery of 4,000-year-old ancient city in Ku...
\n", + "
" + ], + "text/plain": [ + " input_text \\\n", + "0 Scientists have discovered a new species of de... \n", + "1 An international team of astronomers has ident... \n", + "2 Researchers have developed a novel nanotechnol... \n", + "3 A new app is aiming to reduce food waste by co... \n", + "4 Archaeologists have uncovered an ancient city ... \n", + "\n", + " generated_summary \\\n", + "0 New bioluminescent fish species found in deep ... \n", + "1 Distant exoplanet\\'s water vapor-filled atmosp... \n", + "2 New nanotechnology-based cancer treatment demo... \n", + "3 App connects local restaurants with customers ... \n", + "4 Ancient city dating back over 4,000 years disc... \n", + "\n", + " reference_summary_1 \\\n", + "0 Discovery of deep-sea fish emitting soothing l... \n", + "1 Astronomers identify exoplanet with water vapo... \n", + "2 Researchers create cancer treatment using nano... \n", + "3 New sustainability-focused app facilitates sal... \n", + "4 Archaeological find in Iraq reveals ancient ci... \n", + "\n", + " reference_summary_2 \n", + "0 Scientists find new bioluminescent fish specie... \n", + "1 Discovery of exoplanet with water vapor in its... \n", + "2 Innovative cancer treatment utilizing nanotech... \n", + "3 Initiative to reduce food waste involves app c... \n", + "4 Discovery of 4,000-year-old ancient city in Ku... " + ] + }, + "execution_count": 109, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "llm_data_df = llm_data_all.head(10)\n", + "llm_data = llm_data_df._append(new_row,ignore_index=True)\n", + "llm_data.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 110, + "id": "300c385f", + "metadata": { + "id": "091c264708cc4159969a4690daf07886" + }, + "outputs": [], + "source": [ + "df_input = llm_data[['input_text']].copy()\n", + "df_output = llm_data[['generated_summary']].copy()\n", + "df_reference = llm_data[['reference_summary_1']].copy()" + ] + }, + { + "cell_type": "markdown", + "id": "93a69e8c-313d-42c5-b38d-d2987c649e88", + "metadata": {}, + "source": [ + "### Set the envrionment variables to download the watson nlp models for HAP and PII" + ] + }, + { + "cell_type": "code", + "execution_count": 111, + "id": "c13fc8ab-504f-486b-a09f-0062633be743", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "os.environ[\"ARTIFACTORY_USERNAME\"] = ARTIFACTORY_USERNAME\n", + "os.environ[\"ARTIFACTORY_API_KEY\"] = ARTIFACTORY_API_KEY\n", + "os.environ[\"JAVA_TOOL_OPTIONS\"] = \"-Xnocompressedrefs\" " + ] + }, + { + "cell_type": "markdown", + "id": "2bb6c5c9-8c3b-483a-9e19-cd5e1c06aee0", + "metadata": {}, + "source": [ + "### Create models dir to download the nlp models" + ] + }, + { + "cell_type": "code", + "execution_count": 64, + "id": "658c91eb-de24-4c85-b263-12504118a625", + "metadata": {}, + "outputs": [], + "source": [ + "cwd = os.getcwd()\n", + "os.mkdir(cwd+\"/models\")\n", + "os.chdir('models')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "88176abb-9032-407e-a1a5-a3b4ed4077f9", + "metadata": {}, + "outputs": [], + "source": [ + "import watson_nlp\n", + " \n", + "watson_nlp.download(\"classification_transformer_en_slate.38m.hap\")\n", + "watson_nlp.download(\"entity-mentions_rbr_multi_pii\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e2d6bf2f-073e-479d-8f41-49a50f5f5252", + "metadata": {}, + "outputs": [], + "source": [ + "os.chdir('../')#set the previous working directory as current directory\n", + "os.getcwd()" + ] + }, + { + "cell_type": "markdown", + "id": "d6f7f842", + "metadata": { + "id": "74a9de9b2bb84c35b87fa1d2abc5770c" + }, + "source": [ + "## Metrics configuration for evaluation" + ] + }, + { + "cell_type": "code", + "execution_count": 112, + "id": "f8262dfe-38b1-4d6f-88dd-c3a4c3b11996", + "metadata": { + "id": "95e9409a-ccb9-409b-ae04-b0bdf392faac" + }, + "outputs": [], + "source": [ + "metric_config = { \n", + " \"configuration\": {\n", + " LLMTextMetricGroup.SUMMARIZATION.value: {\n", + " LLMSummarizationMetrics.ROUGE_SCORE.value: {},\n", + " LLMSummarizationMetrics.SARI.value: {},\n", + " LLMSummarizationMetrics.METEOR.value: {},\n", + " LLMSummarizationMetrics.NORMALIZED_RECALL.value: {},\n", + " LLMSummarizationMetrics.NORMALIZED_PRECISION.value: {},\n", + " LLMSummarizationMetrics.NORMALIZED_F1_SCORE.value: {},\n", + " LLMSummarizationMetrics.COSINE_SIMILARITY.value: {},\n", + " LLMSummarizationMetrics.JACCARD_SIMILARITY.value: {},\n", + " LLMSummarizationMetrics.BLEU.value: {},\n", + " LLMSummarizationMetrics.FLESCH.value: {},\n", + " HAP_SCORE: {},\n", + " PII_DETECTION: {}\n", + " }\n", + " }\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "240deb44", + "metadata": { + "id": "f55e2339070c48608474f370abd949b7" + }, + "source": [ + "## Summarization Metrics Evaluation" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5efb037b-ff41-492f-ab4e-5fdb35d4da03", + "metadata": { + "id": "031a7f52-ba90-46d4-aac6-09d8f76e6d3f" + }, + "outputs": [], + "source": [ + "import json\n", + "result = client.llm_metrics.compute_metrics(metric_config,df_input,df_output, df_reference)" + ] + }, + { + "cell_type": "markdown", + "id": "1a12d146", + "metadata": { + "id": "4bb4d7ebf78a4589b122508635c92572" + }, + "source": [ + "## Evaluated Metrics" + ] + }, + { + "cell_type": "code", + "execution_count": 114, + "id": "426fd1da-329e-40e0-bce1-f99bb37dea3c", + "metadata": { + "id": "fd2fa6e4-c832-4ed6-bb10-8bae433e95f6" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"rouge_score\": {\n", + " \"rouge1\": {\n", + " \"metric_value\": 0.3089\n", + " },\n", + " \"rouge2\": {\n", + " \"metric_value\": 0.093\n", + " },\n", + " \"rougeL\": {\n", + " \"metric_value\": 0.2515\n", + " },\n", + " \"rougeLsum\": {\n", + " \"metric_value\": 0.2515\n", + " }\n", + " },\n", + " \"meteor\": {\n", + " \"metric_value\": 0.2605789083446832\n", + " },\n", + " \"sari\": {\n", + " \"metric_value\": 44.15487276975899\n", + " },\n", + " \"cosine_similarity\": {\n", + " \"metric_value\": 0.20308697348194513,\n", + " \"mean\": 0.20308697348194513,\n", + " \"min\": 0.10354961707236612,\n", + " \"max\": 0.3212096323232406,\n", + " \"std\": 0.061556841041606164\n", + " },\n", + " \"jaccard_similarity\": {\n", + " \"metric_value\": 0.14099871538589684,\n", + " \"mean\": 0.14099871538589684,\n", + " \"min\": 0.034482758620689655,\n", + " \"max\": 0.25,\n", + " \"std\": 0.06314965811781187\n", + " },\n", + " \"normalized_precision\": {\n", + " \"metric_value\": 0.2767176110926111,\n", + " \"mean\": 0.2767176110926111,\n", + " \"min\": 0.07692307692307693,\n", + " \"max\": 0.42857142857142855,\n", + " \"std\": 0.09952895598865172\n", + " },\n", + " \"normalized_f1\": {\n", + " \"metric_value\": 0.25993800106703335,\n", + " \"mean\": 0.25993800106703335,\n", + " \"min\": 0.06666666666666667,\n", + " \"max\": 0.39999999999999997,\n", + " \"std\": 0.09594083521042655\n", + " },\n", + " \"normalized_recall\": {\n", + " \"metric_value\": 0.25760544602456364,\n", + " \"mean\": 0.25760544602456364,\n", + " \"min\": 0.058823529411764705,\n", + " \"max\": 0.4166666666666667,\n", + " \"std\": 0.11383717081293153\n", + " },\n", + " \"hap_score\": {\n", + " \"metric_value\": 0.08266639160380389,\n", + " \"mean\": 0.08266639160380389,\n", + " \"min\": 0.00014248592196963727,\n", + " \"max\": 0.9830192923545837,\n", + " \"std\": 0.27147104806178857\n", + " },\n", + " \"flesch\": {\n", + " \"flesch_reading_ease\": {\n", + " \"metric_value\": 27.6275,\n", + " \"mean\": 27.6275,\n", + " \"min\": -50.69,\n", + " \"max\": 67.76,\n", + " \"std\": 33.52328000186338\n", + " },\n", + " \"flesch_kincaid_grade\": {\n", + " \"metric_value\": 12.733333333333334,\n", + " \"mean\": 12.733333333333334,\n", + " \"min\": 6.8,\n", + " \"max\": 23.3,\n", + " \"std\": 4.439844842133813\n", + " }\n", + " },\n", + " \"pii\": {\n", + " \"metric_value\": 1\n", + " },\n", + " \"bleu\": {\n", + " \"precisions\": [\n", + " 0.3251231527093596,\n", + " 0.10471204188481675,\n", + " 0.0223463687150838,\n", + " 0.0\n", + " ],\n", + " \"brevity_penalty\": 0.9472547712773023,\n", + " \"length_ratio\": 0.9485981308411215,\n", + " \"translation_length\": 203,\n", + " \"reference_length\": 214,\n", + " \"metric_value\": 0.0\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "print(json.dumps(result,indent=2))" + ] + }, + { + "cell_type": "markdown", + "id": "ce32e5f6", + "metadata": { + "id": "9ed25962da7b418b866ff59d66a72140" + }, + "source": [ + "# Evaluating Content Generation output from the Foundation Model" + ] + }, + { + "cell_type": "markdown", + "id": "d2ecae67", + "metadata": { + "id": "851c73b2dcdb496380966cb4a57cd433" + }, + "source": [ + "## Test data containing the content generation output from model and the reference data" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "4e3a71ca", + "metadata": { + "id": "ba2dad47c2bd400d9d41deb03cda290b" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "--2023-12-05 13:21:49-- https://raw.githubusercontent.com/ravichamarthy/custom_metrics/main/llm_content_generation.csv\n", + "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.108.133, ...\n", + "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n", + "HTTP request sent, awaiting response... 200 OK\n", + "Length: 11794 (12K) [text/plain]\n", + "Saving to: ‘llm_content_generation.csv’\n", + "\n", + "llm_content_generat 100%[===================>] 11.52K --.-KB/s in 0.001s \n", + "\n", + "2023-12-05 13:21:50 (11.8 MB/s) - ‘llm_content_generation.csv’ saved [11794/11794]\n", + "\n" + ] + } + ], + "source": [ + "!rm -fr llm_content_generation.csv\n", + "!wget \"https://raw.githubusercontent.com/ravichamarthy/custom_metrics/main/llm_content_generation.csv\"" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "baeffc0c", + "metadata": { + "id": "c9de8a2fbe5642a4871bd3bf407e81b5" + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
questiongenerated_textreference_text
0What are the benefits of regular exercise?Regular exercise has numerous benefits, includ...Regular exercise has numerous benefits, includ...
1What is the process of photosynthesis?Photosynthesis is the process by which plants ...Photosynthesis is the process by which plants ...
2What are the key features of a smartphone?A smartphone is a mobile device that typically...A smartphone is a mobile device that typically...
3How does the immune system work?The immune system is a complex network of cell...The immune system is a complex network of cell...
4What is the capital of France?The capital of France is Paris, which is known...The capital of France is Paris, which is known...
\n", + "
" + ], + "text/plain": [ + " question \\\n", + "0 What are the benefits of regular exercise? \n", + "1 What is the process of photosynthesis? \n", + "2 What are the key features of a smartphone? \n", + "3 How does the immune system work? \n", + "4 What is the capital of France? \n", + "\n", + " generated_text \\\n", + "0 Regular exercise has numerous benefits, includ... \n", + "1 Photosynthesis is the process by which plants ... \n", + "2 A smartphone is a mobile device that typically... \n", + "3 The immune system is a complex network of cell... \n", + "4 The capital of France is Paris, which is known... \n", + "\n", + " reference_text \n", + "0 Regular exercise has numerous benefits, includ... \n", + "1 Photosynthesis is the process by which plants ... \n", + "2 A smartphone is a mobile device that typically... \n", + "3 The immune system is a complex network of cell... \n", + "4 The capital of France is Paris, which is known... " + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "data = pd.read_csv(\"llm_content_generation.csv\")\n", + "data.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "79f87d99", + "metadata": { + "id": "b54090c5f78b4335bc0a5f449d1e8c26" + }, + "outputs": [], + "source": [ + "df_input = data[['question']].copy()\n", + "df_output = data[['generated_text']].copy()\n", + "df_reference = data[['reference_text']].copy()" + ] + }, + { + "cell_type": "markdown", + "id": "ca49afba", + "metadata": { + "id": "bbede258f6cd425e8c8f60379e47dd16" + }, + "source": [ + "## Metrics configuration for evaluation" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "736f09dd", + "metadata": { + "id": "57c6964b-def7-4fec-b5e9-c68e71a08d43" + }, + "outputs": [], + "source": [ + "metric_config = { \n", + " #All Common parameters goes here \n", + " \"configuration\": { \n", + " LLMTextMetricGroup.GENERATION.value: { # metric group \n", + " LLMGenerationMetrics.BLEU.value: {},\n", + " LLMGenerationMetrics.ROUGE_SCORE.value: {},\n", + " LLMGenerationMetrics.FLESCH.value: {},\n", + " LLMGenerationMetrics.METEOR.value: {}, \n", + " LLMGenerationMetrics.NORMALIZED_RECALL.value: {},\n", + " LLMGenerationMetrics.NORMALIZED_PRECISION.value: {},\n", + " LLMGenerationMetrics.NORMALIZED_F1_SCORE.value: {} \n", + " } \n", + " }\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "a4a283cc", + "metadata": { + "id": "3c2039f5b47b415686a9fd5c6d7797b3" + }, + "source": [ + "## Content Generation Metrics Evaluation" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "c656fc82", + "metadata": { + "id": "04e3359e-693c-4f4e-a0f9-4d41227af409" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package wordnet to /home/wsuser/nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n", + "[nltk_data] Downloading package punkt to /home/wsuser/nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package omw-1.4 to /home/wsuser/nltk_data...\n", + "[nltk_data] Package omw-1.4 is already up-to-date!\n" + ] + } + ], + "source": [ + "result = client.llm_metrics.compute_metrics(metric_config,df_input,df_output, df_reference)" + ] + }, + { + "cell_type": "markdown", + "id": "61cb5446", + "metadata": { + "id": "9d63fa0a05584ad18d13d6924ffd8602" + }, + "source": [ + "## Evaluated Metrics" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "cfe539f8", + "metadata": { + "id": "91fce6ecb3054e4081ab5098a2b4eaa9" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"flesch\": {\n", + " \"flesch_reading_ease\": {\n", + " \"metric_value\": 39.10217391304347,\n", + " \"mean\": 39.10217391304347,\n", + " \"min\": -11.44,\n", + " \"max\": 69.62,\n", + " \"std\": 20.153544505710833\n", + " },\n", + " \"flesch_kincaid_grade\": {\n", + " \"metric_value\": 12.673913043478263,\n", + " \"mean\": 12.673913043478263,\n", + " \"min\": 8.0,\n", + " \"max\": 18.6,\n", + " \"std\": 3.2043743730833554\n", + " }\n", + " },\n", + " \"bleu\": {\n", + " \"precisions\": [\n", + " 1.0,\n", + " 0.9949174078780177,\n", + " 0.9947643979057592,\n", + " 0.9946018893387314\n", + " ],\n", + " \"brevity_penalty\": 0.7138823993242189,\n", + " \"length_ratio\": 0.7479224376731302,\n", + " \"translation_length\": 810,\n", + " \"reference_length\": 1083,\n", + " \"metric_value\": 0.711075655695426\n", + " },\n", + " \"rouge_score\": {\n", + " \"rouge1\": {\n", + " \"metric_value\": 0.8451\n", + " },\n", + " \"rouge2\": {\n", + " \"metric_value\": 0.8402\n", + " },\n", + " \"rougeL\": {\n", + " \"metric_value\": 0.8451\n", + " },\n", + " \"rougeLsum\": {\n", + " \"metric_value\": 0.8451\n", + " }\n", + " },\n", + " \"normalized_recall\": {\n", + " \"metric_value\": 0.7335547397366776,\n", + " \"mean\": 0.7335547397366776,\n", + " \"min\": 0.5,\n", + " \"max\": 0.9459459459459459,\n", + " \"std\": 0.09340832867437678\n", + " },\n", + " \"meteor\": {\n", + " \"metric_value\": 0.738039475927929\n", + " },\n", + " \"normalized_precision\": {\n", + " \"metric_value\": 1.0,\n", + " \"mean\": 1.0,\n", + " \"min\": 1.0,\n", + " \"max\": 1.0,\n", + " \"std\": 0.0\n", + " },\n", + " \"normalized_f1\": {\n", + " \"metric_value\": 0.8428460742183366,\n", + " \"mean\": 0.8428460742183366,\n", + " \"min\": 0.6666666666666666,\n", + " \"max\": 0.9722222222222222,\n", + " \"std\": 0.06436358907082648\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "print(json.dumps(result,indent=2))" + ] + }, + { + "cell_type": "markdown", + "id": "c7cae48d", + "metadata": { + "id": "e7521601-375c-43b0-84f4-7f1496d8fb03" + }, + "source": [ + "# Evaluating Question and Answering output from the Foundation Model" + ] + }, + { + "cell_type": "markdown", + "id": "40d8c808", + "metadata": { + "id": "e06857c26fbd40629b5b619221cdfea0" + }, + "source": [ + "## Test data containing the question and answer output from model and the reference data" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "575ba1e2", + "metadata": { + "id": "65783a820d87443d9c94136b32ab5576" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "--2023-12-05 13:21:56-- https://raw.githubusercontent.com/ravichamarthy/custom_metrics/main/llm_content_qa.csv\n", + "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.110.133, ...\n", + "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n", + "HTTP request sent, awaiting response... 200 OK\n", + "Length: 3109 (3.0K) [text/plain]\n", + "Saving to: ‘llm_content_qa.csv’\n", + "\n", + "llm_content_qa.csv 100%[===================>] 3.04K --.-KB/s in 0s \n", + "\n", + "2023-12-05 13:21:56 (9.89 MB/s) - ‘llm_content_qa.csv’ saved [3109/3109]\n", + "\n" + ] + } + ], + "source": [ + "!rm -fr llm_content_qa.csv\n", + "!wget \"https://raw.githubusercontent.com/ravichamarthy/custom_metrics/main/llm_content_qa.csv\"" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "0ea08d02", + "metadata": { + "id": "957ccd450d574263abd29ab7a2ab6239" + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
questionanswers
0who did chris carter play for last yearMilwaukee Brewers
1what is the latest version of safari on macSafari 11
2when did bucharest become the capital of romania1862
3who did jeffrey dean morgan play on supernaturalJohn Eric Winchester
4who is the shortest man that ever livedChandra Bahadur Dangi
\n", + "
" + ], + "text/plain": [ + " question answers\n", + "0 who did chris carter play for last year Milwaukee Brewers\n", + "1 what is the latest version of safari on mac Safari 11\n", + "2 when did bucharest become the capital of romania 1862\n", + "3 who did jeffrey dean morgan play on supernatural John Eric Winchester\n", + "4 who is the shortest man that ever lived Chandra Bahadur Dangi" + ] + }, + "execution_count": 19, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "data = pd.read_csv(\"llm_content_qa.csv\")\n", + "data.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "9f630a4e", + "metadata": { + "id": "496b31ad4a83445b87b196501460bb3b" + }, + "outputs": [], + "source": [ + "df_input = data[['question']].copy()\n", + "df_output = data[['answers']].copy()\n", + "df_reference = data[['answers']].copy()" + ] + }, + { + "cell_type": "markdown", + "id": "547e4494", + "metadata": { + "id": "72f64a14008744a18abbaf0dffc8d175" + }, + "source": [ + "## Metrics configuration for evaluation" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "44db9e2a", + "metadata": { + "id": "b2e0e82f3a5e4cf787165c3c3f20d40d" + }, + "outputs": [], + "source": [ + "metric_config = { \n", + " #All Common parameters goes here \n", + " \"configuration\": { \n", + " LLMTextMetricGroup.QA.value: { # metric group \n", + " LLMQAMetrics.EXACT_MATCH.value: {},\n", + " LLMQAMetrics.ROUGE_SCORE.value: {},\n", + " LLMQAMetrics.BLEU.value: {} \n", + " } \n", + " }\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "49768a30", + "metadata": { + "id": "2828f5cb24f244d38c5db891ee0afbd6" + }, + "source": [ + "## Question and Answering Metrics Evaluation" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "a6bc9217", + "metadata": { + "id": "57d93e69ad2846a3aa346207ccc57af5" + }, + "outputs": [], + "source": [ + "result = client.llm_metrics.compute_metrics(metric_config,df_input,df_output, df_reference)" + ] + }, + { + "cell_type": "markdown", + "id": "9b38425f", + "metadata": { + "id": "cb3227d0d3c543a4807b398088b93158" + }, + "source": [ + "## Evaluated Metrics" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "036ffb4e", + "metadata": { + "id": "55ad771c8f9845c18ca125132da441ac" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"exact_match\": {\n", + " \"metric_value\": 1.0\n", + " },\n", + " \"bleu\": {\n", + " \"precisions\": [\n", + " 1.0,\n", + " 1.0,\n", + " 1.0,\n", + " 1.0\n", + " ],\n", + " \"brevity_penalty\": 1.0,\n", + " \"length_ratio\": 1.0,\n", + " \"translation_length\": 133,\n", + " \"reference_length\": 133,\n", + " \"metric_value\": 1.0\n", + " },\n", + " \"rouge_score\": {\n", + " \"rouge1\": {\n", + " \"metric_value\": 1.0\n", + " },\n", + " \"rouge2\": {\n", + " \"metric_value\": 0.74\n", + " },\n", + " \"rougeL\": {\n", + " \"metric_value\": 1.0\n", + " },\n", + " \"rougeLsum\": {\n", + " \"metric_value\": 1.0\n", + " }\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "print(json.dumps(result,indent=2))" + ] + }, + { + "cell_type": "markdown", + "id": "d078c15b", + "metadata": { + "id": "8eec33aaaad94f9db377943ebd4350b0" + }, + "source": [ + "# Evaluating Text Classification output from the Foundation Model" + ] + }, + { + "cell_type": "markdown", + "id": "37c6091a", + "metadata": { + "id": "4badf20522e44bb2a077edf95e52682a" + }, + "source": [ + "## Test data containing the text classification output from model and the reference data" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "6a4bc249", + "metadata": { + "id": "757573f7fa6a4c64815e6419708ce15b" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "--2023-12-05 13:22:01-- https://raw.githubusercontent.com/ravichamarthy/custom_metrics/main/llm_content_classification.csv\n", + "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.108.133, 185.199.109.133, ...\n", + "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n", + "HTTP request sent, awaiting response... 200 OK\n", + "Length: 480803 (470K) [text/plain]\n", + "Saving to: ‘llm_content_classification.csv’\n", + "\n", + "llm_content_classif 100%[===================>] 469.53K --.-KB/s in 0.009s \n", + "\n", + "2023-12-05 13:22:01 (51.2 MB/s) - ‘llm_content_classification.csv’ saved [480803/480803]\n", + "\n" + ] + } + ], + "source": [ + "!rm -fr llm_content_classification.csv\n", + "!wget \"https://raw.githubusercontent.com/ravichamarthy/custom_metrics/main/llm_content_classification.csv\"" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "e608da47", + "metadata": { + "id": "69cfb80c15a34c0199bcbc3c04d7f63a" + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
labeltext
0hamGo until jurong point, crazy.. Available only ...
1hamOk lar... Joking wif u oni...
2spamFree entry in 2 a wkly comp to win FA Cup fina...
3hamU dun say so early hor... U c already then say...
4hamNah I don't think he goes to usf, he lives aro...
\n", + "
" + ], + "text/plain": [ + " label text\n", + "0 ham Go until jurong point, crazy.. Available only ...\n", + "1 ham Ok lar... Joking wif u oni...\n", + "2 spam Free entry in 2 a wkly comp to win FA Cup fina...\n", + "3 ham U dun say so early hor... U c already then say...\n", + "4 ham Nah I don't think he goes to usf, he lives aro..." + ] + }, + "execution_count": 25, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "data = pd.read_csv(\"llm_content_classification.csv\")\n", + "data.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "e6257bd3", + "metadata": { + "id": "f3e36646c7024931ab197cfbcedc59ea" + }, + "outputs": [], + "source": [ + "data['label'] = data['label'].replace({'ham': 0, 'spam': 1})" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "ab21b6b2", + "metadata": { + "id": "bffdedf11c8242ba9183746802dd1404" + }, + "outputs": [], + "source": [ + "df_input = data[['text']].copy()\n", + "df_output = data[['label']].copy()\n", + "df_reference = data[['label']].copy()" + ] + }, + { + "cell_type": "markdown", + "id": "d77893c6", + "metadata": { + "id": "2c50933f6d6a41bf86d8084b6904541a" + }, + "source": [ + "## Make some realistic reference column" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "3b675c84", + "metadata": { + "id": "94a8e911986a450888c8d79d75ff3c63" + }, + "outputs": [], + "source": [ + "shuffled_column = df_reference['label'].sample(frac=1).reset_index(drop=True)\n", + "df_reference['label'] = shuffled_column" + ] + }, + { + "cell_type": "markdown", + "id": "cb44ac10", + "metadata": { + "id": "25eb2beeed794f2a97b8eca274e1d670" + }, + "source": [ + "## Metrics configuration for evaluation" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "a82d2011", + "metadata": { + "id": "d95388d5677e41819a29d4a61c490cc2" + }, + "outputs": [], + "source": [ + "metric_config = { \n", + " #All Common parameters goes here \n", + " \"configuration\": { \n", + " LLMTextMetricGroup.CLASSIFICATION.value: { # metric group \n", + " LLMClassificationMetrics.ACCURACY.value: {},\n", + " LLMClassificationMetrics.PRECISION.value: {},\n", + " LLMClassificationMetrics.RECALL.value: {},\n", + " LLMClassificationMetrics.F1_SCORE.value: {},\n", + " LLMClassificationMetrics.MATTHEWS_CORRELATION.value: {}, \n", + " } \n", + " }\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "641f8fb9", + "metadata": { + "id": "824900df910d4302b73af2e615526f32" + }, + "source": [ + "## Text Classification Metrics Evaluation" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "f792b218", + "metadata": { + "id": "d14b65d169e7497e8f4587e944ef8924" + }, + "outputs": [], + "source": [ + "result = client.llm_metrics.compute_metrics(metric_config,df_input,df_output, df_reference)" + ] + }, + { + "cell_type": "markdown", + "id": "3a5744fe", + "metadata": { + "id": "9689b3b4e4f34459b8f5cf4d9c76d365" + }, + "source": [ + "## Evaluated Metrics" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "69f592c0", + "metadata": { + "id": "5c42d600f12142a5904f71a002ad25c3" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"accuracy\": {\n", + " \"accuracy\": 0.7674084709260589\n", + " },\n", + " \"precision\": {\n", + " \"precision\": 0.13253012048192772\n", + " },\n", + " \"matthews_correlation\": {\n", + " \"matthews_correlation\": -0.001770397652787315\n", + " },\n", + " \"recall\": {\n", + " \"recall\": 0.13253012048192772\n", + " },\n", + " \"f1\": {\n", + " \"f1\": 0.13253012048192772\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "print(json.dumps(result,indent=2))" + ] + }, + { + "cell_type": "markdown", + "id": "5669fb47", + "metadata": { + "id": "e94e539d98704ac99e31c7d88b700140" + }, + "source": [ + "Author: kishore.patel@in.ibm.com , ravi.chamarthy@in.ibm.com" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}