Challenge 8 : Visualization and Explainability

Before starting, it is recommended that you familiarize yourself with the provided assets. 

• Download the provided files from the shared folder.
• Inspect the provided DSL and task python files 
• Review the files related to the explainability extension, located under tasks/task6_explainability. At a high level, identify how this task prepares models, datasets, predictions, and labels so that they can be consumed by the visualization and explainability components. 

You are not required to modify any code or DSL files. 

• Follow the same instructions as in Challenge 1 to execute the experiment on proactive. 
• Ensure ProActive is configured as the executionware. 
• Submit the experiment for execution. 

1. In the ProActive UI, locate your experiment and note its experiment ID. 
2. Open the Visualization Dashboard for your experiment at https://vis.extremexp-icom.intracom-telecom.com/<experiment id>. 

Once logged in, the central page of the Visualization Dashboard is the Experiment Monitoring Page. This page provides an overview of the experiment progress and results. Using this page: 

1. Inspect the experiment progress summary bar and verify that all workflows have completed successfully. 
2. Examine the Workflow Execution Table. 
3. Inspect variability points (e.g. model type, hyperparameters). 
4. Review recorded metrics such as accuracy, F1 score, ROC AUC, execution time. 
5. Use sorting, filtering, and grouping controls to explore the results. 

Questions 

• Identify workflows with roc-auc higher than 0.71. 
• Which one of them achieved the highest accuracy ? 
• Which model type (Logistic Regression or Random Forest) performed better on average, based on accuracy ?  

Using the Parallel Coordinates Plot: 

1. Select a performance metric (e.g. accuracy) as the color encoding. 
2. Explore how different parameter values relate to model performance. 
3. Identify sensitivities, trade-offs, and promising parameter regions. 

Questions 

• Which variability points appear to have a strong effect on accuracy ? 

Using the Comparative Analysis Page: 

1. Select multiple the top 5 workflows with respect to accuracy. 
2. Compare them using: 
   1. 1. • metrics visualizations, 
         • model insights (e.g. confusion matrices, ROC curves), 
3. Inspect how these configurations trade-off precision and recall 
4. Group the selected workflows by model type and compare their performance using metrics such as accuracy, precision, recall, F1 score, ROC AUC, as well as execution time.

Questions 

(Based on Steps 1–3) 

• Among the top-performing workflows in terms of accuracy, are there noticeable differences in recall? 

(Based on Step 4) 

• Do the  two model types exhibit systematic differences in performance (e.g. accuracy, F1 score, ROC AUC) or execution time? 

1. Identify the workflow that : 

• belongs to the top-performing group in terms of accuracy, and 
• achieves the highest recall among those workflows. 

2. Navigate to its Workflow Analysis Page and :

• Examine the workflow structure and task-level execution details. 
• Inspect parameters, metrics, and input/output artifacts associated with each task. 

1. Expand the Model Insights section for the workflow from Step 7. 

2. Inspect model performance visualizations such as (select the trained model entry): 

• confusion matrix 
• ROC curve 
• precision–recall curve 
• classification report 

3. Explore instance-level predictions using the Instance View. 

• Identify misclassified instances. 
• Focus on instances where the true label is 1 (churn), but the model predicted 0 (non-churn). 
• Inspect their feature values. 

4. Select one such instance and :

• generate local counterfactual explanations to understand which changes in feature values would lead the model to predict the correct class. 
• generate hyperparameter counterfactuals to explore whether alternative training configurations would change the prediction. 
• inspect and interpret the generated explanations. 
• inspect the results 

5. Generate feature-level explainability results: 

• feature importance 
• PDP 
• ALE. 

6. Explore hyperparameter explainability views to understand how training configurations influence performance. 

Questions 

Instance-level analysis 

• Provide an example of a misclassified instance where the true label is 1 but the model predicted 0. 
  • Which feature changes, suggested by local counterfactual explanations, would lead the model to predict the correct class? 
  • Do hyperparameter counterfactuals suggest that changes in the training configuration could also correct this prediction? If yes, which ones? 

SHAP-based explanations 

• Using SHAP values for the selected instance: 
  • Which features contributed most to the model’s prediction? 
  • In which direction (positive or negative) and how much did they influence the predicted outcome? 

Comparing Feature Importances 

• Compare feature importance and SHAP-based feature importance in the Feature Explainability section: 
  • Are the most important features the same? 
  • If not, how do they differ? 
  • What are the two most important features for the decisions of the model? 

Feature Effects 

• How exactly does the two most important feature affect the prediction of the model?  
• Provide any insights or hypotheses that the PDPs-ALEs reveal about customer behavior. 

Hyperparameter effects 

• Across the workflows that train a Random Forest, analyze how changes in the rfMaxDepth hyperparameter affect different evaluation metrics (e.g. accuracy, recall, precision, F1-score). 
  • Which metric appears to be most sensitive to changes in rfMaxDepth? 
  • What evidence from the visualizations supports your conclusion? 
• Generate a 2D Partial Dependence Plot (2D PDP) involving rfMaxDepth and one additional feature of your choice, using recall as the target metric. 
  • How do the two variables interact to influence recall? 
  • Are there regions of the parameter–feature space where recall improves or degrades significantly?