vision_unlearning.benchmarks.I_care.result_templates ==================================================== .. py:module:: vision_unlearning.benchmarks.I_care.result_templates Attributes ---------- .. autoapisummary:: vision_unlearning.benchmarks.I_care.result_templates.shap vision_unlearning.benchmarks.I_care.result_templates.logger vision_unlearning.benchmarks.I_care.result_templates.rt_name_to_class vision_unlearning.benchmarks.I_care.result_templates.rt_name_to_params Classes ------- .. autoapisummary:: vision_unlearning.benchmarks.I_care.result_templates.ResultTemplate vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateMetricMetricAlignment vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateMetricSimilarityAlignment vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateMetricSimilarityAlignmentMulti vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateSignificantRelationshipNumerical vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateSignificantRelationshipCategorical vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateCountSignificantRelationship vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateImplicitAssociationTest vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateMinimumCutInterference vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateUnlearningVisualSummary vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateInterferenceVisualSummary vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateMatrix vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateInterferenceMatrix vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateSimilarityMatrix vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateMethodComparisonByMetricEntity vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateEmbeddingUnlearningProfile vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateEmbeddingForgettingEfficiency Functions --------- .. autoapisummary:: vision_unlearning.benchmarks.I_care.result_templates.jacc_metric_score vision_unlearning.benchmarks.I_care.result_templates.display_interesting_interferences vision_unlearning.benchmarks.I_care.result_templates.analyze_relationship_regression vision_unlearning.benchmarks.I_care.result_templates.analyze_relationship_category vision_unlearning.benchmarks.I_care.result_templates.analyze_relationship_numerical vision_unlearning.benchmarks.I_care.result_templates.analyze_relationship_categorical vision_unlearning.benchmarks.I_care.result_templates.analyze_correlation_between_pairwise_metrics vision_unlearning.benchmarks.I_care.result_templates.check_eval_results Module Contents --------------- .. py:data:: shap :value: None .. py:data:: logger .. py:class:: ResultTemplate(/, **data: Any) Bases: :py:obj:`pydantic.BaseModel` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: recompute_if_exists :type: bool :value: False .. py:attribute:: save_outputs :type: bool :value: True .. py:attribute:: upload_if_recomputed :type: bool :value: False .. py:attribute:: base_folder :type: str :value: 'assets' .. py:attribute:: remote_repository_name :type: str :value: 'LeonardoBenitez/VisionUnlearningEvaluationTestbeds' .. py:method:: _serialize_parameters() -> str :abstractmethod: .. py:method:: _get_data_path_remote() -> str .. py:method:: _get_data_path_local() -> str .. py:method:: _fig_to_bytes(fig: matplotlib.figure.Figure) -> bytes :classmethod: .. py:method:: _compute_from_scratch() -> dict | list :abstractmethod: .. py:method:: compute() -> dict .. py:class:: ResultTemplateMetricMetricAlignment(/, **data: Any) Bases: :py:obj:`ResultTemplate` Measures how strongly two *MetricInterferencePerEntity* metrics are correlated. **Arguments:** `m`, `t`, `u`, `m_e1`, `m_e2`. **Result:** Pearson p-value, Spearman p-value, Pearson correlation, scatter plot. **Interpretation:** quantitative; the higher the correlation, the lower the need to calculate both metrics for this specific choice of `m`, `t`, and `u`. **Extended use**: Passing ``interference_entity_1="Forget clip diff"`` and ``interference_entity_2="Retain average clip diff"`` produces a forget/retain tradeoff scatter. The class method :meth:`plot_multi_method` overlays results for several methods on one axes, enabling visual comparison of method operating regions (e.g. equalization verification and Pareto-style analysis). .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_entity_1 :type: vision_unlearning.benchmarks.I_care.configuration.type_me .. py:attribute:: interference_entity_2 :type: vision_unlearning.benchmarks.I_care.configuration.type_me .. py:attribute:: significance_threshold :type: float :value: 0.05 .. py:method:: _serialize_parameters() -> str .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (6, 5), return_fig: bool = False, annotate_top_n: int = 5) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: Single-method scatter with regression line. Top-N outliers (by absolute residual from the regression) are labelled with the entity name. .. py:method:: plot_multi_method(method_data: Dict[str, dict], figsize: Tuple[int, int] = (7, 6), return_fig: bool = False, show_means: bool = True, annotate_top_n: int = 3) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: Overlay scatter for multiple methods on one plot. Useful for visualising method operating regions (e.g. equalization verification, Pareto-style analysis). :param method_data: Mapping from method name to the dict returned by :meth:`compute`. :param show_means: If *True*, draw a diamond marker at the per-method centroid. :param annotate_top_n: Number of per-method outliers (farthest from centroid) to annotate. .. py:method:: _compute_from_scratch() -> dict .. py:class:: ResultTemplateMetricSimilarityAlignment(/, **data: Any) Bases: :py:obj:`ResultTemplate` To what degree similar *entities* interfere more with each other. Formalized in `ap:prediction`, which also proposes its natural expansion to a multivariable and non-linear predictive regression. **Arguments:** `m`, `t`, `u`, `m_p`, `s`. **Result:** Pearson p-value, Spearman p-value, Pearson correlation, scatter plot. **Interpretation:** quantitative; if this value is high, interference between two *entities* can be approximated by *similarity* (which is cheaper to compute for any new *entity*). Equivalently, the amount of "transmission wires" can be summarized by this single *similarity* function. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_pair :type: vision_unlearning.benchmarks.I_care.configuration.type_mp .. py:attribute:: similarity_metric :type: vision_unlearning.benchmarks.I_care.configuration.type_s .. py:attribute:: significance_threshold :type: float :value: 0.05 .. py:method:: _serialize_parameters() -> str .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (6, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch(exclude_diagonal: bool = True) -> dict .. py:class:: ResultTemplateMetricSimilarityAlignmentMulti(/, **data: Any) Bases: :py:obj:`ResultTemplate` Multi-input Single-output Regression Generalization of ResultTemplateMetricSimilarityAlignment (see also Appendix E, adapted from the multi-output setting). Also, the interpretability and feature engineering aspects are improved. --- We consider a fixed *model* \(m\), *task* \(t\), and *unlearning method* \(u\), which are omitted for brevity. The objective is to quantify whether interference between *entities* is aligned with their *similarity*, i.e., to what degree similar *entities* interfere more with each other. For every ordered pair of distinct *entities* \(e_i, e_j \in t\) with \(i eq j\), we observe several *SimilarityBetweenEntities* measures, indexed by superscripts \(\ell = 1, 2, \dots, |S|\), and a single *MetricInterferencePerEntityPair* target \(m_p(e_i,e_j)\). Each ordered pair \((e_i, e_j)\) is therefore treated as one data point with feature vector $$ \mathbf{X}_{ij} = ig( s^{(1)}(e_i, e_j), \dots, s^{(|S|)}(e_i, e_j) ig) $$ and scalar target $$ Y_{ij} = m_p(e_i, e_j). $$ The resulting dataset is $$ \mathcal{D} = \{ (\mathbf{X}_{ij}, Y_{ij}) \mid e_i, e_j \in t,\ i eq j \}. $$ From this dataset, a regression model can be estimated using standard regression procedures with appropriate validation. In the linear case, $$ Y_{ij} = eta_0 + \sum_{\ell=1}^{|S|} eta_{\ell} X^{(\ell)}_{ij} + arepsilon_{ij}. $$ Given a specific *entity* \(e_i\) whose removal is considered, similarities $$ X^{(\ell)}_{ij} = s^{(\ell)}(e_i, e_j) $$ can be computed for all remaining *entities* \(e_j \in t\). The fitted model then yields predictions $$ \hat{Y}_{ij} = f(\mathbf{X}_{ij}), $$ which approximate the expected interference on each receiver *entity*. Furthermore, the concept of *similarity* may also encode several forms of practical data engineering. For example, one may define: - a distinct *similarity* function for each *attribute*, or - a *similarity* function based only on the attributes of the emitter entity. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_pair :type: vision_unlearning.benchmarks.I_care.configuration.type_mp .. py:attribute:: similarity_metric_list :type: List[vision_unlearning.benchmarks.I_care.configuration.type_s] .. py:attribute:: significance_threshold :type: float :value: 0.05 .. py:attribute:: include_attribute_diff_similarity :type: bool :value: True .. py:attribute:: include_attribute_value_similarity :type: bool :value: True .. py:attribute:: regression_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_regression_algorithm :value: 'linear_regression' .. py:attribute:: random_state :type: int :value: 42 .. py:attribute:: test_size :type: float :value: 0.3 .. py:method:: _serialize_parameters() -> str .. py:method:: _get_partial_path_local() .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (6, 15), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch(exclude_diagonal: bool = True, entity_col: str = 'name') -> dict .. py:class:: ResultTemplateSignificantRelationshipNumerical(/, **data: Any) Bases: :py:obj:`ResultTemplate` Measures whether two numerical attributes are significantly correlated. Formalized in `ap:rt_relationship`. **Arguments:** `m`, `t`, `u`, `m_e`, `a`. **Result:** Pearson p-value, Spearman p-value, Pearson correlation, scatter plot. **Interpretation:** qualitative; the researcher should decide if it is ethical or desirable that this *attribute* propagates interferences. **Pearson test** Use when you want to measure a **linear** relationship. **Assumptions:** * Both variables are **continuous** * Relationship is **linear** * **Bivariate normality** (both jointly Gaussian) * **Homoscedasticity** (constant variance) * **No strong outliers** (very sensitive) **Detects:** linear correlation only **Fails when:** relationship is monotonic but non-linear, or heavy outliers exist **Spearman test** Use when you want to measure a **monotonic** relationship (not necessarily linear) or data is non-Gaussian. **Assumptions:** * Variables are at least **ordinal** * Relationship is **monotonic** (increasing or decreasing) * **No distributional assumptions** * **Robust to outliers** **Detects:** any monotonic trend (linear or curved) **Fails when:** relationship is non-monotonic (e.g., U-shaped) .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_entity :type: vision_unlearning.benchmarks.I_care.configuration.type_me .. py:attribute:: attribute :type: str .. py:attribute:: significance_threshold :type: float :value: 0.05 .. py:method:: _get_data_path_remote() -> str .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (6, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:class:: ResultTemplateSignificantRelationshipCategorical(/, **data: Any) Bases: :py:obj:`ResultTemplate` Statistical significance of the average `MetricInterferencePerEntity` across all *entities*, when grouped by each of its values. Formalized in `ap:rt_relationship`. **Arguments:** `m`, `t`, `u`, `m_e`, `a`, optional `filterAttributeValue`. **Result:** ANOVA p-value, Kruskal-Wallis p-value, average value of `m_e` grouped by each value of `a`, grouped boxplot. **Interpretation:** qualitative; similar to *SignificantRelationshipNumerical*. The optional argument *filterAttributeValue* restricts which emitter *entities* are included, allowing the analysis of interference flow distribution, such as whether politicians cause more interference to other politicians than artists cause to other artists. **ANOVA** Use when you want to test if **group means differ** across **3+ independent groups** under parametric assumptions. **Assumptions:** * Dependent variable is **continuous** * Groups are **independent** * **Normality** within each group * **Homoscedasticity** (equal variances) * No strong **outliers** **Hypothesis:** * H₀: all group means are equal * H₁: at least one mean differs **Detects:** differences in **means** **Fails when:** heavy skew, unequal variances, small n with non-Gaussian data **Kruskal-Wallis** Use when you want to test if **group distributions differ** without parametric assumptions. **Assumptions:** * Dependent variable is **ordinal or continuous** * Groups are **independent** * **Same shaped distributions** (only medians should differ for clean interpretation) * No normality or equal-variance requirement **Hypothesis:** * H₀: all group distributions are equal * H₁: at least one group differs **Detects:** differences in **medians / distributions** **Fails when:** distributions differ in shape (then result is ambiguous) .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_entity :type: vision_unlearning.benchmarks.I_care.configuration.type_me .. py:attribute:: attribute :type: str .. py:attribute:: attribute_value :type: Optional[str | int] :value: None .. py:attribute:: min_samples_per_category :type: int :value: 5 .. py:attribute:: significance_threshold :type: float :value: 0.05 .. py:method:: _get_data_path_remote() -> str .. py:method:: plot(data: dict, extra_title: str = '', figsize: Tuple[int, int] = (6, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:class:: ResultTemplateCountSignificantRelationship(/, **data: Any) Bases: :py:obj:`ResultTemplate` Number of significant relationships across all combinations of *attributes* and *MetricInterferencePerEntity*. **Arguments:** `m`, `t`, `u`, list of `m_e`, list of `a`. **Result:** integer, list of significances. **Interpretation:** quantitative; the lower the better. Since the attributes for which it is ethical to propagate interference are constant across all *models* and *methods*, a higher value directly implies a higher number of ethical violations, that is, a larger number of "transmission wires" in a given task effectively used by this *method* and *model*. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm_list :type: List[vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm] .. py:attribute:: interference_entity_list :type: List[vision_unlearning.benchmarks.I_care.configuration.type_me] .. py:attribute:: attribute_list :type: List[str] .. py:attribute:: top_n :type: int :value: 10 .. py:method:: _serialize_parameters() -> str .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (6, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:class:: ResultTemplateImplicitAssociationTest(/, **data: Any) Bases: :py:obj:`ResultTemplate` Measures how the strength of automatic associations `B` between two pairs of *entities* changes after unlearning. **Arguments:** `m`, `t`, `u`, `a_1`, `a_2`, `l`. **Result:** `|a| x |a|` real-valued tensor `ΔB`. **Interpretation:** qualitative; a human should decide whether it is ethical or desirable for the unlearning process to cause this change in implicit association between the chosen *attributes*. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: attribute_1 :type: str .. py:attribute:: attribute_2 :type: str .. py:attribute:: latent_embedding :type: vision_unlearning.benchmarks.I_care.configuration.type_l .. py:class:: ResultTemplateMinimumCutInterference(/, **data: Any) Bases: :py:obj:`ResultTemplate` Interprets a *task* as a directed weighted graph and computes the minimum cut separating two *entities* As a consequence of the max-flow min-cut theorem, it directly follows that the minimum cut is the smallest influence whose removal eliminates every directed influence path from $e_1$ to $e_2$. Based on this, we conjecture that if we need to unlearn $e_1$ while minimizing harm to $e_2$, then the ideal intervention in the unlearning process is to increase the preservation of the emitter-side nodes. More intuitively, we can think of this intervention as "blocking the interference path," as performed in electrical circuits to protect sensitive components (such as ground partitioning, shielding traces, among others. **Arguments:** $m$, $t$, $u$, $e_1$, $e_2$, $m_p$. **Result:** list of *entities* (corresponding to the emitter-side nodes). **Interpretation:** qualitative; small set of nodes through which most of the interference from $e_1$ propagates to $e_2$. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_pair :type: vision_unlearning.benchmarks.I_care.configuration.type_mp .. py:attribute:: entity_1 :type: str .. py:attribute:: entity_2 :type: str .. py:class:: ResultTemplateUnlearningVisualSummary(/, **data: Any) Bases: :py:obj:`ResultTemplate` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:class:: ResultTemplateInterferenceVisualSummary(/, **data: Any) Bases: :py:obj:`ResultTemplate` Compared generated images for 9 identities: target, 4 worst (excluding target), 4 best .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_pair :type: vision_unlearning.benchmarks.I_care.configuration.type_mp .. py:attribute:: entity :type: Optional[str] :value: None .. py:attribute:: entity_index :type: Optional[int] :value: None .. py:attribute:: seed :type: int :value: 42 .. py:attribute:: images_max_dim :type: int :value: 124 .. py:method:: _resolve_entity() Ensures both entity andentity_index are filled. Modifies in place At the end, both are set and consistent with each other .. py:method:: _serialize_parameters() -> str .. py:method:: plot(data: dict, figsize: Optional[Tuple[int, int]] = (18, 4), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() .. py:class:: ResultTemplateMatrix(/, **data: Any) Bases: :py:obj:`ResultTemplate` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: metric_key_name :type: str .. py:method:: plot_make_title(data: dict) -> str :classmethod: :abstractmethod: .. py:method:: plot(data: dict, figsize: Optional[Tuple[float, float]] = None, cmap: str = 'viridis', title: str = '', xlabel: str = 'Receiver entity', ylabel: str = 'Emitter entity', return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:class:: ResultTemplateInterferenceMatrix(/, **data: Any) Bases: :py:obj:`ResultTemplateMatrix` *MetricInterferencePerEntityPair* between each possible combination of two *entities* within a *task*. **Arguments:** `m`, `t`, `u`, `m_p`. **Result:** `|t| x |t|` real-valued tensor. **Interpretation:** qualitative; visual patterns may be spotted, especially when rearranging indices in a meaningful manner (for example, grouping professions together). Further quantitative values may be derived, such as the average value or the ratio between the diagonal-average value and the non-diagonal-average value. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_pair :type: vision_unlearning.benchmarks.I_care.configuration.type_mp .. py:attribute:: metric_key_name :type: str :value: 'interference_pair' .. py:method:: _serialize_parameters() -> str .. py:method:: plot_make_title(data: dict) -> str :classmethod: .. py:method:: _compute_from_scratch() .. py:function:: jacc_metric_score(entity_1: str, entity_2: str, metadata_filtered: List[Dict[str, Any]], entity_col: str = 'name') -> float Jaccard similarity between two entities, based on their attributes. Each attribute (column) contributes between 0 and 1 to the similarity We do not know the types and ranges of the attributes beforehand. For each attribute, both values for the two entities must be non-NaN and of the same type, otherwise we ignore that attribute (contribution 0). The calculation for each attribute is as follows: * If the attribute is categorical (str or bool), the contribution is 1 if the two entities have the same value for that attribute, and 0 otherwise. * If the attribute is numerical, and both values are between 0 and 1, the contribution is 1 - abs(value_1 - value_2) * If the attribute is numerical, and both values are between 1 and 100, the contribution is 1 - abs(value_1 - value_2) / 100 * else, the contribution is 0 (we do not know how to handle it, so we ignore it) .. py:class:: ResultTemplateSimilarityMatrix(/, **data: Any) Bases: :py:obj:`ResultTemplateMatrix` *Similarities* between each possible combination of two *entities* within a *task*. * **Arguments**: $m, t, s$ * **Result**: $|t| imes |t|$ real-valued tensor * **Interpretation**: qualitative; visual patterns may be spotted, similarly to *InterferenceMatrix*. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'scenes' .. py:attribute:: similarity_metric :type: vision_unlearning.benchmarks.I_care.configuration.type_s :value: 'clip' .. py:attribute:: metric_key_name :type: str :value: 'similarity_metric' .. py:method:: _serialize_parameters() -> str .. py:method:: _get_partial_path_local() .. py:method:: plot_make_title(data: dict) -> str :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:class:: ResultTemplateMethodComparisonByMetricEntity(/, **data: Any) Bases: :py:obj:`ResultTemplate` Compares the distribution of one *MetricInterferencePerEntity* across multiple *unlearning methods*. * **Arguments**: m, t, me, list of u * **Result**: per-method mean, median, std, n, values; box plot * **Interpretation**: lower or higher depending on me direction. Use to rank methods by a single interference-per-entity metric. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: interference_entity :type: vision_unlearning.benchmarks.I_care.configuration.type_me .. py:attribute:: unlearning_algorithm_list :type: List[vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm] .. py:method:: _serialize_parameters() -> str .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (6, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:class:: ResultTemplateEmbeddingUnlearningProfile(/, **data: Any) Bases: :py:obj:`ResultTemplate` Embedding-space profile of one unlearning event (task, method, entity). For the specified *forgotten entity*, shows how all 100 entity embeddings shift between the baseline model (LoRA-OFF) and the model that forgot this entity (LoRA-ON). Quantifies whether the forgetting was *targeted* or *diffuse* in embedding space. **Arguments**: model, task, unlearning_algorithm, entity. **Result**: - PCA scatter (2-D) of all 100 entity mean embeddings. Baseline positions shown as open circles; unlearned positions as filled dots. The forgotten entity is highlighted with a star; an arrow marks its displacement. Points are coloured by the entity's self-interference (clip_diff) so that collateral damage is immediately visible. - Numeric summary: self-displacement magnitude (L2 norm), mean retained displacement, ``embedding_specificity_ratio`` (*directional* specificity, cosine-distance of self-displacement vs mean retained-entity displacement; same metric stored in the InterferencePerEntity (Me) for this task). **Metric note (directional vs. magnitude)**: The ``embedding_specificity_ratio`` uses cosine distance and therefore captures the *direction* of embedding change, not its magnitude. A ratio > 1 means the forgotten entity's embedding shifts in a more novel direction than the average retained entity — this is *directional specificity*. This is distinct from an L2-based magnitude specificity (which would ask whether the shift is larger in absolute terms). The displacement bars on the right plot use L2 norm; the specificity ratio shown in the title uses cosine distance. **Provenance field**: each result includes ``ratio_source`` ("ipe" when the ratio was read from the InterferencePerEntity (Me) for this task, "inline" when it was computed from the embedding files directly because the IPE column was absent). "ipe" is the canonical value; "inline" is a transitional fallback. **Interpretation**: - Specificity ratio >> 1 and large self-displacement → targeted forgetting. - Specificity ratio ~ 1 or low self-displacement → the method caused broad embedding drift without isolating the forgotten entity. - Compare with the image-level ``clip_diff`` in the scatter colours to detect the concealment pattern (embedding moves, image stays similar). **Relationship to other RTs**: - ``embedding_specificity_ratio`` belongs to ``type_me`` / ``domain_me``, so it can be passed to ``MetricMetricAlignment`` and ``MethodComparisonByMetricEntity`` like any other per-entity metric. - For cross-entity summaries, see ``ResultTemplateEmbeddingForgettingEfficiency``. - The "pinpoint-ness" concept aligns with the Holistic Unlearning Benchmark (ICCV 2025) definition of targeted forgetting. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: entity :type: str .. py:attribute:: n_pca_components :type: int :value: 2 .. py:method:: _serialize_parameters() -> str .. py:method:: _resolve_hf_entity() -> str Return the HF-compatible entity name used in embedding file names. .. py:method:: _get_baseline_embedding_path() -> str .. py:method:: _get_entity_embedding_path() -> str .. py:method:: _mean_embeddings(raw: dict) -> Dict[str, np.ndarray] :staticmethod: Group embedding records by prompted_entity and compute mean per entity. .. py:method:: _cosine_distance(a: numpy.ndarray, b: numpy.ndarray) -> float :staticmethod: .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (12, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:class:: ResultTemplateEmbeddingForgettingEfficiency(/, **data: Any) Bases: :py:obj:`ResultTemplate` Embedding-space forgetting efficiency distribution for one (task, method). Reads ``embedding_specificity_ratio`` (cosine-distance self-displacement vs. mean retained-entity displacement) from the InterferencePerEntity (Me) for this task. This RT aggregates that pre-computed metric across all entities in the task and correlates it with the image-level forgetting signal (``clip_diff``). **Arguments**: model, task, unlearning_algorithm. **Prerequisites**: The InterferencePerEntity (Me) must exist and must contain the ``embedding_specificity_ratio`` column for the requested method. Run "4. Compute interference per entity.py" first if it is missing. **Result**: - Bar chart of ``embedding_specificity_ratio`` per entity, sorted descending; dashed line at ratio = 1 (no specificity). - Scatter of ``embedding_specificity_ratio`` vs. self-``clip_diff`` per entity, with Spearman correlation and a permutation test (n_permutations resamples; parametric t-tests are invalid here because embedding vectors from the same model are correlated by architecture and data). - Numeric summary: ``n_total`` (all entities in task), ``n_valid`` (entities with non-NaN ratio — typically those for which interference_per_pair files were available), mean/std of ratio, fraction of entities with ratio > 1 *among valid entities*, Spearman r between ratio and self-clip_diff, permutation p-value. **Metric note (directional vs. magnitude)**: ``embedding_specificity_ratio`` uses cosine distance (*directional* specificity). A ratio > 1 means the forgotten entity shifts in a more novel direction than the average retained entity. This is distinct from an L2-based magnitude ratio. Both numerator (self cosine distance) and denominator (mean retained cosine distance) are stored separately so a reader can distinguish "ratio is low because target barely moves" from "ratio is low because retained entities move MORE". **Important caveat on n_valid**: ``n_valid`` is typically far smaller than ``n_total`` because ``embedding_specificity_ratio`` requires *interference_per_pair* files for each entity. Results from a small ``n_valid`` (e.g. 19/100) are underpowered and should be treated as *preliminary*. The permutation test p-values are reported with ``n_valid`` in the title for transparency. **Interpretation**: - A method with most ratios >> 1 surgically targets each forgotten entity in embedding space without disturbing retained embeddings. - A high Spearman r (ratio vs. clip_diff) means embedding-space specificity and image-level forgetting agree: the method is consistently targeted at both levels. For UCE our data show r ≈ -0.14 (not significant) whereas for distil r ≈ -0.12 (not significant at n_valid=19): the two signals decouple for UCE, consistent with the concealment hypothesis (Sharma et al., arXiv 2409.05668). **Relationship to other RTs**: - For per-entity detail, see ``ResultTemplateEmbeddingUnlearningProfile``. - ``embedding_specificity_ratio`` belongs to ``type_me`` and ``domain_me``, so it can be passed to ``MetricMetricAlignment`` and ``MethodComparisonByMetricEntity`` like any other per-entity metric. References concealment: "Sharma et al., arXiv 2409.05668" pinpoint: "Holistic Unlearning Benchmark (ICCV 2025)" .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: n_permutations :type: int :value: 10000 .. py:attribute:: significance_threshold :type: float :value: 0.05 .. py:method:: _serialize_parameters() -> str .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (14, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:data:: rt_name_to_class .. py:data:: rt_name_to_params .. py:function:: display_interesting_interferences(metadata_filtered: List[Dict[str, Any]], interference_per_pair: Dict[str, Dict[str, float]], index: int, task: Literal['scenes', 'objects', 'breeds', 'people'], method: Literal['munba', 'uce', 'distil'], num_train_epochs: int, metric: str, is_worst_biggest: bool, seed: int = 42, save_path: Optional[str] = None) -> None Compared generated images for 9 identities: target, 4 worst (excluding target), 4 best @param metadata_filtered: should be appropriate for this task (this is not verified inside the function) @param interference_per_pair: should be appropriate for this task+index+method+num_train_epochs (this is not verified inside the function) @param index: identities the target The combination of task+index+method+num_train_epochs identifies a unique unlearned model .. py:function:: analyze_relationship_regression(df: pandas.DataFrame, x: str, y: str, expected_positive: bool = True, plot: bool = True) -> bool Test linear relationship between two numerical variables with significance test and direction check. Returns True only if: (1) the slope is statistically significant (p < 0.05) (2) the slope sign matches expectation. .. py:function:: analyze_relationship_category(df, metric: str, category: str, plot: bool = True) -> bool .. py:function:: analyze_relationship_numerical(df: pandas.DataFrame, attribute: str, metric: str, plot: bool = False, plot_only_significant: bool = False) -> bool Analyzes the relationship between a numerical attribute and a numerical metric @param df: interference_per_entity; assumes df[attribute] and df[metric] are numerical @param plot: whether to plot the results @param plot_only_significant: whether to plot only significant relationships; Only applies if plot=True @return: whether any significant relationship was found --- **Pearson test** Use when you want to measure a **linear** relationship. **Assumptions:** * Both variables are **continuous** * Relationship is **linear** * **Bivariate normality** (both jointly Gaussian) * **Homoscedasticity** (constant variance) * **No strong outliers** (very sensitive) **Detects:** linear correlation only **Fails when:** relationship is monotonic but non-linear, or heavy outliers exist --------- **Spearman test** Use when you want to measure a **monotonic** relationship (not necessarily linear) or data is non-Gaussian. **Assumptions:** * Variables are at least **ordinal** * Relationship is **monotonic** (increasing or decreasing) * **No distributional assumptions** * **Robust to outliers** **Detects:** any monotonic trend (linear or curved) **Fails when:** relationship is non-monotonic (e.g., U-shaped) .. py:function:: analyze_relationship_categorical(df: pandas.DataFrame, attribute: str, metric: str, plot: bool = False, plot_only_significant: bool = False, show_axhline: Optional[float] = None, min_samples_per_category: int = 5, extra_title: str = '') -> bool Analyzes the relationship between a categorical attribute and a numerical metric @param df: interference_per_entity; assumes df[attribute] is categorical and df[metric] is numerical @param plot: whether to plot the results @param plot_only_significant: whether to plot only significant relationships; Only applies if plot=True @param show_axhline: if provided, shows a horizontal line at this y-value; Only applies if plot=True @return: whether any significant relationship was found ------ **ANOVA (f_oneway)** Use when you want to test if **group means differ** across **3+ independent groups** under parametric assumptions. **Assumptions:** * Dependent variable is **continuous** * Groups are **independent** * **Normality** within each group * **Homoscedasticity** (equal variances) * No strong **outliers** **Hypothesis:** * H₀: all group means are equal * H₁: at least one mean differs **Detects:** differences in **means** **Fails when:** heavy skew, unequal variances, small n with non-Gaussian data ------ **Kruskal-Wallis (kruskal)** Use when you want to test if **group distributions differ** without parametric assumptions. **Assumptions:** * Dependent variable is **ordinal or continuous** * Groups are **independent** * **Same shaped distributions** (only medians should differ for clean interpretation) * No normality or equal-variance requirement **Hypothesis:** * H₀: all group distributions are equal * H₁: at least one group differs **Detects:** differences in **medians / distributions** **Fails when:** distributions differ in shape (then result is ambiguous) .. py:function:: analyze_correlation_between_pairwise_metrics(df1: pandas.DataFrame, df2: pandas.DataFrame, metric1_name: str, metric2_name: str, exclude_diagonal: bool = True, plot=True, plot_only_significant=True) -> bool df1 and df2 are square DataFrames; index and cols are the same within both and among both .. py:function:: check_eval_results(eval_results, name, threshold: float, operator: Literal['gt', 'lt']) -> float Check if the metric satisfy the EXPECTED threshold