vision_unlearning.benchmarks.I_care.result_templates ==================================================== .. py:module:: vision_unlearning.benchmarks.I_care.result_templates Attributes ---------- .. autoapisummary:: vision_unlearning.benchmarks.I_care.result_templates.shap vision_unlearning.benchmarks.I_care.result_templates.logger vision_unlearning.benchmarks.I_care.result_templates.rt_name_to_class vision_unlearning.benchmarks.I_care.result_templates.rt_name_to_params Classes ------- .. autoapisummary:: vision_unlearning.benchmarks.I_care.result_templates.ResultTemplate vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateMetricMetricAlignment vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateMetricSimilarityAlignment vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateMetricSimilarityAlignmentMulti vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateSignificantRelationshipNumerical vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateSignificantRelationshipCategorical vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateCountSignificantRelationship vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateImplicitAssociationTest vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateMinimumCutInterference vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateUnlearningVisualSummary vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateInterferenceVisualSummary vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateMatrix vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateInterferenceMatrix vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateSimilarityMatrix vision_unlearning.benchmarks.I_care.result_templates.ResultTemplateMethodComparisonByMetricEntity Functions --------- .. autoapisummary:: vision_unlearning.benchmarks.I_care.result_templates.jacc_metric_score vision_unlearning.benchmarks.I_care.result_templates.display_interesting_interferences vision_unlearning.benchmarks.I_care.result_templates.analyze_relationship_regression vision_unlearning.benchmarks.I_care.result_templates.analyze_relationship_category vision_unlearning.benchmarks.I_care.result_templates.analyze_relationship_numerical vision_unlearning.benchmarks.I_care.result_templates.analyze_relationship_categorical vision_unlearning.benchmarks.I_care.result_templates.analyze_correlation_between_pairwise_metrics vision_unlearning.benchmarks.I_care.result_templates.check_eval_results Module Contents --------------- .. py:data:: shap :value: None .. py:data:: logger .. py:class:: ResultTemplate(/, **data: Any) Bases: :py:obj:`pydantic.BaseModel` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: recompute_if_exists :type: bool :value: False .. py:attribute:: save_outputs :type: bool :value: True .. py:attribute:: base_folder :type: str :value: 'assets' .. py:attribute:: remote_repository_name :type: str :value: 'LeonardoBenitez/VisionUnlearningEvaluationTestbeds' .. py:method:: _serialize_parameters() -> str :abstractmethod: .. py:method:: _get_data_path_remote() -> str .. py:method:: _get_data_path_local() -> str .. py:method:: _fig_to_bytes(fig: matplotlib.figure.Figure) -> bytes :classmethod: .. py:method:: _compute_from_scratch() -> dict | list :abstractmethod: .. py:method:: compute() -> dict .. py:class:: ResultTemplateMetricMetricAlignment(/, **data: Any) Bases: :py:obj:`ResultTemplate` Measures how strongly two *MetricInterferencePerEntity* metrics are correlated. **Arguments:** `m`, `t`, `u`, `m_e1`, `m_e2`. **Result:** Pearson p-value, Spearman p-value, Pearson correlation, scatter plot. **Interpretation:** quantitative; the higher the correlation, the lower the need to calculate both metrics for this specific choice of `m`, `t`, and `u`. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_entity_1 :type: vision_unlearning.benchmarks.I_care.configuration.type_me .. py:attribute:: interference_entity_2 :type: vision_unlearning.benchmarks.I_care.configuration.type_me .. py:class:: ResultTemplateMetricSimilarityAlignment(/, **data: Any) Bases: :py:obj:`ResultTemplate` To what degree similar *entities* interfere more with each other. Formalized in `ap:prediction`, which also proposes its natural expansion to a multivariable and non-linear predictive regression. **Arguments:** `m`, `t`, `u`, `m_p`, `s`. **Result:** Pearson p-value, Spearman p-value, Pearson correlation, scatter plot. **Interpretation:** quantitative; if this value is high, interference between two *entities* can be approximated by *similarity* (which is cheaper to compute for any new *entity*). Equivalently, the amount of "transmission wires" can be summarized by this single *similarity* function. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_pair :type: vision_unlearning.benchmarks.I_care.configuration.type_mp .. py:attribute:: similarity_metric :type: vision_unlearning.benchmarks.I_care.configuration.type_s .. py:attribute:: significance_threshold :type: float :value: 0.05 .. py:method:: _serialize_parameters() -> str .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (6, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch(exclude_diagonal: bool = True) -> dict .. py:class:: ResultTemplateMetricSimilarityAlignmentMulti(/, **data: Any) Bases: :py:obj:`ResultTemplate` Multi-input Single-output Regression Generalization of ResultTemplateMetricSimilarityAlignment (see also Appendix E, adapted from the multi-output setting). Also, the interpretability and feature engineering aspects are improved. --- We consider a fixed *model* \(m\), *task* \(t\), and *unlearning method* \(u\), which are omitted for brevity. The objective is to quantify whether interference between *entities* is aligned with their *similarity*, i.e., to what degree similar *entities* interfere more with each other. For every ordered pair of distinct *entities* \(e_i, e_j \in t\) with \(i eq j\), we observe several *SimilarityBetweenEntities* measures, indexed by superscripts \(\ell = 1, 2, \dots, |S|\), and a single *MetricInterferencePerEntityPair* target \(m_p(e_i,e_j)\). Each ordered pair \((e_i, e_j)\) is therefore treated as one data point with feature vector $$ \mathbf{X}_{ij} = ig( s^{(1)}(e_i, e_j), \dots, s^{(|S|)}(e_i, e_j) ig) $$ and scalar target $$ Y_{ij} = m_p(e_i, e_j). $$ The resulting dataset is $$ \mathcal{D} = \{ (\mathbf{X}_{ij}, Y_{ij}) \mid e_i, e_j \in t,\ i eq j \}. $$ From this dataset, a regression model can be estimated using standard regression procedures with appropriate validation. In the linear case, $$ Y_{ij} = eta_0 + \sum_{\ell=1}^{|S|} eta_{\ell} X^{(\ell)}_{ij} + arepsilon_{ij}. $$ Given a specific *entity* \(e_i\) whose removal is considered, similarities $$ X^{(\ell)}_{ij} = s^{(\ell)}(e_i, e_j) $$ can be computed for all remaining *entities* \(e_j \in t\). The fitted model then yields predictions $$ \hat{Y}_{ij} = f(\mathbf{X}_{ij}), $$ which approximate the expected interference on each receiver *entity*. Furthermore, the concept of *similarity* may also encode several forms of practical data engineering. For example, one may define: - a distinct *similarity* function for each *attribute*, or - a *similarity* function based only on the attributes of the emitter entity. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_pair :type: vision_unlearning.benchmarks.I_care.configuration.type_mp .. py:attribute:: similarity_metric_list :type: List[vision_unlearning.benchmarks.I_care.configuration.type_s] .. py:attribute:: significance_threshold :type: float :value: 0.05 .. py:attribute:: include_attribute_diff_similarity :type: bool :value: True .. py:attribute:: include_attribute_value_similarity :type: bool :value: True .. py:attribute:: regression_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_regression_algorithm :value: 'linear_regression' .. py:attribute:: random_state :type: int :value: 42 .. py:attribute:: test_size :type: float :value: 0.3 .. py:method:: _serialize_parameters() -> str .. py:method:: _get_partial_path_local() .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (6, 15), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch(exclude_diagonal: bool = True, entity_col: str = 'name') -> dict .. py:class:: ResultTemplateSignificantRelationshipNumerical(/, **data: Any) Bases: :py:obj:`ResultTemplate` Measures whether two numerical attributes are significantly correlated. Formalized in `ap:rt_relationship`. **Arguments:** `m`, `t`, `u`, `m_e`, `a`. **Result:** Pearson p-value, Spearman p-value, Pearson correlation, scatter plot. **Interpretation:** qualitative; the researcher should decide if it is ethical or desirable that this *attribute* propagates interferences. **Pearson test** Use when you want to measure a **linear** relationship. **Assumptions:** * Both variables are **continuous** * Relationship is **linear** * **Bivariate normality** (both jointly Gaussian) * **Homoscedasticity** (constant variance) * **No strong outliers** (very sensitive) **Detects:** linear correlation only **Fails when:** relationship is monotonic but non-linear, or heavy outliers exist **Spearman test** Use when you want to measure a **monotonic** relationship (not necessarily linear) or data is non-Gaussian. **Assumptions:** * Variables are at least **ordinal** * Relationship is **monotonic** (increasing or decreasing) * **No distributional assumptions** * **Robust to outliers** **Detects:** any monotonic trend (linear or curved) **Fails when:** relationship is non-monotonic (e.g., U-shaped) .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_entity :type: vision_unlearning.benchmarks.I_care.configuration.type_me .. py:attribute:: attribute :type: str .. py:attribute:: significance_threshold :type: float :value: 0.05 .. py:method:: _get_data_path_remote() -> str .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (6, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:class:: ResultTemplateSignificantRelationshipCategorical(/, **data: Any) Bases: :py:obj:`ResultTemplate` Statistical significance of the average `MetricInterferencePerEntity` across all *entities*, when grouped by each of its values. Formalized in `ap:rt_relationship`. **Arguments:** `m`, `t`, `u`, `m_e`, `a`, optional `filterAttributeValue`. **Result:** ANOVA p-value, Kruskal-Wallis p-value, average value of `m_e` grouped by each value of `a`, grouped boxplot. **Interpretation:** qualitative; similar to *SignificantRelationshipNumerical*. The optional argument *filterAttributeValue* restricts which emitter *entities* are included, allowing the analysis of interference flow distribution, such as whether politicians cause more interference to other politicians than artists cause to other artists. **ANOVA** Use when you want to test if **group means differ** across **3+ independent groups** under parametric assumptions. **Assumptions:** * Dependent variable is **continuous** * Groups are **independent** * **Normality** within each group * **Homoscedasticity** (equal variances) * No strong **outliers** **Hypothesis:** * H₀: all group means are equal * H₁: at least one mean differs **Detects:** differences in **means** **Fails when:** heavy skew, unequal variances, small n with non-Gaussian data **Kruskal-Wallis** Use when you want to test if **group distributions differ** without parametric assumptions. **Assumptions:** * Dependent variable is **ordinal or continuous** * Groups are **independent** * **Same shaped distributions** (only medians should differ for clean interpretation) * No normality or equal-variance requirement **Hypothesis:** * H₀: all group distributions are equal * H₁: at least one group differs **Detects:** differences in **medians / distributions** **Fails when:** distributions differ in shape (then result is ambiguous) .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_entity :type: vision_unlearning.benchmarks.I_care.configuration.type_me .. py:attribute:: attribute :type: str .. py:attribute:: attribute_value :type: Optional[str | int] :value: None .. py:attribute:: min_samples_per_category :type: int :value: 5 .. py:attribute:: significance_threshold :type: float :value: 0.05 .. py:method:: _get_data_path_remote() -> str .. py:method:: plot(data: dict, extra_title: str = '', figsize: Tuple[int, int] = (6, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:class:: ResultTemplateCountSignificantRelationship(/, **data: Any) Bases: :py:obj:`ResultTemplate` Number of significant relationships across all combinations of *attributes* and *MetricInterferencePerEntity*. **Arguments:** `m`, `t`, `u`, list of `m_e`, list of `a`. **Result:** integer, list of significances. **Interpretation:** quantitative; the lower the better. Since the attributes for which it is ethical to propagate interference are constant across all *models* and *methods*, a higher value directly implies a higher number of ethical violations, that is, a larger number of "transmission wires" in a given task effectively used by this *method* and *model*. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm_list :type: List[vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm] .. py:attribute:: interference_entity_list :type: List[vision_unlearning.benchmarks.I_care.configuration.type_me] .. py:attribute:: attribute_list :type: List[str] .. py:attribute:: top_n :type: int :value: 10 .. py:method:: _serialize_parameters() -> str .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (6, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:class:: ResultTemplateImplicitAssociationTest(/, **data: Any) Bases: :py:obj:`ResultTemplate` Measures how the strength of automatic associations `B` between two pairs of *entities* changes after unlearning. **Arguments:** `m`, `t`, `u`, `a_1`, `a_2`, `l`. **Result:** `|a| x |a|` real-valued tensor `ΔB`. **Interpretation:** qualitative; a human should decide whether it is ethical or desirable for the unlearning process to cause this change in implicit association between the chosen *attributes*. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: attribute_1 :type: str .. py:attribute:: attribute_2 :type: str .. py:attribute:: latent_embedding :type: vision_unlearning.benchmarks.I_care.configuration.type_l .. py:class:: ResultTemplateMinimumCutInterference(/, **data: Any) Bases: :py:obj:`ResultTemplate` Interprets a *task* as a directed weighted graph and computes the minimum cut separating two *entities* As a consequence of the max-flow min-cut theorem, it directly follows that the minimum cut is the smallest influence whose removal eliminates every directed influence path from $e_1$ to $e_2$. Based on this, we conjecture that if we need to unlearn $e_1$ while minimizing harm to $e_2$, then the ideal intervention in the unlearning process is to increase the preservation of the emitter-side nodes. More intuitively, we can think of this intervention as "blocking the interference path," as performed in electrical circuits to protect sensitive components (such as ground partitioning, shielding traces, among others. **Arguments:** $m$, $t$, $u$, $e_1$, $e_2$, $m_p$. **Result:** list of *entities* (corresponding to the emitter-side nodes). **Interpretation:** qualitative; small set of nodes through which most of the interference from $e_1$ propagates to $e_2$. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_pair :type: vision_unlearning.benchmarks.I_care.configuration.type_mp .. py:attribute:: entity_1 :type: str .. py:attribute:: entity_2 :type: str .. py:class:: ResultTemplateUnlearningVisualSummary(/, **data: Any) Bases: :py:obj:`ResultTemplate` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:class:: ResultTemplateInterferenceVisualSummary(/, **data: Any) Bases: :py:obj:`ResultTemplate` Compared generated images for 9 identities: target, 4 worst (excluding target), 4 best .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_pair :type: vision_unlearning.benchmarks.I_care.configuration.type_mp .. py:attribute:: entity :type: Optional[str] :value: None .. py:attribute:: entity_index :type: Optional[int] :value: None .. py:attribute:: seed :type: int :value: 42 .. py:attribute:: images_max_dim :type: int :value: 124 .. py:method:: _resolve_entity() Ensures both entity andentity_index are filled. Modifies in place At the end, both are set and consistent with each other .. py:method:: _serialize_parameters() -> str .. py:method:: plot(data: dict, figsize: Optional[Tuple[int, int]] = (18, 4), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() .. py:class:: ResultTemplateMatrix(/, **data: Any) Bases: :py:obj:`ResultTemplate` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: metric_key_name :type: str .. py:method:: plot_make_title(data: dict) -> str :classmethod: :abstractmethod: .. py:method:: plot(data: dict, figsize: Optional[Tuple[int, int]] = None, cmap: str = 'viridis', title: str = '', xlabel: str = 'Receiver entity', ylabel: str = 'Emitter entity', return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:class:: ResultTemplateInterferenceMatrix(/, **data: Any) Bases: :py:obj:`ResultTemplateMatrix` *MetricInterferencePerEntityPair* between each possible combination of two *entities* within a *task*. **Arguments:** `m`, `t`, `u`, `m_p`. **Result:** `|t| x |t|` real-valued tensor. **Interpretation:** qualitative; visual patterns may be spotted, especially when rearranging indices in a meaningful manner (for example, grouping professions together). Further quantitative values may be derived, such as the average value or the ratio between the diagonal-average value and the non-diagonal-average value. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_pair :type: vision_unlearning.benchmarks.I_care.configuration.type_mp .. py:attribute:: metric_key_name :type: str :value: 'interference_pair' .. py:method:: _serialize_parameters() -> str .. py:method:: plot_make_title(data: dict) -> str :classmethod: .. py:method:: _compute_from_scratch() .. py:function:: jacc_metric_score(entity_1: str, entity_2: str, metadata_filtered: List[Dict[str, Any]], entity_col: str = 'name') -> float Jaccard similarity between two entities, based on their attributes. Each attribute (column) contributes between 0 and 1 to the similarity We do not know the types and ranges of the attributes beforehand. For each attribute, both values for the two entities must be non-NaN and of the same type, otherwise we ignore that attribute (contribution 0). The calculation for each attribute is as follows: * If the attribute is categorical (str or bool), the contribution is 1 if the two entities have the same value for that attribute, and 0 otherwise. * If the attribute is numerical, and both values are between 0 and 1, the contribution is 1 - abs(value_1 - value_2) * If the attribute is numerical, and both values are between 1 and 100, the contribution is 1 - abs(value_1 - value_2) / 100 * else, the contribution is 0 (we do not know how to handle it, so we ignore it) .. py:class:: ResultTemplateSimilarityMatrix(/, **data: Any) Bases: :py:obj:`ResultTemplateMatrix` *Similarities* between each possible combination of two *entities* within a *task*. * **Arguments**: $m, t, s$ * **Result**: $|t| imes |t|$ real-valued tensor * **Interpretation**: qualitative; visual patterns may be spotted, similarly to *InterferenceMatrix*. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'scenes' .. py:attribute:: similarity_metric :type: vision_unlearning.benchmarks.I_care.configuration.type_s :value: 'clip' .. py:attribute:: metric_key_name :type: str :value: 'similarity_metric' .. py:method:: _serialize_parameters() -> str .. py:method:: _get_partial_path_local() .. py:method:: plot_make_title(data: dict) -> str :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:class:: ResultTemplateMethodComparisonByMetricEntity(/, **data: Any) Bases: :py:obj:`ResultTemplate` Compares the distribution of one *MetricInterferencePerEntity* across multiple *unlearning methods*. * **Arguments**: m, t, me, list of u * **Result**: per-method mean, median, std, n, values; box plot * **Interpretation**: lower or higher depending on me direction. Use to rank methods by a single interference-per-entity metric. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: interference_entity :type: vision_unlearning.benchmarks.I_care.configuration.type_me .. py:attribute:: unlearning_algorithm_list :type: List[vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm] .. py:method:: _serialize_parameters() -> str .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (6, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:data:: rt_name_to_class .. py:data:: rt_name_to_params .. py:function:: display_interesting_interferences(metadata_filtered: List[Dict[str, Any]], interference_per_pair: Dict[str, Dict[str, float]], index: int, task: Literal['scenes', 'objects', 'breeds', 'people'], method: Literal['munba', 'uce', 'distil'], num_train_epochs: int, metric: str, is_worst_biggest: bool, seed: int = 42, save_path: Optional[str] = None) -> None Compared generated images for 9 identities: target, 4 worst (excluding target), 4 best @param metadata_filtered: should be appropriate for this task (this is not verified inside the function) @param interference_per_pair: should be appropriate for this task+index+method+num_train_epochs (this is not verified inside the function) @param index: identities the target The combination of task+index+method+num_train_epochs identifies a unique unlearned model .. py:function:: analyze_relationship_regression(df: pandas.DataFrame, x: str, y: str, expected_positive: bool = True, plot: bool = True) -> bool Test linear relationship between two numerical variables with significance test and direction check. Returns True only if: (1) the slope is statistically significant (p < 0.05) (2) the slope sign matches expectation. .. py:function:: analyze_relationship_category(df, metric: str, category: str, plot: bool = True) -> bool .. py:function:: analyze_relationship_numerical(df: pandas.DataFrame, attribute: str, metric: str, plot: bool = False, plot_only_significant: bool = False) -> bool Analyzes the relationship between a numerical attribute and a numerical metric @param df: interference_per_entity; assumes df[attribute] and df[metric] are numerical @param plot: whether to plot the results @param plot_only_significant: whether to plot only significant relationships; Only applies if plot=True @return: whether any significant relationship was found --- **Pearson test** Use when you want to measure a **linear** relationship. **Assumptions:** * Both variables are **continuous** * Relationship is **linear** * **Bivariate normality** (both jointly Gaussian) * **Homoscedasticity** (constant variance) * **No strong outliers** (very sensitive) **Detects:** linear correlation only **Fails when:** relationship is monotonic but non-linear, or heavy outliers exist --------- **Spearman test** Use when you want to measure a **monotonic** relationship (not necessarily linear) or data is non-Gaussian. **Assumptions:** * Variables are at least **ordinal** * Relationship is **monotonic** (increasing or decreasing) * **No distributional assumptions** * **Robust to outliers** **Detects:** any monotonic trend (linear or curved) **Fails when:** relationship is non-monotonic (e.g., U-shaped) .. py:function:: analyze_relationship_categorical(df: pandas.DataFrame, attribute: str, metric: str, plot: bool = False, plot_only_significant: bool = False, show_axhline: Optional[float] = None, min_samples_per_category: int = 5, extra_title: str = '') -> bool Analyzes the relationship between a categorical attribute and a numerical metric @param df: interference_per_entity; assumes df[attribute] is categorical and df[metric] is numerical @param plot: whether to plot the results @param plot_only_significant: whether to plot only significant relationships; Only applies if plot=True @param show_axhline: if provided, shows a horizontal line at this y-value; Only applies if plot=True @return: whether any significant relationship was found ------ **ANOVA (f_oneway)** Use when you want to test if **group means differ** across **3+ independent groups** under parametric assumptions. **Assumptions:** * Dependent variable is **continuous** * Groups are **independent** * **Normality** within each group * **Homoscedasticity** (equal variances) * No strong **outliers** **Hypothesis:** * H₀: all group means are equal * H₁: at least one mean differs **Detects:** differences in **means** **Fails when:** heavy skew, unequal variances, small n with non-Gaussian data ------ **Kruskal-Wallis (kruskal)** Use when you want to test if **group distributions differ** without parametric assumptions. **Assumptions:** * Dependent variable is **ordinal or continuous** * Groups are **independent** * **Same shaped distributions** (only medians should differ for clean interpretation) * No normality or equal-variance requirement **Hypothesis:** * H₀: all group distributions are equal * H₁: at least one group differs **Detects:** differences in **medians / distributions** **Fails when:** distributions differ in shape (then result is ambiguous) .. py:function:: analyze_correlation_between_pairwise_metrics(df1: pandas.DataFrame, df2: pandas.DataFrame, metric1_name: str, metric2_name: str, exclude_diagonal: bool = True, plot=True, plot_only_significant=True) -> bool df1 and df2 are square DataFrames; index and cols are the same within both and among both .. py:function:: check_eval_results(eval_results, name, threshold: float, operator: Literal['gt', 'lt']) -> float Check if the metric satisfy the EXPECTED threshold