vision_unlearning.benchmarks ============================ .. py:module:: vision_unlearning.benchmarks Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/vision_unlearning/benchmarks/I_care/index /autoapi/vision_unlearning/benchmarks/unlearn_canvas/index Attributes ---------- .. autoapisummary:: vision_unlearning.benchmarks.domain_unlearning_algorithm vision_unlearning.benchmarks.domain_task vision_unlearning.benchmarks.domain_attribute vision_unlearning.benchmarks.domain_entity vision_unlearning.benchmarks.domain_model vision_unlearning.benchmarks.domain_mp vision_unlearning.benchmarks.domain_me vision_unlearning.benchmarks.domain_s vision_unlearning.benchmarks.domain_l vision_unlearning.benchmarks.type_task vision_unlearning.benchmarks.type_unlearning_algorithm vision_unlearning.benchmarks.type_model vision_unlearning.benchmarks.type_mp vision_unlearning.benchmarks.type_me vision_unlearning.benchmarks.type_s vision_unlearning.benchmarks.type_l vision_unlearning.benchmarks.GUI_TO_BACKEND vision_unlearning.benchmarks.unlearning_algorithm_to_epochs vision_unlearning.benchmarks.s_to_direction vision_unlearning.benchmarks.EMBEDDING_MODEL vision_unlearning.benchmarks.EMBEDDING_DIM vision_unlearning.benchmarks.rt_name_to_class vision_unlearning.benchmarks.rt_name_to_params Exceptions ---------- .. autoapisummary:: vision_unlearning.benchmarks.InvalidAttributeTypeError vision_unlearning.benchmarks.InsufficientSamplesError Classes ------- .. autoapisummary:: vision_unlearning.benchmarks.InterferencePerEntity vision_unlearning.benchmarks.ResultTemplate vision_unlearning.benchmarks.ResultTemplateMetricMetricAlignment vision_unlearning.benchmarks.ResultTemplateMetricSimilarityAlignment vision_unlearning.benchmarks.ResultTemplateMetricSimilarityAlignmentMulti vision_unlearning.benchmarks.ResultTemplateSignificantRelationshipNumerical vision_unlearning.benchmarks.ResultTemplateSignificantRelationshipCategorical vision_unlearning.benchmarks.ResultTemplateCountSignificantRelationship vision_unlearning.benchmarks.ResultTemplateImplicitAssociationTest vision_unlearning.benchmarks.ResultTemplateMinimumCutInterference vision_unlearning.benchmarks.ResultTemplateUnlearningVisualSummary vision_unlearning.benchmarks.ResultTemplateInterferenceVisualSummary vision_unlearning.benchmarks.ResultTemplateMatrix vision_unlearning.benchmarks.ResultTemplateInterferenceMatrix vision_unlearning.benchmarks.ResultTemplateSimilarityMatrix vision_unlearning.benchmarks.ResultTemplateMethodComparisonByMetricEntity Functions --------- .. autoapisummary:: vision_unlearning.benchmarks.convert_params_from_gui_to_backend vision_unlearning.benchmarks.get_interference_per_pair_path vision_unlearning.benchmarks.get_interference_per_pair vision_unlearning.benchmarks.exists_interference_per_pair vision_unlearning.benchmarks.save_interference_per_pair vision_unlearning.benchmarks.get_interference_per_pair_inverse vision_unlearning.benchmarks.get_interference_per_entity_path vision_unlearning.benchmarks.get_interference_per_entity vision_unlearning.benchmarks.save_interference_per_entity vision_unlearning.benchmarks.choose_metric_column_interference_per_entity vision_unlearning.benchmarks.find_worst_interfered vision_unlearning.benchmarks.metric_of_worst_interfered vision_unlearning.benchmarks.is_worst_interfered_target vision_unlearning.benchmarks.number_of_interfered_worse_than_target vision_unlearning.benchmarks.number_of_interfered_worse_than_threshold vision_unlearning.benchmarks.average_metric vision_unlearning.benchmarks._encode_image_file vision_unlearning.benchmarks._decode_image vision_unlearning.benchmarks.explanation_to_dict vision_unlearning.benchmarks.dict_to_explanation vision_unlearning.benchmarks.load_dino_model vision_unlearning.benchmarks.embed_image_with_dino vision_unlearning.benchmarks.embed_forgetting_session vision_unlearning.benchmarks.embed_forgetting_session_batched vision_unlearning.benchmarks.jacc_metric_score vision_unlearning.benchmarks.display_interesting_interferences vision_unlearning.benchmarks.analyze_relationship_regression vision_unlearning.benchmarks.analyze_relationship_category vision_unlearning.benchmarks.analyze_relationship_numerical vision_unlearning.benchmarks.analyze_relationship_categorical vision_unlearning.benchmarks.analyze_correlation_between_pairwise_metrics vision_unlearning.benchmarks.check_eval_results vision_unlearning.benchmarks.get_metadata_filtered vision_unlearning.benchmarks.get_metadata_filtered_path vision_unlearning.benchmarks.get_target_overwrite vision_unlearning.benchmarks.get_generated_dataset_file vision_unlearning.benchmarks.huggingface_dataset_file_exists vision_unlearning.benchmarks.huggingface_dataset_file_download vision_unlearning.benchmarks.huggingface_dataset_upload vision_unlearning.benchmarks.huggingface_dataset_file_upload vision_unlearning.benchmarks.huggingface_dataset_download Package Contents ---------------- .. py:data:: domain_unlearning_algorithm :value: ['FADE', 'Munba', 'UCE'] .. py:data:: domain_task :value: ['Breeds', 'Scenes', 'People'] .. py:data:: domain_attribute .. py:data:: domain_entity .. py:data:: domain_model :value: ['Stable Diffusion 1.4'] .. py:data:: domain_mp :value: ['Delta Clip', 'Delta Brisque', 'RMSE', 'SSIM'] .. py:data:: domain_me :value: ['Emitter worst interfered brisque diff', 'Emitter worst interfered clip diff', 'Emitter worst... .. py:data:: domain_s :value: ['Clip Cosine Similarity', 'Jacc Similarity'] .. py:data:: domain_l :value: ['Clip Embedding'] .. py:data:: type_task .. py:data:: type_unlearning_algorithm .. py:data:: type_model .. py:data:: type_mp .. py:data:: type_me .. py:data:: type_s .. py:data:: type_l .. py:data:: GUI_TO_BACKEND .. py:data:: unlearning_algorithm_to_epochs .. py:function:: convert_params_from_gui_to_backend(params: Dict[str, Any]) -> Dict[str, Any] Convert GUI values to backend literal values. Unknown keys are passed through unchanged. None stays None. .. py:data:: s_to_direction :type: Dict[type_s, type_direction] .. py:function:: get_interference_per_pair_path(task: Literal['scenes', 'objects', 'breeds', 'people'], index: int, method: Literal['munba', 'uce', 'distil'], num_train_epochs: int, base_folder: str = 'assets') -> str .. py:function:: get_interference_per_pair(task: Literal['scenes', 'objects', 'breeds', 'people'], index: int, method: Literal['munba', 'uce', 'distil'], num_train_epochs: int, max_identities: int = 100, base_folder: str = 'assets') -> Dict[str, Dict[str, float]] .. py:function:: exists_interference_per_pair(task: Literal['scenes', 'objects', 'breeds', 'people'], index: int, method: Literal['munba', 'uce', 'distil'], num_train_epochs: int, base_folder: str = 'assets') -> bool .. py:function:: save_interference_per_pair(interference_per_pair: Dict[str, Dict[str, float]], task: Literal['scenes', 'objects', 'breeds', 'people'], index: int, method: Literal['munba', 'uce', 'distil'], num_train_epochs: int, base_folder: str = 'assets') -> None .. py:function:: get_interference_per_pair_inverse(task: Literal['scenes', 'objects', 'breeds', 'people'], index: int, method: Literal['munba', 'uce', 'distil'], num_train_epochs: int, index_start: int = 0, max_identities: int = 100) -> Dict[str, Dict[str, float]] .. py:function:: get_interference_per_entity_path(task: Literal['scenes', 'objects', 'breeds', 'people'], base_folder: str = 'assets') -> str .. py:function:: get_interference_per_entity(task: Literal['scenes', 'objects', 'breeds', 'people'], max_identities: int = 100, base_folder: str = 'assets') -> List[Dict[str, Any]] .. py:function:: save_interference_per_entity(task: Literal['scenes', 'objects', 'breeds', 'people'], metadata_filtered: List[Dict[str, Any]], base_folder: str = 'assets') -> None .. py:class:: InterferencePerEntity(/, **data: Any) Bases: :py:obj:`pydantic.BaseModel` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: base_folder :type: str :value: 'assets' .. py:attribute:: remote_repository_name :type: str :value: 'LeonardoBenitez/VisionUnlearningEvaluationTestbeds' .. py:attribute:: save_outputs :type: bool :value: True .. py:attribute:: recompute_if_exists :type: bool :value: False .. py:method:: _get_data_path_remote() -> str .. py:method:: _get_data_path_local() -> str .. py:method:: compute() -> List[Dict[str, Any]] .. py:function:: choose_metric_column_interference_per_entity(unlearning_algorithm: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm, interference_entity: vision_unlearning.benchmarks.I_care.configuration.type_me, metric_cols: List[str]) -> str The columns of the interference per entity file are not named in a way that is easy to generate given `unlearning_algorithm` and `interference_entity`, so we need to search for the right one. We assume there is only one match, and we assert it. If there are no matches or more than one match, we raise an error. The names look like this: 'metric_distil_400_emitter_minus_receiver_worst_interfered_ssim (↓)', 'metric_distil_400_emitter_minus_receiver_number_of_interfered_worse_than_target_brisque_diff (↓)', 'metric_distil_400_emitter_minus_receiver_number_of_interfered_worse_than_target_clip_diff (↓)', 'metric_distil_400_emitter_minus_receiver_number_of_interfered_worse_than_target_rmse (↓)', 'metric_distil_400_emitter_minus_receiver_number_of_interfered_worse_than_target_ssim (↓)', 'metric_distil_400_emitter_minus_receiver_number_of_interfered_worse_than_zero_clip_diff (↓)', 'metric_distil_400_emitter_minus_receiver_average_brisque_diff (↓)', 'metric_distil_400_emitter_minus_receiver_average_clip_diff (↑)', 'metric_uce_000_emitter_minus_receiver_average_rmse (↓)', 'metric_munba_100_emitter_minus_receiver_average_ssim (↑)', TODO: these names are defined in `4. Compute interference per entity.ipynb`. There should be a central way of defining them. .. py:function:: find_worst_interfered(interference_per_pair: dict, metric: str, is_worst_biggest: bool) -> Tuple[str, float] .. py:function:: metric_of_worst_interfered(interference_per_pair: dict, metric: str, is_worst_biggest: bool) -> float .. py:function:: is_worst_interfered_target(interference_per_pair: dict, metric: str, is_worst_biggest: bool, target: str) -> bool .. py:function:: number_of_interfered_worse_than_target(interference_per_pair: dict, metric: str, is_worst_biggest: bool, target: str) -> int .. py:function:: number_of_interfered_worse_than_threshold(interference_per_pair: dict, metric: str, is_worst_biggest: bool, threshold: float) -> int .. py:function:: average_metric(interference_per_pair: dict, metric: str) -> float .. py:function:: _encode_image_file(img_path: str, max_dim: int = 1024) -> str Downsample / reduce resolution to limit size before encoding .. py:function:: _decode_image(image_data: str) -> io.BytesIO .. py:function:: explanation_to_dict(expl: Any) -> Dict[str, Any] Serialize a shap.Explanation to a plain dict (JSON-serializable). .. py:function:: dict_to_explanation(d: Dict[str, Any]) -> Any Deserialize a plain dict back to a shap.Explanation. Requires 'shap' package (optional dependency — install with pip install shap or pip install vision-unlearning[testbed]). .. py:exception:: InvalidAttributeTypeError Bases: :py:obj:`ValueError` Inappropriate argument value (of correct type). .. py:exception:: InsufficientSamplesError Bases: :py:obj:`ValueError` Inappropriate argument value (of correct type). .. py:data:: EMBEDDING_MODEL :value: 'dinov2_vits14' .. py:data:: EMBEDDING_DIM :value: 384 .. py:function:: load_dino_model(model_name: str = EMBEDDING_MODEL, force_device: Optional[str] = None) -> Tuple[Any, Any, str] Load DINOv2 model, transform pipeline, and device. Heavy imports (torch, torchvision) happen here, not at module load. :param model_name: DINOv2 model variant (default: 'dinov2_vits14' → 384-dim CLS). :param force_device: If set, use this device string instead of auto-detecting. :returns: (model, transform, device) tuple. model: DINOv2 PyTorch model in eval mode, on device. transform: torchvision.transforms pipeline (resize → crop → normalize). device: device string ('cuda' or 'cpu'). .. py:function:: embed_image_with_dino(image_path: str, model: Any, transform: Any, device: str) -> List[float] Embed a single image using a pre-loaded DINOv2 model. :param image_path: Path to a PNG/JPEG image on disk. :param model: DINOv2 model (from load_dino_model()). :param transform: torchvision transform (from load_dino_model()). :param device: device string ('cuda' or 'cpu'). :returns: 384-dim CLS embedding as a plain Python list of floats. TODO: refactor into batched DataLoader for throughput (currently single-image). .. py:function:: embed_forgetting_session(dataset_folder: str, seeds: List[int], prompts: List[str], metadata_filtered: List[Dict[str, Any]], lora_state: Literal['on', 'off'], task: str, embed_image_fn: Optional[Callable[[str], List[float]]] = None) -> List[Dict[str, Any]] Embed all images from one forgetting session (entity or baseline). Iterates over all (seed, prompt) combinations and embeds each matching image. Images that do not exist on disk are skipped with a warning. :param dataset_folder: Local directory containing the generated images. :param seeds: List of generation seeds (e.g. [0, 1, 2, 3]). :param prompts: Full prompt strings (e.g. "An image of Colin Powell"). :param metadata_filtered: Metadata list used to map prompt index → entity name. metadata_filtered[i]['name'] corresponds to prompts[i]. :param lora_state: 'on' for unlearned model images, 'off' for baseline images. :param task: Task name, passed to get_target_preprocessed(). :param embed_image_fn: Injectable embedding function (image_path → [float]). Required — there is no default. Pass embed_image_with_dino (partially applied) or a test stub. :returns: [ { 'prompted_entity': str, # entity name (preprocessed) 'seed': int, 'prompt': str, 'embedding': List[float], # 384-dim CLS embedding }, ... ] :rtype: List of records .. py:function:: embed_forgetting_session_batched(dataset_folder: str, seeds: List[int], prompts: List[str], metadata_filtered: List[Dict[str, Any]], lora_state: Literal['on', 'off'], task: str, model: Any, transform: Any, device: str, batch_size: int = 32) -> List[Dict[str, Any]] Embed all images for one forgetting session using batched GPU inference. More efficient than embed_forgetting_session() for large image sets. Collects all (path, metadata) pairs first, then processes in batches via a simple loop, amortising Python overhead and maximising GPU utilisation. :param dataset_folder: Local directory containing the generated images. :param seeds: List of generation seeds used. :param prompts: Full prompt strings. :param metadata_filtered: Metadata list: metadata_filtered[i]['name'] → prompts[i]. :param lora_state: 'on' for unlearned model, 'off' for baseline. :param task: Task name, passed to get_target_preprocessed(). :param model: DINOv2 model (from load_dino_model()), on device, in eval mode. :param transform: torchvision transform pipeline (from load_dino_model()). :param device: Torch device string ('cuda' or 'cpu'). :param batch_size: Number of images per GPU forward pass (default 32). TODO: tune based on VRAM; 32 images × 224×224 ≈ 220MB VRAM. :returns: Same structure as embed_forgetting_session(). .. py:class:: ResultTemplate(/, **data: Any) Bases: :py:obj:`pydantic.BaseModel` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: recompute_if_exists :type: bool :value: False .. py:attribute:: save_outputs :type: bool :value: True .. py:attribute:: base_folder :type: str :value: 'assets' .. py:attribute:: remote_repository_name :type: str :value: 'LeonardoBenitez/VisionUnlearningEvaluationTestbeds' .. py:method:: _serialize_parameters() -> str :abstractmethod: .. py:method:: _get_data_path_remote() -> str .. py:method:: _get_data_path_local() -> str .. py:method:: _fig_to_bytes(fig: matplotlib.figure.Figure) -> bytes :classmethod: .. py:method:: _compute_from_scratch() -> dict | list :abstractmethod: .. py:method:: compute() -> dict .. py:class:: ResultTemplateMetricMetricAlignment(/, **data: Any) Bases: :py:obj:`ResultTemplate` Measures how strongly two *MetricInterferencePerEntity* metrics are correlated. **Arguments:** `m`, `t`, `u`, `m_e1`, `m_e2`. **Result:** Pearson p-value, Spearman p-value, Pearson correlation, scatter plot. **Interpretation:** quantitative; the higher the correlation, the lower the need to calculate both metrics for this specific choice of `m`, `t`, and `u`. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_entity_1 :type: vision_unlearning.benchmarks.I_care.configuration.type_me .. py:attribute:: interference_entity_2 :type: vision_unlearning.benchmarks.I_care.configuration.type_me .. py:class:: ResultTemplateMetricSimilarityAlignment(/, **data: Any) Bases: :py:obj:`ResultTemplate` To what degree similar *entities* interfere more with each other. Formalized in `ap:prediction`, which also proposes its natural expansion to a multivariable and non-linear predictive regression. **Arguments:** `m`, `t`, `u`, `m_p`, `s`. **Result:** Pearson p-value, Spearman p-value, Pearson correlation, scatter plot. **Interpretation:** quantitative; if this value is high, interference between two *entities* can be approximated by *similarity* (which is cheaper to compute for any new *entity*). Equivalently, the amount of "transmission wires" can be summarized by this single *similarity* function. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_pair :type: vision_unlearning.benchmarks.I_care.configuration.type_mp .. py:attribute:: similarity_metric :type: vision_unlearning.benchmarks.I_care.configuration.type_s .. py:attribute:: significance_threshold :type: float :value: 0.05 .. py:method:: _serialize_parameters() -> str .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (6, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch(exclude_diagonal: bool = True) -> dict .. py:class:: ResultTemplateMetricSimilarityAlignmentMulti(/, **data: Any) Bases: :py:obj:`ResultTemplate` Multi-input Single-output Regression Generalization of ResultTemplateMetricSimilarityAlignment (see also Appendix E, adapted from the multi-output setting). Also, the interpretability and feature engineering aspects are improved. --- We consider a fixed *model* \(m\), *task* \(t\), and *unlearning method* \(u\), which are omitted for brevity. The objective is to quantify whether interference between *entities* is aligned with their *similarity*, i.e., to what degree similar *entities* interfere more with each other. For every ordered pair of distinct *entities* \(e_i, e_j \in t\) with \(i eq j\), we observe several *SimilarityBetweenEntities* measures, indexed by superscripts \(\ell = 1, 2, \dots, |S|\), and a single *MetricInterferencePerEntityPair* target \(m_p(e_i,e_j)\). Each ordered pair \((e_i, e_j)\) is therefore treated as one data point with feature vector $$ \mathbf{X}_{ij} = ig( s^{(1)}(e_i, e_j), \dots, s^{(|S|)}(e_i, e_j) ig) $$ and scalar target $$ Y_{ij} = m_p(e_i, e_j). $$ The resulting dataset is $$ \mathcal{D} = \{ (\mathbf{X}_{ij}, Y_{ij}) \mid e_i, e_j \in t,\ i eq j \}. $$ From this dataset, a regression model can be estimated using standard regression procedures with appropriate validation. In the linear case, $$ Y_{ij} = eta_0 + \sum_{\ell=1}^{|S|} eta_{\ell} X^{(\ell)}_{ij} + arepsilon_{ij}. $$ Given a specific *entity* \(e_i\) whose removal is considered, similarities $$ X^{(\ell)}_{ij} = s^{(\ell)}(e_i, e_j) $$ can be computed for all remaining *entities* \(e_j \in t\). The fitted model then yields predictions $$ \hat{Y}_{ij} = f(\mathbf{X}_{ij}), $$ which approximate the expected interference on each receiver *entity*. Furthermore, the concept of *similarity* may also encode several forms of practical data engineering. For example, one may define: - a distinct *similarity* function for each *attribute*, or - a *similarity* function based only on the attributes of the emitter entity. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_pair :type: vision_unlearning.benchmarks.I_care.configuration.type_mp .. py:attribute:: similarity_metric_list :type: List[vision_unlearning.benchmarks.I_care.configuration.type_s] .. py:attribute:: significance_threshold :type: float :value: 0.05 .. py:attribute:: include_attribute_diff_similarity :type: bool :value: True .. py:attribute:: include_attribute_value_similarity :type: bool :value: True .. py:attribute:: regression_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_regression_algorithm :value: 'linear_regression' .. py:attribute:: random_state :type: int :value: 42 .. py:attribute:: test_size :type: float :value: 0.3 .. py:method:: _serialize_parameters() -> str .. py:method:: _get_partial_path_local() .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (6, 15), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch(exclude_diagonal: bool = True, entity_col: str = 'name') -> dict .. py:class:: ResultTemplateSignificantRelationshipNumerical(/, **data: Any) Bases: :py:obj:`ResultTemplate` Measures whether two numerical attributes are significantly correlated. Formalized in `ap:rt_relationship`. **Arguments:** `m`, `t`, `u`, `m_e`, `a`. **Result:** Pearson p-value, Spearman p-value, Pearson correlation, scatter plot. **Interpretation:** qualitative; the researcher should decide if it is ethical or desirable that this *attribute* propagates interferences. **Pearson test** Use when you want to measure a **linear** relationship. **Assumptions:** * Both variables are **continuous** * Relationship is **linear** * **Bivariate normality** (both jointly Gaussian) * **Homoscedasticity** (constant variance) * **No strong outliers** (very sensitive) **Detects:** linear correlation only **Fails when:** relationship is monotonic but non-linear, or heavy outliers exist **Spearman test** Use when you want to measure a **monotonic** relationship (not necessarily linear) or data is non-Gaussian. **Assumptions:** * Variables are at least **ordinal** * Relationship is **monotonic** (increasing or decreasing) * **No distributional assumptions** * **Robust to outliers** **Detects:** any monotonic trend (linear or curved) **Fails when:** relationship is non-monotonic (e.g., U-shaped) .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_entity :type: vision_unlearning.benchmarks.I_care.configuration.type_me .. py:attribute:: attribute :type: str .. py:attribute:: significance_threshold :type: float :value: 0.05 .. py:method:: _get_data_path_remote() -> str .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (6, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:class:: ResultTemplateSignificantRelationshipCategorical(/, **data: Any) Bases: :py:obj:`ResultTemplate` Statistical significance of the average `MetricInterferencePerEntity` across all *entities*, when grouped by each of its values. Formalized in `ap:rt_relationship`. **Arguments:** `m`, `t`, `u`, `m_e`, `a`, optional `filterAttributeValue`. **Result:** ANOVA p-value, Kruskal-Wallis p-value, average value of `m_e` grouped by each value of `a`, grouped boxplot. **Interpretation:** qualitative; similar to *SignificantRelationshipNumerical*. The optional argument *filterAttributeValue* restricts which emitter *entities* are included, allowing the analysis of interference flow distribution, such as whether politicians cause more interference to other politicians than artists cause to other artists. **ANOVA** Use when you want to test if **group means differ** across **3+ independent groups** under parametric assumptions. **Assumptions:** * Dependent variable is **continuous** * Groups are **independent** * **Normality** within each group * **Homoscedasticity** (equal variances) * No strong **outliers** **Hypothesis:** * H₀: all group means are equal * H₁: at least one mean differs **Detects:** differences in **means** **Fails when:** heavy skew, unequal variances, small n with non-Gaussian data **Kruskal-Wallis** Use when you want to test if **group distributions differ** without parametric assumptions. **Assumptions:** * Dependent variable is **ordinal or continuous** * Groups are **independent** * **Same shaped distributions** (only medians should differ for clean interpretation) * No normality or equal-variance requirement **Hypothesis:** * H₀: all group distributions are equal * H₁: at least one group differs **Detects:** differences in **medians / distributions** **Fails when:** distributions differ in shape (then result is ambiguous) .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_entity :type: vision_unlearning.benchmarks.I_care.configuration.type_me .. py:attribute:: attribute :type: str .. py:attribute:: attribute_value :type: Optional[str | int] :value: None .. py:attribute:: min_samples_per_category :type: int :value: 5 .. py:attribute:: significance_threshold :type: float :value: 0.05 .. py:method:: _get_data_path_remote() -> str .. py:method:: plot(data: dict, extra_title: str = '', figsize: Tuple[int, int] = (6, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:class:: ResultTemplateCountSignificantRelationship(/, **data: Any) Bases: :py:obj:`ResultTemplate` Number of significant relationships across all combinations of *attributes* and *MetricInterferencePerEntity*. **Arguments:** `m`, `t`, `u`, list of `m_e`, list of `a`. **Result:** integer, list of significances. **Interpretation:** quantitative; the lower the better. Since the attributes for which it is ethical to propagate interference are constant across all *models* and *methods*, a higher value directly implies a higher number of ethical violations, that is, a larger number of "transmission wires" in a given task effectively used by this *method* and *model*. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm_list :type: List[vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm] .. py:attribute:: interference_entity_list :type: List[vision_unlearning.benchmarks.I_care.configuration.type_me] .. py:attribute:: attribute_list :type: List[str] .. py:attribute:: top_n :type: int :value: 10 .. py:method:: _serialize_parameters() -> str .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (6, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:class:: ResultTemplateImplicitAssociationTest(/, **data: Any) Bases: :py:obj:`ResultTemplate` Measures how the strength of automatic associations `B` between two pairs of *entities* changes after unlearning. **Arguments:** `m`, `t`, `u`, `a_1`, `a_2`, `l`. **Result:** `|a| x |a|` real-valued tensor `ΔB`. **Interpretation:** qualitative; a human should decide whether it is ethical or desirable for the unlearning process to cause this change in implicit association between the chosen *attributes*. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: attribute_1 :type: str .. py:attribute:: attribute_2 :type: str .. py:attribute:: latent_embedding :type: vision_unlearning.benchmarks.I_care.configuration.type_l .. py:class:: ResultTemplateMinimumCutInterference(/, **data: Any) Bases: :py:obj:`ResultTemplate` Interprets a *task* as a directed weighted graph and computes the minimum cut separating two *entities* As a consequence of the max-flow min-cut theorem, it directly follows that the minimum cut is the smallest influence whose removal eliminates every directed influence path from $e_1$ to $e_2$. Based on this, we conjecture that if we need to unlearn $e_1$ while minimizing harm to $e_2$, then the ideal intervention in the unlearning process is to increase the preservation of the emitter-side nodes. More intuitively, we can think of this intervention as "blocking the interference path," as performed in electrical circuits to protect sensitive components (such as ground partitioning, shielding traces, among others. **Arguments:** $m$, $t$, $u$, $e_1$, $e_2$, $m_p$. **Result:** list of *entities* (corresponding to the emitter-side nodes). **Interpretation:** qualitative; small set of nodes through which most of the interference from $e_1$ propagates to $e_2$. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_pair :type: vision_unlearning.benchmarks.I_care.configuration.type_mp .. py:attribute:: entity_1 :type: str .. py:attribute:: entity_2 :type: str .. py:class:: ResultTemplateUnlearningVisualSummary(/, **data: Any) Bases: :py:obj:`ResultTemplate` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:class:: ResultTemplateInterferenceVisualSummary(/, **data: Any) Bases: :py:obj:`ResultTemplate` Compared generated images for 9 identities: target, 4 worst (excluding target), 4 best .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_pair :type: vision_unlearning.benchmarks.I_care.configuration.type_mp .. py:attribute:: entity :type: Optional[str] :value: None .. py:attribute:: entity_index :type: Optional[int] :value: None .. py:attribute:: seed :type: int :value: 42 .. py:attribute:: images_max_dim :type: int :value: 124 .. py:method:: _resolve_entity() Ensures both entity andentity_index are filled. Modifies in place At the end, both are set and consistent with each other .. py:method:: _serialize_parameters() -> str .. py:method:: plot(data: dict, figsize: Optional[Tuple[int, int]] = (18, 4), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() .. py:class:: ResultTemplateMatrix(/, **data: Any) Bases: :py:obj:`ResultTemplate` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: metric_key_name :type: str .. py:method:: plot_make_title(data: dict) -> str :classmethod: :abstractmethod: .. py:method:: plot(data: dict, figsize: Optional[Tuple[int, int]] = None, cmap: str = 'viridis', title: str = '', xlabel: str = 'Receiver entity', ylabel: str = 'Emitter entity', return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:class:: ResultTemplateInterferenceMatrix(/, **data: Any) Bases: :py:obj:`ResultTemplateMatrix` *MetricInterferencePerEntityPair* between each possible combination of two *entities* within a *task*. **Arguments:** `m`, `t`, `u`, `m_p`. **Result:** `|t| x |t|` real-valued tensor. **Interpretation:** qualitative; visual patterns may be spotted, especially when rearranging indices in a meaningful manner (for example, grouping professions together). Further quantitative values may be derived, such as the average value or the ratio between the diagonal-average value and the non-diagonal-average value. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: unlearning_algorithm :type: vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm .. py:attribute:: interference_pair :type: vision_unlearning.benchmarks.I_care.configuration.type_mp .. py:attribute:: metric_key_name :type: str :value: 'interference_pair' .. py:method:: _serialize_parameters() -> str .. py:method:: plot_make_title(data: dict) -> str :classmethod: .. py:method:: _compute_from_scratch() .. py:class:: ResultTemplateSimilarityMatrix(/, **data: Any) Bases: :py:obj:`ResultTemplateMatrix` *Similarities* between each possible combination of two *entities* within a *task*. * **Arguments**: $m, t, s$ * **Result**: $|t| imes |t|$ real-valued tensor * **Interpretation**: qualitative; visual patterns may be spotted, similarly to *InterferenceMatrix*. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'scenes' .. py:attribute:: similarity_metric :type: vision_unlearning.benchmarks.I_care.configuration.type_s :value: 'clip' .. py:attribute:: metric_key_name :type: str :value: 'similarity_metric' .. py:method:: _serialize_parameters() -> str .. py:method:: _get_partial_path_local() .. py:method:: plot_make_title(data: dict) -> str :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:class:: ResultTemplateMethodComparisonByMetricEntity(/, **data: Any) Bases: :py:obj:`ResultTemplate` Compares the distribution of one *MetricInterferencePerEntity* across multiple *unlearning methods*. * **Arguments**: m, t, me, list of u * **Result**: per-method mean, median, std, n, values; box plot * **Interpretation**: lower or higher depending on me direction. Use to rank methods by a single interference-per-entity metric. .. py:attribute:: model :type: vision_unlearning.benchmarks.I_care.configuration.type_model :value: 'sd1.4' .. py:attribute:: task :type: vision_unlearning.benchmarks.I_care.configuration.type_task :value: 'people' .. py:attribute:: interference_entity :type: vision_unlearning.benchmarks.I_care.configuration.type_me .. py:attribute:: unlearning_algorithm_list :type: List[vision_unlearning.benchmarks.I_care.configuration.type_unlearning_algorithm] .. py:method:: _serialize_parameters() -> str .. py:method:: plot(data: dict, figsize: Tuple[int, int] = (6, 5), return_fig: bool = False) -> Optional[Tuple[matplotlib.figure.Figure, matplotlib.pyplot.Axes]] :classmethod: .. py:method:: _compute_from_scratch() -> dict .. py:data:: rt_name_to_class .. py:data:: rt_name_to_params .. py:function:: jacc_metric_score(entity_1: str, entity_2: str, metadata_filtered: List[Dict[str, Any]], entity_col: str = 'name') -> float Jaccard similarity between two entities, based on their attributes. Each attribute (column) contributes between 0 and 1 to the similarity We do not know the types and ranges of the attributes beforehand. For each attribute, both values for the two entities must be non-NaN and of the same type, otherwise we ignore that attribute (contribution 0). The calculation for each attribute is as follows: * If the attribute is categorical (str or bool), the contribution is 1 if the two entities have the same value for that attribute, and 0 otherwise. * If the attribute is numerical, and both values are between 0 and 1, the contribution is 1 - abs(value_1 - value_2) * If the attribute is numerical, and both values are between 1 and 100, the contribution is 1 - abs(value_1 - value_2) / 100 * else, the contribution is 0 (we do not know how to handle it, so we ignore it) .. py:function:: display_interesting_interferences(metadata_filtered: List[Dict[str, Any]], interference_per_pair: Dict[str, Dict[str, float]], index: int, task: Literal['scenes', 'objects', 'breeds', 'people'], method: Literal['munba', 'uce', 'distil'], num_train_epochs: int, metric: str, is_worst_biggest: bool, seed: int = 42, save_path: Optional[str] = None) -> None Compared generated images for 9 identities: target, 4 worst (excluding target), 4 best @param metadata_filtered: should be appropriate for this task (this is not verified inside the function) @param interference_per_pair: should be appropriate for this task+index+method+num_train_epochs (this is not verified inside the function) @param index: identities the target The combination of task+index+method+num_train_epochs identifies a unique unlearned model .. py:function:: analyze_relationship_regression(df: pandas.DataFrame, x: str, y: str, expected_positive: bool = True, plot: bool = True) -> bool Test linear relationship between two numerical variables with significance test and direction check. Returns True only if: (1) the slope is statistically significant (p < 0.05) (2) the slope sign matches expectation. .. py:function:: analyze_relationship_category(df, metric: str, category: str, plot: bool = True) -> bool .. py:function:: analyze_relationship_numerical(df: pandas.DataFrame, attribute: str, metric: str, plot: bool = False, plot_only_significant: bool = False) -> bool Analyzes the relationship between a numerical attribute and a numerical metric @param df: interference_per_entity; assumes df[attribute] and df[metric] are numerical @param plot: whether to plot the results @param plot_only_significant: whether to plot only significant relationships; Only applies if plot=True @return: whether any significant relationship was found --- **Pearson test** Use when you want to measure a **linear** relationship. **Assumptions:** * Both variables are **continuous** * Relationship is **linear** * **Bivariate normality** (both jointly Gaussian) * **Homoscedasticity** (constant variance) * **No strong outliers** (very sensitive) **Detects:** linear correlation only **Fails when:** relationship is monotonic but non-linear, or heavy outliers exist --------- **Spearman test** Use when you want to measure a **monotonic** relationship (not necessarily linear) or data is non-Gaussian. **Assumptions:** * Variables are at least **ordinal** * Relationship is **monotonic** (increasing or decreasing) * **No distributional assumptions** * **Robust to outliers** **Detects:** any monotonic trend (linear or curved) **Fails when:** relationship is non-monotonic (e.g., U-shaped) .. py:function:: analyze_relationship_categorical(df: pandas.DataFrame, attribute: str, metric: str, plot: bool = False, plot_only_significant: bool = False, show_axhline: Optional[float] = None, min_samples_per_category: int = 5, extra_title: str = '') -> bool Analyzes the relationship between a categorical attribute and a numerical metric @param df: interference_per_entity; assumes df[attribute] is categorical and df[metric] is numerical @param plot: whether to plot the results @param plot_only_significant: whether to plot only significant relationships; Only applies if plot=True @param show_axhline: if provided, shows a horizontal line at this y-value; Only applies if plot=True @return: whether any significant relationship was found ------ **ANOVA (f_oneway)** Use when you want to test if **group means differ** across **3+ independent groups** under parametric assumptions. **Assumptions:** * Dependent variable is **continuous** * Groups are **independent** * **Normality** within each group * **Homoscedasticity** (equal variances) * No strong **outliers** **Hypothesis:** * H₀: all group means are equal * H₁: at least one mean differs **Detects:** differences in **means** **Fails when:** heavy skew, unequal variances, small n with non-Gaussian data ------ **Kruskal-Wallis (kruskal)** Use when you want to test if **group distributions differ** without parametric assumptions. **Assumptions:** * Dependent variable is **ordinal or continuous** * Groups are **independent** * **Same shaped distributions** (only medians should differ for clean interpretation) * No normality or equal-variance requirement **Hypothesis:** * H₀: all group distributions are equal * H₁: at least one group differs **Detects:** differences in **medians / distributions** **Fails when:** distributions differ in shape (then result is ambiguous) .. py:function:: analyze_correlation_between_pairwise_metrics(df1: pandas.DataFrame, df2: pandas.DataFrame, metric1_name: str, metric2_name: str, exclude_diagonal: bool = True, plot=True, plot_only_significant=True) -> bool df1 and df2 are square DataFrames; index and cols are the same within both and among both .. py:function:: check_eval_results(eval_results, name, threshold: float, operator: Literal['gt', 'lt']) -> float Check if the metric satisfy the EXPECTED threshold .. py:function:: get_metadata_filtered(task: Literal['scenes', 'objects', 'breeds', 'people'], base_folder: str = 'assets') -> List[Dict[str, Any]] .. py:function:: get_metadata_filtered_path(task: Literal['scenes', 'objects', 'breeds', 'people'], base_folder: str = 'assets') -> str .. py:function:: get_target_overwrite(task: Literal['scenes', 'objects', 'breeds', 'people'], method: Literal['munba', 'uce', 'distil'], target: str) -> Tuple[str, str] @return preprocessed target, target_overwrite .. py:function:: get_generated_dataset_file(lora_state: Literal['on', 'off'], seed: int, prompt: str) -> str .. py:function:: huggingface_dataset_file_exists(dataset_repository: str, dataset_path: str, token: Optional[str]) -> bool Checks if a specific file exists in a Hugging Face dataset repository. :param dataset_repository: e.g. "username/dataset_name" :param dataset_path: full path in repo (e.g. "config/file.jsonl") :param token: HF token (can be None for public repos) :return: True if file exists, False otherwise Efficiently checks if a file exists in a Hugging Face dataset repo without listing the entire repository. Could be done more efficiently if we use a new version of the lib, see https://chatgpt.com/share/69edd525-d008-832d-8a0c-ec4560a4fe3b .. py:function:: huggingface_dataset_file_download(folder_datasets: str, dataset_repository: str, file_path: str, token: str, folder_cache: str = '/tmp/huggingface_cache') -> None Download a single file from a dataset in Hugging Face Hub. :param folder_datasets: Local directory where datasets are stored. :param dataset_repository: Hugging Face dataset repository ID :param file_path: Full path of the file within the repository (e.g., "config/data.jsonl") :param token: Hugging Face authentication token :param folder_cache: Cache directory for downloads The file will be saved at os.path.join(folder_datasets, file_path) .. py:function:: huggingface_dataset_upload(folder_datasets: str, dataset_repository: str, dataset_config: str, token: str) Supposes that a folder `dataset_config` exists in `folder_datasets`, and that it contains the dataset files .. py:function:: huggingface_dataset_file_upload(file_path: str, dataset_repository: str, dataset_path: str, token: str) Upload a single file to a specific dataset config in Hugging Face Hub. @param dataset_path: full name of the file in the repository, including the config folder (e.g., "my_config/my_file.jsonl") .. py:function:: huggingface_dataset_download(folder_datasets: str, dataset_repository: str, dataset_config: str, token: str, clean: bool = False, folder_cache: str = '/tmp/huggingface_cache', clean_cache: bool = False) @param clean: If True, the folder will be deleted before downloading