vision_unlearning.evaluator =========================== .. py:module:: vision_unlearning.evaluator Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/vision_unlearning/evaluator/class_unlearning/index /autoapi/vision_unlearning/evaluator/text_to_image/index Attributes ---------- .. autoapisummary:: vision_unlearning.evaluator.logger Classes ------- .. autoapisummary:: vision_unlearning.evaluator.MetricImageTextSimilarity vision_unlearning.evaluator.MetricPaintingStyle vision_unlearning.evaluator.EvaluatorTextToImage Functions --------- .. autoapisummary:: vision_unlearning.evaluator.get_logger vision_unlearning.evaluator.tensorboard_log_image vision_unlearning.evaluator.evaluate_painting_style vision_unlearning.evaluator.log_validation vision_unlearning.evaluator.plot_gradient_conflict_hist vision_unlearning.evaluator.average_metrics vision_unlearning.evaluator._convert_mean_to_std vision_unlearning.evaluator.format_metrics_as_markdown Package Contents ---------------- .. py:class:: MetricImageTextSimilarity(/, **data: Any) Bases: :py:obj:`vision_unlearning.metrics.base.Metric` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: metrics :type: List[Literal['clip']] .. py:attribute:: _clip_metric :type: Optional[torchmetrics.multimodal.clip_score.CLIPScore] :value: None .. py:method:: model_post_init(__context: Optional[dict] = None) -> None Override this method to perform additional initialization after `__init__` and `model_construct`. This is useful if you want to do some validation that requires the entire model to be initialized. .. py:method:: _load_image(image: Union[PIL.Image.Image, numpy.ndarray, str]) -> torch.Tensor .. py:method:: score(image: Union[PIL.Image.Image, numpy.ndarray, str], text: str) -> Dict[str, float] .. py:method:: score_batch(images: List[Union[PIL.Image.Image, numpy.ndarray, str]], texts: List[str]) -> List[Dict[str, float]] Warning: this function don't improve performance. The underlying libraries still work serially. Returns per-pair results in the same order. .. py:method:: score_batch_same_text(images: List[Union[PIL.Image.Image, numpy.ndarray, str]], text: str) -> List[Dict[str, float]] Batch CLIP scoring when all images share the same text prompt. This is meaningfully faster than calling score() N times because the CLIP text encoder runs once for the shared text. Images are processed individually through the CLIP image processor (as in the serial path) but the text encoder forward pass is done only once. Uses _clip_score_update from torchmetrics (private API, tested against torchmetrics 1.x) which returns per-pair scores as a 1-D tensor. The result is numerically equivalent to calling score() N times (max diff < 2e-5 on 512x512 SD1.4 images). NOTE: _clip_score_update is a private torchmetrics symbol — if a future torchmetrics version removes it, fall back to the serial score() loop. :param images: List of N images (PIL Image, np.ndarray, or file path). :param text: Single text caption applied to all images. :returns: float}, one per image in input order. :rtype: List of N dicts {'clip' .. py:class:: MetricPaintingStyle(/, **data: Any) Bases: :py:obj:`MetricImage` Based only on the image itself e.g., image quality, painting style .. py:attribute:: metrics :type: List[Literal['is_desired_style', 'desired_style_confidence']] :value: [] .. py:attribute:: desired_style :type: str .. py:attribute:: top_k :type: int :value: 5 .. py:attribute:: model_path :type: str .. py:attribute:: device :type: Optional[torch.device | str | int] :value: 'cuda' .. py:attribute:: _pipeline :type: Optional[transformers.pipelines.image_classification.ImageClassificationPipeline] :value: None .. py:method:: model_post_init(__context: Optional[dict] = None) -> None Override this method to perform additional initialization after `__init__` and `model_construct`. This is useful if you want to do some validation that requires the entire model to be initialized. .. py:method:: score(image: PIL.Image.Image) -> Dict[str, bool | float] .. py:function:: get_logger(name: str, level=logging.INFO) -> logging.Logger .. py:function:: tensorboard_log_image(tracker, phase_name, prompt, epoch, images) .. py:data:: logger .. py:class:: EvaluatorTextToImage(/, **data: Any) Bases: :py:obj:`pydantic.BaseModel` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: model_config Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict]. .. py:attribute:: pipeline_original :type: Optional[diffusers.StableDiffusionPipeline] .. py:attribute:: pipeline_learned :type: Optional[diffusers.StableDiffusionPipeline] .. py:attribute:: pipeline_unlearned :type: diffusers.StableDiffusionPipeline .. py:attribute:: prompts_forget :type: List[str] .. py:attribute:: prompts_retain :type: List[str] .. py:attribute:: metric_clip :type: vision_unlearning.metrics.MetricImageTextSimilarity .. py:attribute:: compute_runtimes :type: bool :value: True .. py:attribute:: plot_show :type: bool :value: True .. py:method:: evaluate() -> Tuple[List[huggingface_hub.repocard_data.EvalResult], Dict[str, PIL.Image.Image]] .. py:function:: evaluate_painting_style(metadata: List[Dict[str, str]], metric_painting_style: vision_unlearning.metrics.MetricPaintingStyle, dataset_path: str, device: str) -> dict @param metadata: list of dictionaries with keys "file_name" and "text"; follows this schema: Follows this schema: https://huggingface.co/docs/datasets/v2.4.0/en/image_load#image-captioning @return metrics (as float, not yet as EvalResult) Compute metrics from already generated images .. py:function:: log_validation(pipeline, accelerator, epoch, num_validation_images, validation_prompt, seed, is_final_validation=False) -> Dict[str, PIL.Image.Image] Adapted from The HuggingFace Inc. team. All rights reserved. Licensed under the Apache License, Version 2.0. Source: https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py .. py:function:: plot_gradient_conflict_hist(similarities: List[float], title: str, color: str) -> PIL.Image.Image .. py:function:: average_metrics(name_to_value_all: List[Dict[str, float | int | bool]]) -> Dict[str, float] .. py:function:: _convert_mean_to_std(name: str) -> str .. py:function:: format_metrics_as_markdown(name_to_value: Dict[str, Union[float, int, bool]], name_to_value_all: Optional[Dict[str, Dict[str, Union[float, int, bool]]]] = None) -> str Formats metrics as a markdown table. Can display either just overall metrics or include per-class metrics as additional columns. :param name_to_value: Dictionary of overall/average metrics :param name_to_value_all: Optional dictionary of per-class metrics :returns: Markdown formatted table :rtype: str