vision_unlearning.unlearner =========================== .. py:module:: vision_unlearning.unlearner Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/vision_unlearning/unlearner/base/index /autoapi/vision_unlearning/unlearner/fade/index /autoapi/vision_unlearning/unlearner/lora/index /autoapi/vision_unlearning/unlearner/uce_sd_erase/index Attributes ---------- .. autoapisummary:: vision_unlearning.unlearner.logger vision_unlearning.unlearner.logger vision_unlearning.unlearner.logger Classes ------- .. autoapisummary:: vision_unlearning.unlearner.Unlearner vision_unlearning.unlearner.Unlearner vision_unlearning.unlearner.MetricImageTextSimilarity vision_unlearning.unlearner.EvaluatorTextToImage vision_unlearning.unlearner.GradientWeightingMethod vision_unlearning.unlearner.GradientWeightingMethodSimple vision_unlearning.unlearner.UnlearnerLora vision_unlearning.unlearner.UnlearnerLoraDirect vision_unlearning.unlearner.Unlearner vision_unlearning.unlearner.EvaluatorTextToImage vision_unlearning.unlearner.MetricImageTextSimilarity vision_unlearning.unlearner.ConceptType vision_unlearning.unlearner.UCE Functions --------- .. autoapisummary:: vision_unlearning.unlearner.get_logger vision_unlearning.unlearner.plot_gradient_conflict_hist vision_unlearning.unlearner.log_validation vision_unlearning.unlearner.save_model_card vision_unlearning.unlearner.unwrap_model vision_unlearning.unlearner.preprocess_train vision_unlearning.unlearner.collate_fn vision_unlearning.unlearner.unlearn_lora vision_unlearning.unlearner.save_model_card Package Contents ---------------- .. py:function:: get_logger(name: str, level=logging.INFO) -> logging.Logger .. py:data:: logger .. py:class:: Unlearner(/, **data: Any) Bases: :py:obj:`pydantic.BaseModel`, :py:obj:`abc.ABC` performs the actual finetuning One unlearner may have variations/parametrizations that correspond to different unlearning algorithms/methods .. py:method:: train() -> List[huggingface_hub.repocard_data.EvalResult] :abstractmethod: .. py:class:: Unlearner(/, **data: Any) Bases: :py:obj:`pydantic.BaseModel`, :py:obj:`abc.ABC` performs the actual finetuning One unlearner may have variations/parametrizations that correspond to different unlearning algorithms/methods .. py:method:: train() -> List[huggingface_hub.repocard_data.EvalResult] :abstractmethod: .. py:data:: logger .. py:class:: MetricImageTextSimilarity(/, **data: Any) Bases: :py:obj:`vision_unlearning.metrics.base.Metric` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: metrics :type: List[Literal['clip']] .. py:attribute:: _clip_metric :type: Optional[torchmetrics.multimodal.clip_score.CLIPScore] :value: None .. py:method:: model_post_init(__context: Optional[dict] = None) -> None Override this method to perform additional initialization after `__init__` and `model_construct`. This is useful if you want to do some validation that requires the entire model to be initialized. .. py:method:: _load_image(image: Union[PIL.Image.Image, numpy.ndarray, str]) -> torch.Tensor .. py:method:: score(image: Union[PIL.Image.Image, numpy.ndarray, str], text: str) -> Dict[str, float] .. py:method:: score_batch(images: List[Union[PIL.Image.Image, numpy.ndarray, str]], texts: List[str]) -> List[Dict[str, float]] Warning: this function don't improve performance. The underlying libraries still work serially. Returns per-pair results in the same order. .. py:method:: score_batch_same_text(images: List[Union[PIL.Image.Image, numpy.ndarray, str]], text: str) -> List[Dict[str, float]] Batch CLIP scoring when all images share the same text prompt. This is meaningfully faster than calling score() N times because the CLIP text encoder runs once for the shared text. Images are processed individually through the CLIP image processor (as in the serial path) but the text encoder forward pass is done only once. Uses _clip_score_update from torchmetrics (private API, tested against torchmetrics 1.x) which returns per-pair scores as a 1-D tensor. The result is numerically equivalent to calling score() N times (max diff < 2e-5 on 512x512 SD1.4 images). NOTE: _clip_score_update is a private torchmetrics symbol — if a future torchmetrics version removes it, fall back to the serial score() loop. :param images: List of N images (PIL Image, np.ndarray, or file path). :param text: Single text caption applied to all images. :returns: float}, one per image in input order. :rtype: List of N dicts {'clip' .. py:class:: EvaluatorTextToImage(/, **data: Any) Bases: :py:obj:`pydantic.BaseModel` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: model_config Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict]. .. py:attribute:: pipeline_original :type: Optional[diffusers.StableDiffusionPipeline] .. py:attribute:: pipeline_learned :type: Optional[diffusers.StableDiffusionPipeline] .. py:attribute:: pipeline_unlearned :type: diffusers.StableDiffusionPipeline .. py:attribute:: prompts_forget :type: List[str] .. py:attribute:: prompts_retain :type: List[str] .. py:attribute:: metric_clip :type: vision_unlearning.metrics.MetricImageTextSimilarity .. py:attribute:: compute_runtimes :type: bool :value: True .. py:attribute:: plot_show :type: bool :value: True .. py:method:: evaluate() -> Tuple[List[huggingface_hub.repocard_data.EvalResult], Dict[str, PIL.Image.Image]] .. py:function:: plot_gradient_conflict_hist(similarities: List[float], title: str, color: str) -> PIL.Image.Image .. py:function:: log_validation(pipeline, accelerator, epoch, num_validation_images, validation_prompt, seed, is_final_validation=False) -> Dict[str, PIL.Image.Image] Adapted from The HuggingFace Inc. team. All rights reserved. Licensed under the Apache License, Version 2.0. Source: https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py .. py:function:: save_model_card(repo_id: str, base_model: str, dataset_forget_name: str, dataset_retain_name: str, repo_folder: str, images: Dict[str, PIL.Image.Image] = {}, eval_results: List[huggingface_hub.repocard_data.EvalResult] = [], tags: List[str] = [], hyperparameters: dict = {}, similarities_gr: List[float] = [], similarities_gf: List[float] = []) The resulting file looks like this: https://github.com/huggingface/hub-docs/blob/main/modelcard.md This looks hugginface-specific, so you may think it should be in `integrations/huggingface.py`, but it is actually a generic Readme saving Adapted from The HuggingFace Inc. team. All rights reserved. Licensed under the Apache License, Version 2.0. Source: https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py .. py:function:: unwrap_model(model, accelerator) Adapted from The HuggingFace Inc. team. All rights reserved. Licensed under the Apache License, Version 2.0. Source: https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py .. py:function:: preprocess_train(examples, tokenizer, caption_column, image_column, train_transforms, overwrite_column: Optional[str] = None, concept_overwrite: Optional[str] = None) Adapted from The HuggingFace Inc. team. All rights reserved. Licensed under the Apache License, Version 2.0. Source: https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py concept_overwrite: concept to be used for overwriting, described as an textual string (used to modify the prompt). TODO: this handling of concept_overwrite is weird... I wish this were somewhat more structured/organized/clear. For example, the overwriting string may need a more complex prompt than just "an image of f{concept_overwrite}", or with a different article .. py:function:: collate_fn(examples) Adapted from The HuggingFace Inc. team. All rights reserved. Licensed under the Apache License, Version 2.0. Source: https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py .. py:class:: GradientWeightingMethod(/, **data: Any) Bases: :py:obj:`pydantic.BaseModel`, :py:obj:`abc.ABC` Method used to conciliate/harmonize/combine/weight the gradients of the different tasks Inspired by @article{navon2022multi, title={Multi-Task Learning as a Bargaining Game}, author={Navon, Aviv and Shamsian, Aviv and Achituve, Idan and Maron, Haggai and Kawaguchi, Kenji and Chechik, Gal and Fetaya, Ethan}, journal={arXiv preprint arXiv:2202.01017}, year={2022} } Source: https://github.com/AvivNavon/nash-mtl/blob/main/methods/weight_methods.py .. py:method:: weight_grads(grads_forget: List[torch.Tensor], grads_retain: List[torch.Tensor], accelerator) -> torch.Tensor :abstractmethod: @return scaled_grad .. py:class:: GradientWeightingMethodSimple(/, **data: Any) Bases: :py:obj:`GradientWeightingMethod` Fixed weights for each component .. py:attribute:: forget_weight :type: float :value: 1.0 .. py:attribute:: retain_weight :type: float :value: 1.0 .. py:method:: weight_grads(grads_forget: List[torch.Tensor], grads_retain: List[torch.Tensor], accelerator) -> torch.Tensor @return scaled_grad .. py:function:: unlearn_lora(model_original_id: str, model_lora_id: str, device: str, weight_name: str = 'pytorch_lora_weights.safetensors', requires_inversion: bool = True, return_original: bool = True, return_learned: bool = True) -> Tuple[Optional[diffusers.StableDiffusionPipeline], Optional[diffusers.StableDiffusionPipeline], diffusers.StableDiffusionPipeline] id can be both a local dir or a huggingface model id return pipeline_original, pipeline_learned, pipeline_unlearned Inspired by @inproceedings{zhang2023composing, title={Composing Parameter-Efficient Modules with Arithmetic Operations}, author={Zhang, Jinghan and Chen, Shiqi and Liu, Junteng and He, Junxian}, booktitle={Advances in Neural Information Processing Systems}, year={2023} } Source: https://github.com/hkust-nlp/PEM_composition/tree/main/exps/composition_for_unlearning .. py:class:: UnlearnerLora(/, **data: Any) Bases: :py:obj:`vision_unlearning.unlearner.base.Unlearner` Fine-tuning script for Stable Diffusion for text2image with support for LoRA. Strongly based on the huggingface example (see credits in the end) Adapted from The HuggingFace Inc. team. All rights reserved. Licensed under the Apache License, Version 2.0. Source: https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py .. py:attribute:: lora_r :type: int :value: None .. py:attribute:: lora_alpha :type: int :value: None .. py:attribute:: target_modules :type: List[str] :value: None .. py:attribute:: is_lora_negated :type: bool :value: None .. py:attribute:: seed :type: int :value: None .. py:attribute:: model_name_or_path :type: str :value: None .. py:attribute:: revision :type: Optional[str] :value: None .. py:attribute:: variant :type: Optional[str] :value: None .. py:attribute:: dataset_forget_name :type: str :value: None .. py:attribute:: dataset_retain_name :type: str :value: None .. py:attribute:: dataset_forget_config_name :type: Optional[str] :value: None .. py:attribute:: dataset_retain_config_name :type: Optional[str] :value: None .. py:attribute:: image_column :type: str :value: None .. py:attribute:: caption_column :type: str :value: None .. py:attribute:: validation_prompt :type: Optional[str] :value: None .. py:attribute:: num_validation_images :type: int :value: None .. py:attribute:: validation_epochs :type: int :value: None .. py:attribute:: resolution :type: int :value: None .. py:attribute:: center_crop :type: bool :value: None .. py:attribute:: random_flip :type: bool :value: None .. py:attribute:: max_train_samples :type: Optional[int] :value: None .. py:attribute:: dataloader_num_workers :type: int :value: None .. py:attribute:: final_eval_prompts_forget :type: str | List[str] :value: None .. py:attribute:: final_eval_prompts_retain :type: str | List[str] :value: None .. py:attribute:: prediction_type :type: Optional[str] :value: None .. py:attribute:: per_device_train_batch_size :type: int :value: None .. py:attribute:: gradient_accumulation_steps :type: int :value: None .. py:attribute:: num_train_epochs :type: int :value: None .. py:attribute:: learning_rate :type: float :value: None .. py:attribute:: lr_scheduler_type :type: str :value: None .. py:attribute:: should_log :type: bool :value: None .. py:attribute:: local_rank :type: int :value: None .. py:attribute:: device :type: str :value: None .. py:attribute:: n_gpu :type: int :value: None .. py:attribute:: output_dir :type: str :value: None .. py:attribute:: cache_dir :type: Optional[str] :value: None .. py:attribute:: hub_token :type: Optional[str] :value: None .. py:attribute:: hub_model_id :type: Optional[str] :value: None .. py:attribute:: logging_dir :type: str :value: None .. py:attribute:: logging_steps :type: int :value: None .. py:attribute:: save_strategy :type: str :value: None .. py:attribute:: save_total_limit :type: int :value: None .. py:attribute:: gradient_checkpointing :type: bool :value: None .. py:attribute:: enable_xformers_memory_efficient_attention :type: bool :value: None .. py:attribute:: mixed_precision :type: Optional[str] :value: None .. py:attribute:: allow_tf32 :type: bool :value: None .. py:attribute:: use_8bit_adam :type: bool :value: None .. py:attribute:: report_to :type: str :value: None .. py:attribute:: gradient_weighting_method :type: vision_unlearning.utils.gradient_weighting.GradientWeightingMethod :value: None .. py:attribute:: compute_gradient_conflict :type: bool :value: None .. py:attribute:: compute_runtimes :type: bool :value: None .. py:attribute:: compute_memory :type: bool :value: None .. py:attribute:: max_train_steps :type: Optional[int] :value: None .. py:attribute:: lr_warmup_steps :type: int :value: None .. py:attribute:: adam_beta1 :type: float :value: None .. py:attribute:: adam_beta2 :type: float :value: None .. py:attribute:: adam_weight_decay :type: float :value: None .. py:attribute:: adam_epsilon :type: float :value: None .. py:attribute:: max_grad_norm :type: float :value: None .. py:attribute:: checkpointing_steps :type: int :value: None .. py:attribute:: checkpoints_total_limit :type: Optional[int] :value: None .. py:attribute:: resume_from_checkpoint :type: Optional[str] :value: None .. py:attribute:: noise_offset :type: float :value: None .. py:attribute:: _accelerator :type: Optional[accelerate.Accelerator] :value: None .. py:attribute:: _output_dir_checkpoints :type: Optional[str] :value: None .. py:attribute:: _output_dir_lora :type: Optional[str] :value: None .. py:attribute:: _lora_weight_name :type: str :value: 'pytorch_lora_weights.safetensors' .. py:attribute:: _images :type: Dict[str, PIL.Image.Image] .. py:attribute:: _weight_dtype :type: Any :value: Ellipsis .. py:attribute:: _similarities_gr :type: List[float] :value: [] .. py:attribute:: _similarities_gf :type: List[float] :value: [] .. py:attribute:: _noise_scheduler :type: Any :value: None .. py:attribute:: _tokenizer :type: Any :value: None .. py:attribute:: _text_encoder :type: Any :value: None .. py:attribute:: _vae :type: Any :value: None .. py:attribute:: _unet :type: Optional[diffusers.models.unets.unet_2d_condition.UNet2DConditionModel] :value: None .. py:attribute:: _train_forget_dataloader :type: Optional[torch.utils.data.DataLoader] :value: None .. py:attribute:: _train_retain_dataloader :type: Optional[torch.utils.data.DataLoader] :value: None .. py:attribute:: _optimizer :type: Any :value: None .. py:attribute:: _lr_scheduler :type: Any :value: None .. py:attribute:: _lora_layers :type: Any :value: None .. py:attribute:: _peak_mem :type: int :value: 0 .. py:method:: model_post_init(__context: Optional[dict] = None) -> None Override this method to perform additional initialization after `__init__` and `model_construct`. This is useful if you want to do some validation that requires the entire model to be initialized. .. py:method:: _pre_checks() -> None .. py:method:: _get_lora_config() -> peft.LoraConfig .. py:method:: _get_accelerator() .. py:method:: _hook_after_lora_init() .. py:method:: _hook_before_load_model() .. py:method:: _save_lora_layers() Side-effects: modifies self._unet in-place (casts to float32), saves two directories self._output_dir_super and self._output_dir_sub .. py:method:: train() .. py:method:: _prepare_dataloaders() -> Tuple[torch.utils.data.DataLoader, torch.utils.data.DataLoader] :abstractmethod: .. py:method:: _train_one_batch(batch_forget, batch_retain) :abstractmethod: .. py:class:: UnlearnerLoraDirect(/, **data: Any) Bases: :py:obj:`UnlearnerLora` Straight-forward finetuning .. py:method:: _prepare_dataloaders() -> Tuple[torch.utils.data.DataLoader, torch.utils.data.DataLoader] .. py:method:: _train_one_batch(batch_forget, batch_retain) .. py:class:: Unlearner(/, **data: Any) Bases: :py:obj:`pydantic.BaseModel`, :py:obj:`abc.ABC` performs the actual finetuning One unlearner may have variations/parametrizations that correspond to different unlearning algorithms/methods .. py:method:: train() -> List[huggingface_hub.repocard_data.EvalResult] :abstractmethod: .. py:data:: logger .. py:class:: EvaluatorTextToImage(/, **data: Any) Bases: :py:obj:`pydantic.BaseModel` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: model_config Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict]. .. py:attribute:: pipeline_original :type: Optional[diffusers.StableDiffusionPipeline] .. py:attribute:: pipeline_learned :type: Optional[diffusers.StableDiffusionPipeline] .. py:attribute:: pipeline_unlearned :type: diffusers.StableDiffusionPipeline .. py:attribute:: prompts_forget :type: List[str] .. py:attribute:: prompts_retain :type: List[str] .. py:attribute:: metric_clip :type: vision_unlearning.metrics.MetricImageTextSimilarity .. py:attribute:: compute_runtimes :type: bool :value: True .. py:attribute:: plot_show :type: bool :value: True .. py:method:: evaluate() -> Tuple[List[huggingface_hub.repocard_data.EvalResult], Dict[str, PIL.Image.Image]] .. py:class:: MetricImageTextSimilarity(/, **data: Any) Bases: :py:obj:`vision_unlearning.metrics.base.Metric` !!! abstract "Usage Documentation" [Models](../concepts/models.md) A base class for creating Pydantic models. .. attribute:: __class_vars__ The names of the class variables defined on the model. .. attribute:: __private_attributes__ Metadata about the private attributes of the model. .. attribute:: __signature__ The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. attribute:: __pydantic_complete__ Whether model building is completed, or if there are still undefined fields. .. attribute:: __pydantic_core_schema__ The core schema of the model. .. attribute:: __pydantic_custom_init__ Whether the model has a custom `__init__` function. .. attribute:: __pydantic_decorators__ Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. attribute:: __pydantic_generic_metadata__ A dictionary containing metadata about generic Pydantic models. The `origin` and `args` items map to the [`__origin__`][genericalias.__origin__] and [`__args__`][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the `parameter` item maps to the `__parameter__` attribute of generic classes. .. attribute:: __pydantic_parent_namespace__ Parent namespace of the model, used for automatic rebuilding of models. .. attribute:: __pydantic_post_init__ The name of the post-init method for the model, if defined. .. attribute:: __pydantic_root_model__ Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. attribute:: __pydantic_serializer__ The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. attribute:: __pydantic_validator__ The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. attribute:: __pydantic_fields__ A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects. .. attribute:: __pydantic_computed_fields__ A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects. .. attribute:: __pydantic_extra__ A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. attribute:: __pydantic_fields_set__ The names of fields explicitly set during instantiation. .. attribute:: __pydantic_private__ Values of private attributes set on the model instance. .. py:attribute:: metrics :type: List[Literal['clip']] .. py:attribute:: _clip_metric :type: Optional[torchmetrics.multimodal.clip_score.CLIPScore] :value: None .. py:method:: model_post_init(__context: Optional[dict] = None) -> None Override this method to perform additional initialization after `__init__` and `model_construct`. This is useful if you want to do some validation that requires the entire model to be initialized. .. py:method:: _load_image(image: Union[PIL.Image.Image, numpy.ndarray, str]) -> torch.Tensor .. py:method:: score(image: Union[PIL.Image.Image, numpy.ndarray, str], text: str) -> Dict[str, float] .. py:method:: score_batch(images: List[Union[PIL.Image.Image, numpy.ndarray, str]], texts: List[str]) -> List[Dict[str, float]] Warning: this function don't improve performance. The underlying libraries still work serially. Returns per-pair results in the same order. .. py:method:: score_batch_same_text(images: List[Union[PIL.Image.Image, numpy.ndarray, str]], text: str) -> List[Dict[str, float]] Batch CLIP scoring when all images share the same text prompt. This is meaningfully faster than calling score() N times because the CLIP text encoder runs once for the shared text. Images are processed individually through the CLIP image processor (as in the serial path) but the text encoder forward pass is done only once. Uses _clip_score_update from torchmetrics (private API, tested against torchmetrics 1.x) which returns per-pair scores as a 1-D tensor. The result is numerically equivalent to calling score() N times (max diff < 2e-5 on 512x512 SD1.4 images). NOTE: _clip_score_update is a private torchmetrics symbol — if a future torchmetrics version removes it, fall back to the serial score() loop. :param images: List of N images (PIL Image, np.ndarray, or file path). :param text: Single text caption applied to all images. :returns: float}, one per image in input order. :rtype: List of N dicts {'clip' .. py:function:: save_model_card(repo_id: str, base_model: str, dataset_forget_name: str, dataset_retain_name: str, repo_folder: str, images: Dict[str, PIL.Image.Image] = {}, eval_results: List[huggingface_hub.repocard_data.EvalResult] = [], tags: List[str] = [], hyperparameters: dict = {}, similarities_gr: List[float] = [], similarities_gf: List[float] = []) The resulting file looks like this: https://github.com/huggingface/hub-docs/blob/main/modelcard.md This looks hugginface-specific, so you may think it should be in `integrations/huggingface.py`, but it is actually a generic Readme saving Adapted from The HuggingFace Inc. team. All rights reserved. Licensed under the Apache License, Version 2.0. Source: https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py .. py:class:: ConceptType Bases: :py:obj:`str`, :py:obj:`enum.Enum` Enum representing the type of concept to unlearn. .. py:attribute:: Object :value: 'object' .. py:attribute:: Art :value: 'art' .. py:class:: UCE(**data: Any) Bases: :py:obj:`vision_unlearning.unlearner.base.Unlearner` Unified Concept Editing for unlearning in Stable Diffusion models. Adapted from: GitHub: https://github.com/rohitgandikota/unified-concept-editing Arxiv: https://arxiv.org/pdf/2308.14761.pdf Gandikota, R., Orgad, H., Belinkov, Y., Materzyńska, J., & Bau, D. (2024). Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 5111-5120). This unlearner do not use LoRA, and do not perform any fine-tuning (instead, it performs a closed-form weight update). .. py:attribute:: pretrained_model_name_or_path :type: str :value: None .. py:attribute:: erase_scale :type: float :value: None .. py:attribute:: preserve_scale :type: float :value: None .. py:attribute:: lamb :type: float :value: None .. py:attribute:: save_entire_model :type: bool :value: None .. py:attribute:: edit_concepts :type: Optional[str] :value: None .. py:attribute:: guide_concepts :type: Optional[str] :value: None .. py:attribute:: preserve_concepts :type: Optional[str] :value: None .. py:attribute:: concept_type :type: ConceptType :value: None .. py:attribute:: expand_prompts :type: bool :value: True .. py:attribute:: final_eval_prompts_forget :type: str | List[str] :value: None .. py:attribute:: final_eval_prompts_retain :type: str | List[str] :value: None .. py:attribute:: output_dir :type: str :value: None .. py:attribute:: device :type: str :value: 'cuda:0' .. py:attribute:: compute_runtimes :type: bool :value: None .. py:attribute:: hub_model_id :type: Optional[str] :value: None .. py:method:: _collect_text_embeddings(pipe: Any, concepts: list[str], device: str, torch_dtype: torch.dtype) -> dict[str, torch.Tensor] Return dict {concept: last_token_embedding}. .. py:method:: _collect_guide_outputs(concepts: list[str], embeds: dict[str, torch.Tensor], modules: list[torch.nn.Module]) -> dict[str, list[torch.Tensor]] Collect cross-attention outputs for guide/preserve concepts. .. py:method:: _update_weights(original_modules: list[torch.nn.Module], erase_embeds: dict[str, torch.Tensor], guide_outputs: dict[str, list[torch.Tensor]], edit_concepts: list[str], guide_concepts: list[str], preserve_concepts: list[str], erase_scale: float, preserve_scale: float, lamb: float, device: str, torch_dtype: torch.dtype) -> list[torch.nn.Module] Apply the UCE weight update to each module and return new modules. .. py:method:: _save_uce_weights(uce_modules: list[torch.nn.Module], uce_module_names: list[str]) -> None Save updated module weights to a safetensors file. .. py:method:: train() -> List[huggingface_hub.repocard_data.EvalResult] Main UCE training and concept erasure logic. .. py:method:: get_pipeline_from_modified_weights(pretrained_model_name_or_path: str, device: str | torch.device, output_dir: str) -> diffusers.DiffusionPipeline :staticmethod: .. py:method:: evaluate() -> Tuple[List[huggingface_hub.repocard_data.EvalResult], Dict[str, PIL.Image.Image]]