From e44ded48c54b15ab3300d718469647ec596c1c4d Mon Sep 17 00:00:00 2001 From: Aravind N Date: Fri, 12 Dec 2025 17:28:42 -0500 Subject: [PATCH] Added preprocessing steps --- .../README.md | 44 +- .../configs/config_dvm_TIP.yaml | 2 +- .../data_preprocessing.ipynb | 1059 +++++++++++++++++ 3 files changed, 1087 insertions(+), 18 deletions(-) create mode 100644 implementations/multimodal_representation_learning/data_preprocessing.ipynb diff --git a/implementations/multimodal_representation_learning/README.md b/implementations/multimodal_representation_learning/README.md index 89c1c85..ef59883 100644 --- a/implementations/multimodal_representation_learning/README.md +++ b/implementations/multimodal_representation_learning/README.md @@ -17,7 +17,17 @@ The framework combines a Transformer-based tabular encoder with a ResNet-based i The repository includes two primary Jupyter notebooks demonstrating the capabilities and training processes of the TIP model: -1. **`notebook_1_TIP.ipynb`**: **Training and Fine-Tuning** +1. **`data_preprocessing.ipynb`**: **DVM-CAR Dataset Preparation** + + * Guides users through downloading, cleaning, and preprocessing the DVM-CAR dataset. + * **Key Steps**: + + * Downloading raw tabular data and images. + * Cleaning and processing tabular data using provided scripts. + * Converting images to numpy arrays for efficient training. + * Creating missing masks for incomplete-data fine-tuning experiments. + +2. **`notebook_1_TIP.ipynb`**: **Training and Fine-Tuning** * Provides an introduction to TIP and walks through the training process. * **Key Sections**: @@ -28,7 +38,7 @@ The repository includes two primary Jupyter notebooks demonstrating the capabili * **Dataset**: Overview of the multimodal datasets used for training (images + tabular data). * **Fine-Tuning**: Fine-tuning the model in image-only, tabular-only, and multimodal modes. -2. **`notebook_2_TIP.ipynb`**: **Evaluation, Comparison, and Metrics** +3. **`notebook_2_TIP.ipynb`**: **Evaluation, Comparison, and Metrics** * Focuses on evaluating and comparing the performance of the model after fine-tuning. * **Key Sections**: @@ -120,37 +130,37 @@ mkdir -p data/DVM/features #### 2. Clean and Process Tabular Data -Use the provided notebook from the TIP repo: +Run the **`data_preprocessing.ipynb`** notebook included in this repository: -πŸ“„ [create\_dvm\_dataset.ipynb](https://github.com/siyi-wind/TIP/blob/main/data/create_dvm_dataset.ipynb) - -* Open the notebook. +* Open `data_preprocessing.ipynb`. * Update paths to match your local `data/DVM/` setup. -* Run all cells to generate cleaned and processed tabular files. +* Run the tabular data cleaning section to generate cleaned and processed tabular files. --- #### 3. Convert Images to Numpy Arrays -Use the helper script from the TIP repo: - -πŸ“„ [image2numpy.py](https://github.com/siyi-wind/TIP/blob/main/data/image2numpy.py) +Continue with the **`data_preprocessing.ipynb`** notebook: -* Update paths inside the script. -* Run to convert JPG images into numpy format for faster training. +* Run the image conversion section to convert JPG images into numpy format for faster training. +* Update paths as needed within the notebook cells. --- #### 4. Create Missing Masks -For incomplete-data fine-tuning experiments, generate missing masks: +Complete the missing mask generation in **`data_preprocessing.ipynb`**: -πŸ“„ [create\_missing\_mask.ipynb](https://github.com/siyi-wind/TIP/blob/main/data/create_missing_mask.ipynb) - -* Update paths to match `data/DVM/`. -* Run all cells to create missing masks (RVM, RFM, MIFM, LIFM). +* Run the missing mask creation section for incomplete-data fine-tuning experiments. +* This generates missing masks (RVM, RFM, MIFM, LIFM). * Outputs will be stored in `data/DVM/missing_mask/`. +Add the data_base path to **`configs/config_dvm_TIP.yaml`** based on your local setup: + +```yaml +data_base: # TODO: set data base path +``` + --- ### Notes diff --git a/implementations/multimodal_representation_learning/configs/config_dvm_TIP.yaml b/implementations/multimodal_representation_learning/configs/config_dvm_TIP.yaml index 7c3d203..d011fe3 100644 --- a/implementations/multimodal_representation_learning/configs/config_dvm_TIP.yaml +++ b/implementations/multimodal_representation_learning/configs/config_dvm_TIP.yaml @@ -25,7 +25,7 @@ lr_finder_lrs: multitarget: wandb_entity: # Removed this line -data_base: /projects/aieng/multimodal_bootcamp/representation_learning/TIP/data/DVM/features +data_base: # TODO: set data base path num_workers: 10 pin_memory: True # speeds up hostβ†’GPU transfer persistent_workers: True # keeps workers alive between epochs diff --git a/implementations/multimodal_representation_learning/data_preprocessing.ipynb b/implementations/multimodal_representation_learning/data_preprocessing.ipynb new file mode 100644 index 0000000..8109e7a --- /dev/null +++ b/implementations/multimodal_representation_learning/data_preprocessing.ipynb @@ -0,0 +1,1059 @@ +{ + "cells": [ + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Preprocessing DVM dataset\n", + "This notebook preprocesses the DVM-CAR dataset into train/val/test splits and generates tabular, image paths, labels, and auxiliary artifacts needed.\n", + "\n", + "Prepare the DVM-CAR dataset and create train/val/test splits for TIP:\n", + "- Tabular data (.csv)\n", + "- Image paths (.pt)\n", + "- Labels (.pt)\n", + "- Tabular lengths (.pt) β€” number of unique values per column (for encoders)\n", + "\n", + "Reference implementation: MMCL Tabular+Imaging (https://github.com/paulhager/MMCL-Tabular-Imaging/blob/main/data/create_dvm_dataset.ipynb)\n", + "\n", + "### Download data\n", + "Get the resized images and tables from Deep Visual Marketing:\n", + "- Resized images (resized_DVM_v2.zip): https://figshare.com/articles/figure/DVM-CAR_Dataset/19586296/1?file=34792453\n", + "- Tables (tables_V2.0.zip): https://figshare.com/articles/figure/DVM-CAR_Dataset/19586296/2\n", + "\n", + "Place and unzip both into `./data/DVM/`:\n", + "```bash\n", + "mkdir -p ./data/DVM\n", + "# download the two zip files to ./data/DVM (or move them there)\n", + "unzip ./data/DVM/resized_DVM_v2.zip -d ./data/DVM/\n", + "unzip ./data/DVM/tables_V2.0.zip -d ./data/DVM/" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup & Imports\n", + "Configure base paths and import libraries used throughout preprocessing." + ] + }, + { + "cell_type": "code", + "execution_count": 59, + "metadata": {}, + "outputs": [], + "source": [ + "import pandas as pd\n", + "import os\n", + "from os.path import join\n", + "from sklearn.model_selection import train_test_split\n", + "from sklearn.metrics import confusion_matrix\n", + "import seaborn as sns\n", + "from torchvision.io import read_image\n", + "from matplotlib import pyplot as plt\n", + "from tqdm import tqdm\n", + "import torchvision\n", + "import torch\n", + "import random\n", + "import numpy as np\n", + "from sklearn.neighbors import NearestNeighbors\n", + "\n", + "pd.options.display.max_columns = 700\n", + "\n", + "# TODO: change this to the path of your DVM data directory\n", + "BASE = './data/DVM'\n", + "TABLES = join(BASE, 'tables_V2.0')\n", + "FEATURES = join(BASE, 'features') \n", + "\n", + "front_view_only = False\n", + "\n", + "from typing import List\n", + "from sklearn.decomposition import PCA\n", + "from sklearn.linear_model import LogisticRegression\n", + "from sklearn.cluster import k_means, SpectralClustering\n", + "import multiprocessing as mp\n", + "\n", + "ANALYSIS = join(BASE, 'analysis')\n", + "\n", + "def conf_matrix_from_matrices(mat_gt, mat_pred):\n", + " overlap_and = (mat_pred & mat_gt)\n", + " tp = overlap_and.sum()\n", + " fp = mat_pred.sum()-overlap_and.sum()\n", + " fn = mat_gt.sum()-overlap_and.sum()\n", + " tn = mat_gt.shape[0]**2-(tp+fp+fn)\n", + " return tp, fp, fn, tn" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "metadata": {}, + "outputs": [], + "source": [ + "def check_or_save(obj, path, index=None, header=None):\n", + " if isinstance(obj, pd.DataFrame):\n", + " if index is None or header is None:\n", + " raise ValueError('Index and header must be specified for saving a dataframe')\n", + " if os.path.exists(path):\n", + " if not header:\n", + " saved_df = pd.read_csv(path,header=None)\n", + " else:\n", + " saved_df = pd.read_csv(path)\n", + " naked_df = saved_df.reset_index(drop=True)\n", + " naked_df.columns = range(naked_df.shape[1])\n", + " naked_obj = obj.reset_index(drop=not index)\n", + " naked_obj.columns = range(naked_obj.shape[1])\n", + " if naked_df.round(6).equals(naked_obj.round(6)):\n", + " return\n", + " else:\n", + " diff = (naked_df.round(6) == naked_obj.round(6))\n", + " diff[naked_df.isnull()] = naked_df.isnull() & naked_obj.isnull()\n", + " assert diff.all().all(), \"Dataframe is not the same as saved dataframe\"\n", + " else:\n", + " obj.to_csv(path, index=index, header=header)\n", + " else:\n", + " if os.path.exists(path):\n", + " saved_obj = torch.load(path)\n", + " if isinstance(obj, list):\n", + " for i in range(len(obj)):\n", + " check_array_equality(obj[i], saved_obj[i])\n", + " else:\n", + " check_array_equality(obj, saved_obj)\n", + " else:\n", + " print(f'Saving to {path}')\n", + " torch.save(obj, path)\n", + "\n", + "\n", + "def check_array_equality(ob1, ob2):\n", + " if torch.is_tensor(ob1) or isinstance(ob1, np.ndarray):\n", + " assert (ob2 == ob1).all()\n", + " else:\n", + " assert ob2 == ob1" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Create Tabular Dataset" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Read raw CSVs (Ad/Basic/Image/Price/Sales/Trim), clean column names, and derive Adv_ID for unique listing IDs." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "ad_data = pd.read_csv(join(TABLES, 'Ad_table.csv'))\n", + "ad_data.rename(columns={' Genmodel': 'Genmodel', ' Genmodel_ID': 'Genmodel_ID'}, inplace=True)\n", + "\n", + "basic_data = pd.read_csv(join(TABLES, 'Basic_table.csv'))\n", + "\n", + "image_data = pd.read_csv(join(TABLES, 'Image_table.csv'))\n", + "image_data.rename(columns={' Image_ID': 'Image_ID', ' Image_name': 'Image_name', ' Predicted_viewpoint':'Predicted_viewpoint', ' Quality_check':'Quality_check'}, inplace=True)\n", + "\n", + "price_data = pd.read_csv(join(TABLES, 'Price_table.csv'))\n", + "price_data.rename(columns={' Genmodel': 'Genmodel', ' Genmodel_ID': 'Genmodel_ID', ' Year': 'Year', ' Entry_price': 'Entry_price'}, inplace=True)\n", + "\n", + "sales_data = pd.read_csv(join(TABLES, 'Sales_table.csv'))\n", + "sales_data.rename(columns={'Genmodel ': 'Genmodel', 'Genmodel_ID ': 'Genmodel_ID'}, inplace=True)\n", + "\n", + "trim_data = pd.read_csv(join(TABLES, 'Trim_table.csv'))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def parser_adv_id(x):\n", + " split = x[\"Image_ID\"].split('$$')\n", + " return f\"{split[0]}$${split[1]}\"\n", + "\n", + "image_data[\"Adv_ID\"] = image_data.apply(lambda x: parser_adv_id(x), axis=1)\n", + "if front_view_only:\n", + " image_data = image_data[(image_data[\"Quality_check\"]==\"P\")&(image_data[\"Predicted_viewpoint\"]==0)]\n", + "image_data.drop_duplicates(subset=['Adv_ID'], inplace=True)\n", + "image_data" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Join tables to build a unified dataframe with image meta, tabular features, and labels. Drop duplicates and missing rows." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(len(ad_data))\n", + "feature_df = ad_data.merge(price_data[['Genmodel_ID', 'Entry_price', 'Year']], left_on=['Genmodel_ID','Reg_year'], right_on=['Genmodel_ID','Year'])\n", + "print(len(feature_df))\n", + "feature_df" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "data_df = feature_df.merge(image_data[['Adv_ID', 'Image_name', 'Predicted_viewpoint']], left_on=['Adv_ID'], right_on=['Adv_ID'])\n", + "assert data_df[\"Adv_ID\"].is_unique" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def extract_engine_size(x):\n", + " return float(x['Engin_size'][:-1])\n", + "\n", + "data_df.dropna(inplace=True)\n", + "data_df['Engine_size'] = data_df.apply(lambda x: extract_engine_size(x), axis=1)\n", + "data_df.drop(columns=['Engin_size'], inplace=True)\n", + "data_df" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Normalize continuous features and encode categorical features using category codes; concatenate into a single feature frame." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [], + "source": [ + "id_df = data_df.loc[:,'Adv_ID']\n", + "image_name_df = data_df.loc[:,'Image_name']\n", + "viewpoint_df = data_df.loc[:,'Predicted_viewpoint']\n", + "\n", + "continuous_df = data_df.loc[:,(\n", + " 'Adv_year',\n", + " 'Adv_month',\n", + " 'Reg_year',\n", + " 'Runned_Miles',\n", + " 'Price',\n", + " 'Seat_num',\n", + " 'Door_num',\n", + " 'Entry_price', \n", + " 'Engine_size'\n", + " )]\n", + "\n", + "categorical_ids = ['Color',\n", + " 'Bodytype',\n", + " 'Gearbox',\n", + " 'Fuel_type',\n", + " 'Genmodel_ID']\n", + "\n", + "\n", + "\n", + "categorical_df = data_df.loc[:,categorical_ids]\n", + "\n", + "continuous_df['Runned_Miles'] = pd.to_numeric(continuous_df['Runned_Miles'], errors='coerce')\n", + "continuous_df['Price'] = pd.to_numeric(continuous_df['Price'], errors='coerce')\n", + "\n", + "# normalize\n", + "continuous_df=(continuous_df-continuous_df.mean())/continuous_df.std()\n", + "\n", + "categorical_df['Color'] = categorical_df['Color'].astype('category')\n", + "categorical_df['Bodytype'] = categorical_df['Bodytype'].astype('category')\n", + "categorical_df['Gearbox'] = categorical_df['Gearbox'].astype('category')\n", + "categorical_df['Fuel_type'] = categorical_df['Fuel_type'].astype('category')\n", + "categorical_df['Genmodel_ID'] = categorical_df['Genmodel_ID'].astype('category')\n", + "\n", + "cat_columns = categorical_df.select_dtypes(['category']).columns\n", + "\n", + "categorical_df[cat_columns] = categorical_df[cat_columns].apply(lambda x: x.cat.codes)\n", + "\n", + "data_df = pd.concat([id_df, continuous_df, categorical_df, image_name_df, viewpoint_df], axis=1)\n", + "data_df.dropna(inplace=True)" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [], + "source": [ + "minimum_population = 100\n", + "values = (data_df.value_counts(subset=['Genmodel_ID'])>=minimum_population).values\n", + "codes = (data_df.value_counts(subset=['Genmodel_ID'])>=minimum_population).index\n", + "populated_codes = []\n", + "for i, v in enumerate(values):\n", + " if v:\n", + " populated_codes.append(int(codes[i][0]))\n", + "\n", + "len(populated_codes)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Keep classes (Genmodel_ID) with sufficient samples and remap labels to contiguous indices." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "data_df = data_df[data_df['Genmodel_ID'].isin(populated_codes)]\n", + "map = {}\n", + "for i,l in enumerate(data_df['Genmodel_ID'].unique()):\n", + " map[l] = i\n", + "data_df['Genmodel_ID'] = data_df['Genmodel_ID'].map(map)\n", + "data_df" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [], + "source": [ + "bad_indices = []\n", + "for indx, row in data_df.iterrows():\n", + " im_name = row['Image_name']\n", + " split = im_name.split('$$')\n", + " path = join(BASE, 'resized_DVM', split[0], split[1], split[2], split[3], im_name)\n", + " if not os.path.exists(path):\n", + " bad_indices.append(path)\n", + "\n", + "len(bad_indices)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Create stratified splits, persist IDs, labels, and feature CSVs for each split, and record field lengths." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "_ids = list(data_df['Adv_ID'])\n", + "addendum = '_all_views'\n", + "non_feature_columns = ['Adv_ID', 'Image_name', 'Predicted_viewpoint', 'Genmodel_ID']\n", + "if front_view_only:\n", + " train_set_ids, test_ids = train_test_split(_ids, test_size=0.1, random_state=2022)\n", + " train_ids, val_ids = train_test_split(train_set_ids, test_size=0.2, random_state=2022)\n", + " \n", + " bad_indices_train = torch.load(join(FEATURES, f'bad_indices_train{addendum}.pt'))\n", + " bad_indices_val = torch.load(join(FEATURES, f'bad_indices_val{addendum}.pt'))\n", + "\n", + " print(f'Val length before {len(val_ids)}')\n", + " for _id in bad_indices_val:\n", + " val_ids.remove(_id)\n", + " print(f'Val length after {len(val_ids)}')\n", + "\n", + " print(f'Train length before {len(train_ids)}')\n", + " for _id in bad_indices_train:\n", + " train_ids.remove(_id)\n", + " print(f'Train length after {len(train_ids)}')\n", + "else:\n", + " addendum = '_all_views'\n", + " train_set_ids, test_ids = train_test_split(_ids, test_size=0.5, random_state=2022, stratify=data_df['Genmodel_ID'])\n", + " train_ids, val_ids = train_test_split(train_set_ids, test_size=0.2, random_state=2022, stratify=data_df[data_df['Adv_ID'].isin(train_set_ids)]['Genmodel_ID'])\n", + "\n", + "# Make a features folder inside base if it doesn't exist\n", + "os.makedirs(join(FEATURES), exist_ok=True)\n", + "check_or_save(train_ids, join(FEATURES, f'train_ids{addendum}.pt'))\n", + "check_or_save(val_ids, join(FEATURES, f'val_ids{addendum}.pt'))\n", + "check_or_save(test_ids, join(FEATURES, f'test_ids{addendum}.pt'))\n", + "\n", + "train_df = data_df.set_index('Adv_ID').loc[train_ids]\n", + "val_df = data_df.set_index('Adv_ID').loc[val_ids]\n", + "test_df = data_df.set_index('Adv_ID').loc[test_ids]\n", + "\n", + "train_labels_all = list(train_df['Genmodel_ID'])\n", + "val_labels_all = list(val_df['Genmodel_ID'])\n", + "test_labels_all = list(test_df['Genmodel_ID'])\n", + "\n", + "check_or_save(train_labels_all, join(FEATURES,f'labels_model_all_train{addendum}.pt'))\n", + "check_or_save(val_labels_all, join(FEATURES,f'labels_model_all_val{addendum}.pt'))\n", + "check_or_save(test_labels_all, join(FEATURES,f'labels_model_all_test{addendum}.pt'))\n", + "\n", + "check_or_save(train_df.loc[:,~train_df.columns.isin(non_feature_columns)],join(FEATURES,f'dvm_features_train_noOH{addendum}.csv'), index=False, header=False)\n", + "check_or_save(val_df.loc[:,~val_df.columns.isin(non_feature_columns)],join(FEATURES,f'dvm_features_val_noOH{addendum}.csv'), index=False, header=False)\n", + "check_or_save(test_df.loc[:,~test_df.columns.isin(non_feature_columns)],join(FEATURES,f'dvm_features_test_noOH{addendum}.csv'), index=False, header=False)\n", + "\n", + "check_or_save(train_df, join(FEATURES,f'dvm_full_features_train_noOH{addendum}.csv'), index=True, header=True)\n", + "check_or_save(val_df, join(FEATURES,f'dvm_full_features_val_noOH{addendum}.csv'), index=True, header=True)\n", + "check_or_save(test_df, join(FEATURES,f'dvm_full_features_test_noOH{addendum}.csv'), index=True, header=True)\n", + "\n", + "lengths = [1 for i in range(len(continuous_df.columns))]\n", + "\n", + "if 'Genmodel_ID' in categorical_ids:\n", + " categorical_ids.remove('Genmodel_ID')\n", + "max = list(data_df[categorical_ids].max(axis=0))\n", + "max = [i+1 for i in max]\n", + "lengths = lengths + max\n", + "check_or_save(lengths, join(FEATURES, f'tabular_lengths{addendum}.pt'))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Compute on-disk image file paths for each split and save them for lazy loading during training." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def get_paths(df):\n", + " paths = []\n", + " for indx, row in df.iterrows():\n", + " im_name = row['Image_name']\n", + " split = im_name.split('$$')\n", + " path = join(BASE, 'resized_DVM', split[0], split[1], split[2], split[3], im_name)\n", + " paths.append(path)\n", + " return paths\n", + "\n", + "# For big dataset need to save only paths to load live\n", + "addendum = '_all_views'\n", + "train_df = pd.read_csv(join(FEATURES,f'dvm_full_features_train_noOH{addendum}.csv'))\n", + "val_df = pd.read_csv(join(FEATURES,f'dvm_full_features_val_noOH{addendum}.csv'))\n", + "test_df = pd.read_csv(join(FEATURES,f'dvm_full_features_test_noOH{addendum}.csv'))\n", + "\n", + "for df, name in zip([train_df, val_df, test_df], ['train', 'val', 'test']):\n", + " paths = get_paths(df)\n", + " check_or_save(paths, join(FEATURES, f'{name}_paths{addendum}.pt'))" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Create Low Data Splits\n", + "Generate 10% and 1% low-data splits ensuring at least one sample per class; persist corresponding features, ids, paths, and labels." + ] + }, + { + "cell_type": "code", + "execution_count": 42, + "metadata": {}, + "outputs": [], + "source": [ + "def low_data_split(df, nclasses):\n", + " critical_ids = df.groupby('Genmodel_ID', as_index=False).head(n=1)['Adv_ID']\n", + " other_ids = df.loc[~df['Adv_ID'].isin(critical_ids)]['Adv_ID'].values\n", + " to_fill_size = (int(len(df)*0.1)-len(critical_ids))\n", + " stratify = None\n", + " if to_fill_size >= nclasses:\n", + " stratify = df.set_index('Adv_ID').loc[other_ids]['Genmodel_ID']\n", + " if to_fill_size > 0:\n", + " _, low_data_ids = train_test_split(other_ids, test_size=to_fill_size, random_state=2023, stratify=stratify)\n", + " else:\n", + " low_data_ids = []\n", + " new_ids = np.concatenate([critical_ids,low_data_ids])\n", + " return new_ids" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "addendum = '_all_views'\n", + "# data_str = 'images'\n", + "data_str = 'paths'\n", + "location = \"\"\n", + "non_feature_columns = ['Image_name', 'Genmodel_ID', 'Predicted_viewpoint', 'Adv_ID']\n", + "nclasses = 151\n", + "if addendum=='_all_views':\n", + " #data_str = 'paths'\n", + " #location = '_server'\n", + " nclasses = 286\n", + "\n", + "\n", + "for k, prev_k in zip([0.1,0.01],['','_0.1']):\n", + " df = pd.read_csv(join(FEATURES,f'dvm_full_features_train_noOH{addendum}{prev_k}.csv'))\n", + " ids = torch.load(join(FEATURES, f'train_ids{addendum}{prev_k}.pt'))\n", + " ims = torch.load(join(FEATURES, f'train_{data_str}{addendum}{location}{prev_k}.pt'))\n", + " labels = torch.load(join(FEATURES, f'labels_model_all_train{addendum}{prev_k}.pt'))\n", + " low_data_ids = low_data_split(df, nclasses)\n", + " true_false_mask = [i in low_data_ids for i in ids]\n", + " ld = [id for id in ids if id in low_data_ids]\n", + " low_data_ids = ld\n", + " low_data_df = df.loc[true_false_mask]\n", + " if addendum=='_all_views' and not data_str=='images':\n", + " ims = np.array(ims)\n", + " else: \n", + " ims = torch.tensor(ims)\n", + " low_data_ims = ims[true_false_mask]\n", + " low_data_labels = [labels[i] for i in range(len(ids)) if ids[i] in low_data_ids]\n", + "\n", + " \n", + " check_or_save(low_data_df.loc[:,~low_data_df.columns.isin(non_feature_columns)], join(FEATURES,f'dvm_features_train_noOH{addendum}_{k}.csv'), index=False, header=False)\n", + " check_or_save(low_data_df, join(FEATURES,f'dvm_full_features_train_noOH{addendum}_{k}.csv'), index=False, header=True)\n", + " check_or_save(low_data_ims, join(FEATURES, f'train_{data_str}{addendum}{location}_{k}.pt'))\n", + " check_or_save(low_data_ids, join(FEATURES, f'train_ids{addendum}_{k}.pt'))\n", + " check_or_save(low_data_labels, join(FEATURES, f'labels_model_all_train{addendum}_{k}.pt'))\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "split = 'train'\n", + "for k in [0.1, 0.01]:\n", + " print(f'Checking low data split for {split} at {k} FEATURES: {FEATURES}, addendum: {addendum}')\n", + " low_data_ids = torch.load(join(FEATURES, f'{split}_ids{addendum}_{k}.pt'))\n", + " low_data_df = pd.read_csv(join(FEATURES,f'dvm_full_features_{split}_noOH{addendum}_{k}.csv'))\n", + " print(low_data_df.value_counts('Genmodel_ID'))\n", + " print(len(low_data_ids))\n", + "\n", + "print(f'Checking low data split for {split} at FEATURES: {FEATURES}, addendum: {addendum}')\n", + "low_data_ids = torch.load(join(FEATURES, f'{split}_ids{addendum}.pt'))\n", + "low_data_df = pd.read_csv(join(FEATURES,f'dvm_full_features_{split}_noOH{addendum}.csv'))\n", + "print(low_data_df.value_counts('Genmodel_ID'))\n", + "print(len(low_data_ids))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Convert Images to NumPy\n", + "Convert JPEGs to .npy for faster IO in some environments; logs missing paths." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# convert jpg image to numpy array\n", + "def process_DVM(\n", + " DVM_feature_folder = FEATURES):\n", + " missing_paths = []\n", + " for split in ['train', 'val', 'test']:\n", + " img_paths = torch.load(os.path.join(DVM_feature_folder, '{}_paths_all_views.pt'.format(split)))\n", + " np_paths = []\n", + " for path in tqdm(img_paths):\n", + " if not os.path.exists(path):\n", + " missing_paths.append(path)\n", + " continue\n", + " img_np = plt.imread(path)\n", + " save_path = path[:-4] + '.npy'\n", + " np.save(save_path, img_np)\n", + " np_paths.append(save_path)\n", + " # break\n", + " # break\n", + " return\n", + "\n", + "process_DVM()" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## New Physical Features\n", + "Fill missing physical specs (Wheelbase/Length/Width/Height), add jitter, merge into features, and update tabular lengths." + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Adding missing values to physical table" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Fill using other values\n", + "physical_df_orig = pd.read_csv(join(TABLES,'Ad_table (extra).csv'))\n", + "physical_df_orig.rename(columns={' Genmodel_ID':'Genmodel_ID', ' Genmodel':'Genmodel'}, inplace=True)\n", + "\n", + "# Manual touches\n", + "\n", + "# Peugeot RCZ\n", + "physical_df_orig.loc[physical_df_orig['Genmodel_ID'] == '69_36','Wheelbase']=2612\n", + "# Ford Grand C-Max\n", + "physical_df_orig.loc[physical_df_orig['Genmodel_ID'] == '29_20','Wheelbase']=2788 \n", + "\n", + "def fill_from_other_entry(row):\n", + " for attr in ['Wheelbase', 'Length', 'Width', 'Height']:\n", + " if pd.isna(row[attr]) or row[attr]==0:\n", + " other_rows = physical_df_orig.loc[physical_df_orig['Genmodel_ID']==row['Genmodel_ID']]\n", + " other_rows.dropna(subset=[attr], inplace=True)\n", + " other_rows.drop_duplicates(subset=[attr], inplace=True)\n", + " other_rows = other_rows[other_rows[attr]>0]\n", + " if len(other_rows)>0:\n", + " row[attr] = other_rows[attr].values[0]\n", + " return row\n", + "\n", + "physical_df_orig = physical_df_orig.apply(fill_from_other_entry, axis=1)\n", + "\n", + "physical_df_orig.to_csv(join(TABLES,'Ad_table_physical_filled.csv'), index=False)" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Add physical attributes to features" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Add jitter to physical dimensions so they aren't just labels\n", + "def add_jitter(x, jitter=50):\n", + " return x + random.randint(-jitter, jitter)\n", + "\n", + "random.seed(2022)\n", + "physical_df = pd.read_csv(join(TABLES,'Ad_table_physical_filled.csv'))\n", + "for attr in ['Wheelbase', 'Length', 'Width', 'Height']:\n", + " physical_df[attr] = physical_df[attr].apply(add_jitter)\n", + "physical_df.to_csv(join(TABLES,'Ad_table_physical_filled_jittered_50.csv'), index=False)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Ford ranger (29_30) has wrong height. Missing 1 in front... 805.0 instead of 1805.0\n", + "# Mercedes Benz (59_29) wrong wheelbase, 5246.0 instead of 3106\n", + "# Kia Rio (43_9) wrong wheelbase, 4065.0 instead of 2580\n", + "# FIXED\n", + "\n", + "physical_df = pd.read_csv(join(TABLES,'Ad_table_physical_filled_jittered_50.csv'))[['Adv_ID', 'Wheelbase','Height','Width','Length']]\n", + "# TODO k=['', '_0.1', '_0.01'], change k to generate training datasets with different amounts of low data\n", + "k = '_0.01' \n", + "for v in ['_all_views']:\n", + " for split in ['train']: #, ,'val', 'test'\n", + " features_df = pd.read_csv(join(FEATURES,f'dvm_full_features_{split}_noOH{v}{k}.csv'))\n", + " merged_df = features_df.merge(physical_df, on='Adv_ID')\n", + " physical_only_df = merged_df[['Wheelbase','Height','Width','Length','Bodytype']]\n", + "\n", + " for attr in ['Wheelbase','Height','Width','Length']:\n", + " assert merged_df[attr].isna().sum()==0\n", + " assert (merged_df[attr]==0).sum()==0\n", + "\n", + " # normalize physical attributes\n", + " for attr in ['Wheelbase','Height','Width','Length']:\n", + " merged_df[attr] = (merged_df[attr]-merged_df[attr].mean())/merged_df[attr].std()\n", + " physical_only_df[attr] = (physical_only_df[attr]-physical_only_df[attr].mean())/physical_only_df[attr].std()\n", + "\n", + " # Drop unwanted cols\n", + " non_feature_columns = ['Adv_ID', 'Image_name', 'Genmodel_ID']\n", + " if v == '_all_views':\n", + " non_feature_columns.append('Predicted_viewpoint')\n", + " merged_df = merged_df.drop(non_feature_columns, axis=1)\n", + "\n", + " merged_df_cols = merged_df.columns.tolist()\n", + " rearranged_cols = merged_df_cols[-4:]+merged_df_cols[:-4]\n", + " merged_df = merged_df[rearranged_cols]\n", + " check_or_save(merged_df, join(FEATURES,f'dvm_features_{split}_noOH{v}{k}_physical_jittered_50.csv'), index=False, header=False)\n", + " check_or_save(physical_only_df, join(FEATURES,f'dvm_features_{split}_noOH{v}{k}_physical_only_jittered_50.csv'), index=False, header=False)\n", + " lengths = torch.load(join(FEATURES,f'tabular_lengths{v}.pt'))\n", + " new_lengths = [1,1,1,1]\n", + " lengths = new_lengths + lengths\n", + " check_or_save(lengths, join(FEATURES,f'tabular_lengths{v}_physical.pt'))\n", + " lengths = [1,1,1,1,13]\n", + " check_or_save(lengths, join(FEATURES,f'tabular_lengths{v}_physical_only.pt'))\n", + "print(merged_df.columns)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# reorder features to categorical, numerical\n", + "for v in ['_all_views']:\n", + " field_lengths_tabular = torch.load(join(FEATURES, f'tabular_lengths{v}_physical.pt'))\n", + " categorical_ids = []\n", + " continous_ids = []\n", + " for i in range(len(field_lengths_tabular)):\n", + " if field_lengths_tabular[i] == 1:\n", + " continous_ids.append(i)\n", + " else:\n", + " categorical_ids.append(i)\n", + " print('Categorical Index: {}, '.format(len(categorical_ids)), categorical_ids)\n", + " print('Numerical Index: {}, '.format(len(continous_ids)), continous_ids)\n", + "\n", + " reorder_ids = categorical_ids + continous_ids\n", + " reorder_field_lengths_tabular = [field_lengths_tabular[i] for i in reorder_ids]\n", + " check_or_save(reorder_field_lengths_tabular, join(FEATURES, f'tabular_lengths{v}_physical_reordered.pt'),)\n", + " for split in ['train', 'val', 'test']:\n", + " data_tabular = pd.read_csv(join(FEATURES, f'dvm_features_{split}_noOH{v}_physical_jittered_50.csv'), header=None)\n", + " reorder_data_tabular = data_tabular.iloc[:, reorder_ids]\n", + " check_or_save(reorder_data_tabular, join(FEATURES, f'dvm_features_{split}_noOH{v}_physical_jittered_50_reordered.csv'), index=False, header=False)\n", + " for k in ['_0.1', '_0.01']:\n", + " data_tabular = pd.read_csv(join(FEATURES, f'dvm_features_train_noOH{v}{k}_physical_jittered_50.csv'), header=None)\n", + " reorder_data_tabular = data_tabular.iloc[:, reorder_ids]\n", + " check_or_save(reorder_data_tabular, join(FEATURES, f'dvm_features_train_noOH{v}{k}_physical_jittered_50_reordered.csv'), index=False, header=False)\n" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Add Labels to Features\n", + "Add class labels as the last column for train/val features and update lengths accordingly." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "for v in ['_all_views']:\n", + " for split in ['train', 'val']:\n", + " labels = torch.load(join(FEATURES,f'labels_model_all_{split}{v}.pt'))\n", + " features = pd.read_csv(join(FEATURES,f'dvm_features_{split}_noOH{v}_physical_jittered_50.csv'), header=None)\n", + " features['label'] = labels\n", + " check_or_save(features, join(FEATURES,f'dvm_features_{split}_noOH{v}_physical_jittered_50_labeled.csv'), index=False, header=False)\n", + " lengths = torch.load(join(FEATURES,f'tabular_lengths{v}_physical.pt'))\n", + " lengths.append(max(labels)+1)\n", + " check_or_save(lengths, join(FEATURES,f'tabular_lengths{v}_physical_labeled.pt'))\n", + "print(len(lengths))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Remove Adv year\n", + "Demonstrate feature removal and update tabular lengths for ablation experiments." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "tabular_lengths = torch.load(join(FEATURES, f'tabular_lengths_all_views_physical_reordered.pt'))\n", + "print(tabular_lengths)\n", + "tabular_lengths = tabular_lengths[:-1]\n", + "print(len(tabular_lengths))\n", + "check_or_save(tabular_lengths, join(FEATURES, f'tabular_lengths_all_views_physical_reordered_rmAY.pt'))" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [], + "source": [ + "reordered_column_name = [ 'Color', 'Bodytype', 'Gearbox','Fuel_type' ,\n", + " 'Wheelbase', 'Height', 'Width', 'Length', 'Adv_year', 'Adv_month',\n", + " 'Reg_year', 'Runned_Miles', 'Price', 'Seat_num', 'Door_num',\n", + " 'Entry_price', 'Engine_size',]\n", + "column_name = ['Wheelbase', 'Height', 'Width', 'Length', 'Adv_year', 'Adv_month',\n", + " 'Reg_year', 'Runned_Miles', 'Price', 'Seat_num', 'Door_num',\n", + " 'Entry_price', 'Engine_size','Color', 'Bodytype', 'Gearbox','Fuel_type']\n", + "for v in ['', '_0.1', '_0.01']:\n", + " for split in ['train']: # 'train', 'val', 'test'\n", + " reordered_features = pd.read_csv(join(FEATURES,f'dvm_features_{split}_noOH_all_views{v}_physical_jittered_50_reordered.csv'), header=None)\n", + " reordered_features.columns = reordered_column_name \n", + " reordered_features.drop(['Adv_year'], axis=1, inplace=True)\n", + " check_or_save(reordered_features, join(FEATURES, f'dvm_features_{split}_noOH_all_views{v}_physical_jittered_50_reordered_rmAY.csv'), index=False, header=False)\n", + " break\n", + " break\n", + "\n", + "reordered_features" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Create missing mask for train, val, test tabular data\n", + "Create value-level and feature-level missing masks; also generate MI/LI masks based on feature importance." + ] + }, + { + "cell_type": "code", + "execution_count": 68, + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "import pandas as pd\n", + "from os.path import join, dirname\n", + "import torch\n", + "from sklearn.ensemble import RandomForestClassifier\n", + "from sklearn.metrics import roc_auc_score, accuracy_score" + ] + }, + { + "cell_type": "code", + "execution_count": 69, + "metadata": {}, + "outputs": [], + "source": [ + "def create_missing_mask(data_tabular_path, mask_path, random_seed, missing_strategy, missing_rate):\n", + " '''\n", + " missing_strategy: value (random value missingness) or feature (random feature missingness)\n", + " missing_rate: 0.0-1.0\n", + " '''\n", + " data_tabular = np.array(pd.read_csv(data_tabular_path, header=None))\n", + " print(f'data tabular shape: {data_tabular.shape}')\n", + " np.random.seed(random_seed)\n", + " M, N = data_tabular.shape[0], data_tabular.shape[1]\n", + " if missing_strategy == 'value':\n", + " missing_mask_data = np.zeros((M*N), dtype=bool)\n", + " mask_pos = np.random.choice(M*N, size=int(M*N*missing_rate), replace=False)\n", + " missing_mask_data[mask_pos] = True\n", + " missing_mask_data = missing_mask_data.reshape((M,N))\n", + " elif missing_strategy == 'feature':\n", + " missing_mask_data = np.zeros((M,N), dtype=bool)\n", + " mask_pos = np.random.choice(N, size=int(N*missing_rate), replace=False)\n", + " missing_mask_data[:,mask_pos] = True\n", + " else:\n", + " raise print('Only support value and feature missing strategy')\n", + " np.save(mask_path, missing_mask_data)\n", + " print(f'Real missing rate: {missing_mask_data.sum()/missing_mask_data.size}')\n", + " print(f'Save missing mask to {mask_path}')\n", + " return missing_mask_data\n", + "\n", + "def create_certain_missing_mask(data_tabular_path, mask_path, mask_pos_order, missing_strategy, missing_rate):\n", + " '''Create mask according to a mask order list (for MI and LI feature missingness)'''\n", + " data_tabular = np.array(pd.read_csv(data_tabular_path, header=None))\n", + " print(f'data tabular shape: {data_tabular.shape}')\n", + " M, N = data_tabular.shape[0], data_tabular.shape[1]\n", + " assert N == len(mask_pos_order)\n", + " mask_pos = mask_pos_order[:int(N*missing_rate)]\n", + " missing_mask_data = np.zeros((M,N), dtype=bool)\n", + " missing_mask_data[:,mask_pos] = True\n", + " np.save(mask_path, missing_mask_data)\n", + " print(f'Real missing rate: {missing_mask_data.sum()/missing_mask_data.size}')\n", + " print(f'Save missing mask to {mask_path}')\n", + " return missing_mask_data" + ] + }, + { + "cell_type": "code", + "execution_count": 71, + "metadata": {}, + "outputs": [], + "source": [ + "# TODO: change to your own path\n", + "MASK_PATH = join(FEATURES, 'missing_mask')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "missing_strategy = 'value'\n", + "missing_rate = 0.3\n", + "target = 'dvm'\n", + "\n", + "train_name = 'dvm_features_train_noOH_all_views_physical_jittered_50_reordered.csv'\n", + "val_name = 'dvm_features_val_noOH_all_views_physical_jittered_50_reordered.csv'\n", + "test_name = 'dvm_features_test_noOH_all_views_physical_jittered_50_reordered.csv'\n", + "for name, seed, split in zip([train_name, val_name, test_name], [2021,2022,2023], ['train', 'val', 'test']):\n", + " save_mask_path = join(MASK_PATH, f'{name[:-4]}_{target}_{missing_strategy}_{missing_rate}.npy')\n", + " path = join(FEATURES, name)\n", + " # print(path)\n", + " create_missing_mask(path, save_mask_path, seed, missing_strategy, missing_rate)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "missing_strategy = 'feature'\n", + "\n", + "train_name = 'dvm_features_train_noOH_all_views_physical_jittered_50_reordered.csv'\n", + "val_name = 'dvm_features_val_noOH_all_views_physical_jittered_50_reordered.csv'\n", + "test_name = 'dvm_features_test_noOH_all_views_physical_jittered_50_reordered.csv'\n", + "for name, seed, split in zip([train_name, val_name, test_name], [2022,2022,2022], ['train', 'val', 'test']):\n", + " save_mask_path = join(MASK_PATH, f'{name[:-4]}_{target}_{missing_strategy}_{missing_rate}.npy')\n", + " path = join(FEATURES, name)\n", + " # print(path)\n", + " create_missing_mask(path, save_mask_path, seed, missing_strategy, missing_rate)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Check train, val, test to miss the same columns\n", + "train_np = np.load(join(MASK_PATH, f'{train_name[:-4]}_dvm_feature_0.3.npy'))\n", + "val_np = np.load(join(MASK_PATH, f'{val_name[:-4]}_dvm_feature_0.3.npy'))\n", + "test_np = np.load(join(MASK_PATH, f'{test_name[:-4]}_dvm_feature_0.3.npy'))\n", + "print(train_np[0])\n", + "print(val_np[0])\n", + "print(test_np[0])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Baseline: Random Forest on Tabular Data\n", + "Train a simple RF classifier to sanity-check feature utility; compute accuracy and feature importance." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from sklearn.metrics import accuracy_score\n", + "rf = RandomForestClassifier(random_state=2022)\n", + "X_train = pd.read_csv(join(FEATURES, 'dvm_features_train_noOH_all_views_physical_jittered_50_reordered.csv'), header=None)\n", + "X_test = pd.read_csv(join(FEATURES, f'dvm_features_test_noOH_all_views_physical_jittered_50_reordered.csv'), header=None)\n", + "y_train = torch.load(join(FEATURES, 'labels_model_all_train_all_views.pt'))\n", + "y_test = torch.load(join(FEATURES, 'labels_model_all_test_all_views.pt'))\n", + "rf.fit(X_train, y_train)\n", + "\n", + "# Predict classes for the test dataset\n", + "y_pred = rf.predict(X_test)\n", + "\n", + "# Calculate accuracy\n", + "accuracy = accuracy_score(y_test, y_pred)\n", + "\n", + "print(f\"Accuracy on test dataset: {accuracy}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "reordered_column_name = ['Color', 'Bodytype', 'Gearbox','Fuel_type' ,\n", + " 'Wheelbase', 'Height', 'Width', 'Length', 'Adv_year', 'Adv_month',\n", + " 'Reg_year', 'Runned_Miles', 'Price', 'Seat_num', 'Door_num',\n", + " 'Entry_price', 'Engine_size']\n", + "\n", + "# Get feature importances\n", + "importances = rf.feature_importances_\n", + "# Sort feature importances in descending order\n", + "MI_indices = np.argsort(importances)[::-1]\n", + "LI_indices = np.argsort(importances)\n", + "# Get feature names\n", + "MI_feature_name = [reordered_column_name[x] for x in MI_indices]\n", + "print(MI_feature_name)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "missing_rate = 0.3 # Try for 0.1 as well\n", + "\n", + "missing_strategy = 'MI'\n", + "train_name = 'dvm_features_train_noOH_all_views_physical_jittered_50_reordered.csv'\n", + "val_name = 'dvm_features_val_noOH_all_views_physical_jittered_50_reordered.csv'\n", + "test_name = 'dvm_features_test_noOH_all_views_physical_jittered_50_reordered.csv'\n", + "for name, split in zip([train_name, val_name, test_name], ['train', 'val', 'test']):\n", + " save_mask_path = join(MASK_PATH, f'{name[:-4]}_{target}_{missing_strategy}_{missing_rate}.npy')\n", + " path = join(FEATURES, name)\n", + " create_certain_missing_mask(path, save_mask_path, MI_indices, missing_strategy, missing_rate)\n", + "\n", + "missing_strategy = 'LI'\n", + "train_name = 'dvm_features_train_noOH_all_views_physical_jittered_50_reordered.csv'\n", + "val_name = 'dvm_features_val_noOH_all_views_physical_jittered_50_reordered.csv'\n", + "test_name = 'dvm_features_test_noOH_all_views_physical_jittered_50_reordered.csv'\n", + "for name, split in zip([train_name, val_name, test_name], ['train', 'val', 'test']):\n", + " save_mask_path = join(MASK_PATH, f'{name[:-4]}_{target}_{missing_strategy}_{missing_rate}.npy')\n", + " path = join(FEATURES, name)\n", + " create_certain_missing_mask(path, save_mask_path, LI_indices, missing_strategy, missing_rate)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "train_np = np.load(join(MASK_PATH, f'{train_name[:-4]}_dvm_MI_0.3.npy'))\n", + "val_np = np.load(join(MASK_PATH, f'{val_name[:-4]}_dvm_MI_0.3.npy'))\n", + "test_np = np.load(join(MASK_PATH, f'{test_name[:-4]}_dvm_MI_0.3.npy'))\n", + "print(train_np[0])\n", + "print(val_np[0])\n", + "print(test_np[0])" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.15" + }, + "orig_nbformat": 4 + }, + "nbformat": 4, + "nbformat_minor": 2 +}