P3_Cleaning_Content.txt

CLEANING NOTEBOOK

"Concevez une application au service de la santé publique"
0 Preliminaries
	0.0 Importing Packages and Modules
	0.1 Importing a sample of the dataset
	0.2 Memory usage reduction
	0.3 First Overview
1 Feature selection
	1.1 Filtering out non relevant columns
	1.2 Almost empty columns
	1.3 Columns with redundant information
2 Dealing with missing or erroneous data
	2.0 Categorical columns with list of categories
		Preparing for cleaning
		Detecting columns with lists
		Splitting, extracting main, dropping less frequent values
	2.1 'product_name' column
	2.2 'code' column
	2.3 'quantity' column
	2.4 'XXX_100g' nutritional components columns
	2.5 'energy_100g' columns
		2.5.a Getting rid of outliers
		2.5.b Calculating an energy_100g value from main nutritional components:
		2.5.c Determining the unit of energy columns
	2.6 Category columns
		2.6.* Unknown
		2.6.a Beverages
		2.6.b Fruits and vegetables
		2.6.c Fats
		2.6.d Cheese
	2.7 'nutriscore' columns
		2.7.a Checking nutriscore columns
		2.7.b Calculation of preliminary values for Nutriscore calculation
		2.7.c Nutriscore calculation
		2.7.d Nutrigrade determination
3 Dealing with duplicates
	3.1 Duplicates on 'code' column
	3.2 Duplicates on identification key-columns
4 Dropping inoperable rows
	Condition 1:
	Condition 2:
5 Exporting cleaned dataframe