feat(trim): introduce trim operator

[danielelotito]

This PR introduces in ADO the possibility to set up the exploration of a discovery space that stops when a good predictive model for the rest of the space is acquired.
In this iterative modeling, a holdout set is made up the last sampled points and are used to evaluate the performance of the latest acquired model.
As we progress these performance metrics are compared, if the prediction is stable iterative modeling stops.
You can try iterative modeling following the example at examples/trim_custom_experiments/README.md.

My TODOs:

• write extensive documentation
• write more tests
• write a document in which you define more challenging settings and test the expected behavior of the operator, then test the operator in these settings
• add more references to this PR draft

[danielelotito]

@michael-johnston you can have a look at the doc file I have created with docs(example): present trim to a broad public with an example
Next Monday I can do the longer guide in which I explain how to set up parameters different than defaults

[AlessandroPomponio]

initial comments

[AlessandroPomponio]

This should probably have a name

[AlessandroPomponio]

This symbol is already defined in DiscoverySpace, but it's not available in this scope. It could be worth moving it to orchestrator/schema/property.py and importing it from there in both places

[AlessandroPomponio]

Better to have these on separate lines

[AlessandroPomponio]

This function should definitely be revisited - leaving myself a note

[AlessandroPomponio]

I assume this should be the first check and if the df is empty, the function should likely just return

[AlessandroPomponio]

These seem debug functions, right?

[michael-johnston]

@michael-johnston you can have a look at the doc file I have created with docs(example): present trim to a broad public with an example Next Monday I can do the longer guide in which I explain how to set up parameters different than defaults

Looks good. I have some comments but they can wait lets make it appear in the docs first

• add a page for it under website/docs/examples and entry for that page to website/mkdocs.yaml (under nav -> examples)
• include the doc in that page following pattern of e.g. website/docs/examples/random_walk.md
• include the example YAML directly in the doc (see examples/ml-multi-cloud/EXAMPLE_SIMPLE.md) - this avoid having to keep in synch two copies of same thing

[michael-johnston]

Initial comments is we should see if there is a way to add a "update" or "refresh" method to SQLSampleStore that causes an update of the internal cache. This will remove the need for discovery_space_manager everywhere and simplify the code.

@AlessandroPomponio could you check if this is possible. The issue being the cache of entities is created once on init but if a different process add entities to the sample store the instance will not see them.

[michael-johnston]

This can be simplified avoiding the iterator, entities and discovery space manager

Suggested change

	gen = ExplicitEntitySpaceGridSampleGenerator(WalkModeEnum.SEQUENTIAL)
	# VV: We need a list that contains all entities associated with this Discovery Space in its entity source
	# regardless of whether an experiment (on this space or others) have measured any properties using the
	# experiments that this Discovery Space references.
	# To this end, we find all matching entities in the entity source which have at least 1 measurement from at least
	# one of the experiments that this discovery space defines. We then fill in any missing entities by iteratively
	# generating all the entities that this discovery space contains. These "backfill" entities will not be associated
	# with any observed property
	if discoverySpaceManager:
	logger.info("Using manager in getting entities no measurement")
	all_entities = reduce(
	# operator.add, gen.entitySpaceIterator(entitySpace=discoverySpaceManager.discoverySpace.remote()), []
	operator.add,
	gen.entitySpaceIterator(
	entitySpace=ray.get(discoverySpaceManager.entitySpace.remote())
	),
	[],
	)
	else:
	all_entities = reduce(
	operator.add, gen.entitySpaceIterator(entitySpace=space.entitySpace), []
	)
	# same as:
	# all_entities = sum(list(gen.entitySpaceIterator(entitySpace=space.entitySpace)), [])
	cp_ids = [cp.identifier for cp in space.entitySpace.constitutiveProperties]
	list_of_dicts_to_convert = []
	for e in all_entities:
	ed = {}
	ed["identifier"] = e.identifier
	for cp in cp_ids:
	obj = e.valueForConstitutivePropertyIdentifier(identifier=str(cp))
	ed[str(cp)] = obj.value
	list_of_dicts_to_convert.append(ed)
	return pd.DataFrame(list_of_dicts_to_convert)
	cp_ids = [cp.identifier for cp in space.entitySpace.constitutiveProperties]
	# Use sequential_point_iterator to get all points directly and create df without ids
	df = pd.DataFrame(entity_space.sequential_point_iterator(), columns=cp_ids)
	#Note: we will create an variant of this function in entity.py so it doesn't have to done here
	def generate_id(row_dict):
	# Create ConstitutivePropertyValue objects for this point
	property_values = [
	ConstitutivePropertyValue(
	value=value,
	property=ConstitutivePropertyDescriptor(identifier=cp_id),
	)
	for cp_id, value in row_dict.items()
	]
	return Entity.identifier_from_property_values(property_values)
	df["identifier"] = [generate_id(row) for row in df.to_dict("records")]
	return df