@@ -87,7 +87,7 @@ about the nuances, consult the [API documentation](#api-documentation) or see th
8787### YamlConfig
8888The YamlConfig class extends the functionality of the standard Python dataclass module by bundling the dataclass
8989instances with methods to save and load their data to / from .yaml files. Primarily, this functionality is implemented
90- to support storing runtime configuration data in non-volatile and human-readable ( and editable!) format.
90+ to support storing runtime configuration data in a non-volatile, human-readable, and editable format.
9191
9292The YamlConfig class is designed to be ** subclassed** by custom dataclass instances to gain the .yaml saving and
9393loading functionality realized through the inherited ** to_yaml()** and ** from_yaml()** methods:
@@ -130,7 +130,7 @@ The SharedMemoryArray class supports sharing data between multiple Python proces
130130To do so, it implements a shared memory buffer accessed via an n-dimensional NumPy array instance, allowing different
131131processes to read and write any element(s) of the array.
132132
133- #### Array Creation
133+ #### SharedMemoryArray Creation
134134The SharedMemoryArray only needs to be instantiated __ once__ by the main runtime process (thread) and provided to all
135135children processes as an input. The initialization process uses the specified prototype NumPy array and unique buffer
136136name to generate a (new) NumPy array whose data is stored in a shared memory buffer accessible from any thread or
@@ -160,9 +160,9 @@ assert sma.datatype == prototype.dtype
160160print(sma)
161161```
162162
163- #### Array Connection, Disconnection, and Destruction
163+ #### SharedMemoryArray Connection, Disconnection, and Destruction
164164Each process using the SharedMemoryArray instance, __ including__ the process that created it, must use the
165- __ connect()__ method to connect to the array before reading or writing data. At the end of it’s runtime, each connected
165+ __ connect()__ method to connect to the array before reading or writing data. At the end of its runtime, each connected
166166process must call the __ disconnect()__ method to release the local reference to the shared buffer. The ** main** process
167167also needs to call the __ destroy()__ method to destroy the shared memory buffer.
168168```
@@ -191,7 +191,7 @@ assert not sma.is_connected
191191sma.destroy() # For each create_array() call, there has to be a matching destroy() call
192192```
193193
194- #### Reading and Writing Array Data
194+ #### Reading and Writing SharedMemoryArray Data
195195For routine data writing or reading operations, the SharedMemoryArray supports accessing its data via __ indexing__ or
196196__ slicing__ , just like a regular NumPy array. Critically, accessing the data in this way is process-safe, as the
197197instance first acquires an exclusive multiprocessing Lock before interfacing with the data. For more complex access
@@ -236,89 +236,132 @@ sma.disconnect()
236236sma.destroy()
237237```
238238
239- #### Using the Array from Multiple Processes
240- While all methods showcased above run from the same process, the main advantage of the SharedMemoryArray class is that
239+ #### Using SharedMemoryArray from Multiple Processes
240+ While all methods showcased above run in the same process, the main advantage of the SharedMemoryArray class is that
241241it behaves the same way when used from different Python processes. See the [ example] ( examples/shared_memory_array.py )
242242script for more details.
243243
244244### DataLogger
245- The DataLogger class sets up data logger instances running on isolated cores (Processes) and exposes a shared Queue
246- object for buffering and piping data from any other Process to the logger cores. Currently, the logger is only intended
247- for saving serialized byte arrays used by other Ataraxis libraries (notably: ataraxis-video-system and
248- ataraxis-transport-layer).
249-
250- #### Logger creation and use
251- DataLogger is intended to only be initialized once and used by many input processes, which should be enough for most
252- use cases. However, it is possible to initialize multiple DataLogger instances by overriding the default 'instance_name'
253- argument value. The example showcased below is also available as a [ script] ( examples/data_logger.py ) :
245+ The DataLogger class initializes and manages the runtime of a logger process running in an independent Process and
246+ exposes a shared Queue object for buffering and piping data from any other Process to the logger. Currently, the
247+ class is specifically designed for saving serialized byte arrays used by other Ataraxis libraries, most notably the
248+ __ ataraxis-video-system__ and the __ ataraxis-transport-layer__ .
249+
250+ The sections below break down various aspects of working with the DataLogger instance. The individual sections can also
251+ be seen as a combined [ example] ( examples/data_logger.py ) script.
252+
253+ #### Creating and Starting the DataLogger
254+ DataLogger is intended to only be initialized *** once*** in the main runtime thread (Process) and provided to all
255+ children Processes as an input. *** Note!*** While a single DataLogger instance is typically enough for most use cases,
256+ it is possible to use more than a single DataLogger instance at the same time.
254257```
255- import time as tm
256258from pathlib import Path
257259import tempfile
260+ from ataraxis_data_structures import DataLogger
258261
259- import numpy as np
260-
261- from ataraxis_data_structures import DataLogger, LogPackage
262-
263- # Due to the internal use of Process classes, the logger has to be protected by the __main__ guard.
262+ # Due to the internal use of the 'Process' class, each DataLogger call has to be protected by the __main__ guard at
263+ # the highest level of the call hierarchy.
264264if __name__ == "__main__":
265- # As a minimum, each DataLogger has to be given the output folder and the name to use for the shared buffer. The
266- # name has to be unique across all DataLogger instances used at the same time.
267- tempdir = tempfile.TemporaryDirectory() # A temporary directory for illustration purposes
265+
266+ # As a minimum, each DataLogger has to be given the path to the output directory and a unique name to distinguish
267+ # the instance from any other concurrently active DataLogger instance.
268+ tempdir = tempfile.TemporaryDirectory() # Creates a temporary directory for illustration purposes
268269 logger = DataLogger(output_directory=Path(tempdir.name), instance_name="my_name")
269270
270- # The DataLogger will create a new folder : 'tempdir/my_name_data_log' to store logged entries.
271+ # The DataLogger initialized above creates a new directory : 'tempdir/my_name_data_log' to store logged entries.
271272
272- # Before the DataLogger starts saving data, its saver processes need to be initialized.
273+ # Before the DataLogger starts saving data, its saver process needs to be initialized via the start() method.
274+ # Until the saver is initialized, the instance buffers all incoming data in RAM (via the internal Queue object),
275+ # which may eventually exhaust the available memory.
273276 logger.start()
274277
275- # The data can be submitted to the DataLogger via its input_queue. This property returns a multiprocessing Queue
276- # object.
278+ # Each call to the start() method must be matched with a corresponding call to the stop() method. This method shuts
279+ # down the logger process and releases any resources held by the instance.
280+ logger.stop()
281+ ```
282+
283+ #### Data Logging
284+ The DataLogger is explicitly designed to log serialized data of arbitrary size. To enforce the correct data formatting,
285+ all data submitted to the logger *** must*** be packaged into a __ LogPackage__ class instance before it is put into the
286+ DataLoger’s input queue.
287+ ```
288+ from pathlib import Path
289+ import tempfile
290+ import numpy as np
291+ from ataraxis_data_structures import DataLogger, LogPackage, assemble_log_archives
292+ from ataraxis_time import get_timestamp, TimestampFormats
293+
294+ if __name__ == "__main__":
295+ # Initializes and starts the DataLogger.
296+ tempdir = tempfile.TemporaryDirectory()
297+ logger = DataLogger(output_directory=Path(tempdir.name), instance_name="my_name")
298+ logger.start()
299+
300+ # The DataLogger uses a multiprocessing Queue to buffer and pipe the incoming data to the saver process. The queue
301+ # is accessible via the 'input_queue' property of each logger instance.
277302 logger_queue = logger.input_queue
278303
279- # The data to be logged has to be packaged into a LogPackage dataclass before being submitted to the Queue.
280- source_id = np.uint8(1) # Has to be an unit8
281- timestamp = np.uint64(tm.perf_counter_ns()) # Has to be an uint64
282- data = np.array([1, 2, 3, 4, 5], dtype=np.uint8) # Has to be an uint8 numpy array
304+ # The DataLogger is explicitly designed to log serialized data. All data submitted to the logger must be packaged
305+ # into a LogPackage instance to ensure that it adheres to the proper format expected by the logger instance.
306+ source_id = np.uint8(1) # Has to be an unit8 type
307+ timestamp = np.uint64(get_timestamp(output_format=TimestampFormats.INTEGER)) # Has to be an uint64 type
308+ data = np.array([1, 2, 3, 4, 5], dtype=np.uint8) # Has to be an uint8 NumPy array
283309 logger_queue.put(LogPackage(source_id, timestamp, data))
284310
285- # The timer used to timestamp the log entries has to be precise enough to resolve two consecutive datapoints
286- # (timestamps have to differ for the two consecutive datapoints, so nanosecond or microsecond timers are best).
287- timestamp = np.uint64(tm.perf_counter_ns())
311+ # The timer used to timestamp the log entries has to be precise enough to resolve two consecutive data entries.
312+ # Due to these constraints, it is recommended to use a nanosecond or microsecond timer, such as the one offered
313+ # by the ataraxis-time library.
314+ timestamp = np.uint64(get_timestamp(output_format=TimestampFormats.INTEGER))
288315 data = np.array([6, 7, 8, 9, 10], dtype=np.uint8)
289- logger_queue.put(LogPackage(source_id, timestamp, data)) # Same source id
316+ logger_queue.put(LogPackage(source_id, timestamp, data)) # Same source id as the package above
290317
291- # Shutdown ensures all buffered data is saved before the logger is terminated. This prevents all further data
292- # logging until the instance is started again.
318+ # Stops the data logger.
293319 logger.stop()
294320
295- # Verifies two .npy files were created, one for each submitted LogPackage. Note, DataLogger exposes the path to the
296- # log folder via its output_directory property.
321+ # The DataLogger saves the input LogPackage instances as serialized NumPy byte array .npy files. The output
322+ # directory for the saved files can be queried from the DataLogger instance's ' output_directory' property.
297323 assert len(list(logger.output_directory.glob("**/*.npy"))) == 2
324+ ```
298325
299- # The logger also provides a method for compressing all .npy files into .npz archives. This method is intended to be
300- # called after the 'online' runtime is over to optimize the memory occupied by data. To achieve minimal disk space
301- # usage, call the method with the remove_sources argument.
302- logger.compress_logs(remove_sources=True)
326+ #### Log Archive Assembly
327+ To optimize the log writing speed and minimize the time the data sits in the volatile memory, all log entries are saved
328+ to disk as separate NumPy array .npy files. While this format is efficient for time-critical runtimes, it is not
329+ optimal for long-term storage and data transfer. To help with optimizing the post-runtime data storage, the library
330+ offers the __ assemble_log_archives()__ function which aggregates .npy files from the same data source into an
331+ (uncompressed) .npz archive.
303332
304- # The compression creates a single .npz file named after the source_id
305- assert len(list(logger.output_directory.glob("**/*.npy"))) == 0
306- assert len(list(logger.output_directory.glob("**/*.npz"))) == 1
307333```
334+ from pathlib import Path
335+ import tempfile
336+ import numpy as np
337+ from ataraxis_data_structures import DataLogger, LogPackage, assemble_log_archives
338+
339+ if __name__ == "__main__":
340+
341+ # Creates and starts the DataLogger instance.
342+ tempdir = tempfile.TemporaryDirectory()
343+ logger = DataLogger(output_directory=Path(tempdir.name), instance_name="my_name")
344+ logger.start()
345+ logger_queue = logger.input_queue
346+
347+ # Generates and logs 255 data messages. This generates 255 unique .npy files under the logger's output directory.
348+ for i in range(255):
349+ logger_queue.put(LogPackage(np.uint8(1), np.uint64(i), np.array([i, i, i], dtype=np.uint8)))
308350
309- #### Log compression
310- To optimize runtime performance (log writing speed), all log entries are saved to disk as serialized NumPy arrays, each
311- stored in a separate .npy file. While this format is adequate during time-critical runtimes, it is not optimal for
312- long-term storage and data transfer.
351+ # Stops the data logger.
352+ logger.stop()
313353
314- To facilitate long-term log storage, the library exposes a global, multiprocessing-safe, and instance-independent
315- function ` compress_npy_logs() ` . This function behaves exactly like the instance-bound log compression method does, but
316- can be used to compress log entries without the need to have an initialized DataLogger instance. You can
317- use the ` output_directory ` property of an initialized DataLogger instance to get the path to the directory that stores
318- uncompressed log entries, which is a required argument for the instance-independent log compression function.
354+ # Depending on the runtime context, a DataLogger instance can generate a large number of individual .npy files as
355+ # part of its runtime. While having advantages for real-time data logging, this format of storing the data is not
356+ # ideal for later data transfer and manipulation. Therefore, it is recommended to always use the
357+ # assemble_log_archives() function to aggregate the individual .npy files into one or more .npz archives.
358+ assemble_log_archives(log_directory=logger.output_directory, remove_sources=True, memory_mapping=True, verbose=True)
319359
320- Alternatively, you can also use the ` compress_logs ` method exposed by the DataLogger instance to compress the logs
321- immediately after runtime. Overall, it is highly encouraged to compress the logs as soon as possible.
360+ # The archive assembly creates a single .npz file named after the source_id (1_log.npz), using all available .npy
361+ # files. Generally, each unique data source is assembled into a separate .npz archive.
362+ assert len(list(logger.output_directory.glob("**/*.npy"))) == 0
363+ assert len(list(logger.output_directory.glob("**/*.npz"))) == 1
364+ ```
322365
323366___
324367
0 commit comments