Skip to content

Create a new Framesoc Importer

Jump to bottom Edit New page
ycorre edited this page Oct 27, 2015 · 4 revisions

Summary

This page describes how to create a new Framesoc importer. An importer is a plugin who is responsible for converting a trace in a given format into the Framesoc data model. This conversion step is very important since it will define the semantic of the data of the trace.

This tutorial has been developed using Eclipse Luna and Framesoc v1.0.9. The corresponding source code is available in the fr.inria.soctrace.framesoc.tutorials.importer plugin of this repository.

Requirements

Introduction

In this tutorial, we will create a small importer that can read trace data stored in a text file and formatted in CSV.

A Framesoc importer is an Eclipse plugin extending the fr.inria.soctrace.framesoc.core.tool extension point defined in the fr.inria.soctrace.framesoc.core plugin. The plugin will be able to read one or more traces stored in external file(s) and will build the corresponding trace in the Framesoc datamodel. The resulting imported trace is stored in a database (SQLite or MySQL, depending on the Framesoc configuration).

Trace format

For this tutorial, we will use a very simple fictive trace format. The trace will be provided as plain text formatted in CSV. Each line will represent one record of the trace and there are two kinds of records. The first one describes the event producers, which represent the entities producing the events (e.g. processes, threads, computers, etc.). An event producer is described in the following way:

P; parent; name

where P is a tag specifying that this line describes an event producer, parent is the name of the parent event producer (since event producers can form a hierarchy) and name is the name of the event producer.

Events are described in the following way:

E; eventProducer; eventType; timestamp

where E is a tag specifying that this line describes an event, eventProducer the name of the event producer that has produced the event, eventType is the type of the event and timestamp is the date when the event occurred.

Here is a short example of what a trace in that format would look line. We can use it at the end to test our importer:

P;root;process1
P;process1;process1_0
E;process1_0;read;511
E;process1_0;write;489
E;process1_0;read;8878
E;process1_0;read;4997
P;process1_0;process1_1
E;process1_1;read;551
E;process1_1;write;449
E;process1_0;read;6978
E;process1_1;read;5897
P;root;process2
P;process2;process2_0
P;process2;process2_1
E;process2_1;read;411
E;process2_0;write;889
E;process2_1;read;7978
E;process2_1;write;4197
E;process2_0;read;211
E;process2_1;write;489
E;process2_0;read;8078
E;process2_0;write;5697

Tutorial

Create the Framesoc importer plugin

  • Create a new plugin: File -> New -> Plug-in Project.

    • Specify as project name fr.inria.soctrace.framesoc.tutorials.importer and press Next.
    • Do not change anything and press Next again.
    • Uncheck the Create a plug-in using one of the templates option, then press Finish.

  • Open the file META-INF/MANIFEST.MF, if it has not already opened automatically, to add the required dependencies.

    • This file is normally opened using the Plug-in Manifest Editor.

    • In this editor, select the Dependencies tab and press Add.

    • A dialog is shown: in the text field write fr.inria.soctrace.lib and add all the plugins of the Framesoc library.

    • Press Add again, and this time write fr.inria.soctrace.framesoc in order to add the plugins fr.inria.soctrace.framesoc.core and fr.inria.soctrace.framesoc.ui.

    • At this point your list of required plugins should contain:

      org.eclipse.ui
org.eclipse.core.runtime
fr.inria.soctrace.lib.model
fr.inria.soctrace.lib.query
fr.inria.soctrace.lib.search
fr.inria.soctrace.lib.slf4j
fr.inria.soctrace.lib.storage
fr.inria.soctrace.lib.utils
fr.inria.soctrace.framesoc.core
fr.inria.soctrace.framesoc.ui

    • Save the MANIFEST.MF. Note: it is a good practice to save this file immediately after each modification, in order to take advantage at the highest level of the automatic code generation (see below). For the rest of this tutorial, it will be assumed that this file is saved after each modification.

  • Create the importer extension.

    • Open the Extensions tab and press Add.
    • Write fr.inria.soctrace.framesoc.core.tool in the filter, and select the corresponding extension point.
    • Press Finish
    • Right-click on the extension you just created (in the list box), then New -> tool.

  • The documentation of the extension point fields can be accessed selecting the fr.inria.soctrace.framesoc.core.tool line and clicking on Show extension point description on the right.

    id - The ID of the importer. This is NOT the database ID. It is an identifier having the same format of java packages (i.e., x.y.z.k). Note that each importer must have a globally unique id in the system.
class - The main class of the importer, extending the FramesocTool abstract class. See its documentation for more details.
type - The type of the importer.
name - The name of the importer. Note that each importer must have a globally unique name in the system.
doc - Launching documentation. Optional.

  • Fill the the extension element details as follow:

    • id: fr.inria.soctrace.framesoc.tutorials.importer.example
    • class: fr.inria.soctrace.framesoc.tutorials.importer.ExampleImporter
    • type: IMPORT
    • name: Framesoc Tutorial Importer
    • doc: Select a file and click launch.

  • Create the class needed by the extension.

    • Click on the class link.

    • Do not change anything and press Finish

    • The following class should be automatically created:

      package fr.inria.soctrace.framesoc.tutorials.importer;

import fr.inria.soctrace.framesoc.core.tools.model.FramesocTool;
import fr.inria.soctrace.framesoc.core.tools.model.IFramesocToolInput;

public class ExampleImporter extends FramesocTool {

    public ExampleImporter() {
	    // TODO Auto-generated constructor stub
    }

    @Override
    public void launch(IFramesocToolInput input) {
	    // TODO Auto-generated method stub
    }
}

    • Add the following line to the launch() method body:

      System.out.println("Hello World!");

    • In order to check that everything is correctly setup, try to launch the Eclipse Application and test the tool.

      • Right-click on the importer plugin (in the Package explorer on the left), and select Run as -> Eclipse Application.
      • Framesoc -> Trace Analysis -> Import Trace.
      • Select the Framesoc Tutorial Importer and press OK.
      • In the console of the development Eclipse, you should see Hello World! (Don't worry if it appears more than once).

Get the input

Our importer takes as input a simple text file formatted in CSV, so our importer needs to check that a trace file was provided and see if this file is valid. In order to do this, we override the method canLaunch(), which takes as input a file or a list of files, checks the validity of the input and returns a boolean specifying whether the provided input is valid. As long as the canLaunch() method returns false, the validating OK button remains disabled and a message specifying the error can be displayed to provide a feedback to the user.

If the input is more complex than one file or a list of files, or if some options/arguments must be provided, then Framesoc also provides an extension point to build a more complex input interface. However this is not covered in this tutorial, but explanations can be found in section Getting the input of the Framesoc tool tutorial.

For now, we will just check that at least one path to a file that exists and that can be read is provided as input. This is done by adding the following method:

  @Override
  public ParameterCheckStatus canLaunch(IFramesocToolInput input) {
  	// FileInput is the default format for the input of the importer 
  	FileInput args = (FileInput) input;

  	// Check that there is at least one file
  	if (args.getFiles().size() == 0) {
  		return new ParameterCheckStatus(false,
  				"Specify a trace file.");
  	}

  	List<String> theFiles = args.getFiles();
  	ParameterCheckStatus status = new ParameterCheckStatus(true, "");
  	// Check that each file provided exists and can be read 
  	for (String aFileName : theFiles) {
  		File aFile = new File(aFileName);
  		if (!aFile.exists() || !aFile.canRead()) {
  			status.valid = false;
  			status.message = "Invalid trace or illegal arguments passed";
  			return status;
  		}
  	}
  	return status;
  }

If you wish, you can re-run the application to check that the importer can only be launched when a path to a readable file is provided.

Importing the trace

The trace importation consists in several steps:

  1. Open a connection to the Framesoc database and create a new trace database.
  2. Open the trace file.
  3. Read the trace and build the corresponding entities in the Framesoc data model.
  4. Save the newly created elements in the database.

But first, since importing a trace can be a long operation, it must be performed in a concurrent way in order to avoid blocking the interface for the user. Eclipse provides the class Job that allows us to do exactly that. In order to perform the importation processing with this class, we will add a private class TutorialImporterPluginJobBody that implements the interface IPluginToolJobBody. This class must implement the run() method which will perform the importation. The run() method takes as argument a monitor (IProgressMonitor) that can be used to show the user the progress of the importation as well as providing a way for the user to cancel the importation (more explanations about the monitor is provided in section Monitor and cancellation).

Trace database name

In order for the trace database name to be unique, Framesoc provides a method to provide the imported trace with a unique name. This method takes as arguments a prefix (which is used to identify all the traces imported by the importer) and appending it with a timestamp dating the importation of the trace.

  // Use the default trace name provider
  String traceDbName = FramesocManager.getInstance()
  	.getTraceDBName(TRACE_PREFIX);

Of course, using the name provider of Framesoc is optional, and the name of the trace database can be any valid String as long as there is no other trace database with the same name.

System and trace databases

Inside the job, we must instantiate a connection to the Framesoc System database and create a new trace database. The System database is unique and references all the registered traces, but also all the registered tools (including the importers). Consequently, it is accessed in Framesoc through a singleton class that we will use to register our newly imported trace into it.

A trace database is the database object that contains an imported trace. It stores all the events, their types and parameters, all the event producers and potential extra-information on the trace (flag specifying if the trace is the result of an analysis of an existing trace, path to the original trace file(s), etc.). The databases are instantiated like this:

  // Open the System Database
  SystemDBObject systemDB = SystemDBObject.openNewInstance();
  // Create a new Trace Database with the generated name
  TraceDBObject traceDB = new TraceDBObject(traceDbName, DBMode.DB_CREATE);

Once all the importing operations are over, it is essential to close the connections to the databases in order to avoid locking out other tools that would require connections to these databases. One way to make sure that the connections to the database will be closed even if an exception occurs during the importation (e.g. because of an error in the imported trace, cancellation by the user, etc.), is to perform the closing operation inside a block finally which will always be executed, even after a return statement, when the associated try block exits. The connections to the databases are closed with the following methods:

  // Close the trace DB and the system DB (commit)
  DBObject.finalClose(traceDB);
  DBObject.finalClose(sysDB);

The only remaining operation to add in our job is to instantiate the parser (that we will implement in a moment), that takes as inputs the database objects and the trace we want to import.

So if we combine all the operations described above, here is the full code of our job:

  public class TutorialImporterPluginJobBody implements IPluginToolJobBody {

  	// Input provided through the GUI
  	private FileInput input;

  	public TutorialImporterPluginJobBody(IFramesocToolInput input) {
  		this.input = (FileInput) input;
  	}

  	@Override
  	public void run(IProgressMonitor monitor) throws SoCTraceException {

  		// Get all the files provided as arguments
  		List<String> traces = input.getFiles();

  		// For each file
  		for (String traceFile : traces) {

  			// Use the default trace name provider
  			String traceDbName = FramesocManager.getInstance()
  					.getTraceDBName(TRACE_PREFIX);

  			SystemDBObject systemDB = null;
  			TraceDBObject traceDB = null;

  			try {
  				// Open the System Database
  				systemDB = SystemDBObject.openNewInstance();
  				// Create a new Trace Database with the generated name
  				traceDB = new TraceDBObject(traceDbName, DBMode.DB_CREATE);

  				// Instantiate the parser and launch the parsing
  				TutorialParser parser = new TutorialParser(systemDB, traceDB,
  						traceFile);
  				parser.parseTrace(monitor);

  			} catch (SoCTraceException e) {
  				e.printStackTrace();
  				// Use the default import exception handler
  				PluginImporterJob.catchImporterException(e, systemDB, traceDB);
  			} finally {
  				// Close the trace DB and the system DB (commit)
  				DBObject.finalClose(traceDB);
  				DBObject.finalClose(systemDB);
  			}
  		}
  	}
  }

We also need to modify the launch() method in order to run our Job. This is done by calling the method schedule() which adds the job to a waiting queue and executes it when it is at the front of the queue. The method setUser() is just used to specify that the user initiated the action, which will provide a different display in the UI.

  @Override
  public void launch(IFramesocToolInput input) {
  	PluginImporterJob job = new PluginImporterJob("Tutorial Importer",
  			new TutorialImporterPluginJobBody(input));
  	job.setUser(true);
  	job.schedule();
  }

The parser

Now that everything is set up, it is time to perform the actual importation. We will create a new class called TutorialParser, that will perform all the importing operations. The parsing can be decomposed into two operations:

  1. Reading the file(s) and processing it accordingly.
  2. Saving the processed data into the database.

The reading part in our case consists in reading the text file line by line and, for each line, splitting it into the different elements that are separated by a comma. Then, we have to determine if the current line represents the declaration of an Event Producer or an Event. In our format, this is specified by the first element in the line: P for event producer and E for event (alternatively, in our case we could have counted the number of elements on one line: three for a producer and four for an event). So for example, after storing each elements of the line into an array, the instantiation of an event producer is done that way:

  // Instantiate the event producer with a unique ID
  EventProducer ep = new EventProducer(epIdManager.getNextId());
  ep.setName(line[2]);
  ep.setParentId(getEventProducerId(line[1]));
  ep.setLocalId(String.valueOf(ep.getId()));  
  producersMap.add(put(ep.getName(), ep);

Note the presence of an ID manager when instantiating a new Event Producer. This is because all the Event Producers must have a unique ID to identify them in the database. Framesoc provides a class IdManager which is in charge of providing unique IDs for events, events producers and more generally any object requiring a unique ID. You can also note that in our input file format we have a root element as first producer, which is not defined. We will suppose that it is an event producer that is always at the top of the Event producer hierarchy and thus it can be created beforehand in the code, prior to the parsing of the trace file.

Event categories

Before seeing how to instantiate an event, let's see how an event is defined in Framesoc (more information about the Framesoc data model can be found in the technical report SoC-Trace Infrastructure (Inria RT-427)). An event represents something happening in the trace. An event is produced by an identified event producer, happens at a defined timestamp and has a given type. Events can be one of four categories: Punctual Event, State, Link or Variable. The object representation of the different categories of event is identical in the database, the only difference is the interpretation made by Framesoc of two of their fields: a long and a double.

  1. A Punctual Event is the base type of event and the values of the long and the double have no defined interpretation.
  2. A State represents a duration and thus the long value is the end date of the State and the double is the level of imbrication (since states can be imbricated). Note that the end timestamp must be greater or equal to the beginning date, since a negative duration might not be handled correctly by the tools in Framesoc.
  3. A Link represent a link between two event producers (e.g. a communication), the long value being the end date of the link and the double is the ID of the event producer on the receiving end.
  4. A Variable represents a value over time, the long being the ending timestamp of the variable and the double is used to store the value held by the variable.

In our format, we only have punctual events. Consequently events will be instantiate this way:

  // Instantiate with a unique ID
  Event e = new Event(eIdManager.getNextId());
  e.setEventProducer(producersMap.get(line[1]));
  e.setTimestamp(Long.valueOf(line[3]));
  e.setType(getType(line[2], EventCategory.PUNCTUAL_EVENT));
  e.setPage(page);
  eventList.add(e);

The Event Type is also another characteristic that must be stored and later associated to an Event. They can be created on-the-fly, as illustrated in the method below:

  private EventType getType(String name, int category) {
  	// If the type does not exist
  	if (!types.containsKey(name)) {
  		// Create it
  		EventType et = new EventType(etIdManager.getNextId(), category);
  		et.setName(name);
  		types.put(name, et);
  	}
  	return types.get(name);
  }

Saving into database

Once an object is instantiated (event, event producer or event type), it must be saved into the trace database. This is performed through the method:

  traceDB.save(instantiatedObject);

However this command only saves the event, but in order to definitively write the saved data into the database, a commit operation must be performed:

  traceDB.commit();

Since a commit operation is quite costly, it is recommended not to do it too often. But committing too many events at once is also time-consuming. From our experience, we have concluded that performing a commit operation about every 25 000 events provides the best performances. It is therefore necessary to implement a counter of the saved events in order to periodically perform a commit.

Trace metadata

Once all the trace data has been committed into the trace database, it is possible to add some extra data about the trace. To do that we add a new class, TutorialTraceMetadata, that extends the class AbstractTraceMetadataManager. It must contain a method setTraceFields(Trace trace) that will set the trace metadata:

  @Override
  public void setTraceFields(Trace trace) {
  	trace.setAlias(alias);
  	trace.setDbName(dbName);
  	trace.setDescription("Example trace imported " + getCurrentDate());
  	trace.setNumberOfCpus(1);
  	trace.setNumberOfEvents(events);
  	trace.setOutputDevice("Example trace importer");
  	trace.setProcessed(false);
  	trace.setTracedApplication("unknown");
  	trace.setBoard("unknown");
  	trace.setOperatingSystem("unknown");
  }

In our parser, we will instantiate this class and provide its constructor with the values of the metadata. Then we will save those values by calling the methods createMetadata() and saveMetadata():

	TutorialTraceMetadata metadata = new TutorialTraceMetadata(sysDB, traceDB.getDBName(),
  		realAlias, numberOfEvents, minTimestamp, maxTimestamp, timeUnit);
  metadata.createMetadata();
  metadata.saveMetadata();

For some of the metadata, if no value is provided, then Framesoc will try to complete them automatically, but at the cost of potentially time-consuming operations. These trace metadata are: the minimum and maximum timestamps, the number of events and number of event producers.

The complete implementation of the importer can be found in the project directory fr.inria.soctrace.framesoc.tutorials.importer.

Monitor and cancellation

In this section, we will talk about the progress monitor mentioned earlier. The monitor has two functions: the first one is to give feedback to the user that the importation is still in progress and potentially quantify that progress. The second function is to provide a way to the user to cancel the importation. In the importer, the monitor is instantiated automatically when calling the method run() in our implementation of the Job.

If we know how many events we have to process, then it is possible to specify in the monitor the quantity of work to be done and to update the quantity of processed work periodically. These operations are performed through the methods beginTask(String taskName, int quantityOfWork) and worked(int numberOfWorkDone). Another way to provide feedback to the user through the monitor is to display what is the current task ("Processing events", "Saving trace metadata", etc.), through the method subTask(String taskName).

The monitor also offers to the user the possibility to cancel the importation. It is thus necessary to check periodically if a cancellation has been requested with the method isCanceled(). If a cancellation has been requested, then a number of operations should be done in order to cleanly stop the importation. It is up to the importer programmers to decide if they wish to keep the partially imported trace or if they want to discard it completely. In any case, connections to the database should be closed.

Note that these operations on the monitor should not be performed too often, otherwise it would impact the performances. A good practice is to update and to check for cancellation when committing data to the database and after every change in the importing operations.

Test the parser

Now that we have implemented our parser, we can launch Framesoc and try to import the example trace. If everything goes well, the imported trace should appear in the trace viewer (on the left of the Eclipse window) in the folder TutorialTrace. You can check that the displayed metadata are correct, and when opening the trace with the Gantt, you should see something like this:

Imported trace in the Gantt View

Conclusion

In this tutorial, we have seen how to create a simple importer and studied the most important operations that should be performed by the importer. You should of course adapt the processing part of your importer to your trace format. Other parsing techniques are used by the other Framesoc importers and it can be a good practice to study how they perform their importations.

Add a custom footer
Add a custom sidebar
Clone this wiki locally