Formatted Raw Datalog (FRD) file format specification

Document purpose:

This document specifies the FRD file format

Format purpose:

This FRD file format was conceived to standardize a way of passing unprocessed (or semi-processed) datalogs from a simple embedded datalog recording device to a specialized datalog viewing software.

Authors:

Alex Babkin

Philip Tobin

Revision history:

Rev. 0.1 - Jan 26, 2009 - Alex Babkin, Phil Tobin – Finalized version 1 spec.

Rev. 0.2 - Jan 26, 2009 - Phil Tobin – Added tables to make layout clearer.

Trademarks:

* Megasquirt is a trademark of Al Grippo and Bruce Bowling

Definitions:

Controller: any version of Megasquirt* engine management system device

· output: single timestamped burst of uniformly structured data reflecting the controller's state at a particular point in time

· Datalog: a sequence of recorded output'es

· Datalog session: time interval between datalog-on (begin datalog) and datalog-off (stop datalog) user actions

· UNIX Standard 32bit timestamp: number of seconds elapsed since midnight Coordinated Universal Time (UTC) of January 1, 1970, not counting leap seconds

Format structure:

One FRD file associates with (and therefore contains data of) necessarily one datalog session.

FRD file is made up of two sections:

1. Header

2. Formatted Datalog Stream (FDS)

Header

FDS

….

(1.) physically precedes (2.) in terms of position in the datalog file (more rigorously: lseek positions of any bytes of (2.) are all necessarily after the lseek positions of any bytes in (1.) )

1. Header

1.1 Purpose

Header is a data structure containing information about the particular datalog session that takes place in the FDS section of the file (the same file containing the header). This information is extracted from the firmware running on MS unit immediately prior to start of the datalog session and passed, by means of the header section in the FRD file, to the software, which interprets the raw datalog file and converts it into a usable datalog format like the currently used Megasquirt* specific delimited ASCII format.

1.2 Header format fields

- File format (6 bytes): unique identifier for FRD file format (same for any *.frd file)

- Format version (2 bytes): unique identifier of the particular version of the FRD file format

- Date/Time (4 bytes): date/time encoded to Unix 32bit standard (1 second resolution) – This is an Optional field, fill with 0’s if correct time stamp unavailable.

- Controller firmware signature (63 bytes): unique identifier of the firmware version used for the datalog session

- FDS start offset (4 bytes): offset in bytes of where FDS starts in the file, relative to the beginning of the file

- output length (2 bytes): length of output in bytes

Field Name	Offset	Length (bytes)	Required	Value
File Format	0	6	Yes	FRD padded by 0 or in hex x46 x52 x44 x00 x00 x00
Format Version	6	2	Yes	Currently x00 x01, increase values may be used in the future as the format changes
Time Stamp	8	4	No	Unix 32 bit timestamp. Time in seconds since the epoch. If TimeStamp is not available set to all 0’s - x00 x00 x00 x00
Firmware Signature	12	63	Yes	The results of the controller query command. Check you firmware. For mre than one signature if there are CAN controllers, separate with a single x00
Data Begin Index	75	4	Yes	The address of the 1^st byte after the header, with the current format this will always be 81 or x51
Output Length	79	2	Yes	The length of a single outpc or och read

2. Formatted Datalog Stream (FDS)

2.1 Purpose

Contains actual datalog and time markers

2.2 FDS structure

Consists of "back-to-back" sequence of type-data pairs.

2.2.1 Type-Data pair format

- Type (1 byte): unique identifier describing the structure, contents and length of the following data

- Data (length is determined by the type): single data item characterized by the preceding type identifier

Field Name	Offset	Length (bytes)	Required	Value
Block Type	0	1	Yes	Identifier for the type of block, a standard outpc read is type 1 or x01
Counter	1	1	Yes	A rolling 1 byte block counter. Just rolls from 0 to 255 or xFF
Data block	2	Variable	Yes	Contains different data depending on type. See valid types below.

2.2.1.1 Type: output

id: 1

data length: 2 + output length specified in the Header

data format fields:

- running outpc index (1 byte): = output# % 256

- raw output data verbatim as received after sending the output-command to the controller

2.2.1.2 Type: Marker

id: 2

data length: 4 bytes

data description/format: Date/Time timestamp encoded in standard Unix 32bit format – This is an Optional field, fill with 0’s if correct time stamp unavailable.