Output formatter for Parquet output. More...

#include <ParquetFormatter.h>

Inheritance diagram for ParquetFormatter:

Collaboration diagram for ParquetFormatter:

Data Structures
struct	Impl

Public Member Functions
bool	closeTag (std::ostream &into, const std::string &comment="") override
	Closes the most recently opened tag.

OutputFormatterType	getType () const
	Returns the type of formatter being used.

void	openTag (std::ostream &into, const std::string &xmlElement) override
	Keeps track of an open XML tag by adding a new element to the stack.

void	openTag (std::ostream &into, const SumoXMLTag &xmlElement) override
	Keeps track of an open XML tag by adding a new element to the stack.

	ParquetFormatter (const std::string &columnNames, const std::string &compression="", const int batchSize=1000000)
	Constructor.

void	setExpectedAttributes (const SumoXMLAttrMask &expected, const int depth) override
	Which elements are expected and which maximum depth the XML tree has.

template<class T >
void	writeAttr (std::ostream &, const std::string &attr, const T &val, const bool isNull, const bool)
	Writes a named attribute.

template<class T >
void	writeAttr (std::ostream &, const SumoXMLAttr attr, const T &val, const bool isNull, const bool)
	Writes a named attribute.

void	writeAttr (std::ostream &into, const std::string &attr, const double &val, const bool isNull, const bool escape)

void	writeAttr (std::ostream &into, const std::string &attr, const int &val, const bool isNull, const bool escape)

void	writeAttr (std::ostream &into, const SumoXMLAttr attr, const double &val, const bool isNull, const bool escape)
	typed overloads (non-template) – picked by overload resolution over the template

void	writeAttr (std::ostream &into, const SumoXMLAttr attr, const int &val, const bool isNull, const bool escape)

virtual void	writePadding (std::ostream &into, const std::string &val)
	Writes some whitespace to format the output. This method is only implemented for XML output.

virtual void	writePreformattedTag (std::ostream &into, const std::string &val)
	Writes a preformatted tag to the device but ensures that any pending tags are closed. This method is only implemented for XML output.

void	writeTime (std::ostream &into, const SumoXMLAttr attr, const SUMOTime val) override
	Writes a time value.

bool	writeXMLHeader (std::ostream &into, const std::string &rootElement, const std::map< SumoXMLAttr, std::string > &attrs, bool, bool) override
	Writes an "XML header".

bool	wroteHeader () const override
	Whether a complete row has been encountered and triggered writing.

	~ParquetFormatter () override
	Destructor (out-of-line: Impl is incomplete here)

Private Member Functions
void	writeNullAttr (const std::string &attr)

void	writeNullAttr (const SumoXMLAttr attr)

void	writeStringAttr (const std::string &attr, const std::string &val)

void	writeStringAttr (const SumoXMLAttr attr, const std::string &val)
	non-template helpers; defined in the .cpp where arrow/parquet are available

Private Attributes
std::unique_ptr< Impl >	myImpl

const OutputFormatterType	myType
	the type of formatter being used (XML, CSV, Parquet, etc.)

Detailed Description

Output formatter for Parquet output.

All arrow/parquet types are hidden in the Impl so that callers (notably OutputDevice.h) do not pay the include cost.

Definition at line 39 of file ParquetFormatter.h.

Constructor & Destructor Documentation

◆ ParquetFormatter()

ParquetFormatter::ParquetFormatter	(	const std::string &	columnNames,
		const std::string &	compression = `""`,
		const int	batchSize = `1000000`
	)

Constructor.

Definition at line 206 of file ParquetFormatter.cpp.

References myImpl, WRITE_ERRORF, and WRITE_WARNINGF.

◆ ~ParquetFormatter()

ParquetFormatter::~ParquetFormatter ( )

overridedefault

Destructor (out-of-line: Impl is incomplete here)

Member Function Documentation

◆ closeTag()

bool ParquetFormatter::closeTag	(	std::ostream &	into,
		const std::string &	comment = `""`
	)

overridevirtual

Closes the most recently opened tag.

This is where the main action starts. This function determines whether a row is completed and can be written.

Parameters

[in]	into	The output stream to use
[in]	comment	A comment to write after the tag (ignored for Parquet)

Returns: Whether a further element existed in the stack and could be closed

Implements OutputFormatter.

Definition at line 264 of file ParquetFormatter.cpp.

References myImpl, TLF, toString(), WRITE_ERRORF, and WRITE_WARNING.

◆ getType()

OutputFormatterType OutputFormatter::getType ( ) const

inlineinherited

Returns the type of formatter being used.

Returns: the formatter type

Definition at line 152 of file OutputFormatter.h.

References OutputFormatter::myType.

Referenced by OutputDevice::isXML(), OutputDevice::lf(), OutputDevice::writeAttr(), OutputDevice::writeFuncAttr(), and OutputDevice::writeOptionalAttr().

Here is the caller graph for this function:

◆ openTag() [1/2]

void ParquetFormatter::openTag	(	std::ostream &	into,
		const std::string &	xmlElement
	)

overridevirtual

Keeps track of an open XML tag by adding a new element to the stack.

Parameters

[in]	into	The output stream to use (unused)
[in]	xmlElement	Name of the element to open

Implements OutputFormatter.

Definition at line 252 of file ParquetFormatter.cpp.

References myImpl.

Referenced by writeXMLHeader().

Here is the caller graph for this function:

◆ openTag() [2/2]

void ParquetFormatter::openTag	(	std::ostream &	into,
		const SumoXMLTag &	xmlElement
	)

overridevirtual

Keeps track of an open XML tag by adding a new element to the stack.

Parameters

[in]	into	The output stream to use (unused)
[in]	xmlElement	Enum identifier of the element to open

Implements OutputFormatter.

Definition at line 258 of file ParquetFormatter.cpp.

References myImpl, and toString().

◆ setExpectedAttributes()

void ParquetFormatter::setExpectedAttributes	(	const SumoXMLAttrMask &	expected,
		const int	depth
	)

overridevirtual

Which elements are expected and which maximum depth the XML tree has.

This is not necessary for the functionality but very useful for debugging and tracking whether expected attributes are still missing (triggers an error in checkAttr). If expected is empty, no tracking takes place.

The depth parameter is only for performance. If a tag at this depth is closed for the first time, the header is being written and buffered rows may be flushed. Setting it to a large value (which is also the default) means you are on the safe side if more attributes or tags show up later but it may result in buffering the complete output before writing the first line. Setting it to 0 triggers auto detection which means the first time a tag is closed the maximum depth will be determined.

Parameters

[in]	expected	The enum values of the attrs which should be present before a row can be written.
[in]	depth	The maximum expected depth of nested XML elements.

Reimplemented from OutputFormatter.

Definition at line 438 of file ParquetFormatter.cpp.

References myImpl.

◆ writeAttr() [1/6]

template<class T >

void ParquetFormatter::writeAttr	(	std::ostream &	,
		const std::string &	attr,
		const T &	val,
		const bool	isNull,
		const bool
	)

inline

Writes a named attribute.

Parameters

[in]	into	The output stream to use (unused)
[in]	attr	The attribute (name as string value)
[in]	val	The attribute value
[in]	isNull	whether this actually a null value (adds nullptr to myValues)

Definition at line 109 of file ParquetFormatter.h.

References toString(), writeNullAttr(), and writeStringAttr().

◆ writeAttr() [2/6]

template<class T >

void ParquetFormatter::writeAttr	(	std::ostream &	,
		const SumoXMLAttr	attr,
		const T &	val,
		const bool	isNull,
		const bool
	)

inline

Writes a named attribute.

Parameters

[in]	into	The output stream to use (unused)
[in]	attr	The attribute (name as enum value)
[in]	val	The attribute value
[in]	isNull	whether this actually a null value (adds nullptr to myValues)

Definition at line 93 of file ParquetFormatter.h.

References toString(), writeNullAttr(), and writeStringAttr().

Referenced by writeXMLHeader().

Here is the caller graph for this function:

◆ writeAttr() [3/6]

void ParquetFormatter::writeAttr	(	std::ostream &	into,
		const std::string &	attr,
		const double &	val,
		const bool	isNull,
		const bool	escape
	)

Definition at line 367 of file ParquetFormatter.cpp.

References myImpl.

◆ writeAttr() [4/6]

void ParquetFormatter::writeAttr	(	std::ostream &	into,
		const std::string &	attr,
		const int &	val,
		const bool	isNull,
		const bool	escape
	)

Definition at line 380 of file ParquetFormatter.cpp.

References myImpl.

◆ writeAttr() [5/6]

void ParquetFormatter::writeAttr	(	std::ostream &	into,
		const SumoXMLAttr	attr,
		const double &	val,
		const bool	isNull,
		const bool	escape
	)

typed overloads (non-template) – picked by overload resolution over the template

Definition at line 346 of file ParquetFormatter.cpp.

References myImpl, SUMO_ATTR_X, and SUMO_ATTR_Y.

◆ writeAttr() [6/6]

void ParquetFormatter::writeAttr	(	std::ostream &	into,
		const SumoXMLAttr	attr,
		const int &	val,
		const bool	isNull,
		const bool	escape
	)

Definition at line 359 of file ParquetFormatter.cpp.

References myImpl.

◆ writeNullAttr() [1/2]

void ParquetFormatter::writeNullAttr ( const std::string & attr )

private

Definition at line 412 of file ParquetFormatter.cpp.

References myImpl.

◆ writeNullAttr() [2/2]

void ParquetFormatter::writeNullAttr ( const SumoXMLAttr attr )

private

Definition at line 404 of file ParquetFormatter.cpp.

References myImpl.

Referenced by writeAttr(), and writeAttr().

Here is the caller graph for this function:

◆ writePadding()

virtual void OutputFormatter::writePadding	(	std::ostream &	into,
		const std::string &	val
	)

inlinevirtualinherited

Writes some whitespace to format the output. This method is only implemented for XML output.

Parameters

[in]	into	The output stream to use
[in]	val	The whitespace

Reimplemented in PlainXMLFormatter.

Definition at line 138 of file OutputFormatter.h.

References UNUSED_PARAMETER.

Referenced by OutputDevice::writePadding().

Here is the caller graph for this function:

◆ writePreformattedTag()

virtual void OutputFormatter::writePreformattedTag	(	std::ostream &	into,
		const std::string &	val
	)

inlinevirtualinherited

Writes a preformatted tag to the device but ensures that any pending tags are closed. This method is only implemented for XML output.

Parameters

[in]	into	The output stream to use
[in]	val	The preformatted data

Reimplemented in PlainXMLFormatter.

Definition at line 127 of file OutputFormatter.h.

References UNUSED_PARAMETER.

Referenced by OutputDevice::writePreformattedTag().

Here is the caller graph for this function:

◆ writeStringAttr() [1/2]

void ParquetFormatter::writeStringAttr	(	const std::string &	attr,
		const std::string &	val
	)

private

Definition at line 396 of file ParquetFormatter.cpp.

References myImpl.

◆ writeStringAttr() [2/2]

void ParquetFormatter::writeStringAttr	(	const SumoXMLAttr	attr,
		const std::string &	val
	)

private

non-template helpers; defined in the .cpp where arrow/parquet are available

Definition at line 388 of file ParquetFormatter.cpp.

References myImpl.

Referenced by writeAttr(), writeAttr(), and writeTime().

Here is the caller graph for this function:

◆ writeTime()

void ParquetFormatter::writeTime	(	std::ostream &	into,
		const SumoXMLAttr	attr,
		const SUMOTime	val
	)

overridevirtual

Writes a time value.

Currently this writes a string if human readable times are activated and a double otherwise

Parameters

[in]	into	The output stream to use (unused)
[in]	attr	The attribute (name as enum value)
[in]	val	The attribute value

Todo:: use one of Parquet's time types

Implements OutputFormatter.

Definition at line 420 of file ParquetFormatter.cpp.

References gHumanReadableTime, myImpl, STEPS2TIME, time2string(), and writeStringAttr().

◆ writeXMLHeader()

bool ParquetFormatter::writeXMLHeader	(	std::ostream &	into,
		const std::string &	rootElement,
		const std::map< SumoXMLAttr, std::string > &	attrs,
		bool	,
		bool
	)

overridevirtual

Writes an "XML header".

For Parquet output the header is only relevant if it contains additional attributes.

Parameters

[in]	into	The output stream to use
[in]	rootElement	The root element to use
[in]	attrs	Additional attributes to save within the rootElement

Returns: whether something has been written

Reimplemented from OutputFormatter.

Definition at line 234 of file ParquetFormatter.cpp.

References myImpl, openTag(), SUMO_ATTR_SCHEMA_LOCATION, SUMO_ATTR_XMLNS, and writeAttr().

◆ wroteHeader()

bool ParquetFormatter::wroteHeader ( ) const

overridevirtual

Whether a complete row has been encountered and triggered writing.

Returns: Whether the Parquet writer has been initialized and a first row has been written

Implements OutputFormatter.

Definition at line 432 of file ParquetFormatter.cpp.

References myImpl.

Field Documentation

◆ myImpl

std::unique_ptr<Impl> ParquetFormatter::myImpl

private

Definition at line 165 of file ParquetFormatter.h.

Referenced by closeTag(), openTag(), openTag(), ParquetFormatter(), setExpectedAttributes(), writeAttr(), writeAttr(), writeAttr(), writeAttr(), writeNullAttr(), writeNullAttr(), writeStringAttr(), writeStringAttr(), writeTime(), writeXMLHeader(), and wroteHeader().

◆ myType

const OutputFormatterType OutputFormatter::myType

privateinherited

the type of formatter being used (XML, CSV, Parquet, etc.)

Definition at line 170 of file OutputFormatter.h.

Referenced by OutputFormatter::getType().

The documentation for this class was generated from the following files:

Data Structures

Public Member Functions

Private Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ ParquetFormatter()

◆ ~ParquetFormatter()

Member Function Documentation

◆ closeTag()

◆ getType()

◆ openTag() [1/2]

◆ openTag() [2/2]

◆ setExpectedAttributes()

◆ writeAttr() [1/6]

◆ writeAttr() [2/6]

◆ writeAttr() [3/6]

◆ writeAttr() [4/6]

◆ writeAttr() [5/6]

◆ writeAttr() [6/6]

◆ writeNullAttr() [1/2]

◆ writeNullAttr() [2/2]

◆ writePadding()

◆ writePreformattedTag()

◆ writeStringAttr() [1/2]

◆ writeStringAttr() [2/2]

◆ writeTime()

◆ writeXMLHeader()

◆ wroteHeader()

Field Documentation

◆ myImpl

◆ myType