Output formatter for Parquet output. More...

#include <ParquetFormatter.h>

Inheritance diagram for ParquetFormatter:

Collaboration diagram for ParquetFormatter:

Data Structures
struct	Impl

Public Member Functions
bool	closeTag (std::ostream &into, const std::string &comment="") override
	Closes the most recently opened tag and optinally add a comment.

OutputFormatterType	getType ()
	Returns the type of formatter being used.

void	openTag (std::ostream &into, const std::string &xmlElement) override
	Opens an XML tag.

void	openTag (std::ostream &into, const SumoXMLTag &xmlElement) override
	Opens an XML tag.

	ParquetFormatter (const std::string &columnNames, const std::string &compression="", const int batchSize=1000000)
	Constructor.

void	setExpectedAttributes (const SumoXMLAttrMask &expected, const int depth=2) override
	Set the expected attributes to write. This is used for tracking which attributes are expected in table like outputs. This should be not necessary but at least in the initial phase of implementing CSV and Parquet it helps a lot to track errors.

template<class T >
void	writeAttr (std::ostream &, const std::string &attr, const T &val, const bool isNull)

template<class T >
void	writeAttr (std::ostream &, const SumoXMLAttr attr, const T &val, const bool isNull)
	writes a named attribute

void	writeAttr (std::ostream &into, const std::string &attr, const double &val, const bool isNull)

void	writeAttr (std::ostream &into, const std::string &attr, const int &val, const bool isNull)

void	writeAttr (std::ostream &into, const SumoXMLAttr attr, const double &val, const bool isNull)
	typed overloads (non-template) – picked by overload resolution over the template

void	writeAttr (std::ostream &into, const SumoXMLAttr attr, const int &val, const bool isNull)

virtual void	writePadding (std::ostream &into, const std::string &val)
	Writes some whitespace to format the output. This method is only implemented for XML output.

virtual void	writePreformattedTag (std::ostream &into, const std::string &val)
	Writes a preformatted tag to the device but ensures that any pending tags are closed. This method is only implemented for XML output.

void	writeTime (std::ostream &into, const SumoXMLAttr attr, const SUMOTime val) override

virtual bool	writeXMLHeader (std::ostream &into, const std::string &rootElement, const std::map< SumoXMLAttr, std::string > &attrs, bool writeMetadata, bool includeConfig)
	Writes an XML header with optional configuration.

bool	wroteHeader () const override
	Returns whether a header has been written. Useful to detect whether a file is being used by multiple sources.

	~ParquetFormatter () override
	Destructor (out-of-line: Impl is incomplete here)

Private Member Functions
void	writeNullAttr (const std::string &attr)

void	writeNullAttr (const SumoXMLAttr attr)

void	writeStringAttr (const std::string &attr, const std::string &val)

void	writeStringAttr (const SumoXMLAttr attr, const std::string &val)
	non-template helpers; defined in the .cpp where arrow/parquet are available

Private Attributes
std::unique_ptr< Impl >	myImpl

const OutputFormatterType	myType
	the type of formatter being used (XML, CSV, Parquet, etc.)

Detailed Description

Output formatter for Parquet output.

All arrow/parquet types are hidden in the Impl so that callers (notably OutputDevice.h) do not pay the include cost.

Definition at line 39 of file ParquetFormatter.h.

Constructor & Destructor Documentation

◆ ParquetFormatter()

ParquetFormatter::ParquetFormatter	(	const std::string &	columnNames,
		const std::string &	compression = `""`,
		const int	batchSize = `1000000`
	)

Constructor.

Definition at line 200 of file ParquetFormatter.cpp.

References myImpl, WRITE_ERRORF, and WRITE_WARNINGF.

◆ ~ParquetFormatter()

ParquetFormatter::~ParquetFormatter ( )

overridedefault

Destructor (out-of-line: Impl is incomplete here)

Member Function Documentation

◆ closeTag()

bool ParquetFormatter::closeTag	(	std::ostream &	into,
		const std::string &	comment = `""`
	)

overridevirtual

Closes the most recently opened tag and optinally add a comment.

Parameters

[in] into The output stream to use

Returns: Whether a further element existed in the stack and could be closed

Todo:: it is not verified that the topmost element was closed

Implements OutputFormatter.

Definition at line 252 of file ParquetFormatter.cpp.

References myImpl, toString(), WRITE_ERRORF, and WRITE_WARNING.

◆ getType()

OutputFormatterType OutputFormatter::getType ( )

inlineinherited

Returns the type of formatter being used.

Returns: the formatter type

Definition at line 152 of file OutputFormatter.h.

References OutputFormatter::myType.

Referenced by OutputDevice::isXML(), OutputDevice::writeAttr(), OutputDevice::writeFuncAttr(), and OutputDevice::writeOptionalAttr().

Here is the caller graph for this function:

◆ openTag() [1/2]

void ParquetFormatter::openTag	(	std::ostream &	into,
		const std::string &	xmlElement
	)

overridevirtual

Opens an XML tag.

An indentation, depending on the current xml-element-stack size, is written followed by the given xml element ("<" + xmlElement) The xml element is added to the stack, then.

Parameters

[in]	into	The output stream to use
[in]	xmlElement	Name of element to open

Returns: The OutputDevice for further processing

Implements OutputFormatter.

Definition at line 228 of file ParquetFormatter.cpp.

References myImpl, and WRITE_WARNINGF.

◆ openTag() [2/2]

void ParquetFormatter::openTag	(	std::ostream &	into,
		const SumoXMLTag &	xmlElement
	)

overridevirtual

Opens an XML tag.

Helper method which finds the correct string before calling openTag.

Parameters

[in]	into	The output stream to use
[in]	xmlElement	Id of the element to open

Implements OutputFormatter.

Definition at line 240 of file ParquetFormatter.cpp.

References myImpl, toString(), and WRITE_WARNINGF.

◆ setExpectedAttributes()

void ParquetFormatter::setExpectedAttributes	(	const SumoXMLAttrMask &	expected,
		const int	depth = `2`
	)

overridevirtual

Set the expected attributes to write. This is used for tracking which attributes are expected in table like outputs. This should be not necessary but at least in the initial phase of implementing CSV and Parquet it helps a lot to track errors.

Parameters

[in]	expected	which attributes are to be written (at the deepest XML level)
[in]	depth	the maximum XML hierarchy depth (excluding the root)

Reimplemented from OutputFormatter.

Definition at line 401 of file ParquetFormatter.cpp.

References myImpl.

◆ writeAttr() [1/6]

template<class T >

void ParquetFormatter::writeAttr	(	std::ostream &	,
		const std::string &	attr,
		const T &	val,
		const bool	isNull
	)

inline

Definition at line 68 of file ParquetFormatter.h.

References toString(), writeNullAttr(), and writeStringAttr().

◆ writeAttr() [2/6]

template<class T >

void ParquetFormatter::writeAttr	(	std::ostream &	,
		const SumoXMLAttr	attr,
		const T &	val,
		const bool	isNull
	)

inline

writes a named attribute

Generic template: stringifies the value (only when not null, preserving the original lazy evaluation) and forwards to the typed non-template helpers defined in the .cpp.

Definition at line 59 of file ParquetFormatter.h.

References toString(), writeNullAttr(), and writeStringAttr().

◆ writeAttr() [3/6]

void ParquetFormatter::writeAttr	(	std::ostream &	into,
		const std::string &	attr,
		const double &	val,
		const bool	isNull
	)

Definition at line 330 of file ParquetFormatter.cpp.

References myImpl.

◆ writeAttr() [4/6]

void ParquetFormatter::writeAttr	(	std::ostream &	into,
		const std::string &	attr,
		const int &	val,
		const bool	isNull
	)

Definition at line 343 of file ParquetFormatter.cpp.

References myImpl.

◆ writeAttr() [5/6]

void ParquetFormatter::writeAttr	(	std::ostream &	into,
		const SumoXMLAttr	attr,
		const double &	val,
		const bool	isNull
	)

typed overloads (non-template) – picked by overload resolution over the template

Definition at line 309 of file ParquetFormatter.cpp.

References myImpl, SUMO_ATTR_X, and SUMO_ATTR_Y.

◆ writeAttr() [6/6]

void ParquetFormatter::writeAttr	(	std::ostream &	into,
		const SumoXMLAttr	attr,
		const int &	val,
		const bool	isNull
	)

Definition at line 322 of file ParquetFormatter.cpp.

References myImpl.

◆ writeNullAttr() [1/2]

void ParquetFormatter::writeNullAttr ( const std::string & attr )

private

Definition at line 375 of file ParquetFormatter.cpp.

References myImpl.

◆ writeNullAttr() [2/2]

void ParquetFormatter::writeNullAttr ( const SumoXMLAttr attr )

private

Definition at line 367 of file ParquetFormatter.cpp.

References myImpl.

Referenced by writeAttr(), and writeAttr().

Here is the caller graph for this function:

◆ writePadding()

virtual void OutputFormatter::writePadding	(	std::ostream &	into,
		const std::string &	val
	)

inlinevirtualinherited

Writes some whitespace to format the output. This method is only implemented for XML output.

Parameters

[in]	into	The output stream to use
[in]	val	The whitespace

Reimplemented in PlainXMLFormatter.

Definition at line 138 of file OutputFormatter.h.

References UNUSED_PARAMETER.

Referenced by OutputDevice::writePadding().

Here is the caller graph for this function:

◆ writePreformattedTag()

virtual void OutputFormatter::writePreformattedTag	(	std::ostream &	into,
		const std::string &	val
	)

inlinevirtualinherited

Writes a preformatted tag to the device but ensures that any pending tags are closed. This method is only implemented for XML output.

Parameters

[in]	into	The output stream to use
[in]	val	The preformatted data

Reimplemented in PlainXMLFormatter.

Definition at line 127 of file OutputFormatter.h.

References UNUSED_PARAMETER.

Referenced by OutputDevice::writePreformattedTag().

Here is the caller graph for this function:

◆ writeStringAttr() [1/2]

void ParquetFormatter::writeStringAttr	(	const std::string &	attr,
		const std::string &	val
	)

private

Definition at line 359 of file ParquetFormatter.cpp.

References myImpl.

◆ writeStringAttr() [2/2]

void ParquetFormatter::writeStringAttr	(	const SumoXMLAttr	attr,
		const std::string &	val
	)

private

non-template helpers; defined in the .cpp where arrow/parquet are available

Definition at line 351 of file ParquetFormatter.cpp.

References myImpl.

Referenced by writeAttr(), writeAttr(), and writeTime().

Here is the caller graph for this function:

◆ writeTime()

void ParquetFormatter::writeTime	(	std::ostream &	into,
		const SumoXMLAttr	attr,
		const SUMOTime	val
	)

overridevirtual

Implements OutputFormatter.

Definition at line 383 of file ParquetFormatter.cpp.

References gHumanReadableTime, myImpl, STEPS2TIME, time2string(), and writeStringAttr().

◆ writeXMLHeader()

virtual bool OutputFormatter::writeXMLHeader	(	std::ostream &	into,
		const std::string &	rootElement,
		const std::map< SumoXMLAttr, std::string > &	attrs,
		bool	writeMetadata,
		bool	includeConfig
	)

inlinevirtualinherited

Writes an XML header with optional configuration.

If something has been written (myXMLStack is not empty), nothing is written and false returned. The default implementation does nothing and returns false.

Parameters

[in]	into	The output stream to use
[in]	rootElement	The root element to use
[in]	attrs	Additional attributes to save within the rootElement
[in]	includeConfig	whether the current config should be included as XML comment

Returns: whether something has been written

Reimplemented in PlainXMLFormatter.

Definition at line 79 of file OutputFormatter.h.

References UNUSED_PARAMETER.

Referenced by OutputDevice::writeXMLHeader().

Here is the caller graph for this function:

◆ wroteHeader()

bool ParquetFormatter::wroteHeader ( ) const

overridevirtual

Returns whether a header has been written. Useful to detect whether a file is being used by multiple sources.

Returns: Whether a header has been written

Implements OutputFormatter.

Definition at line 395 of file ParquetFormatter.cpp.

References myImpl.

Field Documentation

◆ myImpl

std::unique_ptr<Impl> ParquetFormatter::myImpl

private

Definition at line 97 of file ParquetFormatter.h.

Referenced by closeTag(), openTag(), openTag(), ParquetFormatter(), setExpectedAttributes(), writeAttr(), writeAttr(), writeAttr(), writeAttr(), writeNullAttr(), writeNullAttr(), writeStringAttr(), writeStringAttr(), writeTime(), and wroteHeader().

◆ myType

const OutputFormatterType OutputFormatter::myType

privateinherited

the type of formatter being used (XML, CSV, Parquet, etc.)

Definition at line 170 of file OutputFormatter.h.

Referenced by OutputFormatter::getType().

The documentation for this class was generated from the following files:

Data Structures

Public Member Functions

Private Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ ParquetFormatter()

◆ ~ParquetFormatter()

Member Function Documentation

◆ closeTag()

◆ getType()

◆ openTag() [1/2]

◆ openTag() [2/2]

◆ setExpectedAttributes()

◆ writeAttr() [1/6]

◆ writeAttr() [2/6]

◆ writeAttr() [3/6]

◆ writeAttr() [4/6]

◆ writeAttr() [5/6]

◆ writeAttr() [6/6]

◆ writeNullAttr() [1/2]

◆ writeNullAttr() [2/2]

◆ writePadding()

◆ writePreformattedTag()

◆ writeStringAttr() [1/2]

◆ writeStringAttr() [2/2]

◆ writeTime()

◆ writeXMLHeader()

◆ wroteHeader()

Field Documentation

◆ myImpl

◆ myType