ShreddingAdvice

One path worth promoting to a typed column in a Parquet Variant shredding schema, with the evidence behind the recommendation.

Properties

Link copied to clipboard

how much of the stored document bytes this path accounts for. The number that decides whether shredding it is worth anything: a path holding 0.1% of the bytes cannot save a reader much however typed it is.

Link copied to clipboard

distinct values, exact below the sketch's sparse limit and an estimate above it. See InferredField.distinctIsExact.

Link copied to clipboard

how much of the present values are JSON null.

Link copied to clipboard

the Parquet physical type the observations point at — INT64, DOUBLE, DECIMAL(p, s), BINARY (UTF8), BOOLEAN. Named in Parquet's vocabulary rather than the engine's, because the whole point of this object is to be read by somebody writing a schema.

Link copied to clipboard

the path, with array indices collapsed. See CatalogPath.

Link copied to clipboard

how much of the corpus carries the path, in 0.0..1.0. A shredded field that is usually absent costs a definition level per row and buys little.

Link copied to clipboard

the recommendation in words, for a human reading a report.

Link copied to clipboard

true when the path holds values outside parquetType's family, so the shredded column needs its untyped variant_value fallback populated rather than being a pure typed column. This is the field a hand-written schema gets wrong.

Link copied to clipboard

how much of the path's observed values are of parquetType's family. Below 1.0 the residual values still have to go somewhere — see residual.

Functions

Link copied to clipboard
fun render(): String

The field as a line of a shredding schema, in the specification's own shape.

Link copied to clipboard
open override fun toString(): String