Parquet Dictionary Encoding: What Column Chunks and Pages Actually Do
Learn how Parquet row groups, column chunks, pages, and dictionary encoding work, and why format capabilities do not guarantee query performance.
Publication & corrections: infostoa · Editorial standards
Individual author and specialist reviewer information is not provided for this article.
Apache Parquet is a file format that organizes data into column chunks and pages rather than storing every record as one indivisible row sequence. That organization makes selective column reads and value encodings possible. It does not make every Parquet query fast by itself. The bytes actually read depend on the footer metadata, selected columns, row-group and page layout, encoding, compression, and the behavior of the writer and reader in use.
Footer metadata directs the read path
A Parquet file places file metadata after its data area, followed by the metadata length and a trailing magic number. The metadata records where row groups and column chunks are located. A reader can consult this information first, locate the chunks it needs, and read those chunks sequentially instead of scanning the file as an unstructured byte stream. Understanding the format therefore requires attention to the footer and metadata as well as the encoded values.
A row group is a logical horizontal partition of rows. It contains one column chunk for each column, and a column chunk is divided into pages. Pages are the units at which encoding and compression are described and applied. These levels explain what “columnar” means in practice: values for a column are grouped together, while a row group keeps the same horizontal slice of records across columns. File, row-group, chunk, and page boundaries serve related but different purposes.
This layout can support projection, allowing a query to request a subset of column chunks. Actual reads still depend on the query plan and reader implementation, so inspect measured I/O rather than infer it from the file extension.
Dictionary encoding replaces repeated values with indices
A column chunk may be partly or wholly dictionary-encoded. In that case, the actual values are stored in a dictionary page and data pages store indices that refer to those values. A column containing repeated labels can therefore represent a long string once and refer to it repeatedly instead of writing the full string for every row. The index stream can use run-length encoding or a bit-packed hybrid representation, which can make repeated or small indices compact.
The dictionary itself has a cost. If values are mostly unique, or the dictionary grows large, storing both the dictionary and indices may provide little benefit. Implementations can stop using dictionary encoding when the dictionary grows beyond a useful size and continue with plain encoding. The format does not prescribe one universal point at which a writer should switch: implementation choices and data characteristics matter. For that reason, a dictionary flag alone cannot establish that a file is smaller or faster.
Dictionary encoding is also distinct from compression. Encoding chooses how values are represented; a compression codec then operates on page data according to the file metadata. A repeated value may benefit from dictionary indices, a codec, both, or neither, depending on its distribution and the implementation. Treat these as separate controls when comparing files, and avoid attributing an observed change to the wrong layer.
Pages and codecs have different responsibilities
A data page contains values and, for nested or optional fields, repetition and definition levels. Its header describes encoding and compression. Page boundaries can help a reader avoid data when metadata and query conditions support that choice.
This is not a universal pruning guarantee: engines use page indexes and statistics differently, and filters can read more than expected. Codec changes also do not automatically change dictionary behavior. Record encoding, codec, statistics, and actual I/O separately.
Expert perspective: distinguish what the format permits from what a reader does
Apache Parquet documentation explains how metadata locates column chunks; its format specification defines row groups, pages, and dictionary values and indices. These sources describe structure, not a specific engine’s pruning or speed. Treat the specification as a testable expectation and check it against writer and reader versions, query plans, and measured bytes.
Test performance with a reproducible workload
A query selecting only a few columns may benefit from column-chunk organization when the file and reader allow selective reads. Larger row groups can favor sequential access and parallel work in some settings, while smaller groups create more boundaries and metadata. The trade-offs depend on workload, storage, and engine. Compression also trades storage and transfer against CPU work, so file size alone is not a performance measure.
For comparison, hold data, schema, query, and reader constant while changing one writer setting at a time. Record versions, row-group and codec settings, bytes read, elapsed time, and result correctness. Test representative queries to separate layout, encoding, and reader effects.
Parquet is a flexible column-oriented substrate, not an automatic tuning policy. Its metadata and page structure make optimizations possible, but a reliable conclusion comes from measuring the actual writer-reader-workload combination. The specification is a starting point, not a substitute for that measurement.