qvd2parquet

What it is

qvd2parquet is a command-line converter from Qlik QVD files to Apache Parquet. It was built for a real Qlik to Dremio migration, where a QVD layer was the last thing standing between the Qlik estate and the lakehouse, and where getting those files out correctly turned out to be harder than it looks.

qvd2parquet input.qvd output.parquet

It reads standard, unencrypted QVD files, preserves useful Parquet types instead of stringifying everything, keeps MONEY and FIX columns as exact Parquet decimals, decodes records in parallel, and streams batches into Parquet row groups, so a large file never has to be materialized in memory.

On a real SAP extract

Point it at a directory

Pass --out-dir with one or more files or directories and every .qvd becomes a .parquet of the same name. Add --recursive to descend into subdirectories.

qvd2parquet --out-dir ./parquet --quality-gate numeric --log run.jsonl ./qvds

qvd2parquet: converting 4 file(s)
qvd2parquet: ok   qvds/products.qvd -> parquet/products.parquet (77 rows, 9 columns, 4.6 KiB)
qvd2parquet: ok   qvds/sales.qvd -> parquet/sales.parquet (1,000 rows, 7 columns, 7.7 KiB)
qvd2parquet: FAIL qvds/truncated.qvd: no XML header terminator (0x00) found: not a QVD file?
qvd2parquet: ok   qvds/stock.qvd -> parquet/stock.parquet (120 rows, 19 columns, 7.3 KiB)
converted 3/4 file(s) in 34ms: 1,197 rows, 19.6 KiB

A bad file in the middle does not stop the run. Every input is attempted, failures are listed at the end, and the exit code reports the most actionable one: a schema policy error you can fix outranks a generic read error.

Folder conversion is built in rather than left to a shell loop for one reason. --file-workers converts several files at once and divides the decode workers between them, so the total stays near one per CPU. Four separate processes would each start one worker per core and oversubscribe the machine fourfold.

Exact decimals

SAP figures are not roughly right, so nothing here is carried through a double. MONEY and FIX are always written as Parquet decimals, with values held as scaled integers end to end. Scale comes from the QVD's NumberFormat/nDec, or is inferred from the display strings when it is absent.

QlikView often declares a price as a plain REAL, where the header carries no usable scale at all. By default those columns are promoted to an exact decimal whose scale is derived from the values themselves: the smallest scale at which every value is exactly representable, up to nine decimals. Pure-integer columns stay int64, since decimal(p,0) would gain nothing.

qvd2parquet: schema: Einkaufspreis: REAL with 75 double symbols promoted to
  decimal(5,2); scale 2 inferred from values

A value that does not fit its declared scale is rounded half away from zero, matching what Qlik itself displays, and the rounding is counted and reported rather than done quietly. --decimal-strict turns it into a failure naming the column and the offending value, for a pipeline where an unexpected precision change has to stop the job. A value is never dropped: turning an inexact number into a null would lose data no later check could recover.

Types are resolved, not guessed

The Parquet schema is resolved only after every selected column's symbol table has been read and profiled. That makes mixed-type behaviour explicit, instead of producing an unstable schema that depends on which rows happened to be seen first.

One schema: line is printed per output column, explaining exactly why each type was chosen, and --schema-report writes the same reasoning as JSON. That is the first thing to read when a column does not resolve the way you expected.

Qlik duals

A Qlik dual pairs a number with a display string, and often that string is only the number formatted: 1.234,56 beside 1234.56. Writing it would duplicate the numeric column, so the default drops it. When the string carries something the number does not, such as Open beside 1, it is kept as a separate text column and the reason is stated:

schema: Status: INTEGER with 3 integer symbols, written as int64; 3 of 3 display
  strings carry text the number does not (e.g. "Open" beside 1), so they are kept
  in "Status__text"

A single odd value is enough to keep the column, because the default errs towards preserving data. --dual overrides the call in either direction, and --mixed decides what happens to a column mixing numbers with unrelated text: fail with the counts of each symbol kind, write the whole column as text, keep numerics numeric, or split the two sides into separate columns.

One shape resolves on its own, because nothing is actually being decided. When the numeric symbols are integers and every symbol carries its own display string, the file already states the text for every value, so the column is written as text without inventing a rendering for anything. A part number holding 0901 beside the number 901 is a code, not a quantity, and reading it as 901 would not survive a round trip.

Timestamps stay what Qlik wrote

Qlik stores dates and times as serial day numbers. A serial names no timezone. It is a bare wall-clock reading, and which zone it was recorded in is simply not in the file.

So the default converts nothing. --timezone=none writes the wall clock as-is with no timezone on the column, asserts nothing the QVD does not say, and produces a byte-identical file whatever machine runs the conversion. No zone is guessed behind your back.

When you do know the provenance, naming it is the mode that earns its keep for Parquet. An IANA name asserts that the wall clocks were recorded in that zone and converts them to true instants, which is what makes ordering across a DST change, joins against other instant data, and rendering in a consumer's own zone come out right.

--timezone=none              timestamp[us]            2016-03-01 00:00:00
--timezone=UTC               timestamp[us, tz=UTC]    2016-03-01 00:00:00Z
--timezone=America/Chicago   timestamp[us, tz=UTC]    2016-03-01 06:00:00Z
--timezone=Asia/Tokyo        timestamp[us, tz=UTC]    2016-02-29 15:00:00Z

Why the default is not Local: choosing a zone is a claim about provenance that only the person running the conversion can make, and Local makes it accidentally, using whatever zone the converting machine sits in. Real data shows why that matters. In the Chicago taxi QVDs, 13 March 2016 runs 01:45, then 03:00, with the 02:00 hour absent because the local clock skipped it, which also settles what those readings are. Plenty of local-time data contains that missing hour instead: generated master calendars enumerate every slot whether the zone had it or not, and extracts from systems that do not observe DST write wall clocks the target zone never had. A zoned conversion silently relocates every one of those.

A quality gate, not a hope

--quality-gate validates the written Parquet against metrics collected during conversion. Validation always reads the temporary file before the final rename, so a failed gate never leaves a final-looking output behind.

Integer, decimal and date aggregates are compared exactly, with decimal sums using scaled-integer arithmetic and no floating-point tolerance. The full fingerprint is a multiset digest rather than an ordered stream hash, so it stays valid despite unordered chunk delivery, and nulls are marked explicitly so a null never collides with a zero or an empty string.

A run you can query and audit

--log writes JSON Lines: one record per file, then a summary. The format is chosen so a finished run can be queried rather than read.

duckdb -c "select status, count(*), sum(rows) from read_json_auto('run.jsonl')
           where type='file' group by 1"
duckdb -c "select input, error from read_json_auto('run.jsonl') where status='failed'"

Each record carries row and column counts, output size, elapsed time, throughput, and the quality gate's verdict with any errors, so a batch can be audited without opening every per-file report. Exit codes name the failure class, which is what makes the tool usable from a scheduler:

The Parquet file is the only thing written to stdout's usual place: the identification banner, the per-column schema decisions, progress and the final summary all go to stderr, and stdout stays empty, so the tool composes safely in pipelines and shell substitutions.

Inspect before you convert

--inspect reads the XML header and symbol tables, prints the schema a conversion would produce, and exits without touching the record area, so its cost is independent of row count. On a 29 MiB, 5-million-row file it reads 7.9 KiB and finishes in 0.01s, against 1.58s for the full conversion. Every type policy flag applies, so what you see is what a conversion would write, which makes it a cheap pre-flight check in a pipeline.

SAP field names

QVD field names from SAP extracts are often composite, packing the table, the technical name and a description into one string. --exclude strips QlikView's internal key fields by wildcard, and --field-regex splits the rest:

qvd2parquet \
  --exclude '%*' \
  --field-regex '^[^-]*-\|\|-(?P<name>[^-]*)-\|\|-(?P<comment>.*)$' \
  A057.qvd a057.parquet

The result carries the description as Parquet field metadata and keeps the original QVD name, so nothing is lost:

DATBI  int64             {"comment": "Ende Gültigkeit", "qvd.field": "A057-||-DATBI-||-Ende Gültigkeit"}
KBETR  decimal128(4, 2)  {"comment": "Betrag",           "qvd.field": "A057-||-KBETR-||-Betrag"}

Performance

The record area is fixed-width, so once the symbol tables are read it can be split into contiguous row ranges and decoded concurrently. Each worker owns its Arrow builders, reads its byte range directly, and emits one Arrow record plus chunk-local quality metrics, while a single writer goroutine feeds the Parquet writer.

Measured on an Apple M3 Max over a 200k-row fixture with integer, high-cardinality string, decimal, date and nullable double columns: decoding reaches 15.5M rows/s with one worker per core, and the full pipeline including zstd-compressed Parquet writing runs at 3.3M rows/s. zstd is both the smallest and the fastest option here, which is why it is the default. Nothing in the pipeline scales with total row count: peak memory is the symbol tables of the selected columns plus a bounded amount of builder and writer memory.

One consequence worth knowing: physical row order is not preserved. Chunks are written as workers finish, so the Parquet file holds the same multiset of rows as the QVD but not in the same order. Every quality metric is order-independent, so validation is unaffected, but do not rely on physical row order in downstream queries.

Install

Download the archive for your platform, unpack it, and put qvd2parquet on your PATH. The binaries are pure Go and statically linked, so they have no runtime dependencies: Linux, Windows and macOS, on both amd64 and arm64. Verify the download against the published SHA256SUMS.

shasum -a 256 -c SHA256SUMS --ignore-missing

Or build from source with Go 1.25 or newer:

go install github.com/ralforion/qvd2parquet/cmd/qvd2parquet@latest

From version 1.0.0 the CLI surface and the conversion defaults are stable: a flag will not be removed or change its meaning, and a default will not change what an existing file converts to, outside a major bump. New behaviour arrives behind a new flag. The tool is released under the Apache License 2.0.

Built for a Qlik to Dremio migration

A QVD layer is often a decade of accumulated business logic, and moving it to a lakehouse is where that history either survives or quietly degrades. Stringified numbers, doubles standing in for currency, zero-padded account codes read as integers, and timestamps shifted by whichever machine ran the job are the failures that show up months later in a reconciliation, not on the day of the migration.

That is the job this tool was written for: land a Qlik estate as Parquet in object storage, ready for Dremio to query as Iceberg tables or promoted datasets. The defaults follow from that target. Exact decimals because DMBTR has to reconcile against SAP. Text preserved for zero-padded codes because they are keys, not quantities. Timestamps left as the wall clock Qlik recorded, because Dremio renders the stored value verbatim and applies no session zone, so an unnecessary conversion would be visible in the data and invisible in the schema.

Getting the physical data out correctly is the first half of that move. The second is the business meaning that lived in the Qlik load scripts, which is what the OrionBelt Semantic Layer is for: define dimensions, measures and metrics once in version-controlled YAML, and compile them into correct SQL against Dremio, or against whatever else the lakehouse turns out to hold.

Frequently Asked Questions