3.2 Data quality assessment and classification
The data quality assessment and classification layer provides the diagnostic backbone for Structured Ai’s
routing and processing decisions, ensuring that each workbook is treated according to its actual structural
properties rather than optimistic assumptions about cleanliness. Instead of relying on a few superficial checks,
the framework performs a comprehensive evaluation that examines header organization, data regularity, and
the presence of non-tabular artifacts across all sheets. This richer characterization enables the system to
distinguish, for example, between a well-structured fact table with minor formatting noise and a heavily
formatted management report whose layout carries much of the meaning. A central component of this layer is
the header complexity analysis, which operates at the row level to detect multi-row header structures and
hierarchical label patterns that are common in enterprise spreadsheets. The system scans the top portion of
each sheet, looking for contiguous regions of non-numeric cells, merged spans, and repeated label patterns that
indicate stacked headers rather than ordinary data rows. It interprets explicit formatting indicators such as
bold fonts, shading, and border styles alongside implicit cues like repeated categories and indentation to infer
whether multiple header rows collectively define a logical column name hierarchy. This analysis not only
informs the complexity score but also feeds downstream components that will later consolidate multi-row
headers into machine-usable schema elements. Beyond headers, the framework performs structural regularity
assessment across entire sheets. It evaluates cell population patterns to identify continuous blocks of data,
irregular gaps, and outlier regions that may correspond to embedded charts, comments, or annotation blocks.
It also computes type consistency metrics for each column and row, measuring how frequently a column
switches between numeric, textual, date, or formula-based content. Columns with stable types and minimal
anomalies are strong candidates for relational treatment, whereas columns with highly mixed types or sporadic
formulas may require document-centric handling. Formatting uniformity consistent number formats,
alignments, and styles, is likewise used as a proxy for structural regularity. The system flags irregular
patterns that suggest specialized processing, such as high densities of merged cells, overlapping objects (charts,
shapes), or regions where formulas dominate raw values. These indicators help the classifier understand where
traditional table extraction would fail or where naive flattening would destroy business semantics, prompting
alternative strategies such as Tier 1 document-centric processing or targeted extraction of specific regions.
Crucially, the classification framework is built on the principle that data quality exists on a continuous
spectrum, not as a binary clean/messy label. Each sheet is assigned scores along several dimensions’ header
complexity, type consistency, merged-cell density, and artifact presence, rather than a single yes/no tag. This
nuanced representation allows the routing engine to make more sophisticated decisions, such as treating part
of a workbook as SQL-ready while directing other sections to document-centric pipelines, or selecting hybrid
strategies when sheets fall into intermediate quality bands.
3.3 Natural language query processing pipeline
The natural language query processing pipeline is responsible for converting a user’s free-form question into
an executable plan that respects the structure and semantics of the underlying data sources. Unlike generic
question-answering systems, Structured AI must interpret queries in the context of concrete tables, formulas,
and report layouts, which requires a schema-aware and context-aware approach. To achieve this, the pipeline
integrates database metadata, sampled cell contents, and prior examples into the prompt and validation
process, enabling high-fidelity translations from natural language to SQL or document reasoning tasks.
[4,5]
The pipeline begins with linguistic preprocessing, including tokenization, part-of-speech tagging, and entity
recognition tailored for data analytics terminology (such as measures, dimensions, time periods, and
aggregations). Detected entities are mapped to candidate schema elements table names, column headers, or
derived metrics using fuzzy matching and semantic similarity, taking into account the consolidated headers
produced during document conversion. This mapping phase constrains the search space for subsequent
translation and helps disambiguate overloaded terms like “margin” or “balance” based on their occurrence in
the workbook. Next, the system constructs schema-aware prompts that fuse query text with structural context.
These prompts may include abbreviated schema listings, examples of typical rows, and hints about relevant
relationships (joins, foreign keys), all generated from the inferred schema and data samples. Few-shot learning
is used to inject domain-specific knowledge: the prompt incorporates examples of previously successful
translations for similar query patterns, enabling the model to align new queries with established idioms in that
organization’s reporting practices. Once candidate SQL queries or reasoning specifications are generated, the
pipeline applies a multi-stage validation process to guard against hallucinations and structural errors. Syntax
validation checks that SQL statements are well-formed; semantic validation verifies that all referenced tables
and columns actually exist and that joins follow valid keys; type validation ensures that operations such as
aggregations and comparisons are type-compatible (for instance, not summing text fields or comparing dates
to strings). A feasibility assessment estimates query cost and detects potentially dangerous operations, such as
unbounded Cartesian products or extremely broad scans on massive tables, allowing the system to reject or