Chunk Quality

Quality grades

Every document processed through FlexOrch receives a quality grade:

Grade	Score range	Meaning	RAG recommendation
A	≥ 0.85	Excellent — clean text, complete fields, low noise	Always index
B	≥ 0.65	Good — minor noise or gaps, suitable for production	Index
C	≥ 0.40	Marginal — OCR issues, encoding errors, or sparse content	Review before indexing
D	< 0.40	Poor — garbage content, OCR failure, or nearly empty	Exclude

Recommended minimum for production RAG: grade B.

How the score is computed

The quality score is a composite of three signals:

quality_score = completeness × (0.4 × noise_score + 0.4 × length_score + 0.2)

Signal	Description
`completeness`	Fraction of non-empty lines. Empty = 0.0, all filled = 1.0
`length_score`	`min(avg_length / 500, 1.0)` — penalizes very short texts
`noise_score`	`max(0, 1 − garbage_ratio × 10)` — penalizes symbol/encoding noise

A document with 100% complete fields, healthy average line length, and no noise scores 1.0 → grade A. An OCR-failed scan with short, symbol-heavy lines might score 0.20 → grade D.

OCR confidence penalty

For scanned PDFs and images, an additional OCR confidence check applies:

OCR confidence ≥ 0.70 → no penalty
OCR confidence < 0.70 → quality score capped at 0.74 → maximum grade C

This ensures that readable-but-unreliable OCR output never receives an A or B grade.

Filtering by quality at export

# Export only A and B grade chunks
curl "https://api.flexorch.com/v1/datasets/{id}/export?format=rag&min_quality=B" \
  -H "X-API-KEY: dfx_your_key_here"

# Python SDK — export
client.datasets.export(dataset_id=89, format="rag", path="chunks.json", min_quality="B")

# Python SDK — chunks API
page = client.datasets.get(89).chunks(quality_grade="A,B", page_size=100)

Filtering with flexorch-audit (OSS)

For local files before they reach the platform:

from flexorch_audit import audit, rag_metadata, chunk_text

text = open("contract.txt").read()
result = audit(text)
meta = rag_metadata(result)

print(meta)
# {
#   "should_index": True,          # grade A or B
#   "pii_safe": True,              # no PII detected
#   "quality_gate": "pass",        # "pass" | "warn" | "fail"
#   "recommended_chunk_size": 512, # tokens
#   "estimated_chunks": 6
# }

if meta["should_index"]:
    chunks = chunk_text(
        text,
        strategy="paragraph",
        max_tokens=meta["recommended_chunk_size"],
    )
    for c in chunks:
        print(c["chunk_index"], c["token_count"], c["text"][:80])

Chunking strategies

Strategy	How it splits	Best for
`paragraph` (default)	On blank lines, merges short paragraphs	Prose documents, contracts, reports
`sliding_window`	Fixed-size overlapping windows	Uniform density required, long homogeneous text
`sentence`	On sentence-ending punctuation	Short, factual sentences where context is local

from flexorch_audit import chunk_text

# Paragraph (default)
chunks = chunk_text(text, strategy="paragraph", max_tokens=512)

# Sliding window with 64-token overlap
chunks = chunk_text(text, strategy="sliding_window", max_tokens=512, overlap=64)

# Sentence
chunks = chunk_text(text, strategy="sentence", max_tokens=256)

Each chunk is a dict:

{
    "text": "Invoice FTR-2024-001 was issued by...",
    "token_count": 84,
    "chunk_index": 0
}

Choosing chunk size

Document type	Recommended size	Notes
Invoice / PO / Payroll	Single structured chunk	Keep all fields together
Bank statement	256 tokens	One transaction per chunk
Contract / proposal	512 tokens	Paragraph boundary preferred
Long report	512 – 1024 tokens	Sliding window with overlap
Email thread	256 tokens	Per-message chunks

rag_metadata() returns a recommended_chunk_size based on the detected text length — use it as a starting point.

PII and RAG

PII in chunks can leak sensitive data to your LLM. FlexOrch masks PII automatically:

pii_masked=True on export → [MASKED_EMAIL], [MASKED_NATIONAL_ID_TR], etc.
Filter chunks where metadata.pii_masked == true to guarantee clean RAG context
Check metadata.pii_findings_count — anything > 0 should be reviewed before entering a public-facing chain

# Only load PII-safe chunks
documents = reader.load_data(dataset_id=89, pii_masked_only=True)

RAG pipeline

End-to-end: export chunks or managed search

PII & privacy

Masking strategies and compliance notes

Get Started

Authentication

Guides

Python SDK

TypeScript SDK

Open Source (flexorch-audit)

Connectors

Scheduler

Team

Compliance

MCP Server

Advanced

Quality grades

How the score is computed

OCR confidence penalty

Filtering by quality at export

Filtering with flexorch-audit (OSS)

Chunking strategies

Choosing chunk size

PII and RAG

RAG pipeline

PII & privacy

​Quality grades

​How the score is computed

​OCR confidence penalty

​Filtering by quality at export

​Filtering with flexorch-audit (OSS)

​Chunking strategies

​Choosing chunk size

​PII and RAG

RAG pipeline

PII & privacy

Quality grades

How the score is computed

OCR confidence penalty

Filtering by quality at export

Filtering with flexorch-audit (OSS)

Chunking strategies

Choosing chunk size

PII and RAG