> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flexorch.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Chunk Quality

> How FlexOrch scores and grades chunks, and which thresholds to use for RAG pipelines.

## Quality grades

Every document processed through FlexOrch receives a quality grade:

| Grade | Score range | Meaning                                                   | RAG recommendation     |
| ----- | ----------- | --------------------------------------------------------- | ---------------------- |
| **A** | ≥ 0.85      | Excellent — clean text, complete fields, low noise        | Always index           |
| **B** | ≥ 0.65      | Good — minor noise or gaps, suitable for production       | Index                  |
| **C** | ≥ 0.40      | Marginal — OCR issues, encoding errors, or sparse content | Review before indexing |
| **D** | \< 0.40     | Poor — garbage content, OCR failure, or nearly empty      | Exclude                |

**Recommended minimum for production RAG:** grade **B**.

***

## How the score is computed

The quality score is a composite of three signals:

```
quality_score = completeness × (0.4 × noise_score + 0.4 × length_score + 0.2)
```

| Signal         | Description                                                        |
| -------------- | ------------------------------------------------------------------ |
| `completeness` | Fraction of non-empty lines. Empty = 0.0, all filled = 1.0         |
| `length_score` | `min(avg_length / 500, 1.0)` — penalizes very short texts          |
| `noise_score`  | `max(0, 1 − garbage_ratio × 10)` — penalizes symbol/encoding noise |

A document with 100% complete fields, healthy average line length, and no noise scores 1.0 → grade **A**.

An OCR-failed scan with short, symbol-heavy lines might score 0.20 → grade **D**.

***

## OCR confidence penalty

For scanned PDFs and images, an additional OCR confidence check applies:

* OCR confidence ≥ 0.70 → no penalty
* OCR confidence \< 0.70 → quality score capped at 0.74 → maximum grade **C**

This ensures that readable-but-unreliable OCR output never receives an A or B grade.

***

## Filtering by quality at export

```bash theme={null}
# Export only A and B grade chunks
curl "https://api.flexorch.com/v1/datasets/{id}/export?format=rag&min_quality=B" \
  -H "X-API-KEY: dfx_your_key_here"
```

```python theme={null}
# Python SDK — export
client.datasets.export(dataset_id=89, format="rag", path="chunks.json", min_quality="B")

# Python SDK — chunks API
page = client.datasets.get(89).chunks(quality_grade="A,B", page_size=100)
```

***

## Filtering with flexorch-audit (OSS)

For local files before they reach the platform:

```python theme={null}
from flexorch_audit import audit, rag_metadata, chunk_text

text = open("contract.txt").read()
result = audit(text)
meta = rag_metadata(result)

print(meta)
# {
#   "should_index": True,          # grade A or B
#   "pii_safe": True,              # no PII detected
#   "quality_gate": "pass",        # "pass" | "warn" | "fail"
#   "recommended_chunk_size": 512, # tokens
#   "estimated_chunks": 6
# }

if meta["should_index"]:
    chunks = chunk_text(
        text,
        strategy="paragraph",
        max_tokens=meta["recommended_chunk_size"],
    )
    for c in chunks:
        print(c["chunk_index"], c["token_count"], c["text"][:80])
```

***

## Chunking strategies

| Strategy              | How it splits                           | Best for                                        |
| --------------------- | --------------------------------------- | ----------------------------------------------- |
| `paragraph` (default) | On blank lines, merges short paragraphs | Prose documents, contracts, reports             |
| `sliding_window`      | Fixed-size overlapping windows          | Uniform density required, long homogeneous text |
| `sentence`            | On sentence-ending punctuation          | Short, factual sentences where context is local |

```python theme={null}
from flexorch_audit import chunk_text

# Paragraph (default)
chunks = chunk_text(text, strategy="paragraph", max_tokens=512)

# Sliding window with 64-token overlap
chunks = chunk_text(text, strategy="sliding_window", max_tokens=512, overlap=64)

# Sentence
chunks = chunk_text(text, strategy="sentence", max_tokens=256)
```

Each chunk is a dict:

```python theme={null}
{
    "text": "Invoice FTR-2024-001 was issued by...",
    "token_count": 84,
    "chunk_index": 0
}
```

***

## Choosing chunk size

| Document type          | Recommended size        | Notes                        |
| ---------------------- | ----------------------- | ---------------------------- |
| Invoice / PO / Payroll | Single structured chunk | Keep all fields together     |
| Bank statement         | 256 tokens              | One transaction per chunk    |
| Contract / proposal    | 512 tokens              | Paragraph boundary preferred |
| Long report            | 512 – 1024 tokens       | Sliding window with overlap  |
| Email thread           | 256 tokens              | Per-message chunks           |

`rag_metadata()` returns a `recommended_chunk_size` based on the detected text length — use it as a starting point.

***

## PII and RAG

PII in chunks can leak sensitive data to your LLM. FlexOrch masks PII automatically:

* `pii_masked=True` on export → `[MASKED_EMAIL]`, `[MASKED_NATIONAL_ID_TR]`, etc.
* Filter chunks where `metadata.pii_masked == true` to guarantee clean RAG context
* Check `metadata.pii_findings_count` — anything > 0 should be reviewed before entering a public-facing chain

```python theme={null}
# Only load PII-safe chunks
documents = reader.load_data(dataset_id=89, pii_masked_only=True)
```

<CardGroup cols={2}>
  <Card title="RAG pipeline" icon="route" href="/guides/rag-pipeline">
    End-to-end: export chunks or managed search
  </Card>

  <Card title="PII & privacy" icon="shield" href="/guides/pii-privacy">
    Masking strategies and compliance notes
  </Card>
</CardGroup>