> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flexorch.com/llms.txt
> Use this file to discover all available pages before exploring further.

# RAG Pipeline

> Two approaches to building a RAG pipeline with FlexOrch — export chunks or use the managed search API.

## Overview

FlexOrch supports two RAG approaches:

| Approach                                           | Best for                                                | Plan      |
| -------------------------------------------------- | ------------------------------------------------------- | --------- |
| **Export chunks** (`fmt=rag`, `fmt=hf`)            | Bring-your-own vector store (FAISS, Pinecone, Weaviate) | All plans |
| **Managed search** (`/search` API + `/chunks` API) | Low-ops semantic retrieval, no vector store needed      | Pro+      |

***

## Approach 1 — Export chunks

After building a dataset, export it as RAG-ready chunks:

```bash theme={null}
# RAG JSON — LangChain / LlamaIndex compatible
curl "https://api.flexorch.com/v1/datasets/{id}/export?format=rag&min_quality=B" \
  -H "X-API-KEY: dfx_your_key_here" \
  -o chunks.json

# HuggingFace Arrow — datasets.load_from_disk() compatible
curl "https://api.flexorch.com/v1/datasets/{id}/export?format=hf&min_quality=B" \
  -H "X-API-KEY: dfx_your_key_here" \
  -o dataset.zip
```

The `min_quality` parameter filters out low-quality chunks before export. Pass `A` for only top-grade content, `B` to include solid content (recommended), or omit to include everything.

### RAG chunk format

```json theme={null}
{
  "chunks": [
    {
      "chunk_id": "abc-001",
      "chunk_index": 0,
      "text": "Invoice FTR-2024-001 from [MASKED_NAME]...",
      "token_count": 84,
      "metadata": {
        "doc_type": "invoice",
        "quality_grade": "A",
        "quality_score": 0.91,
        "pii_masked": true,
        "language": "tr"
      }
    }
  ]
}
```

### Load with Python SDK

```python theme={null}
from flexorch_sdk import FlexOrch

client = FlexOrch()

# Export to disk
client.datasets.export(dataset_id=89, format="rag", path="chunks.json", min_quality="B")

# Or load in-memory via the chunks() method
chunks_page = client.datasets.get(89).chunks(
    quality_grade="A,B",
    pii_masked=True,
    page=1,
    page_size=100,
)

for chunk in chunks_page["data"]["chunks"]:
    print(chunk["text"][:80], chunk["metadata"]["quality_grade"])
```

### Load with FlexOrchReader (LlamaIndex)

```python theme={null}
from flexorch_sdk import FlexOrch, FlexOrchReader

client = FlexOrch()
reader = FlexOrchReader(client)

documents = reader.load_data(
    dataset_id=89,
    min_quality="B",
    pii_masked_only=True,
)

from llama_index.core import VectorStoreIndex
index = VectorStoreIndex.from_documents(documents)
```

See the [LlamaIndex guide](/guides/llamaindex) for the full integration.

***

## Approach 2 — Managed search (Pro+)

<Note>
  Semantic and hybrid search require a **Pro or Enterprise** plan.
</Note>

### Step 1 — Index the dataset

```bash theme={null}
curl -X POST "https://api.flexorch.com/v1/datasets/{id}/index" \
  -H "X-API-KEY: dfx_your_key_here"
```

Indexing is idempotent and runs in the background. Poll `GET /v1/datasets/{id}/index/status` until `status` is `"ready"`.

### Step 2 — Search

```bash theme={null}
curl -X POST "https://api.flexorch.com/v1/search" \
  -H "X-API-KEY: dfx_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "invoices over 10000 EUR from Germany",
    "top_k": 5,
    "mode": "hybrid"
  }'
```

### Step 3 — Retrieve chunks for a dataset

```bash theme={null}
curl "https://api.flexorch.com/v1/datasets/89/chunks?quality_grade=A,B&pii_masked=true&page=1&page_size=20" \
  -H "X-API-KEY: dfx_your_key_here"
```

### Use FlexOrchRetriever (LangChain)

```python theme={null}
from flexorch_sdk import FlexOrch, FlexOrchRetriever

client = FlexOrch()

retriever = FlexOrchRetriever(
    client,
    quality_threshold="B",   # include A and B grade chunks
    pii_masked=True,
    top_k=5,
    mode="hybrid",
)

docs = retriever.invoke("payment terms in German invoices")
for doc in docs:
    print(doc.page_content[:80])
    print(doc.metadata["quality_grade"])
```

See the [LangChain guide](/guides/langchain) for a full `RetrievalQA` example.

***

## Chunking strategy by document type

| Document type       | Recommended strategy      | Chunk size             |
| ------------------- | ------------------------- | ---------------------- |
| Invoice / PO        | Single chunk (structured) | all fields             |
| Bank statement      | Per-transaction rows      | 256 tokens             |
| Payroll             | Per-employee row          | 256 tokens             |
| Contract / proposal | Paragraph split           | 512 tokens             |
| Long report         | Sliding window            | 512 tokens, overlap 64 |

The `fmt=rag` export applies FlexOrch's built-in chunker based on `doc_type`. For custom strategies, use the `flexorch-audit` OSS library:

```python theme={null}
from flexorch_audit import chunk_text, rag_metadata, audit

text = open("contract.txt").read()
result = audit(text)
meta = rag_metadata(result)

if meta["should_index"] and meta["pii_safe"]:
    chunks = chunk_text(
        text,
        strategy="paragraph",
        max_tokens=meta["recommended_chunk_size"],
    )
```

See the [flexorch-audit chunk-quality guide](/guides/chunk-quality) for details.

***

## Quality gate for RAG

Not all chunks should enter your vector store. Use quality grades to filter:

| Grade | Score   | Recommendation                       |
| ----- | ------- | ------------------------------------ |
| **A** | ≥ 0.85  | Index always                         |
| **B** | ≥ 0.65  | Index — suitable for production RAG  |
| **C** | ≥ 0.40  | Review before indexing               |
| **D** | \< 0.40 | Exclude — OCR noise, garbage content |

**Recommended minimum:** `B` for production pipelines.

***

## End-to-end example

```python theme={null}
from flexorch_sdk import FlexOrch, FlexOrchRetriever
from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI

client = FlexOrch()

# 1 — Upload and process document
job = client.upload("invoice_batch.pdf")
job.wait()

# 2 — Build dataset
dataset = client.datasets.build(name="invoice-rag", job_ids=[job.id])

# 3 — Index for semantic search (Pro+)
client.datasets.index(dataset.id)

# 4 — Wire into LangChain
retriever = FlexOrchRetriever(client, quality_threshold="B", mode="hybrid")

qa = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(model="gpt-4o-mini"),
    retriever=retriever,
)

print(qa.invoke("What is the total amount on invoice FTR-2024-001?"))
```

<CardGroup cols={2}>
  <Card title="LangChain integration" icon="link" href="/guides/langchain">
    FlexOrchRetriever — full RetrievalQA example
  </Card>

  <Card title="LlamaIndex integration" icon="link" href="/guides/llamaindex">
    FlexOrchReader — VectorStoreIndex example
  </Card>

  <Card title="Chunk quality" icon="chart-bar" href="/guides/chunk-quality">
    Grade thresholds, scoring, and strategies
  </Card>

  <Card title="MCP tools" icon="robot" href="/mcp/tools">
    dataset.index and dataset.chunks for AI agents
  </Card>
</CardGroup>