> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flexorch.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Datasets & Export

> Build datasets from processed documents and export in 9 formats.

## Overview

A dataset is a curated collection of pipeline execution results — structured fields, extracted text, and metadata — ready for LLM training, RAG, or analytics.

Building and exporting datasets does **not** consume credits.

***

## Build a dataset

After your documents are processed (job status `completed`), build a dataset:

```bash theme={null}
curl -X POST https://api.flexorch.com/v1/datasets \
  -H "X-API-KEY: dfx_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"name": "invoices-q1", "execution_ids": ["exec_001", "exec_002"]}'
```

***

## Export formats

| Format            | Extension  | Best for                                              |
| ----------------- | ---------- | ----------------------------------------------------- |
| JSONL             | `.jsonl`   | LLM fine-tuning (OpenAI, Anthropic)                   |
| CSV               | `.csv`     | Spreadsheet analysis                                  |
| Parquet           | `.parquet` | Data pipelines, analytics                             |
| Markdown          | `.md`      | RAG — LlamaIndex, LangChain                           |
| XML               | `.xml`     | Enterprise integrations                               |
| XLSX              | `.xlsx`    | Excel-based workflows                                 |
| HuggingFace Arrow | `zip`      | `datasets.load_from_disk()` — Arrow IPC with metadata |
| RAG chunks        | `.json`    | Semantic chunking — LangChain / LlamaIndex compatible |

***

## Export a dataset

```bash theme={null}
curl "https://api.flexorch.com/v1/datasets/{id}/export?format=jsonl" \
  -H "X-API-KEY: dfx_your_key_here" \
  -o output.jsonl
```

### RAG export with quality filtering

Pass `min_quality` to exclude low-quality chunks before export:

```bash theme={null}
# Export only A and B grade chunks as RAG JSON
curl "https://api.flexorch.com/v1/datasets/{id}/export?format=rag&min_quality=B" \
  -H "X-API-KEY: dfx_your_key_here" \
  -o chunks.json

# Export as HuggingFace Arrow dataset (zip)
curl "https://api.flexorch.com/v1/datasets/{id}/export?format=hf&min_quality=B" \
  -H "X-API-KEY: dfx_your_key_here" \
  -o dataset.zip
```

| `min_quality` value | Included grades |
| ------------------- | --------------- |
| `A`                 | A only          |
| `B` (recommended)   | A, B            |
| `C`                 | A, B, C         |
| omit                | All grades      |

***

## Dataset profile

Get aggregate statistics for a dataset:

```bash theme={null}
curl "https://api.flexorch.com/v1/datasets/{id}/profile" \
  -H "X-API-KEY: dfx_your_key_here"
```

Returns quality grade distribution, PII type summary, average score, and available export formats.

***

## Chunks API

<Note>
  Available on **Pro and Enterprise** plans.
</Note>

Retrieve individual text chunks with quality and PII filters:

```bash theme={null}
curl "https://api.flexorch.com/v1/datasets/{id}/chunks?quality_grade=A,B&pii_masked=true&page=1&page_size=20" \
  -H "X-API-KEY: dfx_your_key_here"
```

**Query parameters:**

| Parameter       | Description                                     |
| --------------- | ----------------------------------------------- |
| `quality_grade` | Comma-separated grades to include, e.g. `A,B`   |
| `pii_masked`    | `true` / `false` — filter by PII masking status |
| `page`          | Page number (1-indexed)                         |
| `page_size`     | Chunks per page, max 100                        |

**Response:**

```json theme={null}
{
  "data": {
    "chunks": [
      {
        "chunk_id": "abc-001",
        "chunk_index": 0,
        "text": "Invoice FTR-2024-001 from [MASKED_NAME]...",
        "token_count": 84,
        "metadata": { "quality_grade": "A", "pii_masked": true }
      }
    ],
    "total": 47,
    "page": 1,
    "page_size": 20,
    "has_more": true
  }
}
```

See the [RAG pipeline guide](/guides/rag-pipeline) for a complete integration walkthrough.

***

## Semantic indexing

<Note>
  Available on **Pro and Enterprise** plans.
</Note>

Index a dataset for semantic search:

```bash theme={null}
curl -X POST "https://api.flexorch.com/v1/datasets/{id}/index" \
  -H "X-API-KEY: dfx_your_key_here"
```

Then search:

```bash theme={null}
curl -X POST "https://api.flexorch.com/v1/search" \
  -H "X-API-KEY: dfx_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"query": "invoices over 10000 EUR from Germany", "top_k": 5, "mode": "hybrid"}'
```

***

## Dataset retention

Datasets are stored for a period defined by your plan:

| Plan       | Retention    |
| ---------- | ------------ |
| Trial      | 7 days       |
| Starter    | 30 days      |
| Pro        | 90 days      |
| Enterprise | Configurable |