Datasets & Export

Overview

A dataset is a curated collection of pipeline execution results — structured fields, extracted text, and metadata — ready for LLM training, RAG, or analytics. Building and exporting datasets does not consume credits.

Build a dataset

After your documents are processed (job status completed), build a dataset:

curl -X POST https://api.flexorch.com/v1/datasets \
  -H "X-API-KEY: dfx_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"name": "invoices-q1", "execution_ids": ["exec_001", "exec_002"]}'

Export formats

Format	Extension	Best for
JSONL	`.jsonl`	LLM fine-tuning (OpenAI, Anthropic)
CSV	`.csv`	Spreadsheet analysis
Parquet	`.parquet`	Data pipelines, analytics
Markdown	`.md`	RAG — LlamaIndex, LangChain
XML	`.xml`	Enterprise integrations
XLSX	`.xlsx`	Excel-based workflows
HuggingFace Arrow	`zip`	`datasets.load_from_disk()` — Arrow IPC with metadata
RAG chunks	`.json`	Semantic chunking — LangChain / LlamaIndex compatible

Export a dataset

curl "https://api.flexorch.com/v1/datasets/{id}/export?format=jsonl" \
  -H "X-API-KEY: dfx_your_key_here" \
  -o output.jsonl

RAG export with quality filtering

Pass min_quality to exclude low-quality chunks before export:

# Export only A and B grade chunks as RAG JSON
curl "https://api.flexorch.com/v1/datasets/{id}/export?format=rag&min_quality=B" \
  -H "X-API-KEY: dfx_your_key_here" \
  -o chunks.json

# Export as HuggingFace Arrow dataset (zip)
curl "https://api.flexorch.com/v1/datasets/{id}/export?format=hf&min_quality=B" \
  -H "X-API-KEY: dfx_your_key_here" \
  -o dataset.zip

`min_quality` value	Included grades
`A`	A only
`B` (recommended)	A, B
`C`	A, B, C
omit	All grades

Dataset profile

Get aggregate statistics for a dataset:

curl "https://api.flexorch.com/v1/datasets/{id}/profile" \
  -H "X-API-KEY: dfx_your_key_here"

Returns quality grade distribution, PII type summary, average score, and available export formats.

Chunks API

Available on Pro and Enterprise plans.

Retrieve individual text chunks with quality and PII filters:

curl "https://api.flexorch.com/v1/datasets/{id}/chunks?quality_grade=A,B&pii_masked=true&page=1&page_size=20" \
  -H "X-API-KEY: dfx_your_key_here"

Query parameters:

Parameter	Description
`quality_grade`	Comma-separated grades to include, e.g. `A,B`
`pii_masked`	`true` / `false` — filter by PII masking status
`page`	Page number (1-indexed)
`page_size`	Chunks per page, max 100

Response:

{
  "data": {
    "chunks": [
      {
        "chunk_id": "abc-001",
        "chunk_index": 0,
        "text": "Invoice FTR-2024-001 from [MASKED_NAME]...",
        "token_count": 84,
        "metadata": { "quality_grade": "A", "pii_masked": true }
      }
    ],
    "total": 47,
    "page": 1,
    "page_size": 20,
    "has_more": true
  }
}

See the RAG pipeline guide for a complete integration walkthrough.

Semantic indexing

Available on Pro and Enterprise plans.

Index a dataset for semantic search:

curl -X POST "https://api.flexorch.com/v1/datasets/{id}/index" \
  -H "X-API-KEY: dfx_your_key_here"

Then search:

curl -X POST "https://api.flexorch.com/v1/search" \
  -H "X-API-KEY: dfx_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"query": "invoices over 10000 EUR from Germany", "top_k": 5, "mode": "hybrid"}'

Dataset retention

Datasets are stored for a period defined by your plan:

Plan	Retention
Trial	7 days
Starter	30 days
Pro	90 days
Enterprise	Configurable

Get Started

Authentication

Guides

Python SDK

TypeScript SDK

Open Source (flexorch-audit)

Connectors

Scheduler

Team

Compliance

MCP Server

Advanced

Datasets & Export

Overview

Build a dataset

Export formats

Export a dataset

RAG export with quality filtering

Dataset profile

Chunks API

Semantic indexing

Dataset retention

​Overview

​Build a dataset

​Export formats

​Export a dataset

​RAG export with quality filtering

​Dataset profile

​Chunks API

​Semantic indexing

​Dataset retention

Overview

Build a dataset

Export formats

Export a dataset

RAG export with quality filtering

Dataset profile

Chunks API

Semantic indexing

Dataset retention