# BYO LLM
Source: https://docs.flexorch.com/advanced/byo-llm
Use your own OpenAI, Anthropic, or Azure OpenAI API key for LLM-assisted extraction.
BYO LLM is an **Enterprise** plan feature.
## Overview
By default, FlexOrch uses a locally hosted LLM (Ollama) for extraction tasks that can't be resolved by deterministic patterns. Enterprise customers can bring their own LLM API key for higher accuracy or to use a specific model.
## Supported providers
| Provider | Models |
| ------------ | ----------------------------------- |
| OpenAI | GPT-4o, GPT-4-turbo, GPT-3.5-turbo |
| Anthropic | claude-sonnet-4-6, claude-haiku-4-5 |
| Azure OpenAI | Deployed model via endpoint |
| Custom | Any OpenAI-compatible endpoint |
## Configure via the platform
Go to **Settings → Integrations → LLM Configuration**.
Enter your provider, model, and API key. Click **Test** to verify the connection before saving.
## Configure via API
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/settings/llm-config \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{
"provider": "openai",
"model": "gpt-4o",
"api_key": "sk-..."
}'
```
Once configured, the BYO LLM is used for all extraction tasks that require an LLM. Deterministic extraction (rule-based) is always attempted first — the LLM is only invoked as a fallback.
# Fine-Tuning
Source: https://docs.flexorch.com/advanced/fine-tuning
Export JSONL datasets and submit fine-tuning jobs to OpenAI.
Fine-tuning job management is a **Pro+** feature. JSONL export for self-managed fine-tuning is available on all plans.
## Export for fine-tuning
Export any dataset as JSONL:
```bash theme={null}
curl "https://api.flexorch.com/v1/datasets/{id}/export?format=jsonl" \
-H "X-API-KEY: dfx_your_key_here" \
-o training_data.jsonl
```
The JSONL file is formatted for direct use with OpenAI's fine-tuning API.
## Managed fine-tuning (Pro+)
Pro and Enterprise customers can submit fine-tuning jobs directly from the platform:
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/fine-tune \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{
"dataset_id": "ds_abc123",
"model": "gpt-4o-mini-2024-07-18",
"suffix": "invoice-extractor"
}'
```
Track job status:
```bash theme={null}
curl https://api.flexorch.com/v1/fine-tune/{job_id} \
-H "X-API-KEY: dfx_your_key_here"
```
# API Reference
Source: https://docs.flexorch.com/api-reference/introduction
FlexOrch REST API — full endpoint reference.
## Base URL
```
https://api.flexorch.com/v1
```
## Authentication
All requests require either an API key or a JWT token:
```bash theme={null}
# API key
-H "X-API-KEY: dfx_your_key_here"
# JWT token
-H "Authorization: Bearer eyJ..."
```
## Response format
All responses follow the same envelope:
```json theme={null}
{
"data": { ... }
}
```
Errors:
```json theme={null}
{
"error": {
"code": "ERROR_CODE",
"message": "Human-readable message"
}
}
```
## Common error codes
| Code | HTTP | Description |
| ----------------------- | ---- | -------------------------------------------------------------------------------------- |
| `INVALID_API_KEY` | 401 | Key not found or revoked |
| `TRIAL_EXPIRED` | 402 | Trial period ended |
| `QUOTA_EXCEEDED` | 429 | Credit limit reached |
| `RATE_LIMIT_EXCEEDED` | 429 | RPM limit reached |
| `PLAN_UPGRADE_REQUIRED` | 403 | Feature not available on your plan |
| `FILE_TOO_LARGE` | 400 | File exceeds the 5 MB limit (flat across all plans) |
| `UNSUPPORTED_FILE` | 400 | File extension not supported |
| `EXTENSION_MISMATCH` | 400 | File content doesn't match its extension, and the detected type isn't supported either |
| `ENCRYPTED_PDF` | 400 | PDF is password-protected — remove the password and re-upload |
| `PDF_TOO_MANY_PAGES` | 400 | PDF exceeds 300 pages |
| `XLSX_TOO_MANY_ROWS` | 400 | XLSX sheet exceeds 50,000 rows |
## Auto-generated reference
The full endpoint reference below is generated directly from the [OpenAPI spec](https://api.flexorch.com/openapi.json) and is always up to date.
# API Keys
Source: https://docs.flexorch.com/authentication/api-keys
Generate, use, and rotate your FlexOrch API keys.
## Overview
FlexOrch uses API keys for authentication. Every request must include a valid key.
API keys follow the format:
```
dfx_
```
Keys are tied to your account. All requests made with a key are scoped to your workspace.
***
## Generating a key
1. Sign in to [app.flexorch.com](https://app.flexorch.com)
2. Go to **Settings → API Keys**
3. Click **Generate New Key**
4. Copy the key immediately — it is shown only once
You can generate multiple keys (e.g., one per environment or integration).
***
## Using a key
Pass the key in the `X-API-KEY` header:
```bash theme={null}
curl https://api.flexorch.com/v1/usage \
-H "X-API-KEY: dfx_your_key_here"
```
Never expose API keys in client-side code, public repositories, or logs.
***
## Revoking a key
1. Go to **Settings → API Keys**
2. Click **Revoke** next to the key you want to remove
Revoked keys stop working immediately.
***
## Security best practices
* Store keys in environment variables, never in source code
* Use separate keys per environment (development, staging, production)
* Rotate keys periodically — revoke old ones after issuing new ones
* Set up a `.env` file and add it to `.gitignore`
```bash theme={null}
# .env
FLEXORCH_API_KEY=dfx_your_key_here
```
```python theme={null}
import os
from flexorch_sdk import FlexOrch
client = FlexOrch(api_key=os.environ["FLEXORCH_API_KEY"])
```
***
## Rate limits
API key requests are subject to plan-based rate limits. If you exceed your limit, the API returns `429 Too Many Requests`.
Check your current usage and limits:
```bash theme={null}
curl https://api.flexorch.com/v1/usage/rate-limits \
-H "X-API-KEY: dfx_your_key_here"
```
```json theme={null}
{
"data": {
"plan": "starter",
"rate_limit": {
"rpm": 60,
"used": 12,
"remaining": 48,
"reset_in_seconds": 34
}
}
}
```
On `429`, wait for `reset_in_seconds` before retrying. The SDK handles this automatically with exponential backoff.
# JWT Authentication
Source: https://docs.flexorch.com/authentication/jwt
Authenticate using email and password with JWT tokens.
## Overview
In addition to API keys, FlexOrch supports JWT-based authentication for browser and user-facing flows. This is used by the platform UI (`app.flexorch.com`) and is available for integrations that need user-level auth.
***
## Sign up
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/auth/signup \
-H "Content-Type: application/json" \
-d '{"email": "you@company.com", "password": "your_password"}'
```
After signup, a verification email is sent. The account must be verified before login.
***
## Log in
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/auth/login \
-H "Content-Type: application/json" \
-d '{"email": "you@company.com", "password": "your_password"}'
```
Response:
```json theme={null}
{
"data": {
"access_token": "eyJ...",
"refresh_token": "eyJ...",
"token_type": "bearer",
"expires_in": 3600
}
}
```
***
## Using a JWT token
Pass the access token as a Bearer header:
```bash theme={null}
curl https://api.flexorch.com/v1/usage \
-H "Authorization: Bearer eyJ..."
```
***
## Refreshing a token
Access tokens expire after 1 hour. Use the refresh token to get a new one:
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/auth/refresh \
-H "Content-Type: application/json" \
-d '{"refresh_token": "eyJ..."}'
```
***
## Choosing between API key and JWT
| Use case | Recommended |
| ----------------------------- | --------------------------- |
| Server-to-server integrations | API key |
| CI/CD pipelines | API key |
| Browser-based apps | JWT |
| Platform UI | JWT (handled automatically) |
For most integrations, **API keys are simpler** — see [API Keys](/authentication/api-keys).
# eIDAS — Electronic Signatures
Source: https://docs.flexorch.com/compliance/eidas
FlexOrch detects electronic signature formats in documents.
## Overview
eIDAS (EU 910/2014) defines standards for electronic signatures. FlexOrch detects the presence of eIDAS-compliant signature formats in processed documents and records this in the execution metadata.
## Detected signature formats
| Format | Description |
| ------ | --------------------------------- |
| XAdES | XML Advanced Electronic Signature |
| PAdES | PDF Advanced Electronic Signature |
| CAdES | CMS Advanced Electronic Signature |
## In the execution result
When an eIDAS signature is detected, the execution metadata includes:
```json theme={null}
{
"eidas_detected": true,
"eidas_formats": ["PAdES"],
"eidas_note": "Document contains a PAdES electronic signature. Validity verification requires a qualified trust service provider (QTSP)."
}
```
FlexOrch detects signature presence and format, but does not validate signature validity or certificate trust chains. Use a qualified trust service provider (QTSP) for legal signature validation.
# GDPR Article 30 (ROPA)
Source: https://docs.flexorch.com/compliance/gdpr-article-30
Generate a Record of Processing Activities from your pipeline data.
## Overview
GDPR Article 30 requires data controllers to maintain a record of processing activities (ROPA). FlexOrch generates a ROPA-compatible record based on your pipeline executions.
## Get the ROPA record
```bash theme={null}
curl https://api.flexorch.com/v1/compliance/record \
-H "X-API-KEY: dfx_your_key_here"
```
Response:
```json theme={null}
{
"data": {
"controller": "your-company@example.com",
"processing_activities": [
{
"category": "document_processing",
"data_subjects": ["employees", "customers", "vendors"],
"data_categories": ["identification", "financial", "contact"],
"retention_period": "30 days",
"technical_measures": ["pseudonymization", "encryption_at_rest"]
}
],
"generated_at": "2024-01-15T10:00:00Z"
}
}
```
The record reflects your actual pipeline activity — document types processed, PII categories found, and retention settings.
## Export as PDF
Coming soon — available for Enterprise plans.
# KVKK Compliance
Source: https://docs.flexorch.com/compliance/kvkk
FlexOrch and Turkish personal data protection law (KVKK).
## Overview
KVKK (Kişisel Verilerin Korunması Kanunu) is Turkey's personal data protection law, aligned with GDPR principles. FlexOrch detects Turkish-specific personal data types and provides masking and audit capabilities to support KVKK compliance.
## Detected Turkish PII types
| Type | Description | KVKK Category |
| ----------------- | ------------------------- | ---------------------- |
| `national_id_tr` | T.C. Kimlik No | Kimlik verisi |
| `tax_id_tr` | Vergi Kimlik No | Finansal veri |
| `phone_tr` | Turkish mobile / landline | İletişim verisi |
| `iban_tr` | Turkish IBAN (TR prefix) | Finansal veri |
| `emeklilik_no` | SGK Emeklilik sicil no | Sosyal güvenlik verisi |
| `isyeri_sicil_no` | SGK İşyeri sicil no | Çalışma verisi |
| `bagkur_no` | Bağ-Kur sicil no | Sosyal güvenlik verisi |
## Masking for KVKK compliance
Enable masking to ensure datasets do not contain raw personal data:
```bash theme={null}
curl https://api.flexorch.com/v1/datasets/{id}/export?format=jsonl \
-H "X-API-KEY: dfx_your_key_here"
```
Masked fields use `[MASKED_NATIONAL_ID_TR]`, `[MASKED_IBAN_TR]` placeholders — the raw values are not included in the export.
## Audit trail
All pipeline executions are logged. The audit log is available via:
```bash theme={null}
curl https://api.flexorch.com/v1/audit/export \
-H "X-API-KEY: dfx_your_key_here"
```
## Privacy policy
FlexOrch Technology processes document data on behalf of its customers as a data processor under KVKK Article 12. For inquiries: [privacy@flexorch.com](mailto:privacy@flexorch.com)
# NIS2 Audit Export
Source: https://docs.flexorch.com/compliance/nis2-audit-export
Export your audit log for NIS2 incident documentation.
Requires an account. Available on all plans.
## Overview
The NIS2 Directive (EU 2022/2555) requires organizations to document security incidents and data processing activities. FlexOrch provides an audit log export covering all pipeline activity.
## Export the audit log
```bash theme={null}
# JSON format
curl "https://api.flexorch.com/v1/audit/export?format=json" \
-H "X-API-KEY: dfx_your_key_here" \
-o audit_log.json
# CSV format
curl "https://api.flexorch.com/v1/audit/export?format=csv" \
-H "X-API-KEY: dfx_your_key_here" \
-o audit_log.csv
```
## Log contents
Each audit record includes:
| Field | Description |
| ------------- | ------------------------------------------------------------------ |
| `timestamp` | Event time (UTC) |
| `event_type` | `job.completed`, `job.failed`, `dataset.built`, `auth.login`, etc. |
| `tenant_id` | Your account identifier |
| `document_id` | Document affected (if applicable) |
| `outcome` | `success` or `failure` |
Raw document content and PII values are never included in the audit log.
# Compliance Overview
Source: https://docs.flexorch.com/compliance/overview
KVKK, GDPR, NIS2, and eIDAS — how FlexOrch supports regulatory requirements.
FlexOrch helps technical teams work with documents that contain personal data, with built-in tooling for KVKK (Turkey), GDPR (EU), NIS2 (EU), and eIDAS (EU) requirements.
## What FlexOrch provides
| Regulation | Support |
| ---------- | ----------------------------------------------------------------------------------------- |
| **KVKK** | PII detection for Turkish personal data types; audit trail; masking before dataset export |
| **GDPR** | 46 EU/US PII types; Article 30 ROPA record; data minimization via masking |
| **NIS2** | Audit log export (JSON + CSV) for incident documentation |
| **eIDAS** | Electronic signature format detection (XAdES, PAdES, CAdES) in documents |
## Important notes
FlexOrch is a technical tool, not a legal compliance service. The outputs assist compliance workflows but do not constitute legal advice. Consult your data protection officer (DPO) for regulatory obligations.
## Data handling
* Uploaded documents are stored only for the retention period defined by your plan
* PII is masked before dataset export by default
* All pipeline executions are logged and auditable
* No document content is shared with third parties
# Core Concepts
Source: https://docs.flexorch.com/concepts
Key terms and concepts you'll encounter when using FlexOrch.
## Document
A document is any file you upload for processing. FlexOrch accepts PDF, DOCX, PPTX, TXT, XLSX, HTML, XML, EML, MSG, JPG, PNG, TIFF, and more.
Each uploaded document receives a unique `document_id` and is stored for the duration defined by your plan's retention policy.
***
## Job
A **job** represents one processing run for a single document. When you upload a file, FlexOrch creates a job and runs the pipeline asynchronously.
Job states:
| State | Meaning |
| ----------- | ---------------------------------------- |
| `queued` | Waiting to be picked up |
| `running` | Pipeline in progress |
| `completed` | Processing finished successfully |
| `failed` | Processing failed — see `failure_reason` |
***
## Pipeline
The pipeline is the sequence of steps FlexOrch runs on every document:
1. **Extract** — Parse raw text from the file (OCR for images/scanned PDFs)
2. **Classify** — Detect document type (invoice, payroll, etc.)
3. **Extract fields** — Pull structured data using deterministic patterns, falling back to LLM for complex fields
4. **Detect PII** — Identify personal and sensitive data across 46 types
5. **Quality score** — Compute a score (0–100) and grade (A–D)
6. **Deliver** — Write results to the pipeline execution record
***
## Execution
A **pipeline execution** is the record of one pipeline run: extracted fields, detected language, quality score, PII summary, and masked text (if PII was found).
***
## Dataset
A **dataset** is a collection of pipeline executions you explicitly build and export. Datasets contain cleaned, structured records from one or more processed documents.
Building a dataset is a separate step — you choose which completed jobs to include.
***
## Quality Grade
Every document receives a quality grade:
| Grade | Score | Meaning |
| ----- | ------ | ------------------------------------------- |
| **A** | 85–100 | High quality — ready for production |
| **B** | 65–84 | Good quality — minor gaps |
| **C** | 45–64 | Moderate — some fields missing or noisy |
| **D** | 0–44 | Low quality — significant extraction issues |
***
## PII Type
A PII type identifies the kind of personal data found (e.g., `email`, `phone_tr`, `national_id_tr`, `iban`). FlexOrch detects 46 types across Turkish, EU, and US jurisdictions.
See [PII Detection](/guides/pii-privacy) for the full type catalog.
***
## Credit
Credits are consumed per processed job. The amount depends on the **processing engine** and **document size**:
| Engine | Credits per job |
| ------------- | --------------- |
| Deterministic | \~1 credit |
| Local AI | \~2–3 credits |
| Premium model | \~5–15 credits |
Dataset builds and exports do not consume credits. Credit usage is tracked per billing period. The [Usage endpoint](/api-reference/usage) shows your current balance.
***
## Connector
A **connector** is a connection to external storage (Amazon S3, Google Cloud Storage, Azure Blob). Documents can be ingested directly from a bucket without manual upload.
***
## Workspace
Each tenant has a single workspace that scopes all their documents, jobs, datasets, and connectors. The workspace identifier is derived from your API key.
# Amazon S3
Source: https://docs.flexorch.com/connectors/amazon-s3
Connect FlexOrch to an S3 bucket.
## Prerequisites
* An AWS account
* An S3 bucket
* An IAM user with `s3:GetObject` and `s3:ListBucket` permissions on the target bucket
## Create the connector
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/connectors \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{
"type": "s3",
"name": "my-s3-connector",
"config": {
"bucket": "my-documents",
"region": "eu-west-1",
"aws_access_key_id": "AKIA...",
"aws_secret_access_key": "..."
}
}'
```
## Ingest files
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/data-process/async \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{
"source": {
"connector_id": 1,
"keys": [
"invoices/jan/inv_001.pdf",
"invoices/jan/inv_002.pdf"
]
}
}'
```
## Config fields
| Field | Required | Description |
| ----------------------- | -------- | ------------------------------------- |
| `bucket` | Yes | S3 bucket name |
| `region` | Yes | AWS region (e.g., `us-east-1`) |
| `aws_access_key_id` | Yes | IAM access key |
| `aws_secret_access_key` | Yes | IAM secret key |
| `prefix` | No | Default key prefix for scheduled sync |
Store AWS credentials securely. The connector config is encrypted at rest.
# Azure Blob Storage
Source: https://docs.flexorch.com/connectors/azure-blob
Connect FlexOrch to an Azure Blob container.
## Prerequisites
* An Azure Storage account
* A blob container
* A storage account access key (or SAS token — coming soon)
## Create the connector
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/connectors \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{
"type": "azure_blob",
"name": "my-azure-connector",
"config": {
"account_name": "mystorageaccount",
"container_name": "documents",
"account_key": "..."
}
}'
```
## Config fields
| Field | Required | Description |
| ---------------- | -------- | -------------------------------------- |
| `account_name` | Yes | Azure Storage account name |
| `container_name` | Yes | Blob container name |
| `account_key` | Yes | Storage account key |
| `prefix` | No | Default blob prefix for scheduled sync |
# Google Cloud Storage
Source: https://docs.flexorch.com/connectors/google-cloud-storage
Connect FlexOrch to a GCS bucket.
## Prerequisites
* A Google Cloud project
* A GCS bucket
* A service account with `storage.objects.get` and `storage.objects.list` roles
* A service account JSON key
## Create the connector
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/connectors \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{
"type": "gcs",
"name": "my-gcs-connector",
"config": {
"project_id": "my-gcp-project",
"bucket": "my-documents",
"credentials_json": "{\"type\": \"service_account\", ...}"
}
}'
```
## Config fields
| Field | Required | Description |
| ------------------ | -------- | ------------------------------------- |
| `project_id` | Yes | GCP project ID |
| `bucket` | Yes | GCS bucket name |
| `credentials_json` | Yes | Service account key JSON (as string) |
| `prefix` | No | Default key prefix for scheduled sync |
# Connectors
Source: https://docs.flexorch.com/connectors/overview
Connect FlexOrch to S3, Google Cloud Storage, or Azure Blob.
Connectors let FlexOrch ingest documents directly from cloud storage — no manual upload required.
## Supported connectors
| Connector | Type |
| -------------------------------------------------------- | ------------ |
| [Amazon S3](/connectors/amazon-s3) | `s3` |
| [Google Cloud Storage](/connectors/google-cloud-storage) | `gcs` |
| [Azure Blob Storage](/connectors/azure-blob) | `azure_blob` |
## Create a connector
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/connectors \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{
"type": "s3",
"name": "production-docs",
"config": {
"bucket": "my-documents",
"region": "us-east-1",
"aws_access_key_id": "AKIA...",
"aws_secret_access_key": "..."
}
}'
```
## Test a connector
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/connectors/{id}/test \
-H "X-API-KEY: dfx_your_key_here"
```
## Use a connector
Pass `source.connector_id` in an upload request:
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/data-process/async \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{"source": {"connector_id": 1, "keys": ["folder/file.pdf"]}}'
```
Connectors can also be used with [Scheduled Sync](/scheduler/overview) to process new files automatically.
# Chunk Quality
Source: https://docs.flexorch.com/guides/chunk-quality
How FlexOrch scores and grades chunks, and which thresholds to use for RAG pipelines.
## Quality grades
Every document processed through FlexOrch receives a quality grade:
| Grade | Score range | Meaning | RAG recommendation |
| ----- | ----------- | --------------------------------------------------------- | ---------------------- |
| **A** | ≥ 0.85 | Excellent — clean text, complete fields, low noise | Always index |
| **B** | ≥ 0.65 | Good — minor noise or gaps, suitable for production | Index |
| **C** | ≥ 0.40 | Marginal — OCR issues, encoding errors, or sparse content | Review before indexing |
| **D** | \< 0.40 | Poor — garbage content, OCR failure, or nearly empty | Exclude |
**Recommended minimum for production RAG:** grade **B**.
***
## How the score is computed
The quality score is a composite of three signals:
```
quality_score = completeness × (0.4 × noise_score + 0.4 × length_score + 0.2)
```
| Signal | Description |
| -------------- | ------------------------------------------------------------------ |
| `completeness` | Fraction of non-empty lines. Empty = 0.0, all filled = 1.0 |
| `length_score` | `min(avg_length / 500, 1.0)` — penalizes very short texts |
| `noise_score` | `max(0, 1 − garbage_ratio × 10)` — penalizes symbol/encoding noise |
A document with 100% complete fields, healthy average line length, and no noise scores 1.0 → grade **A**.
An OCR-failed scan with short, symbol-heavy lines might score 0.20 → grade **D**.
***
## OCR confidence penalty
For scanned PDFs and images, an additional OCR confidence check applies:
* OCR confidence ≥ 0.70 → no penalty
* OCR confidence \< 0.70 → quality score capped at 0.74 → maximum grade **C**
This ensures that readable-but-unreliable OCR output never receives an A or B grade.
***
## Filtering by quality at export
```bash theme={null}
# Export only A and B grade chunks
curl "https://api.flexorch.com/v1/datasets/{id}/export?format=rag&min_quality=B" \
-H "X-API-KEY: dfx_your_key_here"
```
```python theme={null}
# Python SDK — export
client.datasets.export(dataset_id=89, format="rag", path="chunks.json", min_quality="B")
# Python SDK — chunks API
page = client.datasets.get(89).chunks(quality_grade="A,B", page_size=100)
```
***
## Filtering with flexorch-audit (OSS)
For local files before they reach the platform:
```python theme={null}
from flexorch_audit import audit, rag_metadata, chunk_text
text = open("contract.txt").read()
result = audit(text)
meta = rag_metadata(result)
print(meta)
# {
# "should_index": True, # grade A or B
# "pii_safe": True, # no PII detected
# "quality_gate": "pass", # "pass" | "warn" | "fail"
# "recommended_chunk_size": 512, # tokens
# "estimated_chunks": 6
# }
if meta["should_index"]:
chunks = chunk_text(
text,
strategy="paragraph",
max_tokens=meta["recommended_chunk_size"],
)
for c in chunks:
print(c["chunk_index"], c["token_count"], c["text"][:80])
```
***
## Chunking strategies
| Strategy | How it splits | Best for |
| --------------------- | --------------------------------------- | ----------------------------------------------- |
| `paragraph` (default) | On blank lines, merges short paragraphs | Prose documents, contracts, reports |
| `sliding_window` | Fixed-size overlapping windows | Uniform density required, long homogeneous text |
| `sentence` | On sentence-ending punctuation | Short, factual sentences where context is local |
```python theme={null}
from flexorch_audit import chunk_text
# Paragraph (default)
chunks = chunk_text(text, strategy="paragraph", max_tokens=512)
# Sliding window with 64-token overlap
chunks = chunk_text(text, strategy="sliding_window", max_tokens=512, overlap=64)
# Sentence
chunks = chunk_text(text, strategy="sentence", max_tokens=256)
```
Each chunk is a dict:
```python theme={null}
{
"text": "Invoice FTR-2024-001 was issued by...",
"token_count": 84,
"chunk_index": 0
}
```
***
## Choosing chunk size
| Document type | Recommended size | Notes |
| ---------------------- | ----------------------- | ---------------------------- |
| Invoice / PO / Payroll | Single structured chunk | Keep all fields together |
| Bank statement | 256 tokens | One transaction per chunk |
| Contract / proposal | 512 tokens | Paragraph boundary preferred |
| Long report | 512 – 1024 tokens | Sliding window with overlap |
| Email thread | 256 tokens | Per-message chunks |
`rag_metadata()` returns a `recommended_chunk_size` based on the detected text length — use it as a starting point.
***
## PII and RAG
PII in chunks can leak sensitive data to your LLM. FlexOrch masks PII automatically:
* `pii_masked=True` on export → `[MASKED_EMAIL]`, `[MASKED_NATIONAL_ID_TR]`, etc.
* Filter chunks where `metadata.pii_masked == true` to guarantee clean RAG context
* Check `metadata.pii_findings_count` — anything > 0 should be reviewed before entering a public-facing chain
```python theme={null}
# Only load PII-safe chunks
documents = reader.load_data(dataset_id=89, pii_masked_only=True)
```
End-to-end: export chunks or managed search
Masking strategies and compliance notes
# Datasets & Export
Source: https://docs.flexorch.com/guides/datasets
Build datasets from processed documents and export in 9 formats.
## Overview
A dataset is a curated collection of pipeline execution results — structured fields, extracted text, and metadata — ready for LLM training, RAG, or analytics.
Building and exporting datasets does **not** consume credits.
***
## Build a dataset
After your documents are processed (job status `completed`), build a dataset:
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/datasets \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{"name": "invoices-q1", "execution_ids": ["exec_001", "exec_002"]}'
```
***
## Export formats
| Format | Extension | Best for |
| ----------------- | ---------- | ----------------------------------------------------- |
| JSONL | `.jsonl` | LLM fine-tuning (OpenAI, Anthropic) |
| CSV | `.csv` | Spreadsheet analysis |
| Parquet | `.parquet` | Data pipelines, analytics |
| Markdown | `.md` | RAG — LlamaIndex, LangChain |
| XML | `.xml` | Enterprise integrations |
| XLSX | `.xlsx` | Excel-based workflows |
| HuggingFace Arrow | `zip` | `datasets.load_from_disk()` — Arrow IPC with metadata |
| RAG chunks | `.json` | Semantic chunking — LangChain / LlamaIndex compatible |
***
## Export a dataset
```bash theme={null}
curl "https://api.flexorch.com/v1/datasets/{id}/export?format=jsonl" \
-H "X-API-KEY: dfx_your_key_here" \
-o output.jsonl
```
### RAG export with quality filtering
Pass `min_quality` to exclude low-quality chunks before export:
```bash theme={null}
# Export only A and B grade chunks as RAG JSON
curl "https://api.flexorch.com/v1/datasets/{id}/export?format=rag&min_quality=B" \
-H "X-API-KEY: dfx_your_key_here" \
-o chunks.json
# Export as HuggingFace Arrow dataset (zip)
curl "https://api.flexorch.com/v1/datasets/{id}/export?format=hf&min_quality=B" \
-H "X-API-KEY: dfx_your_key_here" \
-o dataset.zip
```
| `min_quality` value | Included grades |
| ------------------- | --------------- |
| `A` | A only |
| `B` (recommended) | A, B |
| `C` | A, B, C |
| omit | All grades |
***
## Dataset profile
Get aggregate statistics for a dataset:
```bash theme={null}
curl "https://api.flexorch.com/v1/datasets/{id}/profile" \
-H "X-API-KEY: dfx_your_key_here"
```
Returns quality grade distribution, PII type summary, average score, and available export formats.
***
## Chunks API
Available on **Pro and Enterprise** plans.
Retrieve individual text chunks with quality and PII filters:
```bash theme={null}
curl "https://api.flexorch.com/v1/datasets/{id}/chunks?quality_grade=A,B&pii_masked=true&page=1&page_size=20" \
-H "X-API-KEY: dfx_your_key_here"
```
**Query parameters:**
| Parameter | Description |
| --------------- | ----------------------------------------------- |
| `quality_grade` | Comma-separated grades to include, e.g. `A,B` |
| `pii_masked` | `true` / `false` — filter by PII masking status |
| `page` | Page number (1-indexed) |
| `page_size` | Chunks per page, max 100 |
**Response:**
```json theme={null}
{
"data": {
"chunks": [
{
"chunk_id": "abc-001",
"chunk_index": 0,
"text": "Invoice FTR-2024-001 from [MASKED_NAME]...",
"token_count": 84,
"metadata": { "quality_grade": "A", "pii_masked": true }
}
],
"total": 47,
"page": 1,
"page_size": 20,
"has_more": true
}
}
```
See the [RAG pipeline guide](/guides/rag-pipeline) for a complete integration walkthrough.
***
## Semantic indexing
Available on **Pro and Enterprise** plans.
Index a dataset for semantic search:
```bash theme={null}
curl -X POST "https://api.flexorch.com/v1/datasets/{id}/index" \
-H "X-API-KEY: dfx_your_key_here"
```
Then search:
```bash theme={null}
curl -X POST "https://api.flexorch.com/v1/search" \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{"query": "invoices over 10000 EUR from Germany", "top_k": 5, "mode": "hybrid"}'
```
***
## Dataset retention
Datasets are stored for a period defined by your plan:
| Plan | Retention |
| ---------- | ------------ |
| Trial | 7 days |
| Starter | 30 days |
| Pro | 90 days |
| Enterprise | Configurable |
# LangChain Integration
Source: https://docs.flexorch.com/guides/langchain
Use FlexOrchRetriever to wire FlexOrch semantic search into any LangChain chain.
This guide covers the **platform SDK** (`flexorch-sdk`). For local file auditing with the zero-dependency OSS library, see [flexorch-audit LangChain](/open-source/langchain).
## Install
```bash theme={null}
pip install flexorch-sdk langchain langchain-openai
```
***
## FlexOrchRetriever
`FlexOrchRetriever` implements LangChain's `BaseRetriever` interface. It calls `POST /v1/search` under the hood and returns `RAGDocument` objects that are duck-type compatible with LangChain's `Document`.
```python theme={null}
from flexorch_sdk import FlexOrch, FlexOrchRetriever
client = FlexOrch() # reads FLEXORCH_API_KEY from env
retriever = FlexOrchRetriever(
client,
quality_threshold="B", # include A and B grade chunks
pii_masked=True, # only return PII-safe chunks
top_k=5,
mode="hybrid", # auto | semantic | hybrid | structured
document_type="invoice", # optional filter
language="tr", # optional ISO 639-1 filter
)
docs = retriever.invoke("invoices over 10000 EUR")
for doc in docs:
print(doc.page_content[:100])
print(doc.metadata)
```
### Parameters
| Parameter | Type | Default | Description |
| ------------------- | -------------- | -------- | ------------------------------------------------------- |
| `client` | `FlexOrch` | required | Authenticated SDK client |
| `quality_threshold` | `str` | `"B"` | Minimum grade — `"A"`, `"B"`, `"C"`, or `"D"` |
| `pii_masked` | `bool \| None` | `None` | Filter to PII-masked chunks only |
| `top_k` | `int` | `5` | Number of results to return |
| `mode` | `str` | `"auto"` | Search mode: `auto`, `semantic`, `hybrid`, `structured` |
| `document_type` | `str \| None` | `None` | Filter by document type |
| `language` | `str \| None` | `None` | Filter by ISO 639-1 language code |
`mode="semantic"` and `mode="hybrid"` require a **Pro or Enterprise** plan. Pass `mode="auto"` to fall back to structured search on lower plans.
***
## RetrievalQA chain
```python theme={null}
from flexorch_sdk import FlexOrch, FlexOrchRetriever
from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI
client = FlexOrch()
retriever = FlexOrchRetriever(client, quality_threshold="B", mode="hybrid")
qa = RetrievalQA.from_chain_type(
llm=ChatOpenAI(model="gpt-4o-mini"),
retriever=retriever,
return_source_documents=True,
)
result = qa.invoke("What is the payment term on invoice FTR-2024-001?")
print(result["result"])
for doc in result["source_documents"]:
print(doc.metadata["quality_grade"], doc.page_content[:60])
```
***
## LCEL chain
```python theme={null}
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import ChatOpenAI
prompt = ChatPromptTemplate.from_template(
"Answer the question based only on the context below.\n\n"
"Context: {context}\n\nQuestion: {question}"
)
def format_docs(docs):
return "\n\n".join(doc.page_content for doc in docs)
chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| ChatOpenAI(model="gpt-4o-mini")
| StrOutputParser()
)
print(chain.invoke("Total amount on the March invoices?"))
```
***
## Quality and language filters
```python theme={null}
# German contracts — grade A only
retriever_de = FlexOrchRetriever(
client,
quality_threshold="A",
document_type="sales_proposal",
language="de",
top_k=10,
)
# Turkish payroll — grade B+, PII masked
retriever_tr = FlexOrchRetriever(
client,
quality_threshold="B",
pii_masked=True,
document_type="payroll",
language="tr",
)
```
***
## Async support
`FlexOrchRetriever` implements `aget_relevant_documents()` for async chains:
```python theme={null}
docs = await retriever.aget_relevant_documents("payment terms")
```
***
## RAGDocument fields
`FlexOrchRetriever.invoke()` returns a list of `RAGDocument` objects:
| Field | Type | Description |
| -------------- | ------ | ------------------------------------------------ |
| `page_content` | `str` | Chunk text (LangChain compatible) |
| `text` | `str` | Alias for `page_content` (LlamaIndex compatible) |
| `metadata` | `dict` | Chunk metadata |
Metadata keys:
| Key | Description |
| --------------- | ------------------------------------- |
| `chunk_id` | Unique chunk identifier |
| `dataset_id` | Source dataset |
| `quality_grade` | `A` \| `B` \| `C` \| `D` |
| `quality_score` | 0.0 – 1.0 |
| `doc_type` | Document classification |
| `language` | ISO 639-1 language code |
| `pii_masked` | `true` if PII was masked |
| `score` | Retrieval relevance score (0.0 – 1.0) |
End-to-end guide: export vs. managed search
FlexOrchReader for VectorStoreIndex
# LlamaIndex Integration
Source: https://docs.flexorch.com/guides/llamaindex
Use FlexOrchReader to load FlexOrch dataset chunks into a LlamaIndex VectorStoreIndex.
This guide covers the **platform SDK** (`flexorch-sdk`). For local file auditing with the zero-dependency OSS library, see [flexorch-audit LlamaIndex](/open-source/llamaindex).
## Install
```bash theme={null}
pip install flexorch-sdk llama-index-core
```
***
## FlexOrchReader
`FlexOrchReader` fetches all chunks from a built FlexOrch dataset with automatic pagination and returns `RAGDocument` objects that are duck-type compatible with LlamaIndex's `Document` (`.text` property, `.metadata` dict).
```python theme={null}
from flexorch_sdk import FlexOrch, FlexOrchReader
client = FlexOrch() # reads FLEXORCH_API_KEY from env
reader = FlexOrchReader(client)
documents = reader.load_data(
dataset_id=89,
min_quality="B", # exclude C and D grade chunks
pii_masked_only=True, # only load PII-safe chunks
page_size=100, # chunks fetched per request
)
for doc in documents:
print(doc.text[:100])
print(doc.metadata["quality_grade"])
```
### Parameters
| Parameter | Type | Default | Description |
| ----------------- | ------ | -------- | ---------------------------------------------------- |
| `dataset_id` | `int` | required | ID of the built FlexOrch dataset |
| `min_quality` | `str` | `"B"` | Minimum grade to include: `"A"`, `"B"`, `"C"`, `"D"` |
| `pii_masked_only` | `bool` | `False` | When `True`, only return chunks where PII was masked |
| `page_size` | `int` | `100` | Chunks fetched per API request (max 100) |
Chunk access via `GET /v1/datasets/{id}/chunks` requires a **Pro or Enterprise** plan.
***
## VectorStoreIndex
```python theme={null}
from flexorch_sdk import FlexOrch, FlexOrchReader
from llama_index.core import VectorStoreIndex
client = FlexOrch()
reader = FlexOrchReader(client)
documents = reader.load_data(dataset_id=89, min_quality="B")
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What are the payment terms in the contracts?")
print(response)
```
***
## Persist and reload
```python theme={null}
from llama_index.core import StorageContext, load_index_from_storage
# Persist
index.storage_context.persist("./storage")
# Reload (no re-fetch needed)
storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context)
```
***
## Streaming query
```python theme={null}
query_engine = index.as_query_engine(streaming=True)
response = query_engine.query("Summarise the Q1 invoices.")
for token in response.response_gen:
print(token, end="", flush=True)
```
***
## Filter by quality at query time
```python theme={null}
from llama_index.core.vector_stores import MetadataFilter, MetadataFilters
filters = MetadataFilters(filters=[
MetadataFilter(key="quality_grade", value="A"),
])
query_engine = index.as_query_engine(filters=filters)
response = query_engine.query("High-quality contract clauses only")
```
***
## Multi-dataset index
Load chunks from several datasets into a single index:
```python theme={null}
all_documents = []
for dataset_id in [89, 90, 91]:
docs = reader.load_data(dataset_id=dataset_id, min_quality="B")
all_documents.extend(docs)
index = VectorStoreIndex.from_documents(all_documents)
```
***
## RAGDocument fields
`FlexOrchReader.load_data()` returns a list of `RAGDocument` objects:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------- |
| `text` | `str` | Chunk text (LlamaIndex compatible) |
| `page_content` | `str` | Alias for `text` (LangChain compatible) |
| `metadata` | `dict` | Chunk metadata |
Metadata keys:
| Key | Description |
| --------------- | --------------------------- |
| `chunk_id` | Unique chunk identifier |
| `chunk_index` | Position within the dataset |
| `dataset_id` | Source dataset |
| `quality_grade` | `A` \| `B` \| `C` \| `D` |
| `quality_score` | 0.0 – 1.0 |
| `doc_type` | Document classification |
| `language` | ISO 639-1 language code |
| `pii_masked` | `true` if PII was masked |
| `token_count` | Estimated token count |
End-to-end guide: export vs. managed search
FlexOrchRetriever for chain-based retrieval
# PII Detection & Privacy
Source: https://docs.flexorch.com/guides/pii-privacy
Personal data types detected by FlexOrch, masking strategies, and compliance outputs.
## Overview
FlexOrch detects personal and sensitive data in your documents before they enter a dataset. Detected PII is masked in the output, giving you clean training data without privacy risk.
Detection covers **46 PII types** across Turkish, EU, and US jurisdictions.
***
## Detected types
### Universal
| Type | Description |
| ------------- | ----------------------------------------------------------- |
| `email` | Email addresses |
| `credit_card` | Credit card numbers (all major schemes, Luhn-validated) |
| `ip` | IPv4 addresses |
| `ip_v6` | IPv6 addresses |
| `iban` | International bank account numbers (IBAN, mod-97 validated) |
### Turkey (TR)
| Type | Description |
| ----------------- | ----------------------------------- |
| `national_id_tr` | T.C. Kimlik No (11 digits) |
| `tax_id_tr` | Vergi Kimlik No (10 digits, VKN) |
| `phone_tr` | Turkish mobile and landline numbers |
| `iban_tr` | Turkish IBANs (TR prefix, 26 chars) |
| `emeklilik_no` | SGK Emeklilik sicil no |
| `isyeri_sicil_no` | SGK İşyeri sicil no |
| `bagkur_no` | Bağ-Kur sicil no |
### Germany (DE)
| Type | Description |
| ---------------- | --------------------------------------- |
| `national_id_de` | Personalausweis number |
| `tax_id_de` | Steueridentifikationsnummer (11 digits) |
| `phone_de` | German phone numbers |
| `iban_de` | German IBANs (DE prefix) |
### France (FR)
| Type | Description |
| ---------------- | -------------------------------------- |
| `national_id_fr` | Numéro de sécurité sociale (15 digits) |
| `tax_id_fr` | Numéro SPI / NIF |
| `phone_fr` | French phone numbers |
| `iban_fr` | French IBANs (FR prefix) |
### Italy (IT)
| Type | Description |
| ---------------- | ----------------------------- |
| `national_id_it` | Codice Fiscale (alphanumeric) |
| `phone_it` | Italian phone numbers |
| `iban_it` | Italian IBANs (IT prefix) |
### Netherlands (NL)
| Type | Description |
| ---------------- | ----------------------------------- |
| `national_id_nl` | Burgerservicenummer (BSN, 9 digits) |
| `tax_id_nl` | BTW-nummer |
| `phone_nl` | Dutch phone numbers |
### Spain (ES)
| Type | Description |
| ---------------- | --------------------- |
| `national_id_es` | DNI / NIE |
| `tax_id_es` | NIF |
| `phone_es` | Spanish phone numbers |
### Poland (PL)
| Type | Description |
| ---------------- | -------------------- |
| `national_id_pl` | PESEL (11 digits) |
| `tax_id_pl` | NIP (10 digits) |
| `phone_pl` | Polish phone numbers |
### United Kingdom (UK) & United States (US)
| Type | Description |
| ---------------- | -------------------------------- |
| `national_id_uk` | National Insurance Number (NINO) |
| `ssn_us` | Social Security Number |
| `phone_uk` | UK phone numbers |
| `phone_us` | US phone numbers |
### Additional EU jurisdictions
Austria, Belgium, Sweden, Denmark, Finland, Portugal — national ID and phone types covered.
***
## Masking strategies
When PII is detected, FlexOrch replaces each finding with a placeholder. The default strategy is `redact`:
| Strategy | Output example | Use case |
| --------- | -------------------- | ---------------------------------- |
| `redact` | `[MASKED_EMAIL]` | Production datasets, fine-tuning |
| `replace` | `person@example.com` | Testing, plausible synthetic data |
| `token` | `` | Preserving structure for NLP tasks |
| `hash` | `a3f2b1...` | Deterministic anonymization |
***
## Results in the API
After processing, PII information is available in the job response:
```json theme={null}
{
"pii_findings_count": 4,
"privacy_applied": true,
"processing_summary": {
"pii_type_summary": {
"email": 2,
"national_id_tr": 1,
"iban": 1
}
}
}
```
The full type breakdown is also available in the dataset profile:
```bash theme={null}
curl https://api.flexorch.com/v1/datasets/{id}/profile \
-H "X-API-KEY: dfx_your_key_here"
```
***
## KVKK & GDPR compliance
FlexOrch generates a compliance report for each dataset:
```bash theme={null}
curl https://api.flexorch.com/v1/datasets/{id}/profile \
-H "X-API-KEY: dfx_your_key_here"
```
The profile includes:
* `gdpr_categories` — GDPR Article 9 special category flags
* `detected_countries` — Jurisdictions found in the data
* `applicable_regulations` — KVKK, GDPR, NIS2, eIDAS as relevant
For the GDPR Article 30 Record of Processing Activities (ROPA), use:
```bash theme={null}
curl https://api.flexorch.com/v1/compliance/record \
-H "X-API-KEY: dfx_your_key_here"
```
***
## Open source
PII detection is also available as a standalone, zero-dependency package — no FlexOrch account required.
See [flexorch-audit](/open-source/overview) for installation and usage.
# Pipeline
Source: https://docs.flexorch.com/guides/pipeline
How FlexOrch processes your documents — step by step.
## Overview
Every uploaded document goes through the same pipeline. Understanding the steps helps you interpret results and troubleshoot low-quality outputs.
```
Upload → Extract text → Classify → Extract fields → Detect PII → Score quality → Deliver
```
***
## Step 1 — Text extraction
FlexOrch reads raw text from the document:
* **PDF (text-based):** Direct text extraction
* **PDF (scanned / image-heavy):** OCR is applied automatically when the text layer is insufficient
* **DOCX / PPTX / TXT / HTML:** Direct parse — table content is included for DOCX and PPTX, not just paragraph text
* **XLSX:** Cell values extracted row by row
* **EML / MSG:** Body + headers, with HTML stripped
* **XML e-invoices:** Structured fields parsed natively (FatturaPA, UBL/Peppol, GİB, XRechnung, ZUGFeRD)
* **Images (JPG, PNG, TIFF):** OCR always applied
The `ocr_confidence` score (0–1) is recorded when OCR runs. Values below 0.7 cap the quality grade at C.
Before OCR runs, scanned pages are automatically deskewed: a coarse 90°/180°/270° orientation check corrects sideways or upside-down scans, then a fine-angle check straightens small tilts (a few degrees) from the scanner or camera. This runs on every scanned page — you don't need to pre-process anything.
FlexOrch also verifies file content against the declared extension before extraction starts, so a mislabeled file (wrong or renamed extension) is routed to the correct parser automatically instead of failing or being misread.
***
## Step 2 — Document classification
FlexOrch identifies the document type using keyword-based classification:
| Type | Examples |
| ----------------- | --------------------------------------------- |
| `invoice` | Sales invoices, e-invoices, purchase invoices |
| `expense_report` | Travel expenses, reimbursement forms |
| `purchase_order` | POs, procurement documents |
| `sales_proposal` | Quotes, proposals, offers |
| `bank_statement` | Account statements, transaction lists |
| `payroll` | Payslips, salary summaries |
| `budget` | Budget plans, financial forecasts |
| `delivery_note` | Shipping documents, delivery confirmations |
| `tax_declaration` | Tax forms, declarations |
| `contract` | Agreements, SOWs |
| `general` | Documents that don't match a specific type |
The `classification_method` field in results indicates whether classification was deterministic (rule-based) or required LLM assistance.
***
## Step 3 — Field extraction
Structured fields are extracted based on the document type. FlexOrch uses deterministic pattern matching as the primary method, with LLM fallback for fields that can't be captured by patterns.
Example fields for an invoice:
| Field | Example value |
| --------------- | ---------------------- |
| `vendor` | Acme Ltd. |
| `document_date` | 2024-01-15 |
| `due_date` | 2024-02-15 |
| `total_amount` | 12500.00 |
| `currency` | EUR |
| `iban` | DE89370400440532013000 |
| `document_no` | INV-2024-00421 |
| `line_items` | Array of |
The `extraction_method_per_field` summary in results shows which fields were filled deterministically vs. by LLM.
***
## Step 4 — PII detection
FlexOrch scans the full document text for personal and sensitive data. Detected items are:
* Counted (`pii_findings_count`)
* Categorized by type (`pii_type_summary`)
* Optionally masked in the output (`privacy_applied`)
If PII is found, the `masked_text` version is used for dataset export by default.
See [PII Detection & Privacy](/guides/pii-privacy) for the full type catalog.
***
## Step 5 — Quality scoring
A quality score (0–100) and grade (A–D) are computed based on:
* Field fill rate (how many expected fields were extracted)
* Noise ratio (proportion of meaningless content)
* OCR confidence (for scanned documents)
| Grade | Score | What it means |
| ----- | ------ | --------------------------------------- |
| A | 85–100 | High quality — all key fields extracted |
| B | 65–84 | Good — minor gaps |
| C | 45–64 | Moderate — notable extraction issues |
| D | 0–44 | Low — significant problems |
***
## Step 6 — Delivery
Results are written to the pipeline execution record, accessible via the job response:
```json theme={null}
{
"data": {
"status": "completed",
"detected_language": "tr",
"quality": {
"score": 91,
"grade": "A"
},
"pii_findings_count": 3,
"privacy_applied": true,
"processing_summary": {
"fields": { "vendor": "Acme Ltd.", "total_amount": 12500.00, ... }
}
}
}
```
***
## Reprocessing
Documents can be reprocessed at any time (e.g., after a pipeline improvement):
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/documents/{document_id}/reprocess \
-H "X-API-KEY: dfx_your_key_here"
```
This creates a new job and execution. Previous results are preserved.
# Quality Scores
Source: https://docs.flexorch.com/guides/quality-scores
How FlexOrch grades documents and what the scores mean for your datasets.
## Overview
Every processed document receives a **quality score** (0–100) and a **grade** (A–D). These help you filter high-quality records and identify documents that may need attention before inclusion in a dataset.
***
## Quality grades
| Grade | Score range | What it means |
| ----- | ----------- | ---------------------------------------------------- |
| **A** | 85–100 | High quality — key fields extracted, low noise |
| **B** | 65–84 | Good — minor gaps or slight noise |
| **C** | 45–64 | Moderate — notable extraction issues or OCR problems |
| **D** | 0–44 | Low — significant problems; review before using |
***
## Factors that affect the score
* **Field fill rate** — How many expected fields for the document type were successfully extracted
* **Noise ratio** — Proportion of the document that is non-informative content (e.g., page numbers, repeated headers)
* **OCR confidence** — For scanned documents, a low OCR confidence score caps the grade at C
***
## Accessing quality data
Quality information is available in the job response:
```json theme={null}
{
"processing_summary": {
"quality": {
"score": 91,
"grade": "A",
"field_fill_rate": 0.94,
"ocr_confidence": null
}
}
}
```
`ocr_confidence` is `null` when OCR was not used (text-based PDF or DOCX).
***
## Quality trend
Track quality over time on the **Usage → Quality** tab in the platform, or via the API:
```bash theme={null}
curl "https://api.flexorch.com/v1/usage/quality-trend?period=30d" \
-H "X-API-KEY: dfx_your_key_here"
```
***
## Filtering by grade
Use the platform's **Datasets → Ready to Build** tab to filter completed jobs by grade before building a dataset. Or filter in your own pipeline:
```python theme={null}
from flexorch_sdk import FlexOrch
client = FlexOrch(api_key="dfx_your_key_here")
# Only include grade A and B jobs in the dataset
good_jobs = [j for j in client.jobs.list() if j.quality_grade in ("A", "B")]
dataset = client.datasets.build(job_ids=[j.id for j in good_jobs])
```
***
## Leaving feedback
If a result looks wrong, you can submit feedback directly from the Jobs page (thumbs up / thumbs down) or via the API:
```bash theme={null}
curl -X POST "https://api.flexorch.com/v1/jobs/{id}/feedback" \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{"rating": "negative", "issue": "wrong_fields"}'
```
# RAG Pipeline
Source: https://docs.flexorch.com/guides/rag-pipeline
Two approaches to building a RAG pipeline with FlexOrch — export chunks or use the managed search API.
## Overview
FlexOrch supports two RAG approaches:
| Approach | Best for | Plan |
| -------------------------------------------------- | ------------------------------------------------------- | --------- |
| **Export chunks** (`fmt=rag`, `fmt=hf`) | Bring-your-own vector store (FAISS, Pinecone, Weaviate) | All plans |
| **Managed search** (`/search` API + `/chunks` API) | Low-ops semantic retrieval, no vector store needed | Pro+ |
***
## Approach 1 — Export chunks
After building a dataset, export it as RAG-ready chunks:
```bash theme={null}
# RAG JSON — LangChain / LlamaIndex compatible
curl "https://api.flexorch.com/v1/datasets/{id}/export?format=rag&min_quality=B" \
-H "X-API-KEY: dfx_your_key_here" \
-o chunks.json
# HuggingFace Arrow — datasets.load_from_disk() compatible
curl "https://api.flexorch.com/v1/datasets/{id}/export?format=hf&min_quality=B" \
-H "X-API-KEY: dfx_your_key_here" \
-o dataset.zip
```
The `min_quality` parameter filters out low-quality chunks before export. Pass `A` for only top-grade content, `B` to include solid content (recommended), or omit to include everything.
### RAG chunk format
```json theme={null}
{
"chunks": [
{
"chunk_id": "abc-001",
"chunk_index": 0,
"text": "Invoice FTR-2024-001 from [MASKED_NAME]...",
"token_count": 84,
"metadata": {
"doc_type": "invoice",
"quality_grade": "A",
"quality_score": 0.91,
"pii_masked": true,
"language": "tr"
}
}
]
}
```
### Load with Python SDK
```python theme={null}
from flexorch_sdk import FlexOrch
client = FlexOrch()
# Export to disk
client.datasets.export(dataset_id=89, format="rag", path="chunks.json", min_quality="B")
# Or load in-memory via the chunks() method
chunks_page = client.datasets.get(89).chunks(
quality_grade="A,B",
pii_masked=True,
page=1,
page_size=100,
)
for chunk in chunks_page["data"]["chunks"]:
print(chunk["text"][:80], chunk["metadata"]["quality_grade"])
```
### Load with FlexOrchReader (LlamaIndex)
```python theme={null}
from flexorch_sdk import FlexOrch, FlexOrchReader
client = FlexOrch()
reader = FlexOrchReader(client)
documents = reader.load_data(
dataset_id=89,
min_quality="B",
pii_masked_only=True,
)
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex.from_documents(documents)
```
See the [LlamaIndex guide](/guides/llamaindex) for the full integration.
***
## Approach 2 — Managed search (Pro+)
Semantic and hybrid search require a **Pro or Enterprise** plan.
### Step 1 — Index the dataset
```bash theme={null}
curl -X POST "https://api.flexorch.com/v1/datasets/{id}/index" \
-H "X-API-KEY: dfx_your_key_here"
```
Indexing is idempotent and runs in the background. Poll `GET /v1/datasets/{id}/index/status` until `status` is `"ready"`.
### Step 2 — Search
```bash theme={null}
curl -X POST "https://api.flexorch.com/v1/search" \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{
"query": "invoices over 10000 EUR from Germany",
"top_k": 5,
"mode": "hybrid"
}'
```
### Step 3 — Retrieve chunks for a dataset
```bash theme={null}
curl "https://api.flexorch.com/v1/datasets/89/chunks?quality_grade=A,B&pii_masked=true&page=1&page_size=20" \
-H "X-API-KEY: dfx_your_key_here"
```
### Use FlexOrchRetriever (LangChain)
```python theme={null}
from flexorch_sdk import FlexOrch, FlexOrchRetriever
client = FlexOrch()
retriever = FlexOrchRetriever(
client,
quality_threshold="B", # include A and B grade chunks
pii_masked=True,
top_k=5,
mode="hybrid",
)
docs = retriever.invoke("payment terms in German invoices")
for doc in docs:
print(doc.page_content[:80])
print(doc.metadata["quality_grade"])
```
See the [LangChain guide](/guides/langchain) for a full `RetrievalQA` example.
***
## Chunking strategy by document type
| Document type | Recommended strategy | Chunk size |
| ------------------- | ------------------------- | ---------------------- |
| Invoice / PO | Single chunk (structured) | all fields |
| Bank statement | Per-transaction rows | 256 tokens |
| Payroll | Per-employee row | 256 tokens |
| Contract / proposal | Paragraph split | 512 tokens |
| Long report | Sliding window | 512 tokens, overlap 64 |
The `fmt=rag` export applies FlexOrch's built-in chunker based on `doc_type`. For custom strategies, use the `flexorch-audit` OSS library:
```python theme={null}
from flexorch_audit import chunk_text, rag_metadata, audit
text = open("contract.txt").read()
result = audit(text)
meta = rag_metadata(result)
if meta["should_index"] and meta["pii_safe"]:
chunks = chunk_text(
text,
strategy="paragraph",
max_tokens=meta["recommended_chunk_size"],
)
```
See the [flexorch-audit chunk-quality guide](/guides/chunk-quality) for details.
***
## Quality gate for RAG
Not all chunks should enter your vector store. Use quality grades to filter:
| Grade | Score | Recommendation |
| ----- | ------- | ------------------------------------ |
| **A** | ≥ 0.85 | Index always |
| **B** | ≥ 0.65 | Index — suitable for production RAG |
| **C** | ≥ 0.40 | Review before indexing |
| **D** | \< 0.40 | Exclude — OCR noise, garbage content |
**Recommended minimum:** `B` for production pipelines.
***
## End-to-end example
```python theme={null}
from flexorch_sdk import FlexOrch, FlexOrchRetriever
from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI
client = FlexOrch()
# 1 — Upload and process document
job = client.upload("invoice_batch.pdf")
job.wait()
# 2 — Build dataset
dataset = client.datasets.build(name="invoice-rag", job_ids=[job.id])
# 3 — Index for semantic search (Pro+)
client.datasets.index(dataset.id)
# 4 — Wire into LangChain
retriever = FlexOrchRetriever(client, quality_threshold="B", mode="hybrid")
qa = RetrievalQA.from_chain_type(
llm=ChatOpenAI(model="gpt-4o-mini"),
retriever=retriever,
)
print(qa.invoke("What is the total amount on invoice FTR-2024-001?"))
```
FlexOrchRetriever — full RetrievalQA example
FlexOrchReader — VectorStoreIndex example
Grade thresholds, scoring, and strategies
dataset.index and dataset.chunks for AI agents
# Uploading Documents
Source: https://docs.flexorch.com/guides/upload
Supported file types, upload methods, and multi-file processing.
## Supported file types
FlexOrch accepts the following formats:
| Category | Extensions |
| ------------- | ---------------------------------------------------------- |
| Documents | `.pdf`, `.docx`, `.txt` |
| Spreadsheets | `.xlsx` |
| Presentations | `.pptx` |
| Web / markup | `.html`, `.htm` |
| Email | `.eml`, `.msg` |
| E-invoices | `.xml` (UBL/Peppol, GİB TR, FatturaPA, XRechnung, ZUGFeRD) |
| Images | `.jpg`, `.jpeg`, `.png`, `.tiff` |
Image files and scanned PDFs are processed with OCR automatically. Scanned pages are also automatically deskewed (rotation and small-angle tilt correction) before OCR runs.
CSV files are not supported for upload — use XLSX for tabular data.
FlexOrch also checks the actual file content, not just the extension — if a
file's content doesn't match its extension (e.g. a PDF renamed to `.docx`),
it's automatically routed to the correct parser when the real type is
supported, or rejected with `EXTENSION_MISMATCH` when it isn't. See
[Common error codes](/api-reference/introduction#common-error-codes).
***
## Single file upload
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/data-process/async \
-H "X-API-KEY: dfx_your_key_here" \
-F "files=@contract.pdf"
```
Response:
```json theme={null}
{
"data": {
"job_ids": ["job_abc123"],
"status": "queued"
}
}
```
***
## Multi-file upload
Send multiple files in a single request:
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/data-process/async \
-H "X-API-KEY: dfx_your_key_here" \
-F "files=@invoice_jan.pdf" \
-F "files=@invoice_feb.pdf" \
-F "files=@payroll_q1.xlsx"
```
Each file gets its own `job_id`. The response includes all IDs:
```json theme={null}
{
"data": {
"job_ids": ["job_001", "job_002", "job_003"],
"status": "queued"
}
}
```
***
## Upload from a connector (S3 / GCS / Azure)
If you have a connector configured, you can process files directly from cloud storage without downloading them first:
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/data-process/async \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{
"source": {
"connector_id": 1,
"keys": ["documents/invoice_2024.pdf", "documents/payroll_q1.xlsx"]
}
}'
```
See [Connectors](/connectors/overview) for setup instructions.
***
## File size limits
The maximum file size is **5 MB per file**, the same across all plans. Files exceeding the limit return `400 FILE_TOO_LARGE`.
Encrypted PDFs are rejected with `ENCRYPTED_PDF` — remove the password before uploading. Very long documents are also capped independently of file size, since a small file can still be complex enough to slow down processing: PDFs over 300 pages return `PDF_TOO_MANY_PAGES`, and XLSX sheets over 50,000 rows return `XLSX_TOO_MANY_ROWS`. Split the file and upload the parts separately if you hit either limit.
***
## Duplicate detection
FlexOrch automatically detects duplicate documents (using a content hash). If you upload a file that was already processed:
* No new credit is consumed
* The existing job and execution IDs are returned
* Processing is skipped
```json theme={null}
{
"data": {
"is_duplicate": true,
"existing_document_id": "doc_xyz",
"existing_job_id": "job_abc"
}
}
```
***
## Polling for results
After upload, poll the job endpoint until `status` is `completed` or `failed`:
```bash theme={null}
curl https://api.flexorch.com/v1/jobs/{job_id} \
-H "X-API-KEY: dfx_your_key_here"
```
The SDK handles polling automatically with `job.wait_until_done()`.
Typical processing time is 5–30 seconds depending on file size and type. Image-heavy PDFs take longer due to OCR.
# Webhooks
Source: https://docs.flexorch.com/guides/webhooks
Receive real-time notifications when jobs complete.
Webhooks require an account. Register and manage webhooks under **Settings → Webhooks**.
## Overview
Webhooks let FlexOrch push notifications to your server when events occur — no polling required.
***
## Supported events
| Event | When it fires |
| --------------- | -------------------------------------- |
| `job.completed` | A processing job finishes successfully |
| `job.failed` | A processing job fails |
| `dataset.ready` | A dataset build completes |
***
## Register a webhook
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/webhooks \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{
"url": "https://your-app.com/hooks/flexorch",
"events": ["job.completed", "job.failed"],
"secret": "your_signing_secret"
}'
```
***
## Payload
```json theme={null}
{
"event": "job.completed",
"job_id": "job_abc123",
"status": "completed",
"quality_grade": "A",
"pii_findings_count": 2,
"timestamp": "2024-01-15T10:30:00Z"
}
```
***
## Verifying signatures
Every request includes a `X-Flexorch-Signature` header. Verify it with your signing secret:
```python theme={null}
import hmac
import hashlib
def verify_signature(payload_body: bytes, secret: str, signature: str) -> bool:
expected = hmac.new(
secret.encode(),
payload_body,
hashlib.sha256
).hexdigest()
return hmac.compare_digest(expected, signature)
```
***
## Automatic S3 export
When registering a webhook for `dataset.ready`, you can configure automatic export to a connector:
```json theme={null}
{
"url": "https://your-app.com/hooks/flexorch",
"events": ["dataset.ready"],
"auto_export": {
"connector_id": 1,
"format": "jsonl",
"prefix": "exports/"
}
}
```
When a dataset is ready, FlexOrch exports it directly to your S3 bucket before firing the webhook.
***
## Retry behavior
If your endpoint returns a non-2xx status, FlexOrch retries once with exponential backoff. Failed deliveries are logged in **Settings → Webhooks → Logs**.
# Introduction
Source: https://docs.flexorch.com/introduction
FlexOrch turns unstructured documents into LLM-ready datasets — automatically.
## What is FlexOrch?
FlexOrch is a document intelligence pipeline that converts unstructured files — PDFs, Word documents, spreadsheets, emails, XML invoices — into clean, structured datasets ready for LLM fine-tuning, RAG, or analytics.
**The core flow:**
```
Upload document → Classify → Extract structured fields
→ Detect & mask PII → Score quality → Export dataset
```
Each step is automated. You get a structured dataset with a quality grade, PII summary, and your choice of export format.
***
## Key capabilities
PDF, DOCX, PPTX, TXT, XLSX, HTML, XML, EML/MSG, JPG/PNG/TIFF — 13 formats processed automatically, with automatic OCR and deskew for scanned pages.
Invoice, expense report, purchase order, payroll, bank statement, sales proposal, budget, delivery note, tax declaration — 9 schema types.
46 PII types across TR, EU, and US jurisdictions. KVKK & GDPR-aligned privacy controls with full audit trail.
Every document gets a quality score (0–100) and grade (A–D) based on extraction completeness, noise ratio, and OCR confidence.
JSONL (fine-tuning), CSV, Parquet, Markdown (RAG), XML, XLSX, HuggingFace Arrow — 9 formats.
`flexorch-audit` — zero-dependency PII detection and quality scoring library for Python and JavaScript.
***
## Plans
| Plan | Credits | Use case |
| -------------- | --------------- | ----------------------------------------------------------- |
| **Trial** | 1,200 / 30 days | Evaluate the platform, no credit card required |
| **Starter** | 1,200 / month | Small teams, regular document processing |
| **Pro** | 6,000 / month | Higher volume, BYO LLM API key, fine-tuning, scheduled sync |
| **Enterprise** | Custom | Team management, compliance reporting, SLA |
***
## Where to start
Process your first document in 5 minutes.
Full endpoint reference, auto-generated from the OpenAPI spec.
Use `flexorch-audit` independently for PII detection and quality scoring.
Install `flexorch-sdk` and start processing in a few lines.
Connect Claude and other AI agents to FlexOrch through the Model Context Protocol.
# MCP Server Overview
Source: https://docs.flexorch.com/mcp/overview
Connect Claude and other AI agents to FlexOrch through the Model Context Protocol.
## What is flexorch-mcp?
`flexorch-mcp` is a stateless proxy that exposes the FlexOrch API as MCP (Model Context Protocol) tools. It lets Claude — and any MCP-compatible agent — process documents, extract structured data, detect PII, and export datasets through natural language tool calls.
**For developers writing code:** use [flexorch-sdk](/sdk/python/quickstart) (Python) or the [TypeScript SDK](/sdk/typescript/quickstart).
**For AI agents:** use `flexorch-mcp`.
```
pip install flexorch-mcp
```
***
## How it works
The MCP server acts as a thin proxy between the agent and the FlexOrch API:
```
Agent (Claude, etc.)
→ MCP tool call
→ flexorch-mcp server
→ FlexOrch API (api.flexorch.com)
→ Processed result back to agent
```
No document content or extracted data is stored by the MCP server. Every tool call goes directly to the FlexOrch API and returns the response. The server is **stateless**.
***
## Typical agent workflow
```
User: "Process this invoice and export it as JSONL."
Agent:
1. process_document(file_url="https://...") → job_id: 1234
2. get_job_status(1234) → completed, execution_id: 567
3. get_extraction_result(567) → vendor, total, IBAN [MASKED], due_date
4. build_dataset(execution_id=567) → job_id: 1235
5. get_job_status(1235) → completed, dataset_id: 89
6. export_dataset(89, format="jsonl") → inline JSONL content
```
***
## 8 tools
| Tool | Description |
| ----------------------- | ------------------------------------------------------------------------------ |
| `process_document` | Download a document from a URL and submit it to the FlexOrch pipeline |
| `get_job_status` | Poll a job until completed or failed |
| `get_extraction_result` | Retrieve structured extracted fields from a completed job |
| `build_dataset` | Build a structured dataset from a completed execution |
| `search_documents` | Full-text and semantic search across indexed datasets |
| `export_dataset` | Export a dataset as JSONL, CSV, JSON, MD, XML, or RAG chunks |
| `dataset.index` | Trigger semantic vector indexing for a dataset (Pro+) |
| `dataset.chunks` | Retrieve LangChain/LlamaIndex-ready text chunks from an indexed dataset (Pro+) |
***
## Security
* **API key stays server-side** — `FLEXORCH_API_KEY` is read from the environment variable, never passed as a tool argument.
* **No local storage** — the MCP server is a stateless proxy. No document content or extracted data is written to disk.
* **PII masking applied by FlexOrch** — extracted fields are masked before the MCP server receives them. Raw PII never reaches the agent.
* **HTTPS only** — all communication with `api.flexorch.com` is encrypted.
* **URL scheme validation** — `process_document` only accepts `http://` and `https://` URLs. Local file paths (`file://`) are rejected.
***
## Links
* [PyPI](https://pypi.org/project/flexorch-mcp/) — `pip install flexorch-mcp`
* [GitHub](https://github.com/flexorch/flexorch-mcp) — MIT license
* [Quickstart](/mcp/quickstart) — Configure Claude Desktop in 2 minutes
* [Tool Reference](/mcp/tools) — Full parameter documentation
# Quickstart
Source: https://docs.flexorch.com/mcp/quickstart
Configure Claude Desktop to use FlexOrch in 2 minutes.
## Prerequisites
* Python 3.10+
* A FlexOrch account and API key — generate one under **Settings → API Keys**
* Claude Desktop (or another MCP-compatible client)
***
## 1. Install
```bash theme={null}
pip install flexorch-mcp
```
Verify the installation:
```bash theme={null}
export FLEXORCH_API_KEY=dfx_your_key_here
flexorch-mcp --check
```
On Windows (PowerShell):
```powershell theme={null}
$env:FLEXORCH_API_KEY = "dfx_your_key_here"
flexorch-mcp --check
```
Expected output:
```
FlexOrch API key : dfx_xxx***
Connection : OK (api.flexorch.com) ✓
Plan : Starter (1,200 credits/mo)
Tools : 8 registered
```
***
## 2. Configure Claude Desktop
Add the following to your Claude Desktop config file (create it if it doesn't exist):
* macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
* Windows: `%APPDATA%\Claude\claude_desktop_config.json`
```json theme={null}
{
"mcpServers": {
"flexorch": {
"command": "flexorch-mcp",
"env": {
"FLEXORCH_API_KEY": "dfx_your_key_here"
}
}
}
}
```
Restart Claude Desktop. You should see "flexorch" appear in the connected servers list.
***
## 3. Configure Cursor
In Cursor's MCP config (`.cursor/mcp.json` or the Cursor settings panel):
```json theme={null}
{
"flexorch": {
"command": "flexorch-mcp",
"env": {
"FLEXORCH_API_KEY": "dfx_your_key_here"
}
}
}
```
***
## 4. Try it
Ask Claude:
> "Process this invoice and tell me the vendor name, total amount, and due date. Mask any PII."
Then provide a URL to a document. Claude will call `process_document`, poll `get_job_status`, and return the extracted fields via `get_extraction_result`.
***
## Plan limits
All FlexOrch plan limits apply to MCP tool calls. Credits are consumed per document processed — the same as direct API usage.
| Plan | Credits/mo | Semantic search |
| ---------- | --------------- | --------------- |
| Trial | 1,200 (30 days) | — |
| Starter | 1,200 | — |
| Pro | 6,000 | ✓ |
| Enterprise | Custom | ✓ |
The `search_documents` tool with `mode="semantic"` or `mode="hybrid"` requires a **Pro plan or above**. Starter and Trial plans fall back to `mode="structured"` automatically when `mode="auto"` is set.
***
## Next steps
* [Tool Reference](/mcp/tools) — full parameter documentation for all 8 tools
* [Connectors](/connectors/overview) — automate ingestion from S3, GCS, or Azure Blob
* [Webhooks](/guides/webhooks) — get notified when a job completes
# Tool Reference
Source: https://docs.flexorch.com/mcp/tools
Complete parameter documentation for all 8 FlexOrch MCP tools.
## process\_document
Download a document from a URL and submit it to the FlexOrch pipeline for classification, extraction, PII detection, and quality scoring.
**Parameters**
| Parameter | Type | Default | Description |
| --------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `file_url` | string | required | Public HTTP/HTTPS URL of the document. Supports PDF, DOCX, PPTX, TXT, XLSX, HTML, XML, EML, MSG, JPG, PNG, TIFF. Max 5 MB. |
| `mask_pii` | boolean | `true` | Mask detected PII in extracted fields. When `false`, raw values are returned. |
| `document_type` | string | `"auto"` | Classification hint. Auto-detected if omitted. Values: `invoice`, `expense_report`, `purchase_order`, `sales_proposal`, `bank_statement`, `payroll`, `auto`. |
**Returns**
```json theme={null}
{
"job_id": 1234,
"status": "queued",
"poll_hint": "Use get_job_status(1234) to check progress..."
}
```
On error:
```json theme={null}
{
"isError": true,
"error": "File exceeds size limit. Max 5 MB."
}
```
***
## get\_job\_status
Poll a processing job until it completes or fails.
**Parameters**
| Parameter | Type | Default | Description |
| --------- | ------- | -------- | --------------------------------------------------------- |
| `job_id` | integer | required | Job ID returned by `process_document` or `build_dataset`. |
**Returns — completed (data\_process)**
```json theme={null}
{
"status": "completed",
"execution_id": 567,
"quality_grade": "A",
"quality_score": 91.0,
"pii_found": true,
"pii_masked": true,
"pii_count": 3,
"row_count": 12,
"has_dataset": false,
"poll_hint": "Use get_extraction_result(567) to retrieve fields."
}
```
**Returns — completed (dataset\_build)**
```json theme={null}
{
"status": "completed",
"dataset_id": 89,
"dataset_name": "q1_invoices",
"row_count": 12,
"poll_hint": "Use export_dataset(89, format='jsonl') to download."
}
```
**Returns — running**
```json theme={null}
{
"status": "running",
"stage": "privacy"
}
```
Pipeline stages: `extract` → `privacy` → `quality` → `dataset`.
**Returns — failed**
```json theme={null}
{
"status": "failed",
"reason": "UNSUPPORTED_FILE"
}
```
***
## get\_extraction\_result
Retrieve structured extracted fields from a completed processing job. Returns up to 100 records inline — for larger documents, use `export_dataset`.
**Parameters**
| Parameter | Type | Default | Description |
| -------------- | ------- | -------- | ------------------------------------------------------ |
| `execution_id` | integer | required | Execution ID from `get_job_status` completed response. |
**Returns**
```json theme={null}
{
"execution_id": 567,
"document_type": "invoice",
"detected_language": "tr",
"quality": { "grade": "A", "score": 91.0, "warnings": [] },
"privacy": { "pii_findings_count": 3, "pii_masked": true },
"row_count": 1,
"columns": ["invoice_number", "vendor_name", "total_amount", "currency", "due_date"],
"fields": [
{
"invoice_number": "FTR-2024-001",
"vendor_name": "[MASKED_NAME]",
"total_amount": 12450.00,
"currency": "EUR",
"due_date": "2024-01-15"
}
]
}
```
When more than 100 records exist:
```json theme={null}
{
"has_more": true,
"has_more_hint": "Showing first 100 of 847 records. Call export_dataset(89, format='jsonl') to retrieve all records at once."
}
```
When no dataset is built yet (records array empty):
```json theme={null}
{
"fields_hint": "Field values are available after building a dataset. Call build_dataset(execution_id=567) then export_dataset(dataset_id, format)."
}
```
Masked fields use the `[MASKED_TYPE]` placeholder format — e.g. `[MASKED_NAME]`, `[MASKED_EMAIL]`, `[MASKED_NATIONAL_ID_TR]`. Raw PII values are never exposed.
***
## build\_dataset
Build a structured, exportable dataset from a completed execution.
**Parameters**
| Parameter | Type | Default | Description |
| -------------- | ------- | -------- | -------------------------------------------------------------- |
| `execution_id` | integer | required | Execution ID from `get_job_status` or `get_extraction_result`. |
| `name` | string | `""` | Dataset name. Auto-generated if omitted. |
| `description` | string | `""` | Optional description. |
**Returns**
```json theme={null}
{
"job_id": 1235,
"status": "queued",
"poll_hint": "Use get_job_status(1235) to check build progress..."
}
```
Poll `get_job_status` until `status` is `"completed"` — the response will include `dataset_id`.
***
## search\_documents
Search across all indexed datasets using structured keyword matching or semantic vector search.
**Parameters**
| Parameter | Type | Default | Description |
| --------------- | ------- | -------- | ---------------------------------------------------------------------------------------------- |
| `query` | string | required | Search query. Max 1000 characters. |
| `top_k` | integer | `5` | Number of results to return. Min 1, max 50. |
| `mode` | string | `"auto"` | Search mode: `auto`, `structured`, `semantic`, `hybrid`. Semantic and hybrid require Pro plan. |
| `document_type` | string | `""` | Filter by document type (optional). |
| `language` | string | `""` | Filter by language ISO 639-1 code, e.g. `"tr"`, `"en"` (optional). |
| `quality_grade` | string | `""` | Filter by quality grade: `A`, `B`, `C`, `D` (optional). |
**Returns**
```json theme={null}
{
"results": [
{
"chunk_id": "abc-123",
"dataset_id": 89,
"dataset_name": "q1_invoices",
"score": 0.94,
"text": "Invoice FTR-2024-001 from [MASKED_NAME] for €12,450...",
"metadata": {
"doc_type": "invoice",
"language": "tr",
"quality_grade": "A",
"mode": "structured"
}
}
],
"total_results": 1,
"mode": "structured",
"query": "invoice 2024"
}
```
`mode="semantic"` and `mode="hybrid"` require a **Pro plan or above**. Trial and Starter plans return a `PLAN_UPGRADE_REQUIRED` error for these modes. Use `mode="auto"` to fall back to structured search automatically.
***
## export\_dataset
Export a built dataset and return its full content as text.
**Parameters**
| Parameter | Type | Default | Description |
| ------------ | ------- | --------- | --------------------------------------------------------------------- |
| `dataset_id` | integer | required | Dataset ID from `get_job_status` (dataset\_build completed). |
| `format` | string | `"jsonl"` | Export format. Supported: `jsonl`, `csv`, `json`, `md`, `xml`, `rag`. |
**Supported formats**
| Format | Best for |
| ------- | ----------------------------------- |
| `jsonl` | LLM fine-tuning (OpenAI, Mistral) |
| `csv` | Spreadsheet analysis, Pandas |
| `json` | General structured data |
| `md` | Markdown-based RAG pipelines |
| `xml` | Enterprise integrations |
| `rag` | LangChain / LlamaIndex chunk arrays |
Binary formats (`parquet`, `hf`) are not supported via MCP — download them directly from `GET /v1/datasets/{id}/export/{format}`.
**Returns**
```json theme={null}
{
"dataset_id": 89,
"format": "jsonl",
"filename": "q1-invoices.jsonl",
"content": "{\"invoice_number\":\"FTR-2024-001\",\"total_amount\":12450.0}\n...",
"byte_count": 4821
}
```
***
## dataset.index
Requires a **Pro or Enterprise** plan.
Trigger semantic vector indexing for a built dataset. Must be called before `dataset.chunks`. Indexing is idempotent — calling it again on an already-indexed dataset is safe.
**Parameters**
| Parameter | Type | Default | Description |
| ------------ | ------- | -------- | ---------------------------------------------------------------------- |
| `dataset_id` | integer | required | ID of the built dataset (from `get_job_status` after `build_dataset`). |
**Returns**
```json theme={null}
{
"dataset_id": 89,
"status": "indexing",
"message": "Indexing started. Check back in 10–60 seconds.",
"index_hint": "Use dataset.chunks(89) after indexing completes to retrieve RAG-ready chunks."
}
```
On plan error:
```json theme={null}
{
"isError": true,
"error": "Semantic indexing requires a Pro plan. Upgrade at app.flexorch.com/settings."
}
```
Indexing typically completes in 10–60 seconds depending on dataset size. There is no async status poll for this tool — wait a few seconds and proceed to `dataset.chunks`.
***
## dataset.chunks
Requires a **Pro or Enterprise** plan. Index the dataset first with `dataset.index`.
Retrieve LangChain/LlamaIndex-ready text chunks from an indexed dataset. Supports quality filtering and pagination.
**Parameters**
| Parameter | Type | Default | Description |
| ----------------- | ------- | -------- | --------------------------------------------------------------------- |
| `dataset_id` | integer | required | ID of the indexed dataset. |
| `min_quality` | string | `"B"` | Minimum quality grade to include. `"B"` returns A and B grade chunks. |
| `pii_masked_only` | boolean | `false` | When `true`, only return chunks where PII was detected and masked. |
| `page` | integer | `1` | Page number (1-indexed). |
| `page_size` | integer | `20` | Chunks per page, max 100. |
**Returns**
```json theme={null}
{
"dataset_id": 89,
"chunks": [
{
"chunk_id": "abc-001",
"chunk_index": 0,
"text": "Invoice FTR-2024-001 from [MASKED_NAME] for €12,450...",
"token_count": 84,
"metadata": {
"quality_grade": "A",
"pii_masked": true,
"doc_type": "invoice",
"language": "tr"
}
}
],
"chunk_count": 20,
"total": 47,
"page": 1,
"page_size": 20,
"has_more": true
}
```
On plan error:
```json theme={null}
{
"isError": true,
"error": "Chunk listing requires a Pro plan. Upgrade at app.flexorch.com/settings."
}
```
**RAG workflow with MCP:**
```
process_document → get_job_status → build_dataset → get_job_status
→ dataset.index → (wait ~30s) → dataset.chunks → embed + query
```
# Compliance Report
Source: https://docs.flexorch.com/open-source/compliance-report
Generate a KVKK and GDPR risk summary with compliance_report().
## Overview
`compliance_report()` analyzes detected PII and returns a structured summary of data categories, applicable regulations, and risk level.
```python Python theme={null}
from flexorch_audit import detect_pii, compliance_report
text = """
Müşteri: Ayşe Kaya
T.C. Kimlik: 12345678901
E-posta: ayse.kaya@sirket.com
IBAN: TR33 0006 1005 1978 6457 8413 26
"""
findings = detect_pii(text)
report = compliance_report(findings)
print(report)
# {
# "risk_level": "high",
# "applicable_regulations": ["KVKK", "GDPR"],
# "gdpr_categories": ["identification", "financial"],
# "detected_countries": ["TR"],
# "pii_type_counts": {
# "national_id_tr": 1,
# "email": 1,
# "iban_tr": 1
# }
# }
```
```javascript JavaScript theme={null}
import { detectPii, complianceReport } from '@flexorch/audit';
const findings = detectPii(text);
const report = complianceReport(findings);
console.log(report.riskLevel); // "high"
console.log(report.applicableRegulations); // ["KVKK", "GDPR"]
```
***
## Risk levels
| Level | Meaning |
| -------- | ------------------------------------------ |
| `none` | No PII detected |
| `low` | Only low-risk types (email, phone) |
| `medium` | Financial or contact data |
| `high` | National IDs, health data, or 3+ PII types |
***
## GDPR categories
Maps detected types to GDPR Article 9 special categories and Article 4 personal data categories:
* `identification` — National IDs, passport numbers
* `financial` — IBAN, credit card, tax ID
* `contact` — Email, phone
* `biometric` — (detected if present)
* `health` — (detected if present)
***
## Platform compliance
For platform users, the full GDPR Article 30 Record of Processing Activities (ROPA) and NIS2 audit export are available via the API. See [GDPR Article 30](/compliance/gdpr-article-30) and [NIS2 Audit Export](/compliance/nis2-audit-export).
# Installation
Source: https://docs.flexorch.com/open-source/installation
Install flexorch-audit for Python or JavaScript.
## Python
```bash theme={null}
pip install flexorch-audit
```
Requires Python 3.10+. No external dependencies.
**Verify:**
```python theme={null}
from flexorch_audit import audit
result = audit("Test email: hello@example.com")
print(result["pii_summary"]["count"]) # 1
```
***
## JavaScript / TypeScript
```bash theme={null}
npm install @flexorch/audit
```
Requires Node.js 18+. No external dependencies.
**Verify:**
```javascript theme={null}
import { audit } from '@flexorch/audit';
const result = await audit('Test email: hello@example.com');
console.log(result.piiSummary.count); // 1
```
Or with CommonJS:
```javascript theme={null}
const { audit } = require('@flexorch/audit');
```
***
## Versions
| Package | Latest | Changelog |
| ------------------------- | ---------------------------------------------------- | --------------------------------------------------------------------------------- |
| `flexorch-audit` (Python) | [PyPI](https://pypi.org/project/flexorch-audit) | [CHANGELOG](https://github.com/flexorch/flexorch-audit/blob/main/CHANGELOG.md) |
| `@flexorch/audit` (JS) | [npm](https://www.npmjs.com/package/@flexorch/audit) | [CHANGELOG](https://github.com/flexorch/flexorch-audit-js/blob/main/CHANGELOG.md) |
# LangChain Integration (OSS)
Source: https://docs.flexorch.com/open-source/langchain
Use AuditedLoader to load local documents with automatic PII detection and quality filtering.
This guide covers the **zero-dependency OSS library** (`flexorch-audit`) for loading local files. To query FlexOrch datasets directly from LangChain using `FlexOrchRetriever`, see [LangChain — Platform SDK](/guides/langchain).
## Overview
`flexorch-audit` ships a LangChain-compatible document loader that audits each document before it enters your chain.
```bash theme={null}
pip install flexorch-audit langchain-community
```
***
## AuditedLoader
```python theme={null}
from flexorch_audit.integrations.langchain import AuditedLoader
loader = AuditedLoader(
file_paths=["contracts/agreement.pdf", "invoices/inv_001.pdf"],
min_grade="B", # Skip documents graded C or D
mask_pii=True, # Replace PII before loading
locales=["universal", "tr", "de"],
)
docs = loader.load()
for doc in docs:
print(doc.metadata["quality_grade"]) # "A"
print(doc.metadata["pii_findings_count"]) # 2
print(doc.page_content[:200]) # PII already masked
```
***
## Parameters
| Parameter | Type | Default | Description |
| ------------ | ---------- | -------- | ----------------------------------------------------- |
| `file_paths` | list\[str] | required | Paths to documents |
| `min_grade` | str | `"D"` | Minimum grade to include (`"A"`, `"B"`, `"C"`, `"D"`) |
| `mask_pii` | bool | `True` | Replace PII with `[MASKED_...]` placeholders |
| `locales` | list\[str] | all | Restrict PII detection to specific jurisdictions |
***
## In a RAG pipeline
```python theme={null}
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from flexorch_audit.integrations.langchain import AuditedLoader
loader = AuditedLoader(file_paths=["docs/"], min_grade="B", mask_pii=True)
docs = loader.load()
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(docs, embeddings)
retriever = vectorstore.as_retriever()
```
Documents with grades below `min_grade` are silently skipped. Check `loader.skipped` for a list of excluded files and their grades.
***
## As a pre-processing step
```python theme={null}
from langchain_core.runnables import RunnableLambda
from flexorch_audit import redact_for_llm
# Use redact_for_llm() inline without a file loader
safe_chain = RunnableLambda(lambda text: redact_for_llm(text)[0]) | your_chain
```
# LlamaIndex Integration (OSS)
Source: https://docs.flexorch.com/open-source/llamaindex
Use AuditedReader to load privacy-safe local documents into LlamaIndex.
This guide covers the **zero-dependency OSS library** (`flexorch-audit`) for loading local files. To load FlexOrch dataset chunks directly using `FlexOrchReader`, see [LlamaIndex — Platform SDK](/guides/llamaindex).
## Overview
`flexorch-audit` ships a LlamaIndex-compatible reader that audits documents and optionally masks PII before indexing.
```bash theme={null}
pip install flexorch-audit llama-index
```
***
## AuditedReader
```python theme={null}
from flexorch_audit.integrations.llamaindex import AuditedReader
reader = AuditedReader(
min_grade="B",
mask_pii=True,
locales=["universal", "tr"],
)
documents = reader.load_data(
file_paths=["contracts/agreement.pdf", "reports/q1.docx"]
)
for doc in documents:
print(doc.metadata["quality_grade"])
print(doc.metadata["pii_findings_count"])
print(doc.text[:200]) # PII masked
```
***
## Parameters
| Parameter | Type | Default | Description |
| ----------- | ---------- | ------- | -------------------------------- |
| `min_grade` | str | `"D"` | Minimum quality grade to include |
| `mask_pii` | bool | `True` | Mask PII before indexing |
| `locales` | list\[str] | all | PII detection scope |
***
## In an index pipeline
```python theme={null}
from llama_index.core import VectorStoreIndex
from flexorch_audit.integrations.llamaindex import AuditedReader
reader = AuditedReader(min_grade="B", mask_pii=True)
documents = reader.load_data(file_paths=["data/"])
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What are the payment terms in the contracts?")
print(response)
```
Low-quality or excluded documents are available in `reader.skipped` with their grade and reason.
# Masking
Source: https://docs.flexorch.com/open-source/masking
Replace detected PII with placeholders using mask().
## Overview
`mask()` takes text and a list of PII findings, and returns the text with each finding replaced according to your chosen strategy.
***
## Basic usage
```python Python theme={null}
from flexorch_audit import detect_pii, mask
text = "Email us at hello@example.com or call +49 30 1234567."
findings = detect_pii(text)
masked = mask(text, findings)
print(masked)
# Email us at [MASKED_EMAIL] or call [MASKED_PHONE_DE].
```
```javascript JavaScript theme={null}
import { detectPii, mask } from '@flexorch/audit';
const text = 'Email us at hello@example.com or call +49 30 1234567.';
const findings = detectPii(text);
const masked = mask(text, findings);
console.log(masked);
// Email us at [MASKED_EMAIL] or call [MASKED_PHONE_DE].
```
***
## Strategies
| Strategy | Example output | Use case |
| ------------------ | ------------------ | --------------------------- |
| `redact` (default) | `[MASKED_EMAIL]` | Production datasets |
| `replace` | `user@example.com` | Synthetic plausible data |
| `token` | `` | Structure-preserving NLP |
| `hash` | `a3f2b19c...` | Deterministic anonymization |
```python Python theme={null}
masked = mask(text, findings, strategy="token")
# Email us at or call .
```
```javascript JavaScript theme={null}
const masked = mask(text, findings, { strategy: 'token' });
```
***
## One-liner
Use `redact_for_llm()` to detect and mask in a single call — optimized for LLM input preparation:
```python Python theme={null}
from flexorch_audit import redact_for_llm
clean_text, summary = redact_for_llm(text)
print(clean_text)
print(summary) # {"count": 2, "types": ["email", "phone_de"]}
```
```javascript JavaScript theme={null}
import { redactForLlm } from '@flexorch/audit';
const { text: cleanText, summary } = await redactForLlm(text);
```
See [redact\_for\_llm](/open-source/redact-for-llm) for full details.
# flexorch-audit
Source: https://docs.flexorch.com/open-source/overview
Zero-dependency PII detection and document quality scoring for Python and JavaScript.
## Overview
`flexorch-audit` is an open source library for PII detection, text masking, and document quality scoring. It runs entirely locally — no API calls, no account required.
`pip install flexorch-audit`
`npm install @flexorch/audit`
***
## Features
* **PII detection** — 46 types across TR, EU, and US jurisdictions
* **Text masking** — 4 strategies: redact, replace, token, hash
* **Quality metrics** — grade (A–D), score, noise ratio, language detection
* **Compliance summary** — KVKK / GDPR risk categories
* **LLM preparation** — `redact_for_llm()` one-liner, token estimation
* **LangChain & LlamaIndex** — drop-in loaders with quality filtering
* **Zero dependencies** — pure Python / pure JavaScript, no heavy ML models
***
## Quick example
```python Python theme={null}
from flexorch_audit import audit
result = audit("""
Invoice from Acme Ltd.
Contact: john.doe@acme.com
IBAN: TR33 0006 1005 1978 6457 8413 26
Amount: €12,500
""")
print(result["quality_grade"]) # "A"
print(result["pii_summary"]["count"]) # 2
print(result["detected_language"]) # "en"
```
```javascript JavaScript theme={null}
import { audit } from '@flexorch/audit';
const result = await audit(`
Invoice from Acme Ltd.
Contact: john.doe@acme.com
IBAN: TR33 0006 1005 1978 6457 8413 26
Amount: €12,500
`);
console.log(result.qualityGrade); // "A"
console.log(result.piiSummary.count); // 2
console.log(result.detectedLanguage); // "en"
```
***
## Relationship to FlexOrch platform
`flexorch-audit` contains the core detection logic used by the FlexOrch platform. The platform adds:
* Structured extraction for 9 document types
* Managed pipeline (async jobs, job history)
* Dataset building and export
* Cloud connectors (S3, GCS, Azure)
* Compliance reports and audit trail
* Team management
Use `flexorch-audit` if you want detection in your own pipeline. Use the FlexOrch platform if you want the full managed workflow.
***
## Links
* [PyPI](https://pypi.org/project/flexorch-audit)
* [npm](https://www.npmjs.com/package/@flexorch/audit)
* [GitHub](https://github.com/flexorch/flexorch-audit)
# PII Detection
Source: https://docs.flexorch.com/open-source/pii-detection
Detect personal data in text with detect_pii() — 46 types, TR/EU/US.
## Installation
```bash Python theme={null}
pip install flexorch-audit
```
```bash JavaScript theme={null}
npm install @flexorch/audit
```
***
## Basic usage
```python Python theme={null}
from flexorch_audit import detect_pii
text = "Contact us at info@company.com or call +90 555 123 4567."
findings = detect_pii(text)
for f in findings:
print(f["type"], f["value"], f["start"], f["end"])
# email info@company.com 14 32
# phone_tr +90 555 123 4567 39 56
```
```javascript JavaScript theme={null}
import { detectPii } from '@flexorch/audit';
const text = 'Contact us at info@company.com or call +90 555 123 4567.';
const findings = detectPii(text);
for (const f of findings) {
console.log(f.type, f.value, f.start, f.end);
}
// email info@company.com 14 32
// phone_tr +90 555 123 4567 39 56
```
***
## Locale filtering
By default, all locales are active. Restrict detection to specific jurisdictions:
```python Python theme={null}
from flexorch_audit import detect_pii
# Only detect universal + Turkish types
findings = detect_pii(text, locales=["universal", "tr"])
# Only EU types
findings = detect_pii(text, locales=["universal", "de", "fr", "it", "nl", "es", "pl"])
```
```javascript JavaScript theme={null}
import { detectPii } from '@flexorch/audit';
// Only detect universal + Turkish types
const findings = detectPii(text, { locales: ['universal', 'tr'] });
```
Available locale values: `"universal"`, `"tr"`, `"de"`, `"fr"`, `"it"`, `"nl"`, `"es"`, `"pl"`, `"uk"`, `"us"`, `"at"`, `"be"`, `"se"`, `"dk"`, `"fi"`, `"pt"`
***
## Finding structure
Each finding contains:
| Field | Type | Description |
| -------- | ------ | ----------------------------------------------------- |
| `type` | string | PII type identifier (e.g., `email`, `national_id_tr`) |
| `value` | string | The matched text |
| `start` | int | Start character position |
| `end` | int | End character position |
| `locale` | string | Jurisdiction (`tr`, `de`, `universal`, …) |
***
## Batch processing
```python Python theme={null}
from flexorch_audit import detect_pii_batch
texts = [
"Customer: john@example.com",
"IBAN: DE89 3704 0044 0532 0130 00",
"No sensitive data here.",
]
results = detect_pii_batch(texts)
# List of finding lists, one per input
```
```javascript JavaScript theme={null}
import { detectPiiBatch } from '@flexorch/audit';
const results = detectPiiBatch([
'Customer: john@example.com',
'IBAN: DE89 3704 0044 0532 0130 00',
'No sensitive data here.',
]);
```
***
## Streaming large files
```python Python theme={null}
from flexorch_audit import audit_stream
for chunk_result in audit_stream("large_document.txt", chunk_size=5000):
print(chunk_result["pii_summary"]["count"])
```
***
## Supported PII types
See [PII Detection & Privacy](/guides/pii-privacy) for the complete list of 46 types across all jurisdictions.
# Quality Metrics
Source: https://docs.flexorch.com/open-source/quality
Score document quality with quality_metrics() and noise_ratio().
## quality\_metrics()
Returns a quality score, grade, and noise ratio for a block of text.
```python Python theme={null}
from flexorch_audit import quality_metrics
text = """
Invoice Date: 2024-01-15
Vendor: Acme GmbH
Total: €12,500.00
IBAN: DE89 3704 0044 0532 0130 00
"""
metrics = quality_metrics(text)
print(metrics)
# {
# "score": 88,
# "grade": "A",
# "noise_ratio": 0.04,
# "detected_language": "en"
# }
```
```javascript JavaScript theme={null}
import { qualityMetrics } from '@flexorch/audit';
const metrics = await qualityMetrics(text);
console.log(metrics.grade); // "A"
console.log(metrics.noiseRatio); // 0.04
```
***
## noise\_ratio()
Returns only the noise ratio (0.0–1.0) — the proportion of the text that is non-informative.
```python Python theme={null}
from flexorch_audit import noise_ratio
ratio = noise_ratio("### !!! @@@ page 1 of 10 ### !!! ###")
print(ratio) # 0.82
```
```javascript JavaScript theme={null}
import { noiseRatio } from '@flexorch/audit';
const ratio = noiseRatio('### !!! @@@ page 1 of 10 ### !!! ###');
console.log(ratio); // 0.82
```
A noise ratio above 0.20 triggers a quality penalty. Above 0.50, a warning is added to the result.
***
## Grade thresholds
| Grade | Score |
| ----- | ------ |
| A | 85–100 |
| B | 65–84 |
| C | 45–64 |
| D | 0–44 |
***
## estimate\_tokens()
Estimate token count before sending to an LLM — no tiktoken required:
```python Python theme={null}
from flexorch_audit import estimate_tokens
count = estimate_tokens(text)
print(count) # 142
```
```javascript JavaScript theme={null}
import { estimateTokens } from '@flexorch/audit';
const count = estimateTokens(text);
console.log(count); // 142
```
# redact_for_llm
Source: https://docs.flexorch.com/open-source/redact-for-llm
One-liner PII removal before sending text to an LLM.
## Overview
`redact_for_llm()` combines detection and masking into a single call optimized for LLM input preparation.
```python Python theme={null}
from flexorch_audit import redact_for_llm
text = """
From: maria.garcia@company.es
Subject: Invoice INV-2024-0042
Please process the payment of €8,500 to IBAN ES91 2100 0418 4502 0005 1332.
"""
clean_text, summary = redact_for_llm(text)
print(clean_text)
# From: [MASKED_EMAIL]
# Subject: Invoice INV-2024-0042
#
# Please process the payment of €8,500 to IBAN [MASKED_IBAN_ES].
print(summary)
# {"count": 2, "types": ["email", "iban_es"]}
```
```javascript JavaScript theme={null}
import { redactForLlm } from '@flexorch/audit';
const { text: cleanText, summary } = await redactForLlm(text);
console.log(cleanText);
// From: [MASKED_EMAIL]
// ...
console.log(summary);
// { count: 2, types: ['email', 'iban_es'] }
```
***
## With locale filtering
```python Python theme={null}
# Only detect Turkish and universal types
clean_text, summary = redact_for_llm(text, locales=["universal", "tr"])
```
```javascript JavaScript theme={null}
const { text: cleanText, summary } = await redactForLlm(text, {
locales: ['universal', 'tr']
});
```
***
## Token estimation
After redacting, estimate token count before sending to the LLM:
```python Python theme={null}
from flexorch_audit import redact_for_llm, estimate_tokens
clean_text, _ = redact_for_llm(text)
tokens = estimate_tokens(clean_text)
print(f"{tokens} tokens")
```
```javascript JavaScript theme={null}
import { redactForLlm, estimateTokens } from '@flexorch/audit';
const { text: cleanText } = await redactForLlm(text);
const tokens = estimateTokens(cleanText);
console.log(`${tokens} tokens`);
```
***
## LangChain integration
Use `redact_for_llm()` as a pre-processing step in a LangChain chain:
```python theme={null}
from langchain_core.runnables import RunnableLambda
from flexorch_audit import redact_for_llm
def safe_input(text: str) -> str:
clean, _ = redact_for_llm(text)
return clean
chain = RunnableLambda(safe_input) | your_llm_chain
```
Or use the ready-made [AuditedLoader](/open-source/langchain) for document loading.
# Quickstart
Source: https://docs.flexorch.com/quickstart
Process your first document and export a dataset in under 5 minutes.
## Prerequisites
* A FlexOrch account ([sign up free](https://app.flexorch.com/signup))
* An API key — generate one under **Settings → API Keys**
***
## Option 1 — Python SDK
```bash theme={null}
pip install flexorch-sdk
```
```python theme={null}
import os
from flexorch_sdk import FlexOrchClient
client = FlexOrchClient(os.environ["FLEXORCH_API_KEY"])
job = client.process("invoice.pdf")
print(job.id) # 1234
```
```python theme={null}
result = job.wait(timeout=120)
print(result.status) # "completed"
print(result.quality_grade) # "A"
print(result.quality_score) # 0.91
print(result.degraded) # False — True if structured extraction
# partially failed but the job still succeeded
```
Building a dataset is a separate, explicit step — a completed job doesn't
have one until you build it.
```python theme={null}
dataset = result.build_dataset().wait().dataset()
# Export as JSONL for fine-tuning
dataset.export("jsonl", path="./output.jsonl")
# Or export as Parquet for analytics
dataset.export("parquet", path="./output.parquet")
```
***
## Option 2 — TypeScript SDK
```bash theme={null}
npm install flexorch-sdk
```
```typescript theme={null}
import { FlexOrchClient } from 'flexorch-sdk';
const client = new FlexOrchClient(process.env.FLEXORCH_API_KEY);
const job = await client.process('invoice.pdf');
const result = await job.wait({ timeout: 120 }); // seconds
console.log(result.qualityGrade); // "A"
console.log(result.qualityScore); // 0.91
```
Building a dataset is a separate, explicit step — a completed job doesn't
have one until you build it.
```typescript theme={null}
import * as fs from 'node:fs/promises';
const built = await result.buildDataset();
const dataset = await (await built.wait()).dataset();
const bytes = await dataset.export('jsonl');
await fs.writeFile('./output.jsonl', bytes);
```
***
## Option 3 — REST API (cURL)
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/data-process/async \
-H "X-API-KEY: dfx_your_key_here" \
-F "files=@invoice.pdf"
```
Response (`202 Accepted` — the same multi-file-capable shape used for
batch uploads, with one entry in `jobs` for a single file):
```json theme={null}
{
"status": "accepted",
"data": {
"accepted": 1,
"rejected": [],
"jobs": [{ "filename": "invoice.pdf", "job_id": 1234, "status": "queued" }]
},
"error": null,
"meta": { "poll": "/v1/jobs/1234" }
}
```
```bash theme={null}
curl https://api.flexorch.com/v1/jobs/1234 \
-H "X-API-KEY: dfx_your_key_here"
```
Completed response includes `processing_summary.quality` (grade, score),
`processing_summary.privacy` (PII findings), `processing_summary.execution_id`
(needed for the next step), and `execution_summary.degraded`.
```bash theme={null}
# Build — execution_id comes from the completed job's processing_summary
curl -X POST https://api.flexorch.com/v1/datasets/build-from-execution/{execution_id} \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{"name": "my-dataset"}'
# Poll the returned dataset_build job_id the same way as step 2, then export.
# fmt is a path segment (json | jsonl | csv | parquet | md | xml | xlsx | rag | hf):
curl https://api.flexorch.com/v1/datasets/{dataset_id}/export/jsonl \
-H "X-API-KEY: dfx_your_key_here" \
-o output.jsonl
```
***
## What's next?
PDF, DOCX, PPTX, XLSX, HTML, XML e-invoices, EML, images — full list.
46 types across TR, EU, and US — TCKN, IBAN, email, phone, passport, and more.
JSONL, CSV, Parquet, Markdown, XML, XLSX, HuggingFace Arrow.
Connect an S3 bucket and process documents automatically.
# Create a Schedule
Source: https://docs.flexorch.com/scheduler/create-schedule
Set up a cron schedule for automatic connector sync.
## Cron syntax
FlexOrch uses standard 5-field cron expressions (UTC timezone):
```
┌───────── minute (0–59)
│ ┌─────── hour (0–23)
│ │ ┌───── day of month (1–31)
│ │ │ ┌─── month (1–12)
│ │ │ │ ┌─ day of week (0–7, 0 and 7 = Sunday)
│ │ │ │ │
* * * * *
```
Common patterns:
| Schedule | Cron |
| ----------------------- | ------------- |
| Every day at 6am UTC | `0 6 * * *` |
| Every Monday at 8am UTC | `0 8 * * 1` |
| Every 6 hours | `0 */6 * * *` |
## Create
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/connectors/{connector_id}/schedules \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{
"cron": "0 6 * * *",
"prefix": "documents/incoming/",
"name": "Daily document sync"
}'
```
## Manual trigger
Trigger a sync immediately without waiting for the schedule:
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/connectors/{connector_id}/schedules/{schedule_id}/trigger \
-H "X-API-KEY: dfx_your_key_here"
```
# Scheduled Sync
Source: https://docs.flexorch.com/scheduler/overview
Automatically process new files from a connector on a cron schedule.
Scheduled sync lets FlexOrch poll a connector (S3, GCS, Azure) on a recurring schedule and automatically process any new files it finds.
## Plan limits
| Plan | Min interval | Max schedules per connector |
| ---------- | ------------- | --------------------------- |
| Trial | Not available | — |
| Starter | 24 hours | 5 |
| Pro | No minimum | 5 |
| Enterprise | Configurable | Configurable |
## Create a schedule
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/connectors/{connector_id}/schedules \
-H "X-API-KEY: dfx_your_key_here" \
-H "Content-Type: application/json" \
-d '{
"cron": "0 6 * * *",
"prefix": "incoming/",
"name": "Daily morning sync"
}'
```
The `cron` field follows standard cron syntax (UTC).
See [Create a Schedule](/scheduler/create-schedule) for details.
# Plan Limits
Source: https://docs.flexorch.com/scheduler/plan-limits
Scheduled sync availability by plan.
| Plan | Available | Min interval | Max schedules |
| ---------- | --------- | ------------ | --------------- |
| Trial | No | — | — |
| Starter | Yes | 24 hours | 5 per connector |
| Pro | Yes | None | 5 per connector |
| Enterprise | Yes | Configurable | Configurable |
Attempting to create a schedule with an interval shorter than the plan minimum returns `429 RATE_LIMIT_EXCEEDED`.
Trial accounts that attempt to create a schedule receive `403 PLAN_UPGRADE_REQUIRED`.
# Sync Logs
Source: https://docs.flexorch.com/scheduler/sync-logs
View sync history and debug sync failures.
## List sync logs
```bash theme={null}
curl "https://api.flexorch.com/v1/connectors/{connector_id}/schedules/{schedule_id}/logs" \
-H "X-API-KEY: dfx_your_key_here"
```
Response includes:
| Field | Description |
| ----------------- | ------------------------------------- |
| `started_at` | Sync start time |
| `completed_at` | Sync completion time |
| `files_found` | Number of files discovered |
| `files_processed` | Number of files successfully queued |
| `status` | `success` or `failed` |
| `error` | Error message if `status` is `failed` |
# Datasets
Source: https://docs.flexorch.com/sdk/python/datasets
Build and export datasets using the Python SDK.
## Get a dataset
```python theme={null}
from flexorch_sdk import FlexOrch
client = FlexOrch()
dataset = client.datasets.get("ds_abc123")
print(dataset.name) # "q1-invoices"
print(dataset.row_count) # 42
print(dataset.status) # "ready"
print(dataset.available_formats) # ["jsonl", "csv", "parquet"]
```
***
## List datasets
```python theme={null}
datasets = client.datasets.list(page=1, page_size=20)
for ds in datasets.items:
print(ds.id, ds.name, ds.row_count)
```
***
## Build a dataset
```python theme={null}
dataset = client.datasets.build(
name="q1-invoices",
job_ids=["job_001", "job_002", "job_003"],
)
```
***
## Export
`client.datasets.export()` writes the file to disk:
```python theme={null}
# JSONL for LLM fine-tuning
client.datasets.export(dataset.id, format="jsonl", path="output.jsonl")
# Parquet for analytics
client.datasets.export(dataset.id, format="parquet", path="output.parquet")
# Markdown for RAG
client.datasets.export(dataset.id, format="md", path="output.md")
```
**Supported formats:** `"json"` | `"jsonl"` | `"csv"` | `"parquet"` | `"md"` | `"xml"` | `"xlsx"` | `"rag"`
***
## Export directly to S3
Push a dataset to a connected S3/GCS/Azure bucket without downloading it first:
```python theme={null}
result = client.datasets.export_to_s3(
dataset_id=dataset.id,
connector_id=1,
format="jsonl",
prefix="exports/datasets/", # optional
)
print(result["s3_key"]) # "exports/datasets/q1-invoices.jsonl"
print(result["size_bytes"]) # 148302
```
***
## Semantic indexing
Available on **Pro and Enterprise** plans.
Index a dataset for semantic search:
```python theme={null}
import time
# Start indexing (non-blocking)
client.datasets.index(dataset.id)
# Poll status
while True:
status = client.datasets.index_status(dataset.id)
if status["status"] != "indexing":
break
time.sleep(3)
print(status["status"]) # "ready"
print(status["total_chunks"]) # 84
```
Then search across indexed datasets:
```python theme={null}
results = client.search(
"invoices over 10000 EUR from Germany",
top_k=5,
filters={"quality_grade": "A", "language": "de"},
)
for r in results:
print(r["score"], r["chunk_text"][:80])
```
***
## Profile
```python theme={null}
profile = client.datasets.profile(dataset.id)
print(profile["quality_distribution"]) # {"A": 12, "B": 5, "C": 1}
print(profile["pii_type_summary"])
```
***
## Dataset fields
| Field | Type | Description |
| ------------------- | ----------- | --------------------------------------- |
| `id` | `str` | Dataset identifier |
| `name` | `str` | Human-readable name |
| `slug` | `str` | URL-safe slug |
| `status` | `str` | `"building"` \| `"ready"` \| `"failed"` |
| `row_count` | `int` | Number of records |
| `created_at` | `str` | ISO 8601 timestamp |
| `available_formats` | `list[str]` | Formats that have been generated |
# Installation
Source: https://docs.flexorch.com/sdk/python/installation
Install the FlexOrch Python SDK.
```bash theme={null}
pip install flexorch-sdk
```
Requires Python 3.10+.
***
## Setup
Set your API key as an environment variable:
```bash theme={null}
export FLEXORCH_API_KEY=dfx_your_key_here
```
The client picks it up automatically:
```python theme={null}
from flexorch_sdk import FlexOrch
client = FlexOrch() # reads FLEXORCH_API_KEY from env
```
Or pass it explicitly:
```python theme={null}
client = FlexOrch(api_key="dfx_your_key_here")
# with options
client = FlexOrch(
api_key="dfx_your_key_here",
timeout=60, # seconds — default 30
max_retries=3, # default 3
)
```
***
## Verify
```python theme={null}
from flexorch_sdk import FlexOrch
client = FlexOrch()
usage = client.usage.get()
print(usage["plan"]) # "trial"
print(usage["credits_used"]) # 0
print(usage["credits_remaining"]) # 1200
```
# Jobs
Source: https://docs.flexorch.com/sdk/python/jobs
Poll, inspect, and manage processing jobs.
## Get a job
```python theme={null}
from flexorch_sdk import FlexOrch
client = FlexOrch()
job = client.jobs.get("job_abc123")
print(job.status) # "completed"
print(job.quality_grade) # "A"
print(job.quality_score) # 91
```
***
## Wait for completion
`job.wait_until_done()` polls until the job finishes and returns the updated `Job`:
```python theme={null}
job = client.jobs.upload("document.pdf")
done = job.wait_until_done()
print(done.quality_grade) # "A"
```
**Options:**
```python theme={null}
done = job.wait_until_done(
timeout=120, # seconds before JobTimeoutError is raised (default: 300)
poll_interval=2, # seconds between polls (default: 2)
)
```
**Errors:**
```python theme={null}
from flexorch_sdk import JobFailedError, JobTimeoutError
try:
done = job.wait_until_done(timeout=60)
except JobFailedError as e:
print("Failed:", e.reason) # e.job_id also available
except JobTimeoutError as e:
print("Timed out:", e.job_id)
```
***
## Job fields
| Field | Type | Description |
| ---------------- | ------------- | -------------------------------------------------------- |
| `id` | `str` | Job identifier |
| `status` | `str` | `"queued"` \| `"running"` \| `"completed"` \| `"failed"` |
| `quality_grade` | `str \| None` | `"A"` – `"D"`, set when completed |
| `quality_score` | `int \| None` | 0–100, set when completed |
| `document_id` | `str \| None` | Associated document ID |
| `has_dataset` | `bool` | Whether a dataset has been built from this job |
| `failure_reason` | `str \| None` | Set when `status == "failed"` |
***
## List jobs
```python theme={null}
jobs = client.jobs.list(page=1, page_size=20)
for job in jobs.items:
print(job.id, job.status, job.quality_grade)
```
***
## Get associated dataset
If `job.has_dataset is True`, retrieve the built dataset:
```python theme={null}
dataset = job.dataset()
if dataset:
print(dataset.id)
print(dataset.row_count)
client.datasets.export(dataset.id, format="jsonl", path="output.jsonl")
```
Returns `None` if no dataset has been built yet.
***
## Submit feedback
```python theme={null}
job.feedback(rating="negative", issue="wrong_fields")
```
**rating:** `"positive"` | `"negative"`
**issue:** `"wrong_fields"` | `"missing_data"` | `"wrong_classification"` | `"pii_not_masked"` | `"other"`
# Quickstart
Source: https://docs.flexorch.com/sdk/python/quickstart
Process your first document and export a dataset in under 5 minutes.
## Prerequisites
* Python 3.10+
* A FlexOrch account and API key — generate one under **Settings → API Keys**
***
## Install
```bash theme={null}
pip install flexorch-sdk
```
***
## Full example
```python theme={null}
from flexorch_sdk import FlexOrchClient
client = FlexOrchClient() # reads FLEXORCH_API_KEY from env
# 1 — Upload and process a document
job = client.process("invoice.pdf")
print(job.id) # 1234
# 2 — Wait for the pipeline to finish
done = job.wait(timeout=120)
print(done.status) # "completed"
print(done.quality_grade) # "A"
# 3 — Build a dataset from this job (a completed job has no dataset yet —
# building one is a separate, explicit step)
build_job = done.build_dataset(name="q1-invoices")
dataset = build_job.wait().dataset()
# 4 — Export as JSONL for fine-tuning
dataset.export("jsonl", path="output.jsonl")
print("Done — output.jsonl written")
```
***
## Error handling
```python theme={null}
from flexorch_sdk import (
FlexOrchClient,
JobFailedError,
TimeoutError,
QuotaError,
)
client = FlexOrchClient()
try:
job = client.process("document.pdf")
done = job.wait(timeout=120)
print(done.quality_grade)
except JobFailedError as e:
print("Processing failed:", e.failure_reason)
except TimeoutError as e:
print("Timed out waiting for job:", e.job_id)
except QuotaError:
print("Credit quota exceeded — upgrade your plan")
```
The SDK's timeout error is exported as `TimeoutError` (shadowing the Python
builtin within this import) — there is no separate `JobTimeoutError` in the
Python package. The TypeScript SDK uses `JobTimeoutError` instead; see the
[TypeScript quickstart](/sdk/typescript/quickstart).
***
## What's next?
Single file, multi-file, from S3.
Poll, inspect, and manage processing jobs.
Build, export, and index datasets.
Full API surface — all classes and methods.
# SDK Reference
Source: https://docs.flexorch.com/sdk/python/reference
Complete Python SDK API surface — all classes and methods.
## FlexOrch
```python theme={null}
from flexorch_sdk import FlexOrch
client = FlexOrch(api_key_or_options=None)
```
**Constructor:**
| Argument | Type | Description |
| ------------- | ----- | ----------------------------------------------- |
| `api_key` | `str` | API key — falls back to `FLEXORCH_API_KEY` env |
| `base_url` | `str` | Override API base URL |
| `timeout` | `int` | Request timeout in seconds (default: `30`) |
| `max_retries` | `int` | Retry count for transient errors (default: `3`) |
***
## Top-level methods
| Method | Returns | Description |
| ---------------------------------------- | ------------ | --------------------------------------- |
| `client.search(query, top_k?, filters?)` | `list[dict]` | Semantic search across indexed datasets |
**search options:**
| Field | Type | Default | Description |
| --------- | ------ | ------- | --------------------------------------------------- |
| `top_k` | `int` | `5` | Number of results to return |
| `filters` | `dict` | — | Filter by `quality_grade`, `language`, `dataset_id` |
***
## client.jobs
| Method | Returns | Description |
| --------------------------------------------------------- | ---------------------- | ----------------------------------- |
| `jobs.upload(path, locale?)` | `Job` | Upload and process a single file |
| `jobs.upload_many(paths, locale?)` | `list[Job]` | Process multiple files sequentially |
| `jobs.upload_bytes(data, filename, locale?)` | `Job` | Upload from bytes |
| `jobs.upload_from_connector(connector_id, keys, locale?)` | `list[Job]` | Ingest files from a connector |
| `jobs.get(job_id)` | `Job` | Fetch a job by ID |
| `jobs.list(page?, page_size?)` | `PaginatedResult[Job]` | List jobs with pagination |
**locale:** language hint (`"tr"`, `"de"`, etc.) or `"und"` for auto-detect (default)
***
## Job
| Field / Method | Type | Description |
| ----------------------------------------------- | ----------------- | -------------------------------------------------------- |
| `id` | `str` | Job ID |
| `status` | `str` | `"queued"` \| `"running"` \| `"completed"` \| `"failed"` |
| `quality_grade` | `str \| None` | `"A"`–`"D"` |
| `quality_score` | `int \| None` | 0–100 |
| `document_id` | `str \| None` | Document ID |
| `has_dataset` | `bool` | Dataset has been built |
| `failure_reason` | `str \| None` | Set on failure |
| `job.wait_until_done(timeout?, poll_interval?)` | `Job` | Poll until completed or failed |
| `job.dataset()` | `Dataset \| None` | Fetch associated dataset |
| `job.feedback(rating, issue?)` | `None` | Submit quality feedback |
**wait\_until\_done options:** `timeout` (seconds, default 300), `poll_interval` (seconds, default 2)
***
## client.datasets
| Method | Returns | Description |
| ------------------------------------------------------------------ | -------------------------- | -------------------------- |
| `datasets.build(name, job_ids)` | `Dataset` | Build a dataset from jobs |
| `datasets.get(dataset_id)` | `Dataset` | Fetch a dataset by ID |
| `datasets.list(page?, page_size?)` | `PaginatedResult[Dataset]` | List datasets |
| `datasets.export(dataset_id, format, path)` | `None` | Export to file |
| `datasets.export_to_s3(dataset_id, connector_id, format, prefix?)` | `dict` | Push export to a connector |
| `datasets.index(dataset_id)` | `dict` | Start semantic indexing |
| `datasets.index_status(dataset_id)` | `dict` | Check indexing progress |
| `datasets.profile(dataset_id)` | `dict` | Quality and PII profile |
***
## Dataset
| Field | Type | Description |
| ------------------- | ----------- | --------------------------------------- |
| `id` | `str` | Dataset ID |
| `name` | `str` | Name |
| `slug` | `str` | URL-safe slug |
| `status` | `str` | `"building"` \| `"ready"` \| `"failed"` |
| `row_count` | `int` | Number of records |
| `created_at` | `str` | ISO 8601 |
| `available_formats` | `list[str]` | Generated export formats |
**ExportFormat:** `"json"` | `"jsonl"` | `"csv"` | `"parquet"` | `"md"` | `"xml"` | `"xlsx"` | `"rag"`
***
## client.connectors
| Method | Returns | Description |
| --------------------------------------- | ------------ | ---------------------- |
| `connectors.create(name, type, config)` | `dict` | Create a connector |
| `connectors.list()` | `list[dict]` | List all connectors |
| `connectors.get(connector_id)` | `dict` | Get a single connector |
| `connectors.delete(connector_id)` | `None` | Delete a connector |
| `connectors.test(connector_id)` | `dict` | Test connectivity |
**ConnectorType:** `"s3"` | `"gcs"` | `"azure_blob"`
***
## client.usage
| Method | Returns | Description |
| --------------------- | ------- | ------------------------------- |
| `usage.get()` | `dict` | Current credit balance and plan |
| `usage.rate_limits()` | `dict` | RPM limits and remaining |
***
## client.webhooks
| Method | Returns | Description |
| --------------------------------------- | ------------ | ------------------------ |
| `webhooks.list()` | `list[dict]` | List registered webhooks |
| `webhooks.create(url, events, secret?)` | `dict` | Register a webhook |
| `webhooks.delete(webhook_id)` | `None` | Remove a webhook |
**WebhookEvent:** `"job.completed"` | `"job.failed"` | `"dataset.ready"`
***
## Errors
| Class | Raised when |
| ----------------- | ----------------------------------------------------------- |
| `AuthError` | Invalid or revoked API key (401) |
| `QuotaError` | Credit limit exceeded (402/429) |
| `RateLimitError` | RPM limit exceeded (429) |
| `NotFoundError` | Resource not found (404) |
| `ValidationError` | Bad request (400/422) |
| `ServerError` | Server-side error (5xx) |
| `JobFailedError` | `job.wait_until_done()` — job ended with status `"failed"` |
| `JobTimeoutError` | `job.wait_until_done()` — timeout elapsed before completion |
All errors extend `FlexOrchError`.
# Upload
Source: https://docs.flexorch.com/sdk/python/upload
Upload documents for processing using the Python SDK.
## Single file
Use `client.jobs.upload()` — the standard entry point for uploading and queuing a single document:
```python theme={null}
from flexorch_sdk import FlexOrch
client = FlexOrch()
job = client.jobs.upload("invoice.pdf")
print(job.id) # "job_abc123"
print(job.status) # "queued"
```
The returned `Job` object can be awaited with `job.wait_until_done()`.
***
## Multiple files
```python theme={null}
jobs = client.jobs.upload_many([
"invoices/jan.pdf",
"invoices/feb.pdf",
"payroll/q1.xlsx",
])
print(len(jobs)) # 3
print([j.id for j in jobs]) # ["job_001", "job_002", "job_003"]
```
Each file gets its own `Job`. Files are processed sequentially.
***
## With options
```python theme={null}
job = client.jobs.upload("document.pdf", locale="tr")
# locale hint: 'und' = auto-detect (default), 'tr', 'de', 'fr', etc.
```
***
## From bytes
```python theme={null}
with open("document.pdf", "rb") as f:
job = client.jobs.upload_bytes(f.read(), filename="document.pdf")
```
***
## From a connector (S3 / GCS / Azure)
If you have a connector configured, process files directly from cloud storage:
```python theme={null}
jobs = client.jobs.upload_from_connector(
connector_id=1,
keys=[
"documents/invoice_2024.pdf",
"documents/payroll_q1.xlsx",
],
)
for job in jobs:
done = job.wait_until_done()
print(done.quality_grade)
```
See [Connectors](/connectors/overview) for setup.
***
## Polling manually
If you need fine-grained control over polling:
```python theme={null}
import time
job = client.jobs.upload("document.pdf")
while job.status in ("queued", "running"):
time.sleep(2)
job = client.jobs.get(job.id)
if job.status == "completed":
print("Grade:", job.quality_grade)
else:
print("Failed:", job.failure_reason)
```
Or use `job.wait_until_done()` — it handles this automatically with configurable timeout and poll interval.
# Datasets
Source: https://docs.flexorch.com/sdk/typescript/datasets
Build and export datasets using the TypeScript SDK.
## Get a dataset
```typescript theme={null}
import { FlexOrchClient } from 'flexorch-sdk';
const client = new FlexOrchClient();
const dataset = await client.datasets.get('ds_abc123');
console.log(dataset.name); // "q1-invoices"
console.log(dataset.rowCount); // 42
console.log(dataset.status); // "ready"
console.log(dataset.availableFormats); // ["jsonl", "csv", "parquet"]
```
***
## List datasets
```typescript theme={null}
const datasets = await client.datasets.list({ page: 1, pageSize: 20 });
for (const ds of datasets) {
console.log(ds.id, ds.name, ds.rowCount);
}
```
***
## Export
`dataset.export(format)` returns a `Uint8Array`. Write it to disk with `fs.writeFile`:
```typescript theme={null}
import { writeFile } from 'node:fs/promises';
const dataset = await client.datasets.get('ds_abc123');
// JSONL for LLM fine-tuning
const jsonl = await dataset.export('jsonl');
await writeFile('output.jsonl', jsonl);
// Parquet for analytics
const parquet = await dataset.export('parquet');
await writeFile('output.parquet', parquet);
// Markdown for RAG
const md = await dataset.export('md');
await writeFile('output.md', md);
```
**Supported formats:** `"json"` | `"jsonl"` | `"csv"` | `"parquet"` | `"md"` | `"xml"` | `"xlsx"` | `"rag"`
***
## Export directly to S3
Push a dataset to a connected S3/GCS/Azure bucket without downloading it first:
```typescript theme={null}
const result = await dataset.exportToS3(
'connector_id_here',
'jsonl',
'exports/datasets/', // optional prefix
);
console.log(result.s3Key); // "exports/datasets/q1-invoices.jsonl"
console.log(result.sizeBytes); // 148302
```
***
## Semantic indexing
Available on **Pro and Enterprise** plans.
Index a dataset for semantic search:
```typescript theme={null}
// Start indexing (async, returns immediately)
await dataset.index();
// Poll status
let status = await dataset.indexStatus();
while (status.status === 'indexing') {
await new Promise(r => setTimeout(r, 3000));
status = await dataset.indexStatus();
}
console.log(status.status); // "ready"
console.log(status.totalChunks); // 84
```
Then search across indexed datasets:
```typescript theme={null}
const results = await client.search('invoices over 10000 EUR from Germany', {
topK: 5,
filters: { qualityGrade: 'A', language: 'de' },
});
for (const r of results) {
console.log(r.score, r.chunkText.slice(0, 80));
}
```
***
## Dataset fields
| Field | Type | Description |
| ------------------ | ---------- | --------------------------------------- |
| `id` | `string` | Dataset identifier |
| `name` | `string` | Human-readable name |
| `slug` | `string` | URL-safe slug |
| `status` | `string` | `"building"` \| `"ready"` \| `"failed"` |
| `rowCount` | `number` | Number of records |
| `createdAt` | `string` | ISO 8601 timestamp |
| `availableFormats` | `string[]` | Formats that have been generated |
# Installation
Source: https://docs.flexorch.com/sdk/typescript/installation
Install the FlexOrch TypeScript / JavaScript SDK.
```bash theme={null}
npm install flexorch-sdk
```
Requires Node.js 18+. Works with ESM and CommonJS.
***
## Setup
Set your API key as an environment variable:
```bash theme={null}
export FLEXORCH_API_KEY=dfx_your_key_here
```
The client picks it up automatically:
```typescript theme={null}
import { FlexOrchClient } from 'flexorch-sdk';
const client = new FlexOrchClient(); // reads FLEXORCH_API_KEY from env
```
Or pass it explicitly:
```typescript theme={null}
const client = new FlexOrchClient('dfx_your_key_here');
// options object form
const client = new FlexOrchClient({
apiKey: 'dfx_your_key_here',
timeout: 60, // seconds — default 30
maxRetries: 3, // default 3
});
```
***
## CommonJS
```javascript theme={null}
const { FlexOrchClient } = require('flexorch-sdk');
const client = new FlexOrchClient();
```
***
## TypeScript types
All exports are fully typed. Import types alongside the client:
```typescript theme={null}
import { FlexOrchClient } from 'flexorch-sdk';
import type { ExportFormat } from 'flexorch-sdk';
const format: ExportFormat = 'jsonl'; // "json" | "jsonl" | "csv" | "parquet" | "md" | "xml" | "xlsx" | "rag"
```
***
## Verify
```typescript theme={null}
import { FlexOrchClient } from 'flexorch-sdk';
const client = new FlexOrchClient();
const usage = await client.usage.get();
console.log(usage.plan); // "trial"
console.log(usage.creditsUsed); // 0
console.log(usage.creditsRemaining); // 1200
```
# Jobs
Source: https://docs.flexorch.com/sdk/typescript/jobs
Poll, inspect, and manage processing jobs.
## Get a job
```typescript theme={null}
import { FlexOrchClient } from 'flexorch-sdk';
const client = new FlexOrchClient();
const job = await client.jobs.get('job_abc123');
console.log(job.status); // "completed"
console.log(job.qualityGrade); // "A"
console.log(job.qualityScore); // 91
```
***
## Wait for completion
`job.wait()` polls until the job finishes and returns the updated `Job`:
```typescript theme={null}
const job = await client.process('document.pdf');
const done = await job.wait();
console.log(done.qualityGrade); // "A"
```
**Options:**
```typescript theme={null}
const done = await job.wait({
timeout: 120, // seconds before JobTimeoutError is thrown (default: 300)
pollInterval: 2, // seconds between polls (default: 2)
});
```
**Errors:**
```typescript theme={null}
import { JobFailedError, JobTimeoutError } from 'flexorch-sdk';
try {
const done = await job.wait({ timeout: 60 });
} catch (err) {
if (err instanceof JobFailedError) {
console.error('Failed:', err.reason); // err.jobId also available
} else if (err instanceof JobTimeoutError) {
console.error('Timed out:', err.jobId);
}
}
```
***
## Job fields
| Field | Type | Description |
| --------------- | ---------------- | -------------------------------------------------------- |
| `id` | `string` | Job identifier |
| `status` | `string` | `"queued"` \| `"running"` \| `"completed"` \| `"failed"` |
| `qualityGrade` | `string \| null` | `"A"` – `"D"`, set when completed |
| `qualityScore` | `number \| null` | 0–100, set when completed |
| `documentId` | `string \| null` | Associated document ID |
| `hasDataset` | `boolean` | Whether a dataset has been built from this job |
| `failureReason` | `string \| null` | Set when `status === "failed"` |
***
## List jobs
```typescript theme={null}
const jobs = await client.jobs.list({ page: 1, pageSize: 20 });
for (const job of jobs) {
console.log(job.id, job.status, job.qualityGrade);
}
```
***
## Get associated dataset
If `job.hasDataset === true`, retrieve the built dataset:
```typescript theme={null}
const dataset = await job.dataset();
if (dataset) {
console.log(dataset.id);
console.log(dataset.rowCount);
const bytes = await dataset.export('jsonl');
}
```
Returns `null` if no dataset has been built yet.
# Quickstart
Source: https://docs.flexorch.com/sdk/typescript/quickstart
Process your first document and export a dataset in under 5 minutes.
## Prerequisites
* Node.js 18+
* A FlexOrch account and API key — generate one under **Settings → API Keys**
***
## Install
```bash theme={null}
npm install flexorch-sdk
```
***
## Full example
```typescript theme={null}
import { FlexOrchClient } from 'flexorch-sdk';
import { writeFile } from 'node:fs/promises';
const client = new FlexOrchClient(process.env.FLEXORCH_API_KEY!);
// 1 — Upload and process a document
const job = await client.process('invoice.pdf');
console.log(job.id); // 1234
// 2 — Wait for the pipeline to finish
const done = await job.wait({ timeout: 120 }); // seconds
console.log(done.status); // "completed"
console.log(done.qualityGrade); // "A"
// 3 — Build a dataset from this job (a completed job has no dataset yet —
// building one is a separate, explicit step)
const buildJob = await done.buildDataset({ name: 'q1-invoices' });
const dataset = await (await buildJob.wait()).dataset();
// 4 — Export as JSONL for fine-tuning
const bytes = await dataset!.export('jsonl');
await writeFile('output.jsonl', bytes);
console.log('Done — output.jsonl written');
```
***
## Error handling
```typescript theme={null}
import {
FlexOrchClient,
JobFailedError,
JobTimeoutError,
QuotaError,
} from 'flexorch-sdk';
const client = new FlexOrchClient();
try {
const job = await client.process('document.pdf');
const done = await job.wait({ timeout: 120 });
console.log(done.qualityGrade);
} catch (err) {
if (err instanceof JobFailedError) {
console.error('Processing failed:', err.failureReason);
} else if (err instanceof JobTimeoutError) {
console.error('Timed out waiting for job:', err.jobId);
} else if (err instanceof QuotaError) {
console.error('Credit quota exceeded — upgrade your plan');
} else {
throw err;
}
}
```
***
## What's next?
Single file, multi-file, from S3.
Poll, inspect, and manage processing jobs.
Build, export, and index datasets.
Full API surface — all classes and methods.
# SDK Reference
Source: https://docs.flexorch.com/sdk/typescript/reference
Complete TypeScript SDK API surface — all classes and methods.
## FlexOrchClient
```typescript theme={null}
import { FlexOrchClient } from 'flexorch-sdk';
const client = new FlexOrchClient(apiKeyOrOptions?)
```
**Constructor:**
| Argument | Type | Description |
| ----------------- | --------------------------------- | --------------------------------- |
| `apiKeyOrOptions` | `string \| FlexOrchClientOptions` | API key string, or options object |
**FlexOrchClientOptions:**
| Field | Type | Default | Description |
| ------------ | -------- | ----------------------------- | -------------------------------- |
| `apiKey` | `string` | `FLEXORCH_API_KEY` env | Your API key |
| `baseUrl` | `string` | `https://api.flexorch.com/v1` | Base URL override |
| `timeout` | `number` | `30` | Request timeout in seconds |
| `maxRetries` | `number` | `3` | Retry count for transient errors |
***
## Top-level methods
| Method | Returns | Description |
| ------------------------------------------------ | ------------------------- | --------------------------------------- |
| `client.process(filePath, opts?)` | `Promise` | Upload and process a single file |
| `client.processMany(filePaths, opts?)` | `Promise` | Process multiple files sequentially |
| `client.processFromS3(connectorId, keys, opts?)` | `Promise` | Ingest files from a connector |
| `client.search(query, opts?)` | `Promise` | Semantic search across indexed datasets |
**process / processMany options:**
| Field | Type | Default | Description |
| ---------------- | ------------------------- | ------- | --------------------------------------------------------------- |
| `locale` | `string` | `"und"` | Language hint (`"tr"`, `"de"`, etc.) or `"und"` for auto-detect |
| `pipelineConfig` | `Record` | — | Advanced pipeline overrides |
***
## client.jobs
| Method | Returns | Description |
| ------------------ | ---------------- | ------------------------- |
| `jobs.get(jobId)` | `Promise` | Fetch a job by ID |
| `jobs.list(opts?)` | `Promise` | List jobs with pagination |
**list options:** `{ page?: number, pageSize?: number }`
***
## Job
| Field / Method | Type | Description |
| ----------------- | -------------------------- | -------------------------------------------------------- |
| `id` | `string` | Job ID |
| `status` | `string` | `"queued"` \| `"running"` \| `"completed"` \| `"failed"` |
| `qualityGrade` | `string \| null` | `"A"`–`"D"` |
| `qualityScore` | `number \| null` | 0–100 |
| `documentId` | `string \| null` | Document ID |
| `hasDataset` | `boolean` | Dataset has been built |
| `failureReason` | `string \| null` | Set on failure |
| `job.wait(opts?)` | `Promise` | Poll until completed or failed |
| `job.dataset()` | `Promise` | Fetch associated dataset |
**wait options:** `{ timeout?: number, pollInterval?: number }` (seconds)
***
## client.datasets
| Method | Returns | Description |
| ------------------------- | -------------------- | ----------------------------- |
| `datasets.get(datasetId)` | `Promise` | Fetch a dataset by ID |
| `datasets.list(opts?)` | `Promise` | List datasets with pagination |
***
## Dataset
| Field / Method | Type | Description |
| -------------------------------------------------- | ------------------------------------------------- | --------------------------------------- |
| `id` | `string` | Dataset ID |
| `name` | `string` | Name |
| `slug` | `string` | URL-safe slug |
| `status` | `string` | `"building"` \| `"ready"` \| `"failed"` |
| `rowCount` | `number` | Number of records |
| `createdAt` | `string` | ISO 8601 |
| `availableFormats` | `string[]` | Generated export formats |
| `dataset.export(format)` | `Promise` | Download export as bytes |
| `dataset.exportToS3(connectorId, format, prefix?)` | `Promise<{ s3Key, sizeBytes }>` | Push export to a connector |
| `dataset.index()` | `Promise<{ status, message }>` | Start semantic indexing |
| `dataset.indexStatus()` | `Promise<{ status, chunksIndexed, totalChunks }>` | Check indexing progress |
**ExportFormat:** `"json" | "jsonl" | "csv" | "parquet" | "md" | "xml" | "xlsx" | "rag"`
***
## client.connectors
| Method | Returns | Description |
| --------------------------------------- | ------------------------------ | ---------------------- |
| `connectors.create(name, type, config)` | `Promise` | Create a connector |
| `connectors.list()` | `Promise` | List all connectors |
| `connectors.get(connectorId)` | `Promise` | Get a single connector |
| `connectors.delete(connectorId)` | `Promise` | Delete a connector |
| `connectors.test(connectorId)` | `Promise` | Test connectivity |
**ConnectorType:** `"s3" | "gcs" | "azure_blob"`
***
## client.usage
| Method | Returns | Description |
| ------------- | ------------------------ | ------------------------------- |
| `usage.get()` | `Promise` | Current credit balance and plan |
***
## client.webhooks
| Method | Returns | Description |
| --------------------------------------- | -------------------- | ------------------------ |
| `webhooks.list()` | `Promise` | List registered webhooks |
| `webhooks.create(url, events, secret?)` | `Promise` | Register a webhook |
| `webhooks.delete(webhookId)` | `Promise` | Remove a webhook |
**WebhookEvent:** `"job.completed" | "job.failed" | "dataset.ready"`
***
## Errors
| Class | Thrown when |
| ----------------- | ------------------------------------------------ |
| `AuthError` | Invalid or revoked API key (401) |
| `QuotaError` | Credit limit exceeded (402/429) |
| `RateLimitError` | RPM limit exceeded (429) |
| `NotFoundError` | Resource not found (404) |
| `ValidationError` | Bad request (400/422) |
| `ServerError` | Server-side error (5xx) |
| `JobFailedError` | `job.wait()` — job ended with status `"failed"` |
| `JobTimeoutError` | `job.wait()` — timeout elapsed before completion |
All errors extend `FlexOrchError`.
# Upload
Source: https://docs.flexorch.com/sdk/typescript/upload
Upload documents for processing using the TypeScript SDK.
## Single file
Use `client.process()` — the top-level shortcut for uploading and queuing a single document:
```typescript theme={null}
import { FlexOrchClient } from 'flexorch-sdk';
const client = new FlexOrchClient();
const job = await client.process('invoice.pdf');
console.log(job.id); // "job_abc123"
console.log(job.status); // "queued"
```
The returned `Job` object can be awaited with `job.wait()`.
***
## Multiple files
```typescript theme={null}
const jobs = await client.processMany([
'invoices/jan.pdf',
'invoices/feb.pdf',
'payroll/q1.xlsx',
]);
console.log(jobs.length); // 3
console.log(jobs.map(j => j.id)); // ["job_001", "job_002", "job_003"]
```
Each file gets its own `Job`. `processMany` processes them sequentially.
***
## With options
```typescript theme={null}
const job = await client.process('document.pdf', {
locale: 'tr', // hint the pipeline language; 'und' = auto-detect (default)
});
```
***
## From a connector (S3 / GCS / Azure)
If you have a connector configured, process files directly from cloud storage:
```typescript theme={null}
const jobs = await client.processFromS3(
'connector_id_here',
[
'documents/invoice_2024.pdf',
'documents/payroll_q1.xlsx',
],
);
for (const job of jobs) {
const done = await job.wait();
console.log(done.qualityGrade);
}
```
See [Connectors](/connectors/overview) for setup.
***
## Polling manually
If you need fine-grained control over polling:
```typescript theme={null}
let job = await client.process('document.pdf');
while (job.status === 'queued' || job.status === 'running') {
await new Promise(r => setTimeout(r, 2000)); // 2s
job = await client.jobs.get(job.id);
}
if (job.status === 'completed') {
console.log('Grade:', job.qualityGrade);
} else {
console.error('Failed:', job.failureReason);
}
```
Or use `job.wait()` — it handles this automatically with configurable timeout and poll interval.
# Invite Members
Source: https://docs.flexorch.com/team/invite-members
Add team members by email.
## Send an invitation
```bash theme={null}
curl -X POST https://api.flexorch.com/v1/team/invite \
-H "Authorization: Bearer eyJ..." \
-H "Content-Type: application/json" \
-d '{"email": "colleague@company.com", "role": "member"}'
```
An email is sent to the invitee with an acceptance link. The link expires after 7 days.
## Accept an invitation
The invitee clicks the link in the email, creates an account (or signs in), and is added to the team.
## Remove a member
```bash theme={null}
curl -X DELETE https://api.flexorch.com/v1/team/members/{user_id} \
-H "Authorization: Bearer eyJ..."
```
# Team Management
Source: https://docs.flexorch.com/team/overview
Invite team members and manage roles.
Team management is available on **Pro and Enterprise** plans.
FlexOrch supports multi-member teams scoped to your account. All members share the same workspace — documents, jobs, datasets, and connectors are visible to all members with appropriate roles.
## Roles
| Role | Description |
| -------- | -------------------------------------------------- |
| `owner` | Full access, including billing and team management |
| `admin` | Full access except billing |
| `member` | Can upload, process, and export |
| `viewer` | Read-only access |
## Manage your team
Go to **Settings → Team** in the platform to invite members, update roles, and revoke access.
See [Invite Members](/team/invite-members) and [Roles](/team/roles) for details.
# Roles & Permissions
Source: https://docs.flexorch.com/team/roles
What each role can do.
| Action | Viewer | Member | Admin | Owner |
| ------------------------------ | ------ | ------ | ----- | ----- |
| View documents, jobs, datasets | ✓ | ✓ | ✓ | ✓ |
| Upload documents | — | ✓ | ✓ | ✓ |
| Build datasets and export | — | ✓ | ✓ | ✓ |
| Manage webhooks | — | — | ✓ | ✓ |
| Manage connectors | — | — | ✓ | ✓ |
| Invite / remove members | — | — | ✓ | ✓ |
| Change member roles | — | — | ✓ | ✓ |
| Manage billing | — | — | — | ✓ |
| Delete account | — | — | — | ✓ |
## Update a member's role
```bash theme={null}
curl -X PATCH https://api.flexorch.com/v1/team/members/{user_id}/role \
-H "Authorization: Bearer eyJ..." \
-H "Content-Type: application/json" \
-d '{"role": "admin"}'
```
The team owner's role cannot be changed. There must always be at least one owner.