What is Scale AI?
Scale AI is a San Francisco company that supplies training data, model evaluations and applied AI systems. Its about page states the mission "to develop reliable AI systems for the world's most important decisions" and groups the work into three areas: Data at Scale (training data, annotations and RLHF), Evaluations (model evaluations and red-teaming) and Applied AI (full-stack systems for enterprises and governments). The same page lists 2016 as the founding year, more than 1,000 employees and headquarters in San Francisco, CA.
The product line on scale.com consists of three named offerings. The Scale Data Engine covers data collection, curation and annotation for text, image, video and 3D sensor data, plus a Generative AI Data Engine for prompt-response generation, RLHF, red teaming and evaluation datasets. The Scale GenAI Portfolio (SGP, documented as the Scale GenAI Platform) is a platform for building, deploying, evaluating and monitoring AI agents on enterprise data. Scale Donovan is the public sector offering for building and deploying agents on government networks.
Scale is a commercial, closed-source vendor service. Customers reach it through the web dashboard at dashboard.scale.com, a REST API versioned at https://api.scale.com/v1/ and SDKs for Python (scaleapi on PyPI and conda-forge) and JavaScript (scaleapi on npm). SGP is additionally documented for deployment into a customer AWS account, Azure subscription or GCP project, using Scale-managed Terraform modules and a Kubernetes operator called System Manager. Donovan is listed for GovCloud at DISA IL4 and FedRAMP High, for classified networks and as a Kubernetes containerized platform.
Scale also runs SEAL, a research lab that publishes public leaderboards at labs.scale.com. The leaderboard site lists more than 20 benchmarks covering agentic coding, frontier reasoning and safety, and more than 100 evaluated models. The SGP product page refers to SEAL as the in-house frontier benchmarking lab used for its evaluation work.
No public price list exists: the scale.com pricing URL redirects to the demo request form, and the customer support page directs prospective customers to sales. Existing Pro and Nucleus customers are supported through an Engagement Manager or Field Engineer during business hours, 9:00 to 17:00 Pacific Time, Monday to Friday. The security page lists SOC 2 Type II, ISO/IEC 27001:2022, a DoD IL4 Provisional Authorization and FedRAMP High authorization, with reports in the Scale Trust Center. Related listings on iThub include Weights & Biases for experiment tracking and evaluation and LangChain for agent development.
Scale AI screenshots
Explore the interface. Open a screenshot to view it at full size.
SGP Dashboards view from the Scale documentation, with demo data comparing agent versions by accuracy scores.
Official SGP overview diagram: Agentex handles agent execution, AgentOps covers management, tracing, evaluation and improvement.
Evaluation results overview in SGP, from the LLM as a Judge guide in the Scale documentation.
Configuring an LLM judge in SGP: model, optional system prompt and a rubric with item variables.
Quality tab of the Scale Pro customer dashboard, from the Scale documentation.
Source: official Scale AI project page.
Key capabilities
- Scale Pro data labeling Scale Pro is the managed labeling service in the Data Engine. Tasks are submitted through the API or the dashboard, labeled by Scale with machine learning pre-labeling and human review, and delivered on a schedule agreed with the customer. Dedicated Engagement Managers set up projects, and the documentation overview lists complex 3D and sensor fusion formats and SLA-backed quality as part of Pro.
- Supported data types The Data Engine page lists text (document processing, natural language processing, transcription, content and language), images (electro-optical, infrared, transcription), video (full motion video) and 3D (sensor fusion and LiDAR). Annotation types defined in a project taxonomy include box, polygon, point, ellipse, cuboid, event, text response, list selection, tree selection, date, linear scale and ranking.
- Projects, tasks and batches A task is one unit of work mapped to one item of data, with a Scale-generated ID. Tasks belong to a project that is tied to a single task type and a single set of instructions; projects cannot be deleted and their task type cannot be changed afterwards. On Pro, batches divide a project further, for example by dataset or weekly submission.
- Customer dashboard The dashboard has Overview, Tasks and Quality tabs. Tasks can be filtered by creation or completion date, status (Completed, In Progress, Queued, Error, Canceled, Redo) and audit status. JSON export from the Tasks page is limited to 1,000 tasks; larger exports use the API. Only queued tasks can be canceled, and canceling is done through the API.
- Quality audits The Quality tab audits a random sample of tasks, all completed tasks in a date range or tasks completed in a given week. Each audit is Approve, Make Changes or Reject, and the callback body then carries
customer_review_status,customer_review_commentsandprior_responses. Reports include a task level report, an auditor tracker for batches and per-auditor quality. - Scale REST API and SDKs The API is resource oriented, accepts form-encoded request bodies and returns JSON. A sandbox mode is selected by the API key used. The API does not support bulk updates: each request acts on a single object. Official libraries are
pip install --upgrade scaleapi, the conda-forge package andnpm install scaleapi, per the API introduction. - API limits and compatibility The technical limits page gives 30 task creation requests per second, 2,500 attributes, 10 KB task metadata, 8 KB file upload metadata and 120 MB per attachment, with no limit on tasks per batch. The compatibility policy states that only v1 is supported and that breaking changes happen only in a new API version.
- Data hosting integrations Attachments can be fetched from AWS S3, Google Cloud Storage or Azure Blob Storage. For S3, Scale assumes an IAM role named
ScaleAI-Integrationin the customer account with an external ID, or reads through a cross-account bucket policy. Non-AWS customers can allowlist a published set of static IP addresses used for fetching data and sending callbacks. - Generative AI Data Engine Datasets for model builders are produced by vetted subject matter experts. The GenAI documentation describes project archetypes for evals, RLHF, rubrics and supervised fine-tuning, and an API built around projects, batches, tasks, threads, turns, messages and annotations. Pre-built expert datasets are also exposed as Dataset resources.
- Nucleus dataset management Nucleus brings data, labels and model predictions together. The documentation lists visualising data at scale, curating dataset segments, reviewing annotations and measuring model performance. Its full documentation is hosted separately at nucleus.scale.com.
- SGP agent building with Agentex Agentex is the agent building and execution layer of SGP. It provides a CLI to run an agent locally, a local UI for testing and a deployment pipeline that pushes changes to a hosted agent, with deployment history visible in the platform. The product page lists long-running asynchronous workflows, multi-agent coordination and model choice across providers.
- SGP evaluations Evaluations combine data, optionally a reusable dataset, with tasks. Documented task types are auto evaluation with an LLM judge, agent evaluation, Agentex evaluation, contributor evaluation by human annotators and custom Python scoring functions. The evaluation guide uses the
scale-gp-betaPython package, which runs on the V5 API. - Evaluation dashboards Dashboards are scoped to a single evaluation or an evaluation group. Nine widget types exist: metric, table, bar chart, histogram, donut chart, scatter plot, timeseries, markdown and heading. Data widgets are driven by a SQL-like JSON query with aggregations such as AVG, SUM, COUNT, PERCENTILE and COUNT_DISTINCT.
- AgentOps tracing and monitoring Agent Operations covers managing agents, reviewing traces of live interactions and feeding human feedback back into improvement. The SGP page describes semantic layer monitoring, evaluation scoring, trace transparency, audit trails and source-cited outputs. An AgentOps SDK for Python is documented alongside the Agentex SDK.
- Deployment in a customer cloud The AWS deployment guide requires Terraform 1.9.0 or later, kubectl, helm, jq and the
sgpctlCLI, plus Terraform modules, a workspace ID and a registration token from Scale. Equivalent guides exist for Azure and GCP. Optional components include the Agentex service, Model Engine, Spellbook prompt management and Document Understanding. - Scale Donovan Donovan offers a no-code Agent Factory for building agents from data sources and knowledge bases, model comparison with metrics such as factual accuracy and hallucination rate, and an Agent Arsenal of prebuilt agents. It connects to sources such as GDELT and runs on Unclassified/CUI GovCloud at IL4 and FedRAMP High, on classified networks and as a cloud agnostic Kubernetes deployment.
- SEAL leaderboards The SEAL leaderboard site publishes rankings on private benchmarks in agentic, safety and frontier categories, such as Humanity's Last Exam, MCP Atlas and SWE-Bench Pro. Scores are shown with confidence intervals. The leaderboards are public pages; the underlying private benchmark sets are run by Scale.
Where Scale AI fits
Labeling training data for perception models
An ML team creates a project for one task type, such as 3D cuboid annotation on LiDAR and camera data, defines the taxonomy and instructions, and submits tasks through the API with attachments stored in S3, GCS or Azure Blob Storage. Scale Pro labels the data and the team retrieves completed task responses through callbacks, the Tasks tab or the List Tasks endpoint. The documentation describes this as the core Data Engine workflow.
Auditing delivered annotations
Reviewers use the Quality tab to sample completed tasks, then approve, fix or reject them. Fixes update the callback response immediately and keep earlier responses in prior_responses; rejections send feedback to Scale's taskers. Weekly reports aggregate tasks completed from Monday to Sunday Pacific Time for the last four weeks, as described in the Quality tab documentation.
Producing post-training data for language models
Model developers commission RLHF preference data, supervised fine-tuning examples, rubric-based ratings and evaluation sets through the Generative AI Data Engine. Work is organised into projects and batches, and delivered tasks are structured as threads, turns and messages with annotations. Batches can be finalized, paused, resumed or canceled through the GenAI API.
Building and operating enterprise agents
An enterprise team connects sources such as Confluence, SharePoint or S3 to SGP, develops agents with Agentex locally and deploys them to the hosted environment. Live interactions appear in tracing, and evaluations with LLM judges, agent judges or human contributors score agent output. SGP can run inside the customer's own AWS, Azure or GCP environment.
Comparing models and agent versions
Evaluation groups and dashboards compare metrics across runs, for example accuracy averages per agent name and version or response length. Timeseries widgets track metrics across evaluation runs. The same evaluation can mix automatic scoring with contributor evaluation and auditing by human reviewers.
Deploying agents on government networks
Public sector teams use Donovan to assemble agents with a no-code interface, compare models such as Llama variants on factual accuracy, hallucination rate and latency, and deploy on GovCloud, classified or air-gapped networks. Donovan integrates with SGP for agent building, according to the Donovan product page.
Install
pip install --upgrade scaleapiReviews of Scale AI
Be the first to share your experience with Scale AI.
Write a review
Loading review form…
Alternatives to Scale AI
Weights & BiasesWeights & Biases tracks ML experiments and manages models and datasets, with Weave for LLM and agent evaluation, now offered within CoreWeave Forge.
LangChainLangChain is an MIT-licensed open-source framework (Python/JS) for building LLM applications and agents via LangGraph; LangSmith adds optional hosted tracing, evaluation and deployment.
CohereCohere offers Command language models, Embed and Rerank via API, plus the North platform, with free rate-limited trial keys and pay-as-you-go production keys.