Choose K3 without an A/B test when the request genuinely needs more than K2.6’s 256K context or when the direct API must enforce tool_choice="required". Choose K2.6 without an A/B test when the workflow requires true non-thinking operation. When both models are eligible, run the same tasks and select the lowest cost per accepted result—not the model with the largest parameter count or the most persuasive single answer.
Experiment disclosure: this page provides a controlled protocol and worked price envelopes. It does not claim that kimi-ai.free measured live latency or quality scores. Publish a winner only after running the harness, saving outputs, and attaching the result sheet.
Use deterministic eligibility gates first
Some model decisions are structural and should not be blurred by an average benchmark score. Check the complete request budget—not only source documents—against the context window. Include system instructions, tool schemas, history, media tokens, tool results, and the requested completion allowance. Then check whether the workflow needs a parameter one model does not support. If a hard gate decides the route, skip the comparison and document the reason.
| Gate | K2.6 | K3 | Routing result |
|---|---|---|---|
| Total relevant request budget is over 262,144 tokens | Ineligible | Eligible up to 1,048,576 tokens | K3, after estimating tokens and reserving headroom |
| Direct API must require at least one tool call | tool_choice="required" unsupported | Supports required | K3 or enforce the action outside the model |
| Thinking must be disabled | Supported | Always reasons | K2.6 |
| Work is a repository-centered coding-agent task | General-purpose candidate | Flagship candidate | Add K2.7 Code; this two-model page is not the complete candidate set |
| Both fit and no unique parameter is required | Eligible | Eligible | Run the controlled experiment |
A hard gate should be based on a real requirement. Do not claim “needs 1M context” because a folder contains many files; retrieve and estimate the relevant working set first. Do not require a tool call when the application can safely make the deterministic call itself. The purpose of the gate is to remove impossible configurations, not to manufacture a reason to pick K3.
Pre-register the decision before seeing outputs
Write the decision rules before the first model call. Otherwise it is easy to reward whichever answer looks better and change the criteria afterward. The pre-registration should define the workload, source set, expected output, hard failures, quality rubric, repetition count, cache condition, reasoning configuration, time and cost budgets, human-review rate, and final selection rule. Store it in the same repository as the prompts and result files.
- Define the unit of work. Example: one contract set, one repository issue, or one support-ticket batch—not “general intelligence.”
- Freeze the input. Preserve source checksums, prompt text, schema, tools, and expected answer keys.
- Define hard failures. Invalid schema, unsupported claim, missed mandatory evidence, unsafe tool request, or failed test should not be averaged away.
- Choose two configuration tracks. Compare quality-oriented settings separately from low-latency settings.
- Set repetition counts. Use at least five runs for an initial quality view and twenty or more comparable runs for latency percentiles.
- Set the selection rule. Choose on cost per accepted result plus reviewer correction time, subject to minimum pass rate and latency limits.
- Lock the document. Do not change weights or thresholds after reading model outputs; create a new experiment version instead.
Keep the full technical details on the owner pages: Kimi K3 Technical Dossier and Kimi K2.6 Cost–Latency–Capability Guide. This comparison needs only the differences that can change the routing decision.
Build a shared task set
A useful set contains tasks that are common enough for both models and boundary cases that expose structural limits. Do not use one prompt and declare a universal winner. Each task should have its own answer key or review rubric. Estimate exact tokens with the official tokenizer endpoint; the sizes below are planning bands, not promises about a specific file.
| Task | Planned budget | What it tests | Eligibility |
|---|---|---|---|
| T1 — Support-ticket classification | ≈10K uncached input + ≤1K output | Schema validity, instruction following, throughput, and low-cost routing | Both models |
| T2 — Evidence extraction from a stable manual | ≈50K cached input + ≤2K output | Prefix caching, traceable extraction, and repeated-query economics | Both models |
| T3 — Cross-file design review | ≈200K uncached input + ≤20K output | Long-context synthesis, contradiction handling, and reviewer correction time | Both models, near the K2.6 operating ceiling |
| T4 — Large evidence dossier | ≈700K uncached input + ≤20K output | Retrieval and synthesis beyond 256K | K3 only |
| T5 — Mandatory first-turn tool workflow | Task-dependent | Guaranteed tool use and tool-result continuity | K3 only on the direct API with required |
For T1 and T2, provide a machine-checkable JSON schema and an answer key. For T3 and T4, require source IDs for every material claim, include controlled contradictions, and add one unanswerable question to test abstention. For T5, use a harmless read-only tool in the benchmark, validate arguments, return a result for every tool-call ID, and log the complete assistant message. If your primary workload is code editing, replace or supplement these tasks with the K2.7 Code repository fixture.
Run two configuration tracks
K3 always reasons while K2.6 can disable thinking, so there is no single perfectly identical reasoning setting. Avoid hiding that asymmetry. Run two business-relevant tracks and label them clearly. The quality track asks what happens when both models receive a meaningful reasoning budget. The efficiency track asks for the lowest-cost configuration that can still pass the task.
| Control | Quality track | Efficiency track |
|---|---|---|
| K2.6 reasoning | thinking: enabled | thinking: disabled when the task supports it |
| K3 reasoning | reasoning_effort: high | reasoning_effort: low |
| Prompt and sources | Identical | Identical within the track |
| Output schema | Identical strict schema | Identical strict schema |
| Completion limit | Same task-specific ceiling | Same task-specific ceiling |
| Sampling | Do not pass unsupported values; models use documented fixed settings | Same |
| Cache condition | Cold and warm runs separated | Cold and warm runs separated |
| Quality repetitions | At least 5 per task/model | At least 5 per task/model |
| Latency sample | At least 20 comparable calls if p95 is used | At least 20 comparable calls if p95 is used |
Use clean sessions when changing model or reasoning configuration. K3 documentation notes that changing reasoning-effort levels can invalidate prefix-cache hits. Preserve complete assistant messages in multi-turn tests. Run both models through the same region, account conditions, client, retry policy, timeout, and concurrency level whenever possible. Record exceptions instead of pretending the environment was identical.
Price the experiment before running it
Current direct-API rates verified August 30, 2026 are: K3 at $0.30 per million cached input tokens, $3.00 per million uncached input tokens, and $15.00 per million output tokens; K2.6 at $0.16, $0.95, and $4.00 respectively. The table below converts the shared task bands into planning envelopes. It excludes taxes, paid tools, infrastructure, retries, and human review, and it uses actual output assumptions rather than the maximum possible output.
| Task envelope | Estimated K2.6 model cost | Estimated K3 model cost | Interpretation |
|---|---|---|---|
| 10K uncached input + 1K output | $0.0135 | $0.0450 | K3 must prevent enough failures or rework to justify roughly $0.0315 additional model cost per run |
| 50K cached input + 2K output | $0.0160 | $0.0450 | Caching narrows absolute cost, but K2.6 remains cheaper under the verified rates |
| 200K uncached input + 20K output | $0.2700 | $0.9000 | Reviewer time and acceptance rate become central at this scale |
| 700K uncached input + 20K output | Ineligible by context | $2.4000 | This is a structural K3 route, not an A/B case |
Compute two operational metrics after the run:
cost_per_accepted_result = total_model_cost / accepted_runs
adjusted_cost_per_accepted_result = (
total_model_cost
+ external_tool_fees
+ reviewer_minutes / 60 * loaded_reviewer_hourly_cost
) / accepted_runsThe adjusted figure reveals why a more expensive model can still be economical if it dramatically reduces correction, and why a cheap request can be costly when it fails often. Keep the authoritative full rate table and pricing disclaimers on Kimi API Pricing; this page includes only the values required for the comparison.
A reproducible A/B harness
The harness below runs one prompt repeatedly against both models, saves raw outputs, records usage and total elapsed time, and calculates model cost from the current rates. It does not score semantic quality automatically; attach a task-specific scorer or complete the blinded review sheet afterward. For interactive applications, add streaming instrumentation for time to first token. Store the API key only in the server environment.
import json
import os
import time
from pathlib import Path
from typing import Any
from openai import OpenAI
client = OpenAI(
api_key=os.environ["MOONSHOT_API_KEY"],
base_url="https://api.moonshot.ai/v1",
)
RATES = {
"kimi-k2.6": {"cached": 0.16, "uncached": 0.95, "output": 4.00},
"kimi-k3": {"cached": 0.30, "uncached": 3.00, "output": 15.00},
}
def estimate_cost(model: str, usage: Any) -> float:
prompt = int(getattr(usage, "prompt_tokens", 0) or 0)
cached = int(getattr(usage, "cached_tokens", 0) or 0)
output = int(getattr(usage, "completion_tokens", 0) or 0)
uncached = max(prompt - cached, 0)
rate = RATES[model]
return (
cached / 1_000_000 * rate["cached"]
+ uncached / 1_000_000 * rate["uncached"]
+ output / 1_000_000 * rate["output"]
)
def run_once(model: str, prompt: str, track: str, run_id: int) -> dict[str, Any]:
request: dict[str, Any] = {
"model": model,
"messages": [{"role": "user", "content": prompt}],
"max_completion_tokens": 4096,
"response_format": {"type": "json_object"},
}
if model == "kimi-k3":
request["reasoning_effort"] = "high" if track == "quality" else "low"
else:
request["extra_body"] = {
"thinking": {"type": "enabled" if track == "quality" else "disabled"}
}
started = time.perf_counter()
response = client.chat.completions.create(**request)
elapsed = time.perf_counter() - started
usage = response.usage
row = {
"task": Path("task.md").stem,
"track": track,
"model": model,
"run_id": run_id,
"elapsed_seconds": round(elapsed, 3),
"prompt_tokens": int(getattr(usage, "prompt_tokens", 0) or 0),
"cached_tokens": int(getattr(usage, "cached_tokens", 0) or 0),
"completion_tokens": int(getattr(usage, "completion_tokens", 0) or 0),
"estimated_model_cost_usd": round(estimate_cost(model, usage), 8),
"finish_reason": response.choices[0].finish_reason,
"output": response.choices[0].message.content,
# Complete these after blinded scoring:
"hard_gate_pass": None,
"quality_score": None,
"reviewer_minutes": None,
"review_notes": None,
}
return row
prompt = Path("task.md").read_text(encoding="utf-8")
rows: list[dict[str, Any]] = []
for track in ("quality", "efficiency"):
for model in ("kimi-k2.6", "kimi-k3"):
for run_id in range(1, 6):
rows.append(run_once(model, prompt, track, run_id))
Path("ab-results.json").write_text(json.dumps(rows, indent=2), encoding="utf-8")
print("Saved", len(rows), "unscored runs. Do not publish a winner before review.")For strict structured output, replace json_object with the same json_schema definition for both models. If the task uses tools, save every assistant tool-call message, tool arguments, tool result, and final response. Do not silently retry only one model; record every retry and include its cost. For latency percentiles, run a separate larger sample after the prompt and scoring process are stable.
Score accepted outcomes, not polished prose
Apply hard gates before assigning points. A response with an invented source, invalid schema, failed test, or unauthorized tool action is rejected even if its explanation is fluent. Then use a task-specific score. Blind reviewers to the model name where practical, randomize output order, and resolve disagreement with the pre-written answer key.
| Decision layer | Example measure | How it affects selection |
|---|---|---|
| Eligibility | Context and required parameter support | An ineligible model is not scored for that task |
| Hard acceptance | Schema valid, evidence supported, required fields complete, no unsafe action | Any hard failure rejects the run |
| Task quality | 0–100 rubric tied to the answer key or tests | Use pass rate and score distribution, not the best single answer |
| Latency | Time to first token, total time, p50, p95 | Must remain under the product’s user-experience limit |
| Cost | Actual model and tool cost | Calculate cost per accepted result |
| Human correction | Minutes to make the result publishable or mergeable | Convert to loaded labor cost or report separately |
| Reliability | Retries, 429/5xx events, truncation, malformed output | Include in the operational failure rate |
An example selection rule is: the candidate must achieve at least a 90% hard-gate pass rate, remain below the task’s p95 latency ceiling, and meet the minimum quality score. Among candidates that pass, choose the lowest adjusted cost per accepted result. If K3 improves pass rate by a meaningful pre-set margin or sharply reduces reviewer time, route that task class to K3. If outcomes are equivalent within the pre-set tolerance, keep K2.6. Change these example thresholds to match your risk level before testing—not afterward.
Interpret results by task class
Do not collapse all tasks into one winner. A router can send T1 classification to K2.6, T3 design review to K3, T4 large dossiers to K3 by eligibility, and repository repair to K2.7 Code. Report the model, configuration, and date with every result because Kimi’s platform, serving routes, and prices can change. Keep vendor benchmark data separate from your field measurements.
| Observed pattern | Recommended interpretation |
|---|---|
| Both pass; K2.6 has lower adjusted cost and acceptable latency | Route that task class to K2.6 |
| K3 has materially higher pass rate or much lower correction time | Route that task class to K3 and document the value of the uplift |
| K2.6 non-thinking fails but K2.6 thinking passes | Keep K2.6 with a mode router rather than escalating models automatically |
| K2.6 hits context or required-tool constraint | Route to K3 because of structural eligibility |
| Both fail the same evidence requirement | Improve retrieval, prompt, schema, tools, or source quality before changing models |
| Coding task remains unreliable on both | Evaluate K2.7 Code with the repository field-test protocol |
| Results vary widely between repetitions | Increase the sample, inspect prompt ambiguity, and avoid a production decision |
Publish median and distribution information, not only averages. Include failures, sample size, complete task definition, and whether the prefix was cached. A statement such as “K3 was better” without the acceptance gate, configuration, model date, and workload is not reusable evidence and will quickly become stale.
Decision tree for production routing
The final router should be simple enough to audit. Start with deterministic conditions, then use the experiment result for the ambiguous middle. Do not let a user-facing model name substitute for a policy.
- Estimate total tokens. If the relevant request plus completion budget exceeds 262,144 and remains within K3’s limit, route to K3.
- Check tool enforcement. If the direct API must guarantee at least one tool call, route to K3 or move the deterministic action into application code.
- Check reasoning policy. If thinking must be disabled, route to K2.6.
- Check workload family. If it is repository-centered engineering, evaluate K2.7 Code rather than limiting the router to two general candidates.
- Look up the task-class experiment. Route to the configuration with the lowest adjusted cost per accepted result that meets pass-rate and latency gates.
- Use K2.6 as the bounded-work fallback. If no experiment exists and no hard K3 gate applies, start with K2.6 under strict validation.
- Log and review. Store the route reason, model, mode, tokens, cost, latency, acceptance result, and human correction so the policy can be re-evaluated.
Do not retry a random failed K2.6 request on K3 without recording that escalation. An unobserved fallback hides the true cost and can create inconsistent behavior. Escalate only for documented failure classes, cap the number of attempts, and surface a safe failure when neither route passes.
Roll out the winner without locking the site to it
Even a controlled experiment is a snapshot. Release the routing policy behind a feature flag, preserve a rollback path, and monitor acceptance rate, cost, latency, errors, tool failures, and reviewer corrections by task class. Re-run the experiment when model parameters, prices, context limits, serving routes, application prompts, tools, or source data materially change. A quarterly review is useful for stable workloads, but a documented platform change should trigger an earlier review.
- Shadow test: send a small sample to the non-serving candidate without using its answer, subject to privacy and cost rules.
- Canary route: expose a small percentage of eligible traffic and compare accepted outcomes.
- Budget alarm: alert on cost per accepted result, not only total spend.
- Quality alarm: monitor unsupported claims, schema failures, failed tests, and correction minutes.
- Version log: record prompt, tool, model, parameter, and scoring changes in the experiment repository.
- Rollback: return the task class to the previous route when hard failures or operational limits are exceeded.
The goal is not to crown a permanent winner. It is to keep each task on the least expensive configuration that reliably meets its requirements while preserving an evidence trail. That is how the comparison page remains useful without duplicating the two model profiles or becoming obsolete after the next release.
Frequently asked questions
Which is better, Kimi K3 or K2.6?
Neither is universally better. K3 has structural advantages for context beyond 256K and required tool choice. K2.6 supports non-thinking operation and has lower current direct-API unit rates. When both are eligible, select by controlled pass rate, latency, cost per accepted result, and reviewer correction time.
Is K3 always more accurate?
Do not assume so for every task. K3 is the flagship and is designed for harder long-horizon work, but a bounded extraction or classification task may produce equivalent accepted outcomes on K2.6. Test the actual workload with a fixed rubric.
Is K2.6 always faster?
No universal latency claim is reliable. K2.6 can disable thinking, which may benefit suitable tasks, but actual timing depends on input, output, cache, account, region, load, concurrency, and retries. Measure the production configuration.
How should cached tokens be handled in the comparison?
Separate cold and warm runs. Record cached_tokens, keep the prefix stable, and calculate cost with the cached rate. Do not combine cache hits and misses in one unexplained average.
Why is K2.7 Code mentioned in a K3 vs K2.6 comparison?
Because a repository-centered coding task may be misrouted if the candidate set includes only two general choices. The comparison page links out rather than reproducing the K2.7 technical and field-test content.
How often should the experiment be repeated?
Repeat after a material model, price, parameter, prompt, tool, or workload change, and periodically for important stable routes. Record the date and configuration so readers do not treat an old result as permanent.
Main sources used
Technical claims in this guide were checked against the following primary sources on August 30, 2026. Model behavior, availability, pricing, limits, and documentation can change; verify the official pages before a production deployment or purchase.
- Kimi API model list — current model roles and context windows
- Kimi model parameter reference — thinking, reasoning effort, tool choice, and fixed-parameter differences
- Kimi API pricing — current direct-API cached input, uncached input, and output rates
- Kimi K3 API guide — K3 context, reasoning, tool use, message preservation, and limits
- Kimi K2.6 API guide — K2.6 multimodal use, context, thinking modes, and tool behavior
- Kimi API benchmark best practices — reproducibility, repeated runs, streaming, retries, and context management
