How to Handle Invalid JSON From LLMs: A 10,000-Case TypeScript Test

Invalid JSON from LLMs is not always a parsing problem. An LLM returns a JSON object. JSON.parse() succeeds. The integration still breaks because confidence is 1.7, an enum contains an invented action, or a required field is missing.

That is the trap: valid JSON is not necessarily valid data.

I built a small TypeScript benchmark to test four ways of handling invalid JSON from LLMs. The experiment replayed the same 10,000 seeded payloads through a raw parser, an object extractor, a conservative syntax repairer, and a schema gate with one retry. The schema-gated path accepted 99.3% as valid, accepted no schema-invalid records, and used 1.15 model calls per task in the controlled test.

The more interesting result was lower down the table. Syntax repair recovered far more payloads than JSON.parse() alone, but it accepted the same 7% of semantically invalid objects. Repair made the strings parseable. It did not make the values correct.

Editorial note: I designed the experiment, corruption mix, code, and conclusions. AI tools assisted with code scaffolding and editorial refinement. The benchmark is a deterministic simulation of output boundaries, not a comparison of model vendors.

Why invalid JSON from LLMs fails in different ways

“Invalid JSON” usually bundles together at least three separate problems:

1. Packaging errors: the object is wrapped in prose or a Markdown code fence.
2. Syntax errors: trailing commas, smart quotes, bare keys, or truncation prevent parsing.
3. Contract errors: the JSON parses, but a field has the wrong type, range, enum value, or business meaning.

A parser detects the second category. A little extraction logic can handle some of the first. Neither proves that the object satisfies the application contract.

JSON Schema exists for that missing job. The official JSON Schema documentation describes it as a declarative way to validate document structure, constraints, and types. It also notes that sufficiently complex formats may require both structural validation and a second semantic phase in application code.

OpenAI’s API documentation recommends JSON Schema response formats on models that support them and exposes strict schema adherence. That should be the first choice when the provider and model support it. Applications still need a boundary check because refusals, truncation, tool results, older models, proxy layers, and non-OpenAI providers can produce paths outside the ideal response.

The 10,000-case experiment

The test generated 10,000 task objects with this intended contract:

const PayloadSchema = z.object({  taskId: z.string().regex(/^task-\d+$/),  confidence: z.number().min(0).max(1),  actions: z    .array(z.enum(["read", "write", "review"]))    .min(1)    .max(3), }).strict();

Each valid source object was converted into one of seven output conditions:

The mix is declared rather than presented as a universal production distribution. Its purpose is to exercise distinct recovery branches under repeatable conditions. A seeded random generator creates the same trial set on every run.

I then applied four strategies:

• JSON.parse() only: parse the entire response and accept anything that parses.
• Extract object: keep the substring between the first { and last }, then parse.
• Repair syntax: extract, normalize smart punctuation, quote simple bare keys, remove trailing commas, then parse.
• Schema plus retry: apply conservative repair, validate with Zod, and make one bounded retry if parsing or validation fails.

Results: parse rate can hide bad data

Extraction recovered the fenced and prose-wrapped objects, lifting valid acceptance by 15.8 percentage points. Syntax repair added another 28.5 points by handling the deliberately limited punctuation and key-format errors.

Neither strategy changed the invalid-acceptance rate. All parseable objects with a confidence value of 1.7 still passed through. If the monitoring dashboard reported only “JSON parsed successfully,” the repair strategy would look excellent while quietly preserving every semantic error in the test.

The schema gate changed the decision boundary. It rejected parseable but invalid values and retried only the 14.5% of first attempts that could not be safely accepted. That is why average model calls rose from 1.00 to only 1.15 rather than doubling.

The remaining 0.7% failed after the retry. A real application should route those cases to a typed error or human fallback. It should not invent missing data to make the schema pass.

A conservative recovery pipeline

The safest implementation separates repair from validation.

The repair function in the benchmark is intentionally small:

function repair(input: string): string {  return extract(input)    .replaceAll("“", '"')    .replaceAll("”", '"')    .replaceAll("：", ":")    .replace(/([{,])\s*([A-Za-z]+)\s*:/g, '$1"$2":')    .replace(/,\s*([}\]])/g, "$1"); }

This code fixes representation problems that have a reasonably unambiguous interpretation. It does not close a truncated object, create required fields, coerce arbitrary strings into numbers, or substitute enum values. Those operations would cross from repairing syntax into guessing intent.

The boundary handler then parses and validates:

type ParseResult<T> =  | { ok: true; value: T }  | { ok: false; reason: "syntax" | "schema"; issues?: unknown }; function parsePayload(input: string): ParseResult<Payload> {  let candidate: unknown;  try {    candidate = JSON.parse(repair(input));  } catch {    return { ok: false, reason: "syntax" };  }  const checked = PayloadSchema.safeParse(candidate);  return checked.success    ? { ok: true, value: checked.data }    : { ok: false, reason: "schema", issues: checked.error.issues }; }

That discriminated union matters. Downstream code cannot accidentally treat a failed parse as a valid payload without handling the ok: false branch.

Make retries useful, not blind

A retry should include the failure reason and the target contract. Repeating the original prompt unchanged may reproduce the same output.

For a syntax failure, the correction can say that the previous response was not parseable and request one JSON object with no surrounding prose. For a schema failure, provide compact field-level feedback such as confidence must be <= 1 or actions[0] must be one of read, write, review.

Keep the retry budget small. If the second attempt still fails, return an explicit application error or escalate. Infinite repair loops increase latency and make failures harder to diagnose.

Also record these metrics separately:

• first-attempt parse success
• first-attempt schema success
• failures by syntax category
• schema failures by field and constraint
• retry success rate
• invalid outputs blocked before side effects
• latency and token cost added by recovery

This is more actionable than one combined “structured output success” number.

Production rules I would keep

Use provider-native structured outputs when available, but validate again at your trust boundary. Keep schemas strict enough to reject unexpected fields. Add business invariants after structural validation when relationships between values matter.

Treat extraction and repair as compatibility layers, not proof. Preserve the original response in a trace so a repaired value can be audited. Never log sensitive payloads without redaction and retention controls.

Most importantly, validate before a side effect. A malformed research summary is inconvenient. A parseable but invalid payment, permission, deletion, or publishing command can be costly.

Conclusion

The benchmark’s syntax repairer looked strong if measured by parse rate: it recovered 85.5% of valid objects. Yet it accepted all 7% of the deliberately schema-invalid records. Parsing answered only whether the string looked like JSON.

The schema-gated workflow answered the question the application actually cared about: whether the value satisfied the contract. One bounded retry recovered most rejected cases, reaching 99.3% valid acceptance with 1.15 average model calls and no invalid acceptances in this controlled run.

The practical pattern is straightforward: request structured output, repair only unambiguous formatting defects, parse, validate, check business invariants, and retry within a budget. If that still fails, stop safely.

References

• OpenAI API, Structured output response formats
• JSON Schema, What is JSON Schema?
• JSON Schema, Creating your first schema
• JSON Schema, Structural and semantic validation
• Zod, DocumentationAn LLM returns a JSON object. JSON.parse() succeeds. The integration still breaks because confidence is 1.7, an enum contains an invented action, or a required field is missing.

That is the trap: valid JSON is not necessarily valid data.

I built a small TypeScript benchmark to test four ways of handling invalid JSON from LLMs. The experiment replayed the same 10,000 seeded payloads through a raw parser, an object extractor, a conservative syntax repairer, and a schema gate with one retry. The schema-gated path accepted 99.3% as valid, accepted no schema-invalid records, and used 1.15 model calls per task in the controlled test.

The more interesting result was lower down the table. Syntax repair recovered far more payloads than JSON.parse() alone, but it accepted the same 7% of semantically invalid objects. Repair made the strings parseable. It did not make the values correct.

Editorial note: I designed the experiment, corruption mix, code, and conclusions. AI tools assisted with code scaffolding and editorial refinement. The benchmark is a deterministic simulation of output boundaries, not a comparison of model vendors.

Why LLM JSON fails in different ways

“Invalid JSON” usually bundles together at least three separate problems:

1. Packaging errors: the object is wrapped in prose or a Markdown code fence.
2. Syntax errors: trailing commas, smart quotes, bare keys, or truncation prevent parsing.
3. Contract errors: the JSON parses, but a field has the wrong type, range, enum value, or business meaning.

A parser detects the second category. A little extraction logic can handle some of the first. Neither proves that the object satisfies the application contract.

JSON Schema exists for that missing job. The official JSON Schema documentation describes it as a declarative way to validate document structure, constraints, and types. It also notes that sufficiently complex formats may require both structural validation and a second semantic phase in application code.

OpenAI’s API documentation recommends JSON Schema response formats on models that support them and exposes strict schema adherence. That should be the first choice when the provider and model support it. Applications still need a boundary check because refusals, truncation, tool results, older models, proxy layers, and non-OpenAI providers can produce paths outside the ideal response.

The 10,000-case experiment

The test generated 10,000 task objects with this intended contract:

const PayloadSchema = z.object({  taskId: z.string().regex(/^task-\d+$/),  confidence: z.number().min(0).max(1),  actions: z    .array(z.enum(["read", "write", "review"]))    .min(1)    .max(3), }).strict();

Each valid source object was converted into one of seven output conditions:

The mix is declared rather than presented as a universal production distribution. Its purpose is to exercise distinct recovery branches under repeatable conditions. A seeded random generator creates the same trial set on every run.

I then applied four strategies:

• JSON.parse() only: parse the entire response and accept anything that parses.
• Extract object: keep the substring between the first { and last }, then parse.
• Repair syntax: extract, normalize smart punctuation, quote simple bare keys, remove trailing commas, then parse.
• Schema plus retry: apply conservative repair, validate with Zod, and make one bounded retry if parsing or validation fails.

Results: parse rate can hide bad data

Extraction recovered the fenced and prose-wrapped objects, lifting valid acceptance by 15.8 percentage points. Syntax repair added another 28.5 points by handling the deliberately limited punctuation and key-format errors.

Neither strategy changed the invalid-acceptan

An LLM returns a JSON object. JSON.parse() succeeds. The integration still breaks because confidence is 1.7, an enum contains an invented action, or a required field is missing.

That is the trap: valid JSON is not necessarily valid data.

I built a small TypeScript benchmark to test four ways of handling invalid JSON from LLMs. The experiment replayed the same 10,000 seeded payloads through a raw parser, an object extractor, a conservative syntax repairer, and a schema gate with one retry. The schema-gated path accepted 99.3% as valid, accepted no schema-invalid records, and used 1.15 model calls per task in the controlled test.

The more interesting result was lower down the table. Syntax repair recovered far more payloads than JSON.parse() alone, but it accepted the same 7% of semantically invalid objects. Repair made the strings parseable. It did not make the values correct.

Editorial note: I designed the experiment, corruption mix, code, and conclusions. AI tools assisted with code scaffolding and editorial refinement. The benchmark is a deterministic simulation of output boundaries, not a comparison of model vendors.

Why LLM JSON fails in different ways

“Invalid JSON” usually bundles together at least three separate problems:

1. Packaging errors: the object is wrapped in prose or a Markdown code fence.
2. Syntax errors: trailing commas, smart quotes, bare keys, or truncation prevent parsing.
3. Contract errors: the JSON parses, but a field has the wrong type, range, enum value, or business meaning.

A parser detects the second category. A little extraction logic can handle some of the first. Neither proves that the object satisfies the application contract.

JSON Schema exists for that missing job. The official JSON Schema documentation describes it as a declarative way to validate document structure, constraints, and types. It also notes that sufficiently complex formats may require both structural validation and a second semantic phase in application code.

OpenAI’s API documentation recommends JSON Schema response formats on models that support them and exposes strict schema adherence. That should be the first choice when the provider and model support it. Applications still need a boundary check because refusals, truncation, tool results, older models, proxy layers, and non-OpenAI providers can produce paths outside the ideal response.

The 10,000-case experiment

The test generated 10,000 task objects with this intended contract:

const PayloadSchema = z.object({  taskId: z.string().regex(/^task-\d+$/),  confidence: z.number().min(0).max(1),  actions: z    .array(z.enum(["read", "write", "review"]))    .min(1)    .max(3), }).strict();

Each valid source object was converted into one of seven output conditions:

The mix is declared rather than presented as a universal production distribution. Its purpose is to exercise distinct recovery branches under repeatable conditions. A seeded random generator creates the same trial set on every run.

I then applied four strategies:

• JSON.parse() only: parse the entire response and accept anything that parses.
• Extract object: keep the substring between the first { and last }, then parse.
• Repair syntax: extract, normalize smart punctuation, quote simple bare keys, remove trailing commas, then parse.
• Schema plus retry: apply conservative repair, validate with Zod, and make one bounded retry if parsing or validation fails.

Results: parse rate can hide bad data

Extraction recovered the fenced and prose-wrapped objects, lifting valid acceptance by 15.8 percentage points. Syntax repair added another 28.5 points by handling the deliberately limited punctuation and key-format errors.

Neither strategy changed the invalid-acceptan

ce rate. All parseable objects with a confidence value of 1.7 still passed through. If the monitoring dashboard reported only “JSON parsed successfully,” the repair strategy would look excellent while quietly preserving every semantic error in the test.

The schema gate changed the decision boundary. It rejected parseable but invalid values and retried only the 14.5% of first attempts that could not be safely accepted. That is why average model calls rose from 1.00 to only 1.15 rather than doubling.

The remaining 0.7% failed after the retry. A real application should route those cases to a typed error or human fallback. It should not invent missing data to make the schema pass.

A conservative recovery pipeline

The safest implementation separates repair from validation.

The repair function in the benchmark is intentionally small:

function repair(input: string): string {  return extract(input)    .replaceAll("“", '"')    .replaceAll("”", '"')    .replaceAll("：", ":")    .replace(/([{,])\s*([A-Za-z]+)\s*:/g, '$1"$2":')    .replace(/,\s*([}\]])/g, "$1"); }

This code fixes representation problems that have a reasonably unambiguous interpretation. It does not close a truncated object, create required fields, coerce arbitrary strings into numbers, or substitute enum values. Those operations would cross from repairing syntax into guessing intent.

The boundary handler then parses and validates:

type ParseResult<T> =  | { ok: true; value: T }  | { ok: false; reason: "syntax" | "schema"; issues?: unknown }; function parsePayload(input: string): ParseResult<Payload> {  let candidate: unknown;  try {    candidate = JSON.parse(repair(input));  } catch {    return { ok: false, reason: "syntax" };  }  const checked = PayloadSchema.safeParse(candidate);  return checked.success    ? { ok: true, value: checked.data }    : { ok: false, reason: "schema", issues: checked.error.issues }; }

That discriminated union matters. Downstream code cannot accidentally treat a failed parse as a valid payload without handling the ok: false branch.

Make retries useful, not blind

A retry should include the failure reason and the target contract. Repeating the original prompt unchanged may reproduce the same output.

For a syntax failure, the correction can say that the previous response was not parseable and request one JSON object with no surrounding prose. For a schema failure, provide compact field-level feedback such as confidence must be <= 1 or actions[0] must be one of read, write, review.

Keep the retry budget small. If the second attempt still fails, return an explicit application error or escalate. Infinite repair loops increase latency and make failures harder to diagnose.

Also record these metrics separately:

• first-attempt parse success
• first-attempt schema success
• failures by syntax category
• schema failures by field and constraint
• retry success rate
• invalid outputs blocked before side effects
• latency and token cost added by recovery

This is more actionable than one combined “structured output success” number.

Production rules I would keep

Use provider-native structured outputs when available, but validate again at your trust boundary. Keep schemas strict enough to reject unexpected fields. Add business invariants after structural validation when relationships between values matter.

Treat extraction and repair as compatibility layers, not proof. Preserve the original response in a trace so a repaired value can be audited. Never log sensitive payloads without redaction and retention controls.

Most importantly, validate before a side effect. A malformed research summary is inconvenient. A parseable but invalid payment, permission, deletion, or publishing command can be costly.

Conclusion

The benchmark’s syntax repairer looked strong if measured by parse rate: it recovered 85.5% of valid objects. Yet it accepted all 7% of the deliberately schema-invalid records. Parsing answered only whether the string looked like JSON.

The schema-gated workflow answered the question the application actually cared about: whether the value satisfied the contract. One bounded retry recovered most rejected cases, reaching 99.3% valid acceptance with 1.15 average model calls and no invalid acceptances in this controlled run.

The practical pattern is straightforward: request structured output, repair only unambiguous formatting defects, parse, validate, check business invariants, and retry within a budget. If that still fails, stop safely.

References

• OpenAI API, Structured output response formats
• JSON Schema, What is JSON Schema?
• JSON Schema, Creating your first schema
• JSON Schema, Structural and semantic validation
• Zod, Documentationce rate. All parseable objects with a confidence value of 1.7 still passed through. If the monitoring dashboard reported only “JSON parsed successfully,” the repair strategy would look excellent while quietly preserving every semantic error in the test.

The schema gate changed the decision boundary. It rejected parseable but invalid values and retried only the 14.5% of first attempts that could not be safely accepted. That is why average model calls rose from 1.00 to only 1.15 rather than doubling.

The remaining 0.7% failed after the retry. A real application should route those cases to a typed error or human fallback. It should not invent missing data to make the schema pass.

A conservative recovery pipeline

The safest implementation separates repair from validation.

The repair function in the benchmark is intentionally small:

function repair(input: string): string {  return extract(input)    .replaceAll("“", '"')    .replaceAll("”", '"')    .replaceAll("：", ":")    .replace(/([{,])\s*([A-Za-z]+)\s*:/g, '$1"$2":')    .replace(/,\s*([}\]])/g, "$1"); }

This code fixes representation problems that have a reasonably unambiguous interpretation. It does not close a truncated object, create required fields, coerce arbitrary strings into numbers, or substitute enum values. Those operations would cross from repairing syntax into guessing intent.

The boundary handler then parses and validates:

type ParseResult<T> =  | { ok: true; value: T }  | { ok: false; reason: "syntax" | "schema"; issues?: unknown }; function parsePayload(input: string): ParseResult<Payload> {  let candidate: unknown;  try {    candidate = JSON.parse(repair(input));  } catch {    return { ok: false, reason: "syntax" };  }  const checked = PayloadSchema.safeParse(candidate);  return checked.success    ? { ok: true, value: checked.data }    : { ok: false, reason: "schema", issues: checked.error.issues }; }

That discriminated union matters. Downstream code cannot accidentally treat a failed parse as a valid payload without handling the ok: false branch.

Make retries useful, not blind

A retry should include the failure reason and the target contract. Repeating the original prompt unchanged may reproduce the same output.

For a syntax failure, the correction can say that the previous response was not parseable and request one JSON object with no surrounding prose. For a schema failure, provide compact field-level feedback such as confidence must be <= 1 or actions[0] must be one of read, write, review.

Keep the retry budget small. If the second attempt still fails, return an explicit application error or escalate. Infinite repair loops increase latency and make failures harder to diagnose.

Also record these metrics separately:

• first-attempt parse success
• first-attempt schema success
• failures by syntax category
• schema failures by field and constraint
• retry success rate
• invalid outputs blocked before side effects
• latency and token cost added by recovery

This is more actionable than one combined “structured output success” number.

Production rules I would keep

Use provider-native structured outputs when available, but validate again at your trust boundary. Keep schemas strict enough to reject unexpected fields. Add business invariants after structural validation when relationships between values matter.

Treat extraction and repair as compatibility layers, not proof. Preserve the original response in a trace so a repaired value can be audited. Never log sensitive payloads without redaction and retention controls.

Most importantly, validate before a side effect. A malformed research summary is inconvenient. A parseable but invalid payment, permission, deletion, or publishing command can be costly.

Conclusion

The benchmark’s syntax repairer looked strong if measured by parse rate: it recovered 85.5% of valid objects. Yet it accepted all 7% of the deliberately schema-invalid records. Parsing answered only whether the string looked like JSON.

The schema-gated workflow answered the question the application actually cared about: whether the value satisfied the contract. One bounded retry recovered most rejected cases, reaching 99.3% valid acceptance with 1.15 average model calls and no invalid acceptances in this controlled run.

The practical pattern is straightforward: request structured output, repair only unambiguous formatting defects, parse, validate, check business invariants, and retry within a budget. If that still fails, stop safely.

References

• OpenAI API, Structured output response formats
• JSON Schema, What is JSON Schema?
• JSON Schema, Creating your first schema
• JSON Schema, Structural and semantic validation
• Zod, Documentation