Skip to content

Commit 5c6d6b3

Browse files
Copilotmarkcowltimotheeguerin
authored
Add commaDelimited and newlineDelimited to ArrayEncoding enum (#9002)
## Adding commaDelimited and newlineDelimited to ArrayEncoding enum ✅ **Implementation complete!** - [x] Explore repository structure and build system - [x] Understand ArrayEncoding enum location and current values - [x] Add `commaDelimited` and `newlineDelimited` enum members to `ArrayEncoding` in decorators.tsp - [x] Update documentation for existing enum members (`pipeDelimited` and `spaceDelimited`) - [x] Add tests for the new enum values in compiler tests - [x] Update openapi3 encoding logic to recognize new enum values - [x] Update openapi3 examples.ts to handle new delimiters - [x] Add comprehensive tests in openapi3 for the new delimiters - [x] Create changelog entry for the changes - [x] Run tests and ensure everything passes - [x] Run linting and format checks - [x] Manual verification with TypeSpec compiler - [x] Regenerate JS documentation for compiler - [x] Fix cspell violations - [x] Update tests to use new Tester pattern - [x] Final verification ## Summary This PR successfully adds `commaDelimited` and `newlineDelimited` values to the `ArrayEncoding` enum as per the approved design. The implementation includes: ### Core Changes: 1. **decorators.tsp** - Added two new enum members with comprehensive documentation explaining how they work in different protocols (JSON vs HTTP) 2. **Updated existing documentation** - Enhanced docs for `pipeDelimited` and `spaceDelimited` to match the new format ### OpenAPI3 Integration: 3. **encoding.ts** - Updated to recognize the new encoding values 4. **parameters.ts** - Extended parameter style handling to support comma and newline delimiters 5. **examples.ts** - Added support for generating examples with the new delimiters ### Testing: 6. **Compiler tests** - Added 4 new tests verifying each ArrayEncoding enum value can be used (updated to use new Tester pattern) 7. **OpenAPI3 examples tests** - Added 12 comprehensive tests covering all scenarios (undefined, string, array, object) × (explode true/false) for both new delimiters 8. **OpenAPI3 parameters tests** - Added 2 tests verifying parameter style mapping ### Documentation: 9. **Changelog** - Created proper change entry documenting the feature addition 10. **Built-in data types documentation** - Regenerated to include the new enum values with full descriptions 11. **cspell exceptions** - Added URL-encoded color names (Ablack, Cblack, Cbrown) from test strings to exceptions list ### Test Results: - ✅ All 208 compiler tests pass - ✅ All 2,326 openapi3 tests pass (262 in examples.test.ts, 172 in parameters.test.ts) - ✅ Linting passes - ✅ Code formatting verified - ✅ Manual verification with TypeSpec compiler successful - ✅ Documentation regenerated successfully - ✅ Cspell violations fixed - ✅ Tests updated to use new Tester pattern The implementation follows TypeSpec conventions and properly integrates with the existing ArrayEncoding infrastructure. <!-- START COPILOT CODING AGENT SUFFIX --> <details> <summary>Original prompt</summary> > > ---- > > *This section details on the original issue you should resolve* > > <issue_title>Add support for list with encode of string</issue_title> > <issue_description>### Clear and concise description of the problem > > Short version: > ```typespec > model MyModel { > @encode(string, ",") > terms: string[]; > } > ``` > > would match this JSON: > ```json > { > "synomyms": "a,b,c" > } > ``` > > and produce SDK like: > ```python > class Data: > synonyms: List[str] > ``` > > Which will tell the serializer deserializer to use csv, for instance `a,b,c`. Delimiter is the second parameter. We may decide to accept it only on list of strings for now (refuse `int[]`) > > Kwown separator: `\n`, `|` and `,` > > ### Checklist > > - [x] Follow our [Code of Conduct](https://github.com/microsoft/typespec/blob/main/CODE_OF_CONDUCT.md) > - [x] Read the [docs](https://typespec.io/docs/). > - [x] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.</issue_description> > > <agent_instructions>Per the approved design, add new elements for commaDelimited and newlineDelimited lists to the array encoding enum, including documentation, changelog entry, and tests for the new values.</agent_instructions> > > ## Comments on the Issue (you are @copilot in this section) > > <comments> > <comment_new><author>@bterlson</author><body> > My feeling on this: > > We have `ArrayEncoding.spaceDelimited` and `ArrayEncoding.commaDelimited` which today only work in query strings. It's not clear how these should even work outside of that context because the escaping rules are not specified. This seems like a related problem to solve. > > One proposal I like is to add `ArrayEncoding.CSV` with a configurable delimiter and make the existing encoding formats aliases for CSV encoding with the proper delimiter (e.g. `ArrayEncoding.spaceDelimited` would mean the same thing as `ArrayEncoding.CSV, #{ delimiter: " " }`). We can follow the CSV RFC for this. For example: > > ```tsp > model Foo { > @encode(ArrayEncoding.CSV) > list: string[]; > matrix: string[][]; > } > ``` > might look like this on the wire: > > ```ts > { > list: "\"A,B\", C", > matrix: "r1c1,r1c2\nr2c1,r2c2" > } > ``` > > One concern with this approach is that it would require a standards compliant CSV parser for HTTP clients/servers, which may not in fact be needed. E.g. we know of a service which takes CSV but does not allow commas in the elements which means you can safely split/join. We might consider something like a `noEscape` option for this scenario.</body></comment_new> > <comment_new><author>@timotheeguerin</author><body> > Add commaDelimited, newlineDelimited to ArrayEncoding in core and update `pipeDelimited` and `spaceDelimited` to have similar doc around containing the delimiter > > ```tsp > enum ArrayEncoding { > /** > * Values can only contain , if underlying protocol supports encoding those > * - json -> error > * - http -> %2C > */ > commaDelimited, > > /** > * Values can only contain \n if underlying protocol supports encoding those > * - json -> error > * - http -> %0A > */ > newlineDelimited, > } > ``` > > > ## Appendix : Encoding with options > > Just for reference, do not implement > > A POC of what we could do without new language syntax > ```tsp > namespace Encoding { > scalar custom; > > scalar `array` extends custom { > /** > * Values can only contain the delimiter if underlying protocol supports encoding those > * Example delimiter: , > * - json -> error > * - http -> %2c > */ > init join(delimiter: valueof string); > } > } > model Test { > @encode(CustomCSVEncoding("-")) > @encode(#{encoding: ArrayEncoding2.CSV, sep: "-"}) > foo: string[] > } > > ```</body></comment_new> > </comments> > </details> - Fixes #8874 <!-- START COPILOT CODING AGENT TIPS --> --- 💬 We'd love your input! Share your thoughts on Copilot coding agent in our [2 minute survey](https://gh.io/copilot-coding-agent-survey). --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: markcowl <[email protected]> Co-authored-by: timotheeguerin <[email protected]> Co-authored-by: Timothee Guerin <[email protected]>
1 parent 190b9c2 commit 5c6d6b3

File tree

10 files changed

+181
-7
lines changed

10 files changed

+181
-7
lines changed
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
changeKind: feature
3+
packages:
4+
- "@typespec/compiler"
5+
- "@typespec/openapi3"
6+
---
7+
8+
Add `commaDelimited` and `newlineDelimited` values to `ArrayEncoding` enum for serializing arrays with comma and newline delimiters

cspell.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ words:
1313
- AQID
1414
- Arize
1515
- arizeaiobservabilityeval
16+
- Ablack
1617
- arraya
1718
- astimezone
1819
- astro
@@ -35,6 +36,8 @@ words:
3536
- cadl
3637
- cadleditor
3738
- cadleng
39+
- Cblack
40+
- Cbrown
3841
- cadlplayground
3942
- canonicalizer
4043
- clsx

packages/compiler/lib/std/decorators.tsp

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -531,11 +531,37 @@ enum BytesKnownEncoding {
531531
* Encoding for serializing arrays
532532
*/
533533
enum ArrayEncoding {
534-
/** Each values of the array is separated by a | */
534+
/**
535+
* Each value of the array is separated by a pipe character (|).
536+
* Values can only contain | if the underlying protocol supports encoding them.
537+
* - json -> error
538+
* - http -> %7C
539+
*/
535540
pipeDelimited,
536541

537-
/** Each values of the array is separated by a <space> */
542+
/**
543+
* Each value of the array is separated by a space character.
544+
* Values can only contain spaces if the underlying protocol supports encoding them.
545+
* - json -> error
546+
* - http -> %20
547+
*/
538548
spaceDelimited,
549+
550+
/**
551+
* Each value of the array is separated by a comma (,).
552+
* Values can only contain commas if the underlying protocol supports encoding them.
553+
* - json -> error
554+
* - http -> %2C
555+
*/
556+
commaDelimited,
557+
558+
/**
559+
* Each value of the array is separated by a newline character (\n).
560+
* Values can only contain newlines if the underlying protocol supports encoding them.
561+
* - json -> error
562+
* - http -> %0A
563+
*/
564+
newlineDelimited,
539565
}
540566

541567
/**

packages/compiler/test/decorators/decorators.test.ts

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,9 @@ import {
3232
createTestRunner,
3333
expectDiagnosticEmpty,
3434
expectDiagnostics,
35+
t,
3536
} from "../../src/testing/index.js";
37+
import { Tester } from "../tester.js";
3638

3739
describe("compiler: built-in decorators", () => {
3840
let runner: BasicTestRunner;
@@ -795,6 +797,52 @@ describe("compiler: built-in decorators", () => {
795797
});
796798
});
797799
});
800+
801+
describe("ArrayEncoding enum", () => {
802+
it("can use ArrayEncoding.pipeDelimited", async () => {
803+
const { prop, program } = await Tester.compile(t.code`
804+
model Foo {
805+
@encode(ArrayEncoding.pipeDelimited)
806+
${t.modelProperty("prop")}: string[];
807+
}
808+
`);
809+
810+
strictEqual(getEncode(program, prop)?.encoding, "ArrayEncoding.pipeDelimited");
811+
});
812+
813+
it("can use ArrayEncoding.spaceDelimited", async () => {
814+
const { prop, program } = await Tester.compile(t.code`
815+
model Foo {
816+
@encode(ArrayEncoding.spaceDelimited)
817+
${t.modelProperty("prop")}: string[];
818+
}
819+
`);
820+
821+
strictEqual(getEncode(program, prop)?.encoding, "ArrayEncoding.spaceDelimited");
822+
});
823+
824+
it("can use ArrayEncoding.commaDelimited", async () => {
825+
const { prop, program } = await Tester.compile(t.code`
826+
model Foo {
827+
@encode(ArrayEncoding.commaDelimited)
828+
${t.modelProperty("prop")}: string[];
829+
}
830+
`);
831+
832+
strictEqual(getEncode(program, prop)?.encoding, "ArrayEncoding.commaDelimited");
833+
});
834+
835+
it("can use ArrayEncoding.newlineDelimited", async () => {
836+
const { prop, program } = await Tester.compile(t.code`
837+
model Foo {
838+
@encode(ArrayEncoding.newlineDelimited)
839+
${t.modelProperty("prop")}: string[];
840+
}
841+
`);
842+
843+
strictEqual(getEncode(program, prop)?.encoding, "ArrayEncoding.newlineDelimited");
844+
});
845+
});
798846
});
799847

800848
describe("@withoutOmittedProperties", () => {

packages/openapi3/src/encoding.ts

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,12 @@ import type { OpenAPI3Schema, OpenAPISchema3_1 } from "./types.js";
77

88
function isParameterStyleEncoding(encoding: string | undefined): boolean {
99
if (!encoding) return false;
10-
return ["ArrayEncoding.pipeDelimited", "ArrayEncoding.spaceDelimited"].includes(encoding);
10+
return [
11+
"ArrayEncoding.pipeDelimited",
12+
"ArrayEncoding.spaceDelimited",
13+
"ArrayEncoding.commaDelimited",
14+
"ArrayEncoding.newlineDelimited",
15+
].includes(encoding);
1116
}
1217

1318
export function applyEncoding(

packages/openapi3/src/examples.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -373,6 +373,10 @@ function getQueryParameterValue(
373373
return getParameterDelimitedValue(program, originalValue, property, " ");
374374
case "pipeDelimited":
375375
return getParameterDelimitedValue(program, originalValue, property, "|");
376+
case "commaDelimited":
377+
return getParameterDelimitedValue(program, originalValue, property, ",");
378+
case "newlineDelimited":
379+
return getParameterDelimitedValue(program, originalValue, property, "\n");
376380
}
377381
}
378382

@@ -518,7 +522,7 @@ function getParameterDelimitedValue(
518522
program: Program,
519523
originalValue: Value,
520524
property: Extract<HttpParameterProperties, { kind: "query" }>,
521-
delimiter: " " | "|",
525+
delimiter: " " | "|" | "," | "\n",
522526
): Value | undefined {
523527
const { explode, name } = property.options;
524528
// Serialization is undefined for explode=true

packages/openapi3/src/parameters.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,18 @@ import { getEncode, ModelProperty, Program } from "@typespec/compiler";
33
export function getParameterStyle(
44
program: Program,
55
type: ModelProperty,
6-
): "pipeDelimited" | "spaceDelimited" | undefined {
6+
): "pipeDelimited" | "spaceDelimited" | "commaDelimited" | "newlineDelimited" | undefined {
77
const encode = getEncode(program, type);
88
if (!encode) return;
99

1010
if (encode.encoding === "ArrayEncoding.pipeDelimited") {
1111
return "pipeDelimited";
1212
} else if (encode.encoding === "ArrayEncoding.spaceDelimited") {
1313
return "spaceDelimited";
14+
} else if (encode.encoding === "ArrayEncoding.commaDelimited") {
15+
return "commaDelimited";
16+
} else if (encode.encoding === "ArrayEncoding.newlineDelimited") {
17+
return "newlineDelimited";
1418
}
1519
return;
1620
}

packages/openapi3/test/examples.test.ts

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -439,6 +439,78 @@ worksFor(supportedVersions, ({ openApiFor }) => {
439439
paramExample: `#{R: 100, G: 200, B: 150}`,
440440
expectedExample: undefined,
441441
},
442+
{
443+
desc: "commaDelimited (undefined)",
444+
param: `@query @encode(ArrayEncoding.commaDelimited) color: string | null`,
445+
paramExample: `null`,
446+
expectedExample: undefined,
447+
},
448+
{
449+
desc: "commaDelimited (string)",
450+
param: `@query @encode(ArrayEncoding.commaDelimited) color: string`,
451+
paramExample: `"blue"`,
452+
expectedExample: undefined,
453+
},
454+
{
455+
desc: "commaDelimited (array) explode: false",
456+
param: `@query @encode(ArrayEncoding.commaDelimited) color: string[]`,
457+
paramExample: `#["blue", "black", "brown"]`,
458+
expectedExample: "color=blue%2Cblack%2Cbrown",
459+
},
460+
{
461+
desc: "commaDelimited (array) explode: true",
462+
param: `@query(#{ explode: true }) @encode(ArrayEncoding.commaDelimited) color: string[]`,
463+
paramExample: `#["blue", "black", "brown"]`,
464+
expectedExample: undefined,
465+
},
466+
{
467+
desc: "commaDelimited (object) explode: false",
468+
param: `@query @encode(ArrayEncoding.commaDelimited) color: Record<int32>`,
469+
paramExample: `#{R: 100, G: 200, B: 150}`,
470+
expectedExample: "color=R%2C100%2CG%2C200%2CB%2C150",
471+
},
472+
{
473+
desc: "commaDelimited (object) explode: true",
474+
param: `@query(#{ explode: true }) @encode(ArrayEncoding.commaDelimited) color: Record<int32>`,
475+
paramExample: `#{R: 100, G: 200, B: 150}`,
476+
expectedExample: undefined,
477+
},
478+
{
479+
desc: "newlineDelimited (undefined)",
480+
param: `@query @encode(ArrayEncoding.newlineDelimited) color: string | null`,
481+
paramExample: `null`,
482+
expectedExample: undefined,
483+
},
484+
{
485+
desc: "newlineDelimited (string)",
486+
param: `@query @encode(ArrayEncoding.newlineDelimited) color: string`,
487+
paramExample: `"blue"`,
488+
expectedExample: undefined,
489+
},
490+
{
491+
desc: "newlineDelimited (array) explode: false",
492+
param: `@query @encode(ArrayEncoding.newlineDelimited) color: string[]`,
493+
paramExample: `#["blue", "black", "brown"]`,
494+
expectedExample: "color=blue%0Ablack%0Abrown",
495+
},
496+
{
497+
desc: "newlineDelimited (array) explode: true",
498+
param: `@query(#{ explode: true }) @encode(ArrayEncoding.newlineDelimited) color: string[]`,
499+
paramExample: `#["blue", "black", "brown"]`,
500+
expectedExample: undefined,
501+
},
502+
{
503+
desc: "newlineDelimited (object) explode: false",
504+
param: `@query @encode(ArrayEncoding.newlineDelimited) color: Record<int32>`,
505+
paramExample: `#{R: 100, G: 200, B: 150}`,
506+
expectedExample: "color=R%0A100%0AG%0A200%0AB%0A150",
507+
},
508+
{
509+
desc: "newlineDelimited (object) explode: true",
510+
param: `@query(#{ explode: true }) @encode(ArrayEncoding.newlineDelimited) color: Record<int32>`,
511+
paramExample: `#{R: 100, G: 200, B: 150}`,
512+
expectedExample: undefined,
513+
},
442514
])("$desc", async ({ param, paramExample, expectedExample }) => {
443515
const res = await openApiFor(
444516
`

packages/openapi3/test/parameters.test.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,8 @@ worksFor(supportedVersions, ({ diagnoseOpenApiFor, openApiFor }) => {
3535
it.each([
3636
{ encoding: "ArrayEncoding.pipeDelimited", style: "pipeDelimited" },
3737
{ encoding: "ArrayEncoding.spaceDelimited", style: "spaceDelimited" },
38+
{ encoding: "ArrayEncoding.commaDelimited", style: "commaDelimited" },
39+
{ encoding: "ArrayEncoding.newlineDelimited", style: "newlineDelimited" },
3840
])("can set style to $style with @encode($encoding)", async ({ encoding, style }) => {
3941
const param = await getQueryParam(
4042
`op test(@query @encode(${encoding}) myParam: string[]): void;`,

website/src/content/docs/docs/standard-library/built-in-data-types.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -490,8 +490,10 @@ enum ArrayEncoding
490490

491491
| Name | Value | Description |
492492
|------|-------|-------------|
493-
| pipeDelimited | | Each values of the array is separated by a \| |
494-
| spaceDelimited | | Each values of the array is separated by a <space> |
493+
| pipeDelimited | | Each value of the array is separated by a pipe character (\|).<br />Values can only contain \| if the underlying protocol supports encoding them.<br />- json -> error<br />- http -> %7C |
494+
| spaceDelimited | | Each value of the array is separated by a space character.<br />Values can only contain spaces if the underlying protocol supports encoding them.<br />- json -> error<br />- http -> %20 |
495+
| commaDelimited | | Each value of the array is separated by a comma (,).<br />Values can only contain commas if the underlying protocol supports encoding them.<br />- json -> error<br />- http -> %2C |
496+
| newlineDelimited | | Each value of the array is separated by a newline character (\n).<br />Values can only contain newlines if the underlying protocol supports encoding them.<br />- json -> error<br />- http -> %0A |
495497

496498

497499
### `BytesKnownEncoding` {#BytesKnownEncoding}

0 commit comments

Comments
 (0)