Input Format

1. Chat Completion

Example Payload

{ "model": "/opt/ml/model",
"messages": [
{"role": "system", "content": "You are a helpful medical assistant."},
{"role": "user", "content": "What should I do if I have a fever and body aches?"}
],
"max_tokens": 1024,
"temperature": 0.7
}

2. Text Completion

Single Prompt Example

{ "model": "/opt/ml/model",
"prompt": "How can I maintain good kidney health?",
"max_tokens": 512,
"temperature": 0.6
}

Multiple Prompts Example

{ "model": "/opt/ml/model",
"prompt": [
"How can I maintain good kidney health?",
"What are the best practices for kidney care?"
],
"max_tokens": 512,
"temperature": 0.6
}

Important Notes:

• Streaming Responses: Add "stream": true to your request payload to enable streaming
• Model Path Requirement: Always set "model": "/opt/ml/model" (SageMaker's fixed model location)

For addistional details check the documentation here

Output Format

The API delivers responses in two modes:

• Non-streaming:

The complete response is returned as a single JSON object once the model finishes generating the output. This occurs when "stream": false (default) is set in the request payload.

• Streaming:

The response is delivered incrementally as JSON Lines (JSONL) chunks, each prefixed with data: and ending with a newline. The stream concludes with data: [DONE]. This mode is activated by setting "stream": true in the request payload. This section details the structure and fields of the output for both chat completion and text completion endpoints in each mode, reflecting the behavior of a model hosted on Amazon SageMaker with the fixed path "/opt/ml/model".

Non-Streaming Responses

1. Chat Completion

Description:

The chat completion response contains the model’s reply to a series of input messages (e.g., from "system" and "user" roles), as shown in the user’s example payload.

Example:

{ "id": "chatcmpl-1d202501a96e4580b6352ba7064e6bb8", "object": "chat.completion", "created": 1743488701, "model": "/opt/ml/model", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "The patient presents with symptoms of a ...", "tool_calls": [] }, "logprobs": null, "finish_reason": "stop", "stop_reason": null, } ], "usage": { "prompt_tokens": 206, "completion_tokens": 356, "total_tokens": 562, "prompt_tokens_details": null }, "prompt_logprobs": null }

2. Text Completion

Description:

The text completion response contains the model’s generated text based on a single prompt or an array of prompts, as shown in the user’s single and multiple prompt examples.

Example (Single Prompt):

{ "id": "cmpl-a6d9952b95dc4c0dbea4cf9deeb46560", "object": "text_completion", "created": 1743488720, "model": "/opt/ml/model", "choices": [ { "index": 0, "text": "If you have a fever and body aches ...", "logprobs": null, "finish_reason": "stop", "stop_reason": null, "prompt_logprobs": null } ], "usage": { "prompt_tokens": 14, "completion_tokens": 368, "total_tokens": 382, "prompt_tokens_details": null } }

Example (Multiple Prompts):

{ "id": "cmpl-86c6f7fe2ead4dc79ba5942eecfb9930", "object": "text_completion", "created": 1743489812, "model": "/opt/ml/model", "choices": [ { "index": 0, "text": "To maintain good kidney health ...", "logprobs": null, "finish_reason": "stop", "stop_reason": null, "prompt_logprobs": null }, { "index": 1, "text": "Best practices for kidney care include ...", "logprobs": null, "finish_reason": "stop", "stop_reason": null, "prompt_logprobs": null } ], "usage": { "prompt_tokens": 20, "completion_tokens": 50, "total_tokens": 70, "prompt_tokens_details": null } }

Streaming Responses

In streaming mode ("stream": true), the API delivers the response as a series of JSON Lines (JSONL) chunks, each prefixed with data: and terminated with a newline. The stream ends with data: [DONE]. This aligns with the user’s streaming examples using invoke_streaming_endpoint.

1. Chat Completion (Streaming)

Description:

Each chunk contains a portion of the assistant’s message. The full response is reconstructed by concatenating the content fields from the delta objects in the order received.

Example

data: {"id":"chatcmpl-5a398898be0b4014b7eb9fb15798a006","object":"chat.completion.chunk","created":1743433744,"model":"/opt/ml/model","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null,"stop_reason":null}]} data: {"id":"chatcmpl-5a398898be0b4014b7eb9fb15798a006","object":"chat.completion.chunk","created":1743433744,"model":"/opt/ml/model","choices":[{"index":0,"delta":{"content":"If"},"logprobs":null,"finish_reason":null,"stop_reason":null}]} data: {"id":"chatcmpl-5a398898be0b4014b7eb9fb15798a006","object":"chat.completion.chunk","created":1743433744,"model":"/opt/ml/model","choices":[{"index":0,"delta":{"content":" you"},"logprobs":null,"finish_reason":null,"stop_reason":null}]} data: {"id":"chatcmpl-5a398898be0b4014b7eb9fb15798a006","object":"chat.completion.chunk","created":1743433744,"model":"/opt/ml/model","choices":[{"index":0,"delta":{"content":" have"},"logprobs":null,"finish_reason":"length","stop_reason":null}]} data: [DONE]

2. Text Completion (Streaming)

Description:

Each chunk contains a portion of the generated text. The full response is reconstructed by concatenating the text fields from each chunk in the order received.

Example:

data: {"id":"cmpl-1318a788635e47a58bafeaf18a2816c2","object":"text_completion","created":1743433786,"model":"/opt/ml/model","choices":[{"index":0,"text":"If","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null} data: {"id":"cmpl-1318a788635e47a58bafeaf18a2816c2","object":"text_completion","created":1743433786,"model":"/opt/ml/model","choices":[{"index":0,"text":" you","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null} data: {"id":"cmpl-1318a788635e47a58bafeaf18a2816c2","object":"text_completion","created":1743433786,"model":"/opt/ml/model","choices":[{"index":0,"text":" have","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null} data: {"id":"cmpl-1318a788635e47a58bafeaf18a2816c2","object":"text_completion","created":1743433786,"model":"/opt/ml/model","choices":[{"index":0,"text":" a","logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":null} data: [DONE]