| Parameter |
Parameter Description |
Parameter Type |
Options |
Default Value |
input |
The data to be predicted, supporting multiple input types (required). |
Python Var|str|list |
- Python Var: Image data represented by
numpy.ndarray.
- str: Local path of an image file or PDF file, such as
/root/data/img.jpg; URL link, such as the network URL of an image file or PDF file: example; Local directory, which must contain images to be predicted, such as the local path: /root/data/ (currently, prediction of PDF files in directories is not supported; PDF files must be specified with an exact file path).
- List: The elements of the list must be of the above types, such as
[numpy.ndarray, numpy.ndarray], ["/root/data/img1.jpg", "/root/data/img2.jpg"], ["/root/data1", "/root/data2"].
|
None |
device |
The inference device for the pipeline. |
str|None |
- CPU: For example,
cpu indicates using the CPU for inference;
- GPU: For example,
gpu:0 indicates using the first GPU for inference;
- NPU: For example,
npu:0 indicates using the first NPU for inference;
- XPU: For example,
xpu:0 indicates using the first XPU for inference;
- MLU: For example,
mlu:0 indicates using the first MLU for inference;
- DCU: For example,
dcu:0 indicates using the first DCU for inference;
- None: If set to
None, the parameter value initialized by the pipeline will be used by default. During initialization, the local GPU device 0 will be prioritized; if unavailable, the CPU device will be used.
|
None |
prompt_type |
The type of prompt used during model inference. |
str |
- box: Use bounding boxes as prompt inputs. If set to
box, the input prompt must be in the form of list[list[float, float, float, float]].
- point: Use points as prompt inputs. If set to
point, the input prompt must be in the form of list[list[float, float]].
|
None |
prompt |
The specific prompt used during model inference. |
list[list[float]] |
- list[list[float]]: Must be set according to the type specified by
prompt_type.
|
None |
(3) Process the prediction results. The prediction result for each sample is of the `dict` type and supports operations such as printing, saving as an image, and saving as a `json` file:
| Method |
Description |
Parameter |
Parameter Type |
Parameter Description |
Default Value |
print() |
Print the result to the terminal |
format_json |
bool |
Whether to format the output content using JSON indentation |
True |
indent |
int |
Specify the indentation level to beautify the output JSON data, making it more readable. Effective only when format_json is True |
4 |
ensure_ascii |
bool |
Control whether to escape non-ASCII characters to Unicode. When set to True, all non-ASCII characters will be escaped; False will retain the original characters. Effective only when format_json is True |
False |
save_to_json() |
Save the result as a JSON file |
save_path |
str |
Path to save the file. When it is a directory, the saved file name is consistent with the input file type naming |
None |
indent |
int |
Specify the indentation level to beautify the output JSON data, making it more readable. Effective only when format_json is True |
4 |
ensure_ascii |
bool |
Control whether to escape non-ASCII characters to Unicode. When set to True, all non-ASCII characters will be escaped; False will retain the original characters. Effective only when format_json is True |
False |
save_to_img() |
Save the result as an image file |
save_path |
str |
Path to save the file, supports directory or file path |
None |
- Calling the `print()` method will print the result to the terminal, with the printed content explained as follows:
- `input_path`: `(str)` The input path of the image to be predicted
- `page_index`: `(Union[int, None])` If the input is a PDF file, it indicates which page of the PDF it is, otherwise it is `None`
- `prompts`: `(dict)` The original prompt information used for predicting the image
- `masks`: `...` The actual predicted masks of the segmentation model. Due to the large data size, it is replaced with `...` for printing. You can save the prediction results as an image using `res.save_to_img` or as a JSON file using `res.save_to_json`.
- `mask_infos`: `(list)` Segmentation result information corresponding to the elements in `masks`, with the same length as `masks`. Each element is a dictionary containing the following fields
- `label`: `(str)` The type of prompt used to predict the corresponding element in `masks`, such as `box_prompt` indicating that the corresponding mask was obtained using a bounding box as the prompt
- `prompt`: `list` The specific prompt information used for predicting the corresponding element in `masks`
- Calling the `save_to_json()` method will save the above content to the specified `save_path`. If specified as a directory, the saved path will be `save_path/{your_img_basename}_res.json`; if specified as a file, it will be saved directly to that file. Since JSON files do not support saving numpy arrays, the `numpy.array` types will be converted to lists.
- Calling the `save_to_img()` method will save the visualization results to the specified `save_path`. If specified as a directory, the saved path will be `save_path/{your_img_basename}_res.{your_img_extension}`; if specified as a file, it will be saved directly to that file.
* Additionally, it also supports obtaining visualized images and prediction results through attributes, as follows:
API Reference
For the main operations provided by the service:
- The HTTP request method is POST.
- Both the request body and response body are JSON data (JSON objects).
- When the request is processed successfully, the response status code is
200, and the response body has the following attributes:
| Name |
Type |
Meaning |
logId |
string |
The UUID of the request. |
errorCode |
integer |
Error code. Fixed at 0. |
errorMsg |
string |
Error description. Fixed at "Success". |
result |
object |
The result of the operation. |
- When the request is not processed successfully, the response body has the following attributes:
| Name |
Type |
Meaning |
logId |
string |
The UUID of the request. |
errorCode |
integer |
Error code. Same as the response status code. |
errorMsg |
string |
Error description. |
The main operations provided by the service are as follows:
Perform object segmentation on an image.
POST /open-vocabulary-segmentation
- The attributes of the request body are as follows:
| Name |
Type |
Meaning |
Required |
image |
string |
The URL of the image file accessible by the server or the Base64 encoded result of the image file content. |
Yes |
prompt |
list[list[float]] |
The prompt used for prediction. |
Yes |
prompt_type |
str |
The type of prompt used for prediction. |
Yes |
- When the request is processed successfully, the
result in the response body has the following attributes:
| Name |
Type |
Meaning |
masks |
array |
The segmentation prediction results. |
mask_infos |
array |
Corresponds one-to-one with the elements in the masks field, recording the prompts used for the corresponding segmentation results in masks. |
image |
string |
The segmentation result image. The image is in JPEG format and encoded in Base64. |
Note: Considering network transmission, the segmentation results recorded in the masks field are encoded with rle. To obtain the original segmentation results, you need to decode them using pycocotools.mask.decode.
result example is as follows:
{
'masks': [rle_mask1, rle_mask2, rle_mask3]
'mask_infos': [
{'label': 'box_prompt', 'prompt': [112.9, 118.4, 513.8, 382.1]},
{'label': 'box_prompt', 'prompt': [4.6, 263.6, 92.2, 336.6]},
{'label': 'box_prompt', 'prompt': [592.4, 260.9, 607.2, 294.2]}
]
}
Multi-language Service Call Example
Python
import base64
import requests
API_URL = "http://localhost:8080/open-vocabulary-segmentation" # Service URL
image_path = "./open_vocabulary_segmentation.jpg"
output_image_path = "./out.jpg"
# Encode the local image with Base64
with open(image_path, "rb") as file:
image_bytes = file.read()
image_data = base64.b64encode(image_bytes).decode("ascii")
payload = {
"image": image_data, # Base64-encoded file content or image URL
"prompt_type": "box",
"prompt": [[112.9,118.4,513.8,382.1],[4.6,263.6,92.2,336.6],[592.4,260.9,607.2,294.2]]
}
# Call the API
response = requests.post(API_URL, json=payload)
# Process the returned data from the API
assert response.status_code == 200
result = response.json()["result"]
image_base64 = result["image"]
image = base64.b64decode(image_base64)
with open(output_image_path, "wb") as file:
file.write(base64.b64decode(result["image"]))
print(f"Output image saved at {output_image_path}")
print("\nresult(with rle encoded binary mask):")
print(result)
C++
#include <iostream>
#include "cpp-httplib/httplib.h" // https://github.com/Huiyicc/cpp-httplib
#include "nlohmann/json.hpp" // https://github.com/nlohmann/json
#include "base64.hpp" // https://github.com/tobiaslocker/base64
int main() {
httplib::Client client("localhost:8080");
const std::string imagePath = "./demo.jpg";
const std::string outputImagePath = "./out.jpg";
httplib::Headers headers = {
{"Content-Type", "application/json"}
};
// Encode the local image with Base64
std::ifstream file(imagePath, std::ios::binary | std::ios::ate);
std::streamsize size = file.tellg();
file.seekg(0, std::ios::beg);
std::vector<char> buffer(size);
if (!file.read(buffer.data(), size)) {
std::cerr << "Error reading file." << std::endl;
return 1;
}
std::string bufferStr(reinterpret_cast<const char*>(buffer.data()), buffer.size());
std::string encodedImage = base64::to_base64(bufferStr);
nlohmann::json jsonObj;
jsonObj["image"] = encodedImage;
std::string body = jsonObj.dump();
// Call the API
auto response = client.Post("/small-object-segmentation", headers, body, "application/json");
// Process the returned data from the API
if (response && response->status == 200) {
nlohmann::json jsonResponse = nlohmann::json::parse(response->body);
auto result = jsonResponse["result"];
encodedImage = result["image"];
std::string decodedString = base64::from_base64(encodedImage);
std::vector<unsigned char> decodedImage(decodedString.begin(), decodedString.end());
std::ofstream outputImage(outPutImagePath, std::ios::binary | std::ios::out);
if (outputImage.is_open()) {
outputImage.write(reinterpret_cast<char*>(decodedImage.data()), decodedImage.size());
outputImage.close();
std::cout << "Output image saved at " << outPutImagePath << std::endl;
} else {
std::cerr << "Unable to open file for writing: " << outPutImagePath << std::endl;
}
auto detectedObjects = result["detectedObjects"];
std::cout << "\nDetected objects:" << std::endl;
for (const auto& category : detectedObjects) {
std::cout << category << std::endl;
}
} else {
std::cout << "Failed to send HTTP request." << std::endl;
return 1;
}
return 0;
}
Java
import okhttp3.*;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.Base64;
public class Main {
public static void main(String[] args) throws IOException {
String API_URL = "http://localhost:8080/small-object-segmentation"; // Service URL
String imagePath = "./demo.jpg"; // Local image path
String outputImagePath = "./out.jpg"; // Output image path
// Encode the local image in Base64
File file = new File(imagePath);
byte[] fileContent = java.nio.file.Files.readAllBytes(file.toPath());
String imageData = Base64.getEncoder().encodeToString(fileContent);
ObjectMapper objectMapper = new ObjectMapper();
ObjectNode params = objectMapper.createObjectNode();
params.put("image", imageData); // Base64-encoded file content or image URL
// Create an OkHttpClient instance
OkHttpClient client = new OkHttpClient();
MediaType JSON = MediaType.Companion.get("application/json; charset=utf-8");
RequestBody body = RequestBody.Companion.create(params.toString(), JSON);
Request request = new Request.Builder()
.url(API_URL)
.post(body)
.build();
// Call the API and process the response data
try (Response response = client.newCall(request).execute()) {
if (response.isSuccessful()) {
String responseBody = response.body().string();
JsonNode resultNode = objectMapper.readTree(responseBody);
JsonNode result = resultNode.get("result");
String base64Image = result.get("image").asText();
JsonNode detectedObjects = result.get("detectedObjects");
byte[] imageBytes = Base64.getDecoder().decode(base64Image);
try (FileOutputStream fos = new FileOutputStream(outputImagePath)) {
fos.write(imageBytes);
}
System.out.println("Output image saved at " + outputImagePath);
System.out.println("\nDetected objects: " + detectedObjects.toString());
} else {
System.err.println("Request failed with code: " + response.code());
}
}
}
}
Go
package main
import (
"bytes"
"encoding/base64"
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
)
func main() {
API_URL := "http://localhost:8080/small-object-segmentation"
imagePath := "./demo.jpg"
outputImagePath := "./out.jpg"
// Encode the local image in Base64
imageBytes, err := ioutil.ReadFile(imagePath)
if err != nil {
fmt.Println("Error reading image file:", err)
return
}
imageData := base64.StdEncoding.EncodeToString(imageBytes)
payload := map[string]string{"image": imageData} // Base64-encoded file content or image URL
payloadBytes, err := json.Marshal(payload)
if err != nil {
fmt.Println("Error marshaling payload:", err)
return
}
// Call the API
client := &http.Client{}
req, err := http.NewRequest("POST", API_URL, bytes.NewBuffer(payloadBytes))
if err != nil {
fmt.Println("Error creating request:", err)
return
}
res, err := client.Do(req)
if err != nil {
fmt.Println("Error sending request:", err)
return
}
defer res.Body.Close()
// Process the response data
body, err := ioutil.ReadAll(res.Body)
if err != nil {
fmt.Println("Error reading response body:", err)
return
}
type Response struct {
Result struct {
Image string `json:"image"`
DetectedObjects []map[string]interface{} `json:"detectedObjects"`
} `json:"result"`
}
var respData Response
err = json.Unmarshal([]byte(string(body)), &respData)
if err != nil {
fmt.Println("Error unmarshaling response body:", err)
return
}
outputImageData, err := base64.StdEncoding.DecodeString(respData.Result.Image)
if err != nil {
fmt.Println("Error decoding base64 image data:", err)
return
}
err = ioutil.WriteFile(outputImagePath, outputImageData, 0644)
if err != nil {
fmt.Println("Error writing image to file:", err)
return
}
fmt.Printf("Image saved at %s.jpg\n", outputImagePath)
fmt.Println("\nDetected objects:")
for _, category := range respData.Result.DetectedObjects {
fmt.Println(category)
}
}
C#
using System;
using System.IO;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Threading.Tasks;
using Newtonsoft.Json.Linq;
class Program
{
static readonly string API_URL = "http://localhost:8080/small-object-segmentation";
static readonly string imagePath = "./demo.jpg";
static readonly string outputImagePath = "./out.jpg";
static async Task Main(string[] args)
{
var httpClient = new HttpClient();
// Base64 encode the local image
byte[] imageBytes = File.ReadAllBytes(imagePath);
string image_data = Convert.ToBase64String(imageBytes);
var payload = new JObject{ { "image", image_data } }; // Base64 encoded file content or image URL
var content = new StringContent(payload.ToString(), Encoding.UTF8, "application/json");
// Call the API
HttpResponseMessage response = await httpClient.PostAsync(API_URL, content);
response.EnsureSuccessStatusCode();
// Process the API response
string responseBody = await response.Content.ReadAsStringAsync();
JObject jsonResponse = JObject.Parse(responseBody);
string base64Image = jsonResponse["result"]["image"].ToString();
byte[] outputImageBytes = Convert.FromBase64String(base64Image);
File.WriteAllBytes(outputImagePath, outputImageBytes);
Console.WriteLine($"Output image saved at {outputImagePath}");
Console.WriteLine("\nDetected objects:");
Console.WriteLine(jsonResponse["result"]["detectedObjects"].ToString());
}
}
Node.js
const axios = require('axios');
const fs = require('fs');
const API_URL = 'http://localhost:8080/small-object-segmentation'
const imagePath = './demo.jpg'
const outputImagePath = "./out.jpg";
let config = {
method: 'POST',
maxBodyLength: Infinity,
url: API_URL,
data: JSON.stringify({
'image': encodeImageToBase64(imagePath) // Base64 encoded file content or image URL
})
};
// Base64 encode the local image
function encodeImageToBase64(filePath) {
const bitmap = fs.readFileSync(filePath);
return Buffer.from(bitmap).toString('base64');
}
// Call the API
axios.request(config)
.then((response) => {
// Process the API response
const result = response.data["result"];
const imageBuffer = Buffer.from(result["image"], 'base64');
fs.writeFile(outputImagePath, imageBuffer, (err) => {
if (err) throw err;
console.log(`Output image saved at ${outputImagePath}`);
});
console.log("\nDetected objects:");
console.log(result["detectedObjects"]);
})
.catch((error) => {
console.log(error);
});
PHP
<?php
$API_URL = "http://localhost:8080/small-object-segmentation"; // Service URL
$image_path = "./demo.jpg";
$output_image_path = "./out.jpg";
// Base64 encode the local image
$image_data = base64_encode(file_get_contents($image_path));
$payload = array("image" => $image_data); // Base64 encoded file content or image URL
// Call the API
$ch = curl_init($API_URL);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($payload));
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);
// Process the API response
$result = json_decode($response, true)["result"];
file_put_contents($output_image_path, base64_decode($result["image"]));
echo "Output image saved at " . $output_image_path . "\n";
echo "\nDetected objects:\n";
print_r($result["detectedObjects"]);
?>
The general open vocabulary segmentation pipeline includes an open vocabulary segmentation module. You can choose the model based on the benchmark data below.
If you prioritize model accuracy, choose a model with higher accuracy; if you prioritize inference speed, choose a model with faster inference speed; if you prioritize storage size, choose a model with a smaller storage size.
