Basic Usage Tutorial

In the Quickstart section, we introduced how to use fake_port to test the simulated environment. This section explains how to evaluate your own models on GenManipBench.

Overview

GenManip is designed so that users can evaluate their models with minimal concern about the internal details of Isaac Sim or GenManip itself. Once your environment is correctly configured, you only need to communicate with GenManip through the defined protocol — that is, make sure your input and output follow the required communication format.

On top of this, GenManip supports a variety of benchmarks that evaluate models across multiple dimensions. You can find detailed descriptions of each benchmark in the Benchmarks section. For any benchmark, you can follow this general workflow:

# Download assets
python standalone_tools/download_assets.py --dataset benchmark_name
python standalone_tools/download_assets.py --dataset benchmark_name-layout

# Launch your model service
python standalone_tools/fake_port.py

# Run the evaluation script
python eval_V3.py -cfg configs/tasks/benchmark_name.yml

Detailed benchmark-specific information can be found in the corresponding Benchmarks pages.

Integrating Your Model

You can refer to the file standalone_tools/fake_port.py, which outlines the communication logic. This script retrieves data from the Isaac Sim benchmark environment and returns a static action (e.g., example joint positions and end-effector poses for different robot arms). For more information about the interface and communication protocol, please refer to the Communication Protocol section.

It is important to note that the fake_port.py script is completely standalone — it has no external dependencies. This makes it very easy to integrate into your own project and add your model’s inference logic. For example, you could modify the main loop as follows:

...
if __name__ == "__main__":
    send_socket = create_send_port_and_wait(port=args.send_port)
    time.sleep(1)
    receive_socket = create_receive_port_and_attach(port=args.receive_port)

    while True:
        data = wait_message(receive_socket)
        actions = {"action": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}
        processed_data = process_data(data)
        actions["action"] = model.inference(processed_data)
        send_message(send_socket, actions)