RTX Lidar point cloud becomes geometrically inconsistent during rapid sensor motion

[yuheisugano]

Hi Isaac Sim team,
I am observing the following issue with RTX lidar simulation in a moving-vehicle setup. I would appreciate any guidance or suggestions.

Description

I observed a geometric inconsistency issue in RTX lidar point clouds when the sensor undergoes rapid motion.
The issue can be observed directly in the Isaac Sim viewport using the DebugDrawPointCloud visualization. During aggressive vehicle motion, point clouds no longer remain spatially aligned with static landmark surfaces and instead appear to slide or smear relative to the geometry.

The issue was also reproduced through ROS2 point cloud publication. I recorded a ROS2 bag containing:

• point cloud messages
• lidar odometry (sensor pose with respect to the world frame)
  I then replayed the bag and visualized the data in RViz using a custom ROS2 node that transforms each point cloud into the world frame using the corresponding sensor pose at each frame. The same geometric inconsistency was observed in RViz as well.

Step to reproduce

I created a minimal reproducible repository containing the simulation setup and instructions: https://github.com/yuheisugano/isaacsim-lidar-vehicle-repro.git
Please clone the repository and follow the instructions in README.md.

Isaac Sim Version

• 5.1.0
• 6.0-dev2

Operating System (OS)

• Ubuntu 24.04.3 LTS

GPU Name

• NVIDIA GeForce RTX 4090

GPU Driver and CUDA versions

• Driver Version: 580.159.03
• CUDA Version: 13.0

Logs

A log file is in logs/ within the reproduction repository.

Additional Information

• The goal is to simulate lidar point clouds for moving-vehicle dataset generation in SLAM / odometry experiments.
• The vehicle motion is simulated using VehicleSDK in order to model more realistic vehicle dynamics, including tire-ground interaction and slipping behavior.
• I experimented with several potentially relevant parameters, including:
  • simulation_app.enable_motion_bvh
  • sim_time.physics_dt
  • sim_time.rendering_dt
  • omni:sensor:Core:outputMotionCompensationState
  • omni:sensor:Core:scanRateBaseHz
  • omni:sensor:Core:patternFiringRateHz
    However, I was unable to resolve the issue.
• The issue appears more prominently when:
  • vehicle speed increases
  • steering amplitude becomes larger
  • sensor motion becomes more aggressive
• Screen recordings of the simulation and rviz visualization are in media/ within the reproduction repository.

Please let me know if additional logs, videos, or reduced test cases would be helpful.

[michaellin6]

Hi @yuheisugano, Thank you for positing this issue and also for sharing the repro.

On first thought if your rendering rate matches your lidar scan rate exactly — both at 10 Hz. With rendering_dt: 0.1 and scanRateBaseHz: 10.0 (the Example_Rotary default), one full 360° sweep is built from a single GPU render frame. All rays are cast from the sensor's pose at that one instant, but the motion compensation pipeline treats them as if they were emitted progressively across the 100 ms sweep at interpolated poses. At ~5 m/s with steering, that 100 ms of "phantom" pose variation introduces the sliding/smearing you're seeing.
If you have not tried it, try rendering_dt = 0.01 (100 Hz) or rendering_dt = 0.02 (50 Hz) while keeping scanRateBaseHz: 10.0. This gives the sensor 5–10 distinct poses per sweep, so raycasts actually come from different positions. In the meantime, I am also trying your repro and checking for what may be going on.

[michaellin6]

Following up on this — I built a minimal kinematic version of your scene (no Vehicle SDK, no ROS2, just an Xform translating the lidar past a static wall at 8 m/s with sinusoidal yaw) and I'm getting the same sliding behavior. The cause is your sim_time.rendering_dt: 0.1 exactly matching the Example_Rotary scan rate of 10 Hz — one render per full scan rotation, which is the worst case for the per-frame consumer pattern. The annotator IsaacExtractRTXSensorPointCloudNoAccumulator returns points referenced to the sensor pose at scan time, but the lidar prim's USD ComputeLocalToWorldTransform at the moment your publisher reads it is already one rendering_dt later — so the cloud ends up offset by ~2 × rendering_dt × sensor_velocity in the direction of motion (about 1.6 m for your parameters).

Fix: drop rendering_dt to 0.005 (200 Hz, 20 renders per scan rotation). Everything else — enable_motion_bvh: true, outputMotionCompensationState: COMPENSATED, outputFrameOfReference: SENSOR, zero_delay.kit, the ROS2 stamp source — can stay as you have it.

Numbers from a kinematic isolation test (8 m/s longitudinal + ±0.4 rad sinusoidal yaw, wall at world x = 17.9 m, consumer reads IsaacExtractRTXSensorPointCloudNoAccumulator.data and applies Xformable.ComputeLocalToWorldTransform(0.0)):

rendering_dt = 0.100 s  -> wall centroid mean = 19.64 m   (residual +1.74 m, max +1.89 m)
rendering_dt = 0.005 s  -> wall centroid mean = 18.10 m   (residual +0.19 m, max +0.32 m)

The remaining ~7 cm in fix-mode is the wall's own 0.2 m thickness averaged front/back as the sensor passes through.

Background reading: the RTX Sensor Annotators page describes the same artifact in the "dragging" warning. If you'd rather not raise your rendering rate, the alternative is to switch from IsaacExtractRTXSensorPointCloudNoAccumulator to IsaacCreateRTXLidarScanBuffer and consume its Transform output (lidar→world at end-of-scan) instead of Xformable.ComputeLocalToWorldTransform — that output is the correct reference for the buffered data. The per-frame extractor doesn't expose a Transform output, so the consumer has to obtain the lidar pose another way, and reading it from USD at the wrong time is what produces the sliding.

Attached are both my simplified repro and fix scripts.
fix.py
repro.py

[yuheisugano]

Hi @michaellin6, Thank you so much for the detailed investigation and repro scripts. The explanation regarding the synchronization between rendering rate and scan generation was very helpful.

After testing the suggested fixes, I now understand that reducing rendering_dt does significantly improve the geometric consistency of the scan.

However, the remaining issue for my use case is the worst-case distortion (e.g., the max residual +0.32m wall position in the shared case). Even after accounting for wall thickness, the remaining error can still become noticeable during aggressive sensor motion, especially under high angular velocity.

I also tested:

• IsaacCreateRTXLidarScanBuffer
• consuming the annotator Transform output directly
• much smaller values such as rendering_dt = physics_dt = 0.00001 s

The overall behavior improves substantially with smaller rendering_dt, but some shadowing/blurring artifacts still remain in aggressive-motion scenarios.

Since my primary use case is SLAM/odometry research, I would like to prioritize geometric consistency over simulation speed.
Therefore, I would like to ask:
Is there any way to generate a geometrically consistent or artifact-free scan?
For example:

• generating one vertical scan column per rendering frame and manually accumulating them,
• generating an instantaneous whole scan within a single rendering frame (without motion distortion),
• or another method that guarantees geometrically consistent scans even under aggressive motion.

I understand that an instantaneous scan may not be physically realistic for a rotating LiDAR, but such an option would be extremely valuable for controlled perception experiments and pose-estimation benchmarking.

Thank you again for your help and detailed investigation. I really appreciate the time and effort you put into reproducing and analyzing the issue.

[yuheisugano]

Hi @michaellin6, sorry to follow up again.

I wanted to kindly check whether you might have any further thoughts on my follow-up question above. I understand you may be busy, but this issue is still blocking my RTX lidar simulation experiments, so any guidance or suggestions would be greatly appreciated.

Thank you again for your help.

[michaellin6]

Hi @yuheisugano, sorry for the slow reply, I am checking with the internal team that works on the LiDAR motion compensation component. @adevalla-nvidia Can you help follow up on this?

[michaellin6]

internal ticket: 6313423