Add proxy-coupled MJWarp+VBD solver and Franka cable lifting task#5657
Draft
mmichelis wants to merge 175 commits into
Draft
Add proxy-coupled MJWarp+VBD solver and Franka cable lifting task#5657mmichelis wants to merge 175 commits into
mmichelis wants to merge 175 commits into
Conversation
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <copilot@github.com>
…ractive scene) Co-authored-by: Copilot <copilot@github.com>
…pawners, schemas)
…article sync Create isaaclab_experimental/deformable/ module containing: - DeformableObject and DeformableObjectData (Newton backend) - Warp kernels for particle gather/scatter operations - CoupledSolver for rigid-body + VBD cloth interaction - VBDSolverCfg, CoupledSolverCfg, NewtonModelCfg configs - Solver factory functions registered with NewtonManager - Particle sync for USD/Fabric viewport rendering - Cloner hooks for deformable body replication - Model cfg hook for post-finalize parameter application Bug fixes included: - init_pos/init_rot zeroed after Xform bake to prevent double-application - vis_mesh_prim fallback for empty vis_candidates (surface cloth) - No weakref in data class (direct references for particle_q/qd) - model_cfg always applied (not gated behind contact attributes)
Add test_deformable_object.py and test_rigid_deformable_coupling.py under isaaclab_experimental/test/deformable/ with imports adapted for the experimental module layout. Introduce register_hooks() in the deformable __init__ so hooks survive NewtonManager.clear() across test fixtures. Include pre-commit formatting fixes.
Add two deformable object manipulation tasks to isaaclab_tasks_experimental: - Isaac-Pick-Cloth-Direct-v0: Franka robot + cloth (shirt) with coupled MJWarp + VBD solver - Isaac-Pick-VBD-Cube-Direct-v0: Franka robot + deformable cube with coupled MJWarp + VBD solver Both tasks use DeformableNewtonCfg (NewtonCfg subclass with model_cfg field) and import isaaclab_experimental.deformable to trigger hook registration. Also add isaaclab_tasks_experimental import to zero_agent.py for task discovery.
Update the deformable tutorial to support both PhysX and Newton backends via --backend argument. Use core isaaclab.assets imports for backend-agnostic DeformableObject, and import VBDSolverCfg from isaaclab_experimental for the Newton backend path.
Add _get_deformable_ignore_paths() to skip deformable sim/visual mesh prims in builder.add_usd() calls, preventing Newton from creating redundant static mesh colliders for deformable bodies. Also add rebuild_bvh() call in _simulate_physics_only() for solvers that require BVH rebuilds (e.g. VBD cloth).
Gracefully handle particle-only scenes (no rigid bodies) by logging a warning instead of raising RuntimeError when body_label/body_key is empty. Add Fabric particle sync setup for deformable bodies: resize visual mesh topology to match sim mesh, and create per-instance particle offset attributes so the sync kernel can write points.
Defer heavy imports in isaaclab_experimental.deformable behind __getattr__ so pxr is not loaded before Kit starts. Ensure register_hooks() always re-registers after NewtonManager.clear() by resetting the _hooks_registered guard on each explicit call. Fix SurfaceDeformableBodyMaterialCfg class-identity mismatch in from_files.py that caused isinstance() to always fail, making PhysX create volume hierarchies for surface cloth. In deformable_object.py, prefer the original authored Mesh over the PhysX sim_mesh proxy (which has no points at construction time) when a surface deformable material is configured. In newton_replicate.py, pass deformable prim paths as ignore_paths to add_usd so Newton's USD importer skips PhysX proxy meshes.
Replace bare `import isaaclab_experimental.deformable` with explicit `register_hooks()` calls since deferred imports no longer trigger hook registration on import.
The Newton deformable implementation (solvers, kernels, particle sync, cloner hooks) and its tests now live under isaaclab_contrib. All import references in tutorials, task envs, and the isaaclab_newton re-export stub are updated accordingly.
Moves the two deformable task environments from isaaclab_tasks_experimental to isaaclab_tasks so they are discovered by zero_agent.py without needing an extra import of the experimental package.
… code Restore dt=0.01 in SimulationCfg, update env prim comment to explain Newton's env_\d+ naming requirement, restore original reset() ordering, and remove stale commented-out loop.
Remove TetMesh class, spawn_tet_mesh_cuboid, spawn_tet_mesh_from_file, spawn_mesh_from_file, _spawn_tet_mesh_geom_from_tet_mesh and their config counterparts. None are used by any task, test, or tutorial. Also restore Newton docstring in modify_deformable_body_properties.
The isaaclab_newton/assets/deformable_object/__init__.py stub re-exported DeformableObject from isaaclab_contrib, creating a reverse dependency. This is unnecessary since register_hooks() already registers the Newton backend with the DeformableObject factory directly.
Extract the 50-line inline block that sets up Fabric attributes for deformable particle sync from NewtonManager.start_simulation() into setup_fabric_particle_sync() in isaaclab_contrib/deformable/particle_sync. Registered via a new _post_start_simulation_fn hook, keeping isaaclab_newton free of deformable-specific logic.
Cherry-pick Mike's fixes from xiangdonglai#4: - Move visual mesh topology overwrite to define_deformable_body_properties in schemas.py so it happens at USD authoring time, not simulation start. - Add per-instance newton:particleCount Fabric attribute so the sync kernel supports heterogeneous deformable bodies instead of assuming a single particles_per_body for all instances. - Simplify setup_fabric_particle_sync() to only create Fabric attributes.
…e logging - Replace CPU-bound Python for-loop in write_nodal_kinematic_target_to_sim_index with vectorized torch advanced indexing (eliminates per-env GPU-CPU sync). - Log sync_particles_to_usd failures at warning level on first occurrence, then throttle to debug for subsequent failures. - Cache usdrt module import to avoid re-importing every render frame. - Add clarifying comment on particle_q_prev aliasing in two-way coupling friction (known limitation, acceptable for one-way primary use case). - Document _MAX_REACTION_CONTACTS fixed upper bound in coupled_solver.
mmichelis
force-pushed
the
mym/proxycoupling
branch
from
590fae0 to
29d172d
Compare
mmichelis
force-pushed
the
mym/proxycoupling
branch
from
d87e448 to
3f11ccc
Compare
mmichelis
force-pushed
the
mym/proxycoupling
branch
from
3f11ccc to
49215b5
Compare
eval_run.py runs the Isaac-Lift-Cable-Franka-v0 state machine for one cycle under user-supplied dotted-path overrides, captures per-step state to metrics.parquet, and writes an aggregated summary.json suitable for scoring. Failures (NaN tensors, env.step exceptions, or unexpected exceptions) cleanly exit with nan_flag=1 in the summary so the driver can score them.
eval_run: cfg now exposes 'object_pose' (not 'cable_pose'); the demo script in scripts/environments/state_machine/lift_franka_soft.py still references cable_pose, so the inline state-machine driver in eval_run uses the actual cfg name. tune.py: walks an 11-parameter schedule (solver + cable material), sweeps 3-4 trial values per parameter in fresh eval_run subprocesses, keeps the lowest-cost value, appends to tuning_log.jsonl, and writes final_best.json when done.
mmichelis
force-pushed
the
mym/proxycoupling
branch
from
f1d527c to
470f2ac
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add a proxy-coupled MJWarp + VBD solver (wrapping Newton's
SolverProxyCoupled) and aIsaac-Lift-Cable-Franka-v0task where the Franka picks up a Newton cable. The arm lives in the MJWarp entry, the cable particles in the VBD entry, and the gripper fingers are exposed as virtual proxies so VBD detects them as contacts on the cable and returns feedback wrenches to MuJoCo via lagged impulses.Builds on top of PR #5641. Requires newton branch: gdaviet/coupled-solver-framework
Changes
New: proxy-coupled MJWarp + VBD solver
NewtonProxyCoupledMJWarpVBDManager(isaaclab_contrib/deformable/proxy_coupled_mjwarp_vbd_manager.py): wrapsnewton.solvers.SolverProxyCoupledwith MuJoCo Warp + VBD entries. Bodies/joints/shapes are partitioned between the entries fromSceneEntityCfgselectors; joints inherit their child body's owner; static shapes (body == -1) always go to VBD. Proxy bodies are filtered to those owning at least oneCOLLIDE_SHAPES-flagged shape.ProxyCoupledMJWarpVBDSolverCfg: declarative cfg withmjwarp_bodies/vbd_bodies/proxy_bodies(alllist[SceneEntityCfg]), plusproxy_mode("lagged"/"staggered"),proxy_iterations,proxy_collide_interval,proxy_mass_scale. Body selection usesre.fullmatchagainst body short names, scoped by the asset'sprim_path.Newton cfg consolidation
CoupledNewtonCfgreplacesDeformableNewtonCfg. Adds ascene_cfgslot so coupled solvers can resolveSceneEntityCfgselectors at solver-build time; envs now doself.sim.physics = CoupledNewtonCfg(solver_cfg=..., scene_cfg=self.scene).solver_typestrings from VBD / coupled cfgs.Bug fixes
NewtonManager.step()now always runseval_fkafter env resets, arm + binary gripper actions; 10-segment cable; goal sampled in robot root frame; reach / lift / goal-tracking rewards.Test plan
./isaaclab.sh -p scripts/environments/state_machine/lift_franka_soft.py --task Isaac-Lift-Cable-Franka-v0— gripper picks up the cable and reaches the goal sphere.Type of change
Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there