Converting between Screen Coordinates and World Coordinates

Discussion in 'Visualization' started by Frederik Ebert, Mar 1, 2018.

  1. I'd like to convert between screen and world coordinates in mujoco 1.5 (I'm using the mujoco-py bindings).

    In the old python bindings for mujoco 1.3 it is possible to do this conversion by obtaining the GL_MODELVIEW_MATRIX and GL_PROJECTION_MATRIX from python OpenGL like this and then computing the conversion manually.

    However in the new mujoco 1.5 bindings, python OpenGL is not used anymore and instead it seems like images are obtained directly from mujoco's internal renderer like here.

    Is there a way to obtain the projection matrices from mujoco directly, or is there any other way to compute this conversion?

  2. Emo Todorov

    Emo Todorov Administrator Staff Member

    MuJoCo does not have a native function for doing this, but it should be possible to extend the mujoco-py 1.50 wrapper to include OpenGL functionality (like 1.31) and get the modelview and projection matrices. Alternatively you can compute these matrices yourself using the camera field of view, clipping planes and screen aspect ratio and then apply the frustum computations. This is standard OpenGL, there should be plenty of information online how to do it.

    As a side note, MuJoCo was designed to be used from C/C++ where the user can call OpenGL functions without any wrappers getting in the way. It turns however that out most people prefer a wrapper that gets in their way :) OpenAI as well as DeepMind have released such wrappers, but they cannot be expected to provide the full flexibility of writing C code.
  3. Hi Emo,

    I'm also having the same issue. I was able to the camera field of view angle , but I couldn't find information regarding clipping planes and aspect ratio. Will it be possible for you to guide me where to look for it?