PyQt Agnostic Launcher II

As part of the improvements i have been carrying on to the VFX pipeline we are developing i wanted to dig deeper into the problem of executing a PySide Maya tool outside the DCC, this is, as standalone, as well as being able to execute it inside Maya without doing any changes to the code. I already came out with a first version of the launcher which you can see in http://jiceq.blogspot.com.es/2016/08/pyqt-agnostic-tool-launcher.html  . This basically detects whether there isn´t a Qt host application running and if not, we assume it is Maya running.

 @contextlib.contextmanager  
 def application():    
   if not QtGui.qApp:  
     app = QtGui.QApplication(sys.argv)  
     parent = None  
     yield parent  
     app.exec_()  
   else:  
     parent = get_maya_main_window()  
     yield parent  

This works fine for the beginnings of a VFX pipeline, mostly based in Maya. But as soon as you face the need to integrate other heterogeneous packages (that ship with any version of Python and PyQt, which is becoming a standard in the industry. see: http://www.vfxplatform.com/  ) you will probably want to be able to, at least, run the same GUI embedded in different packages as well as standalone. So the need to distinguish between host apps arises and this first solution falls short.

One poor solution is to query the Operating System whether the maya.exe/maya.bin or nuke.exe/nuke.bin processes were running. In the following fashion, for example:

  
def tdfx_is_maya_process_running():    
   return tdfx_is_process_running('maya')
def tdfx_is_nuke_process_running():    
   return tdfx_is_process_running('nuke')
def tdfx_is_process_running(process_name):
   if os.platform() == 'windows':
      ''' specific os code here '''
      return is_running    
   elif os.platform() == 'linux':
      ''' specific os code here '''
      return is_running
   return False

And use this instead in the previous if-statement to retrieve the corresponding QApplication main window instance.

This is a very poor solution, if we can call it a solution. It doesnt work well: you may have an instance of Maya or Nuke running, but you may want to run in standalone mode your custom script from your preferred IDE. The above functions will both return True, first problem. Second, it will depend on order of evaluation, so if you are testing first "tdfx_is_maya_process_running()" then your launcher will attempt to get the Maya main window instance. And third and most important, your launcher wont work because internally it is detecting Maya, so it is reporting the presence of a QApplication.qApp pointer, when you are in standalone mode and there is no qApp pointer actually!

So basically, this approach is not valid. What we really want to query is not the processes running, but more specifically if my current script is running embedded in a qt host application or not, and if so, i want to be able to know which one is.

I googled a little bit and was surprised that some people had faced this problem and meanly resolved it their own -not so great and elegant- way. I just thought there must be some way in Qt to query the host application. I just cant acknowledge something so basic wasnt taken into account in the framework. After some looking into the documentation..eureka, i found this line:

QtWidgets.QApplication.applicationName()

which returns the name of the host application. In standalone Qt apps, it is a parameter that must be set by the programmer.

def tdfx_qthostapp_is_maya():
    return tdfx_qthostapp_is('Maya-2017')

def tdfx_qthostapp_is_nuke():
    return tdfx_qthostapp_is('Nuke')

def tdfx_qthostapp_is(dcc_name):
    from PySide2 import QtWidgets
    hostappname = QtWidgets.QApplication.applicationName()
    if hostappname == dcc_name:
        return True
    return False

Consequently my new contextmanager version takes the following form:

 @contextlib.contextmanager  
 def application():    
   if tdfx_qthostapp_is_none():  
     app = QtGui.QApplication(sys.argv)  
     parent = None  
     yield parent  
     app.exec_()  
   elif tdfx_qthostapp_is_maya():  
     parent = get_maya_main_window()  
     yield parent
   elif tdfx_qthostapp_is_nuke():
     parent = get_nuke_main_window()
     yield parent  

This is a step improvement towards easing the integration of other PyQt-API-based DCCs in a VFX pipeline and easing the task of the programmer, thus avoiding to produce GUI application-specific code. Nonetheless, there is still some work to do that i will deal with when i have more time. This is, making the GUI code fully portable between PySide2 and PySide (or Qt4 and Qt5). There are already some solutions out there like the "Qt.py module" that intends to abstract the GUI from the Qt4 to Qt5 big jump in recent Maya 2017 Python API.

TACTIC Python API Tweak: Hack To Report Copied Byte Amount To Qt Widget

During the development of some Maya Tools that used the Southpaw Tactic Python API I bumped into the following, at first simple, problem: I wanted to give a visual report of the uploading progress process. Each artist had to check-in their work to the asset management system via internet.

The first version of the tool only gave report of the progress by means of a progress bar that visually was enough to notify when the upload had finished. This worked ideally for multiple tiny files. But soon Groom & Hair artists, as well as VFX artist where generating a lot of huge simulating data that needed to be uploaded.

We were working remotely and uploading the artist's work could easily take a couple of hours. The first approach was to use HTTP protocol to transfer those huge amounts of files. There we found a bug in the Python API of Tactic v4.4.04 that limited the file size to 10 MB (10*1024*104 bytes) that forced us to look in the documentation and upgrade to a newer version of Tactic that had this bug fixed. But that's another story.

What interests me here is that the Python Tactic API upload functions dont give any report of the number of bytes uploaded. It only gives a report of when an entire file has been checked-in, this is, by doing a Piecewise Check-In.

So we changed the upload method to use Tactic's handoff dir which consists basically on replacing the HTTP protocol by a protocol like CIFS or NFS where you just perform a copy from your local to the server's directory just like you would between two directories on your local filesystem.

That was the first step.

Now once, definitely using the most powerful transfer method. I only needed to have a look at the API. The "tactic_client_stub.py" module and the "TacticServerStub" class. The Piecewise Check-in works as explained here.

You can see that the API uses the "shutil.copy" and "shutil.move" methods to upload. I cannot tweak the "shutil" module, since it's a built-in one that comes by default with the Maya Python Interpreter. But i can build my own :))!!

My goal is to be able to report the amount of bytes transferred using a Qt Widget so basically i have to simulate a Signal/Slot behaviour from the copy/move methods. It would be nice if i could add a callback inside that method that triggered a Qt Signal, isnt it?!

A LEAST INTRUSIVE SOLUTION

The shutil module uses a lot of different methods to copy files considering the metadata, creation and last modification time, user owner and group owner and the permissions bits, etc. It is explained here.

All of them at last, call the "copyfileobj" method. That's the method i want to tweak.

Now, what kind of function can trigger a Qt Signal?? what are its requisites??

I remembered all Qt Classes inherit from the QObject Class.. A quick look at the PyQt Documentation explains it.

"The central feature in this model is a very powerful mechanism for seamless object communication called signals and slots"

So basically, the only thing i need is to define a class that inherits from QObject, define a custom signal and have the callback method to emit the signal!!. The following is not production code, it is just an example of how it would work.

All that is left is to catch the signal in the proper QWidget, with this information you can compute the time left for the upload to finish and hence give an estimate based on internet speed.

This solution is simple, straightforward and doesnt imply rewriting the TacticServerStub Class. Maybe if i find myself in the need of tweaking again i would consider writing my own TacticServerStub class.

Comments & Critics Welcome!!

Python Idiosyncrasies (I)

I want to post here some Python language characteristics i find different and interesting for anyone coming from more traditional ones such as C. So far i've used them every once and a while in the time i've been coding tools here in Madrid and i intend to do several related posts in the future as i learn new features.

I have to say that this only relates to Python 2.7 which is the version Mayapy 2015 comes with.

Composite conditions

Here we find the two key words: "any" and "all"

Both can receive as input a list of expressions that return a boolean and in turn the result is a boolean

"any" equivalent: "if sentence1 or sentence2 or sentence3.... or sentenceN"

"all" equivalent: "if sentence1 and sentence2 and sentence3... and sentenceN"

This way those complex expressions in the conditional can be reduced in combination with a list comprehension. For example this snippet of code shows how to check if some letters appear in a string:

1:  if any(letter in "somestring" for letter in ['a','b','c','s']):  

would return True whereas:

1:  if all(letter in "somestring" for letter in ['a','b','c','s'])  

would return False. 

Iterating multiple lists (I)

Use here the zip() command as stated here:

1:  for elementA, elementB in zip(listA,listB):  

It is worth noting that the zip command iterates until the last element of the shortest of the lists. Though many times both lists have same length, it might not always be the case. Furthermore, you might encounter the need to iterate until the longest list, returning a predefined value for empty element. There is another command for this from the itertools module:

1:  import itertools  
2:  list1 = ['a1','b1']  
3:  list2 = ['a2','b2','c2']  
4:  for element1, elemen2 in itertools.izip_longest(list1,list2)   

By default empty indices' value are set to 'None'.

What happpens if you want this behaviour but also be able to use the element's index?

There is the "enumerate" command that returns a list's current index and element.

1:  for index,element in enumerate(list1):  

And even more: we can combine iterating through multiple lists with the index of the element. In this case we will have one index and two variables holding each current element like so:

 for same_index, elementA, elementB in enumerate(zip(listA,listB)):  

 

 Iterating multiple lists (II)

What we have seen previously is okay, but there is a better way to do the same things in terms of execution time and memory consumption and it relies again on the itertools module.

Using a context manager i have measured the execution time of both functions hereby:

 def slow():  
          for i, (x,y) in enumerate(zip(range(10000000),range(10000000))):  
              pass  
 def quick():  
          for i, x,y in itertools.izip(itertools.count(), range(10000000),range(10000000)):  
              pass  
 measure(slow)  
 measure(quick)  

 Slow() function took 3.32 seconds whereas quick() took 1.30!! A good reason to take into account the itertools module.

Joining strings

Difference between os.path.join() and str.join() both are similar in behaviour.

os.path.join('path','to','directory')

takes a variable number of string arguments to assemble them all into a string with os.sep (operating system path separator). Note that it won't put an os.sep character at the beginning of the string neither at the end. But, if it's an absolute path you can always set as first argument the os.sep character.

if you have the arguments in a list instead of comma separated you can expand the list into it using the * operator thus this would also work:

os.path.join(*['path','to','directory'] 


 Now the str.join method does take a tuple or list as argument and the 'str' string or character will be the separator thus this:

'-'.join(['a','b','c'])

yields the following string : 'a-b-c'. Note again that there is no '-' at the beginining nor at the end.   

Advanced Multithreading GUI Programming with PySide

One of the tasks i was in charge of was to look for a video player in linux that met several requirements from the art director. Some of them were:

- available for linux
- free
- frame by frame backwards and forward playing
- hotkeys for the above, not only buttons
- able to play several clips in a row
- no blank frame between clips
- able to play the audio of the video as well
- autoloop feature
- plays .mov, .mp4, .avi, compatible with the most possible codecs out there.

The autodesk RV Player is the best one according to some experienced animators but we couldn't afford licenceses for each individual.

After testing the obvious ones such as VLC (with jump in time extension), Openshot, MPlayer (with SMPlayer front-end)... All of them were lacking any of the listed features. Concretely i couldnt restrain my disappointment when i found that VLC's extension wasnt working as claimed on the web.

Several days passed by, and in the meantime i was busy doing other stuff and the animators had to deal with openshot which is not properly a video player but rather a video editor.

Anyways i found out that openshot was built upon the media lovin't toolkit video engine (MELT from now on)

So i decided to install it and give it a go.

Couldn't compile the last version 0.9.6 so i tried the 0.9.2 and yeah it worked!!

I was testing it and realized it met all the primary requisites only drawback: it had to be launched from the terminal and this was not as much user-friendly as it should for an animator.

So i was talking to my boss and he suggested coding a gui for maya that listed all the videos in the work directory and launch the player through Maya. That's when i got "hands on".

CODING THE TOOL

I was getting pretty advanced with the gui programming in PySide/Python time to do a test and launch the player came.

And here i discovered a big problem for me that took me some time to resolve. The problem was that the video player was correctly launched if done manually from an opened terminal but when done from the python script with:

import subprocess

command = ["gnome-terminal", "-x", "bash", "-c", "/usr/bin/melt <videofilefortest>"]
subprocess.call(command, shell = False, env = os.environ.copy())

it was giving an error regarding one of the dynamic link libraries (the famous "DLL" files in windows).

TWO WAYS TO SOLVE IT

Further investigation revealed that doing "ldconfig" in that terminal and launching again the player was a solution. "ldconfig" apparently rebuilds a cache file that specifies the name and path of each dynamic library so the system is able to find them. That opened a way to go. I just had to be able to login as root in the script, do a "ldconfig" and then launch MELT.

Another logical way was to mimic the same environment configuration between the two terminals setting and unsetting the ENV variables until both matched. It was clear that both terminals had different configuration. A "printenv > env_good.txt" and "printenv > env_bad.txt" helped to list the ENV variables of both terminals.

SOLUTION

I was jumping from one strategy to the other always testing. Had some difficulties trying to login as root in a script and also i wasn't confortable hardcoding the root password for obvious security reasons.

Finally i noticed that some important ENV variables werent set in the "good terminal" such as PYTHONHOME PYTHONPATH etc. So I decided to UNSET them prior to launching the player. With some help from a linux forum a user suggested also to UNSET not only those variables but also PATH, LD_LIBRARY_PATH. I agreed and finally it worked!!

EXPLANATION

Apparently, the way subprocess.call works is it executes the commands in a child process. Child processes inherit by default the environment from the parent process which in this case is Maya. But Maya itself is launched with specific configuration that doesnt necessarily match the default one.

It was the LD_LIBRARY_PATH what made it work. Makes sense since the concrete problem was the system wasnt finding where the libraries were.


ADVANCED GUI PROGRAMMING

That problem solved the rest was straightforward.

I noticed that the "search-the-filesystem-for-videos" was taking a bit too much time causing the tool to be apparently "frozen". So I decided to add something showing the tool was busy while searching. Something like a "living" waiting pattern.

This obviously led to the use of multithreading: one showing the waiting state while the other performs the search.

First i was wrong because i tried to run in a background process the GUI waiting icon. And i was wrong because a simple search in google showed that all the GUI processing has to be done in the main thread, and not in secondary ones. That is a good thing to know. I just had to switch the tasks and.. voilá.

I was really proud of it, not that it mattered too much since it was more like an aesthetic thing but i have to admit i haven't done much multithreading programming in all my years of programming.

Visual Studio Setup for Maya Python and Qt4 programming

I'm following a course on 3d using Maya and having worked as a programmer for a couple of years i want to learn to code some python tools for Maya as well as some C++ plug-ins. So the first thing to do is to choose the development environment.

I ve heard in some places that people tend to use SublimeText and that it's really easy to configure for Maya python programming. Well the truth is i'm rather used to VS since i ve programmed in C++ and C# doing some things on my own. Plus, lately i ve been programming some stuff in python and pygame and i used VS with the python tools so this is my selected IDE.

Now i will explain how to setup Visual Studio 2013 for Maya Python programming.

VISUAL STUDIO 2013 SETUP FOR MAYA TOOLS PROGRAMMING IN PYTHON

1. Download and install Python Tools for Visual Studio (PTVS).You can get them here.

2. Download and install PyQt4 for python 2.7 (since Maya 2015 is using python version 2.7.3). You can get it here. You need to install PyQt4 in the following path:

C:\Program Files\Autodesk\Maya2015\Python\Lib\site-packages

3. Open VS 2013 and create a Python Application project.
Under project properties->debug you need to add the following folders to the search path:

C:\Program Files\Autodesk\Maya2015\devkit\other\pymel\extras\completion\py
C:\Program Files\Autodesk\Maya2015\Python\Lib\site-packages\maya
C:\Program Files\Autodesk\Maya2015\Python\Lib\site-packages\pymel
C:\Program Files\Autodesk\Maya2015\Python\Lib\site-packages\PySide
C:\Program Files\Autodesk\Maya2015\Python\Lib\site-packages\PyQt4

Now we have autocompletion for maya.cmds, pyqt4 etc!!

Under "interpreter path" add the line:

C:\Program Files\Autodesk\Maya2015\bin\mayapy.exe

You can have the mayapy.exe interpreter if you want or the one in python27 folder if you prefer. We ve made sure both python version are the same 2.7

4. Now we will add the possibility to execute code from VS directly into Maya.
Download the fast script execution (toMaya) by Josbalcaen here.

- once installed, restart VS 2013
- go to Tools->Options->Keyboard
- search for "ToMaya" and create a new shortcut for example ALT+SPACE

5. write the following code in the Maya script editor

1 2 3 4

import maya.cmds as cmds try: cmds.commandPort(name="127.0.0.1:6000", close=True, echoOutput=True) except:    pass cmds.commandPort(name="127.0.0.1:6000", echoOutput=True)

and put it in a custom shelf or configure your userSetup.mel file to execute the script on Maya startup.

Click on the script, now Maya is listening to this port which is the one ToMaya uses for sending the scripts via VS.

That's all, we should have maya.cmds, PyQt and Python auto completion along with direct execution in Maya!!.

If you prefer, i ve gathered all the necessary stuff needed and put it here. (except for the PTVS, sorry!!)