Github repository

INTRODUCTION

The idea of opening up my own github repository has been around my head for some years now. I ve been putting it off for a long time, the reason being i ve always had my personal backup of script code and utilities ive developped through the years at home but the fear of several hard disks crashing including my home NAS finally made up my mind and give github a shot.

It's purpose is primarily for personal use, im not too keen on spending time uploading the unit tests or even commenting unless it is fuzzy enough not to understand.

FIRST PROJECT: SERIALIZER

https://github.com/juan-cristobal-quesada/serializer

In python we have several built-in methods to serialize objects: json, pickle, .. etc. The JSON module serializes only basic types and some built-in datastructures whereas Pickle/cPickle attempts to serialize all custom class objects.

There are several further modules implemented in the python repository that intend to solve different issues with serialization. My current implementation relies on cPickle because of it speed but leverages the final serialization object by limiting the type of variables that are serializable. This comes specially handy if you intend to send the serialized object over a network. Fine tuning which objects get serialized and which dont allows more control over the size.

The serializer in this project allows for basic types serialization including basic lists and dicts datastructures which covers pretty much the core data of the objects we needed to send as well as a special class called Serializable intended for any custom class to inherit from in order to be serialized. In the process the path of the module is appended so that it can be correctly reconstructed at the endpoint.

The resulting object is then base 64 encoded so that it an be ascii compliant, for example allowing to be passed to another subprocess as an environment variable.


FURTHER IMPROVEMENTS

- extend the base serialization and add a readable format such as json notably for debug purposes.
- add a zip compression functionality
- add an encryption functionality so that the serialized object is protected when traveling through the network.
- add support for more built-in basic types such as OrderedDict and others.
- add unit testing cases to showcase the usage.

PyQt Agnostic Launcher II

As part of the improvements i have been carrying on to the VFX pipeline we are developing i wanted to dig deeper into the problem of executing a PySide Maya tool outside the DCC, this is, as standalone, as well as being able to execute it inside Maya without doing any changes to the code. I already came out with a first version of the launcher which you can see in http://jiceq.blogspot.com.es/2016/08/pyqt-agnostic-tool-launcher.html  . This basically detects whether there isn´t a Qt host application running and if not, we assume it is Maya running.

 @contextlib.contextmanager  
 def application():    
   if not QtGui.qApp:  
     app = QtGui.QApplication(sys.argv)  
     parent = None  
     yield parent  
     app.exec_()  
   else:  
     parent = get_maya_main_window()  
     yield parent  

This works fine for the beginnings of a VFX pipeline, mostly based in Maya. But as soon as you face the need to integrate other heterogeneous packages (that ship with any version of Python and PyQt, which is becoming a standard in the industry. see: http://www.vfxplatform.com/  ) you will probably want to be able to, at least, run the same GUI embedded in different packages as well as standalone. So the need to distinguish between host apps arises and this first solution falls short.

One poor solution is to query the Operating System whether the maya.exe/maya.bin or nuke.exe/nuke.bin processes were running. In the following fashion, for example:

  
def tdfx_is_maya_process_running():    
   return tdfx_is_process_running('maya')
def tdfx_is_nuke_process_running():    
   return tdfx_is_process_running('nuke')
def tdfx_is_process_running(process_name):
   if os.platform() == 'windows':
      ''' specific os code here '''
      return is_running    
   elif os.platform() == 'linux':
      ''' specific os code here '''
      return is_running
   return False

And use this instead in the previous if-statement to retrieve the corresponding QApplication main window instance.

This is a very poor solution, if we can call it a solution. It doesnt work well: you may have an instance of Maya or Nuke running, but you may want to run in standalone mode your custom script from your preferred IDE. The above functions will both return True, first problem. Second, it will depend on order of evaluation, so if you are testing first "tdfx_is_maya_process_running()" then your launcher will attempt to get the Maya main window instance. And third and most important, your launcher wont work because internally it is detecting Maya, so it is reporting the presence of a QApplication.qApp pointer, when you are in standalone mode and there is no qApp pointer actually!

So basically, this approach is not valid. What we really want to query is not the processes running, but more specifically if my current script is running embedded in a qt host application or not, and if so, i want to be able to know which one is.

I googled a little bit and was surprised that some people had faced this problem and meanly resolved it their own -not so great and elegant- way. I just thought there must be some way in Qt to query the host application. I just cant acknowledge something so basic wasnt taken into account in the framework. After some looking into the documentation..eureka, i found this line:

QtWidgets.QApplication.applicationName()

which returns the name of the host application. In standalone Qt apps, it is a parameter that must be set by the programmer.

def tdfx_qthostapp_is_maya():
    return tdfx_qthostapp_is('Maya-2017')

def tdfx_qthostapp_is_nuke():
    return tdfx_qthostapp_is('Nuke')

def tdfx_qthostapp_is(dcc_name):
    from PySide2 import QtWidgets
    hostappname = QtWidgets.QApplication.applicationName()
    if hostappname == dcc_name:
        return True
    return False

Consequently my new contextmanager version takes the following form:

 @contextlib.contextmanager  
 def application():    
   if tdfx_qthostapp_is_none():  
     app = QtGui.QApplication(sys.argv)  
     parent = None  
     yield parent  
     app.exec_()  
   elif tdfx_qthostapp_is_maya():  
     parent = get_maya_main_window()  
     yield parent
   elif tdfx_qthostapp_is_nuke():
     parent = get_nuke_main_window()
     yield parent  

This is a step improvement towards easing the integration of other PyQt-API-based DCCs in a VFX pipeline and easing the task of the programmer, thus avoiding to produce GUI application-specific code. Nonetheless, there is still some work to do that i will deal with when i have more time. This is, making the GUI code fully portable between PySide2 and PySide (or Qt4 and Qt5). There are already some solutions out there like the "Qt.py module" that intends to abstract the GUI from the Qt4 to Qt5 big jump in recent Maya 2017 Python API.

Thinking In Design

This week ive started to refactor our pipeline code. We are migrating to Maya 2017 among other things and i wasnt proud of how the development process was held during the last 7 months. To understand it a bit, 7 months ago we were facing a hurry in all aspects. We needed to produce a teaser in barely 4-5 months of strong, intense workload because we bet everything to reach to the AFM with something cool enough to raise some funds and produce the desired movie. The working conditions in terms of organization and qualified staff were a disadvantage. Something everyone of us had had to bear with. Nevertheless there were big pros, we all had passion and were totally committed to the project. I was going to say "Luckily the project worked out really well", but it was due to all our efforts and all the muscle we put into.

Anyways, from the point of view of the pipeline, which is what interests me here, besides the lack of organization we were dealing with a new Digital Asset Management tool where there is little documentation, so at the beginning we didnt have a precise idea of what where the capabilities, the pipeline was being developped at the same time the production started.... Briefly, i had no time to think properly about a good design. Don't misunderstand me, the code produced at that moment was completely functional and i have some testimonies claiming the tools were working well. But that step was necessary to explore the needs and can dos of the pipeline we were conceiving. Now we know how some things were done, we can improve them based on something that already works.

A REFACTORING EXAMPLE

As a little example, there is always the need to use a class that manages some common parameters with some common methods and functions. The way we did it first is just define a Singleton class, inherit from it and start to add parameters and their getters and setters, which in Python can be defined as @property . The manipulation of this data consists among other things, of storing their values , and loading them into memory, by means of some kind of persistence system. It could be a database or something as simple as a text file.

But this approach is really bad design because each time you define a new parameter, you need to change all the methods that input and output the parameters.... Really not very scalable!

Another constraint we didnt take into account is some of the parameters could be classified together. This is, they were related and could be interesting to group them. Some of them dont mean anything on their own if they are not accompanied with their corresponding mate.For example, a login consists of the username and the password. Having the username does not make sense if you havent defined also the password. Under the preceding approach every parameter is independent from the others. And there is no trace of those relationships.

Under those conditions i redesigned the system by making heavy use of inheritance. The related paraemeters could be grouped under a specific class wich derives from the Group Class called here "Section". This class is the atomic class responsible for managing a group of related parameters. So each time i want to expand with a new group of parameters, i only have to define a derived class that inherits from Section and define the parameter keys. Anyother functionality is already present in the base class.

Moreover, i can force from the base classe that the derived ones implement a PARAMS (param1,param2,etc) tuple which are automatically managed. This way i economize work as well as i ensure nobody misuses the class and understands how it is made. It is the same mechanism when we enforce the implementation of an abstract method in the class that inherits from the interface by raising a NotImplemented Exception.

The result is a much more easy to use and therefore extendable manager. Each time i want to create a new group of parameters i just need to define them in a new ConcreteSection class and no more worries than registering the section in the __init__ method. No any other changes to the manager!!

Enough talking, here is a UML class diagram exposing the generic final design.

TACTIC Python API Tweak: Hack To Report Copied Byte Amount To Qt Widget

During the development of some Maya Tools that used the Southpaw Tactic Python API I bumped into the following, at first simple, problem: I wanted to give a visual report of the uploading progress process. Each artist had to check-in their work to the asset management system via internet.

The first version of the tool only gave report of the progress by means of a progress bar that visually was enough to notify when the upload had finished. This worked ideally for multiple tiny files. But soon Groom & Hair artists, as well as VFX artist where generating a lot of huge simulating data that needed to be uploaded.

We were working remotely and uploading the artist's work could easily take a couple of hours. The first approach was to use HTTP protocol to transfer those huge amounts of files. There we found a bug in the Python API of Tactic v4.4.04 that limited the file size to 10 MB (10*1024*104 bytes) that forced us to look in the documentation and upgrade to a newer version of Tactic that had this bug fixed. But that's another story.

What interests me here is that the Python Tactic API upload functions dont give any report of the number of bytes uploaded. It only gives a report of when an entire file has been checked-in, this is, by doing a Piecewise Check-In.

So we changed the upload method to use Tactic's handoff dir which consists basically on replacing the HTTP protocol by a protocol like CIFS or NFS where you just perform a copy from your local to the server's directory just like you would between two directories on your local filesystem.

That was the first step.

Now once, definitely using the most powerful transfer method. I only needed to have a look at the API. The "tactic_client_stub.py" module and the "TacticServerStub" class. The Piecewise Check-in works as explained here.

You can see that the API uses the "shutil.copy" and "shutil.move" methods to upload. I cannot tweak the "shutil" module, since it's a built-in one that comes by default with the Maya Python Interpreter. But i can build my own :))!!

My goal is to be able to report the amount of bytes transferred using a Qt Widget so basically i have to simulate a Signal/Slot behaviour from the copy/move methods. It would be nice if i could add a callback inside that method that triggered a Qt Signal, isnt it?!

A LEAST INTRUSIVE SOLUTION

The shutil module uses a lot of different methods to copy files considering the metadata, creation and last modification time, user owner and group owner and the permissions bits, etc. It is explained here.

All of them at last, call the "copyfileobj" method. That's the method i want to tweak.

Now, what kind of function can trigger a Qt Signal?? what are its requisites??

I remembered all Qt Classes inherit from the QObject Class.. A quick look at the PyQt Documentation explains it.

"The central feature in this model is a very powerful mechanism for seamless object communication called signals and slots"

So basically, the only thing i need is to define a class that inherits from QObject, define a custom signal and have the callback method to emit the signal!!. The following is not production code, it is just an example of how it would work.

All that is left is to catch the signal in the proper QWidget, with this information you can compute the time left for the upload to finish and hence give an estimate based on internet speed.

This solution is simple, straightforward and doesnt imply rewriting the TacticServerStub Class. Maybe if i find myself in the need of tweaking again i would consider writing my own TacticServerStub class.

Comments & Critics Welcome!!

Python Multithreaded Asset Downloader

We are getting towards the end of the production of El Viaje Imposible 's teaser. A mixed 3d/real image project i ve been working on for the last few months along with other fantastic and experienced co-workers.

For the time we ve been developing the pipeline and artists using it, everyday there were some kind of issue and lately the problem was due to the http protocol we used to do the transfers. This was set from the very beginning hoping to review the different uploading methods our pipe allows in the future where the need to send massive amounts of files aroused.

Well, this time has come, Cloth and Hair Artists are already working and in order to pass on their work to the lighters they need to export caches files. Taking into account that our Hair plugin generates  a cache file per frame (even though one can choose to do inter frame caching also, i.e. to avoid flickering), that there may be a couple of plugin nodes that read/export cache multiplied by the number of characters in a shot this makes hundreds of files if not thousands of files to be sent to the server. Hence the need of a good bulletproof protocol.

Forgot to say, a lot of artists are working remotely! With all the inconvenientes this implies, you see.

This week we have improved a lot our checkin/checkout pipeline. We dont use anymore HTTP but have relied now on Samba as our audiovisual project management system allows this.

From my part, one of the improvements i ve done this week is to "parallelize" the assets downloader tool. The first release was running a unique thread in the background and downloaded each pipeline task assets sequentially. 

This was unbearable when we got deeper in the production as more advanced tasks depended upon all the previous  tasks. This means in order to perform a task, an artist should wait until near more than a hundred tasks were checked taking as long as 10 min sitting just with crossed arms.

IMPLEMENTATION

The goal was to substitute the sequential background thread with a configurable number of independent threads each in charge of checking the assets of a unique task. For this, we identify a class Job that is responsible for holding its own connection through Tactic API and all the metadata needed to tell to Tactic what are the assets it is looking for.

Then we define our Worker Class that will be sharing a thread-safe Queue. This Worker class will ask for the current job indefinitely while there are still jobs in the queue. Actually this is a variant of the Producer/Consumer problem where we fulfill the queue with jobs from the beginning, so there is no need for a producer thread.

class Worker(QtCore.QThread):
        
        '''
        define different signals to emit
        '''
        def __init__(self, queue, report):
            QtCore.QThread.__init__(self)
            self.queue = queue
            self.abort = False
            '''
            rest of variables
            '''        
        def run(self):

            while not self.abort and not self.queue.empty():
                job = self.queue.get()
                
                try:
                                
                    response = job.execute()
                    
                except Exception,e:
                    
                    process(e)

                self.queue.task_done()          

One of the problems left then is how to shutdown all the threads when closing the QDialog. I had quite a hard time figuring out the best way of doing it. 

Googleing a bit, people asked the same questions when your thread is running a while True sort of loop. Most people tend to confirm that the most elegant way is to put a "semaphor" also called "sentinel" which no any other thing that a boolean that is checked within every iteration. This allows to set this boolean from outside the thread, so next time it iterates it will jump out of the loop.

Another possibility is to put a Job None Object in the queue, so that immediately after retrieving it from the queue the thread checks its value and exits accordingly. This would work for a single thread, if we spawn 10 threads we should put 10 None Job Objects in the queue.

This leaves the question..¿how to terminate a specific thread? It's not needed here but rather something to think of later...

I resorted to the first elegant solution, that's the reason of the self.abort. So here is the code that overrides the closeEvent()

    def closeEvent(self, *args, **kwargs):
        
        for t in self.threads:
            if t.isRunning():
                t.abort = True
        import time
        time.sleep(2)
        for t in self.td.threads:
            t.terminate()
       
        return QtGui.QDialog.closeEvent(self, *args, **kwargs)

As you can see, before closing, we set the semaphore of each thread to True. The interesting thing about this code is that if you inmediately after try to terminate the thread (im not gonna discuss here the correctness of terminating/killing a thread) the window gets hung. Not sure why this is happening. All we need to do is sleep() a sufficient amount of time to give all the threads the chance to get out of the job.execute() and check for the semaphore.

My only concern with this solution is: what happens if one of the threads is downloading say 1 GB of data? would 2 seconds like in the example be enough time for it to get to the semaphore checking and then exit ? 

That's why i would really want to tweak the Tactic API and get low level for each downloaded chunk of data for example 10 MB. In 10MB slices this problem would disappear... but i'm stuck with the API for now and its interface.

IS IT REALLY PARALLEL?

Well not really. This same code in C or C++ would work totally parallelized but we are bumping into the GIL here, the Global Interpreter Lock of the MayaPy and CPython interpreters. You can have a look at all the posts regarding this in google. Basically, the GIL is a mechanism that forbids the python code to run more than one thread at the same time. This is to prevent the Interpreter's memory gets corrupted.

if we want full parallelization, we should go into multiprocessing which differs from multithreading in that each spawned process has its own memory space. Ideal when you dont need to share objects between processes for example or the need is little. Apart from the fact that, a lot of benchmarks that some people have done, come to the conclusion that in Python, multithreading tends to take more time in CPU-bound tasks that the same code running in single thread.

So if your task is CPU expensive, then try to go Multiprocessing rather than multithreading. But, if your tasks are I/O, networking ,etc (like it is the case here) i find multithreading more suitable. 

Nevertheless, i would like to give it another spin to the code a run benchmarks this time using the multiprocessing module.

PyQt Agnostic Tool Launcher

After some weeks of resting and a new challenge at Drakhar Studio where i am developing the nuts & bolts of a pipeline software that communicates with TACTIC, i have now some spare time to talk about one of my recent discoveries concerning Python programming.

I'm always looking on how to improve my code (in general, no matter what the programming language is), although it's certainly true Python is one in a million because of many reasons, one of them being that a bunch of design patterns are an intrinsic part of the language, such as decorators and context managers.

More specifically, i was looking for a way to run my tools regardless or whether it was standalone (this is, running its own QtApplication, or embedded into Maya's). In the past, i didnt have much time to dig into this and consequently, i used two separate launchers.

Last week this was solved by the use of a context manager. I realised i could make good use of it, since in both cases (running standalone or in a host app) i had to make the same two calls:

 tool_window = Gui_class(parent)  
 tool_window.show()  

Where Gui_class() is the main GUI PyQt Class. The difference is the parent argument where in one case it must be None and in the other it must be a pointer to the Maya GUI Main Window.

This difference in the parent argument could be handled just the same way as for example the open() context manager works:

 with open(filepath, 'wb') as f:  
    src_lines = f.readlines()  

In this case, the advantage is the user doesnt have to remember to close the file stream and the open() method yields the stream as 'f'.


LAUNCHER IMPLEMENTATION

 @contextlib.contextmanager  
 def application():    
   if not QtGui.qApp:  
     app = QtGui.QApplication(sys.argv)  
     parent = None  
     yield parent  
     app.exec_()  
   else:  
     parent = get_maya_main_window()  
     yield parent  

This is my implementation of my custom context manager, it is basically an if statement that checks whether the tool is being lauched from a host application or in his own QApplication. This function could be already the main launcher, the only need is to put the previous two lines showed where the yield statement goes. But this goes against one of the OOP principles, Dont Repeat Yourself (DRY).

So if we push a little bit further we end up with:

 def launcher(cls):      
   with application() as e:  
     window = cls(parent=e)  
     window.show()  

Where cls is the name of the main GUI class. This way, we have an agnostic launcher prepared to work as standalone and within a host app like maya.

Needless to say that some pieces of the tool only will work inside Maya but this way at least we can launch the tool for GUI refinement and development.

No more separate launchers with duplicate parts of the code!!

Simple Procedural Texture Generator and Visualizer

INTRODUCTION

I've been thinking about coding something related with perlin noise, just something that could be used as a justification. Normally i would have coded it in C++ with Qt but since ive been digging into the guts of python and PySide/PyQt for the last year, together with the fact that python GUI with PyQt is not that hard like in C++ (something it really does not have much interest once you get how the layouts, widgets, etc, work).

My only concern was performance because i wanted to do all the calculation and send the vertices data to the gpu each time you changed any of the parameter values governing the shape of the noise, the size, the visualization,..etc. I was willing to accept a little lag.

I wont explain deeply how Perlin noise works. For this you can have a look at the wikipedia or in a book i consider very useful: Texturing & Modeling: A Procedural Approach 

My approach basically consists of a function that generates values for a given octave. Then the final result will be a superposition of those octaves depending on the number specified.

INTERPOLATION

One of the options i wanted to explore was to obtain a more organic feel to the noise. With linear interpolation you can get some artifacts horizontally and vertically which really doesnt look well.

Here are the three interpolation methods:

1. linear

2. cosine

3. cubic 

All of the form "interpolate(x0,x1,t)"

 def Linear(a,b,t):  
   return a * (1 - t) + b * t  
 def Cosine(a,b,t):  
   t2 = (1 - math.cos(t * math.pi)) / 2.0  
   return (a * (1 - t2) + b * t2);  
 def Spline(x0,x1,t):  
   a = x0 - x1  
   b = -1.5 * x0 + 1.5 * x1  
   c = -0.5 * x0 + 0.5 * x1  
   d = x0  
   t2= t * t  
   return a * t2 * t + b * t2 + c * t + d  

The spline or cubic interpolation was used in a simplified manner. Normally the cubic interpolation formula uses information of 4 points: the two in the middle plus the rightmost and leftmost of them. For coding purposes, just to simplify, we assumed p0=p1 and p2=p3, hence the above code.

I will quote this page for the cubic interpolation just in case it disappears.

If the values of a function f(x) and its derivative are known at x=0 and x=1, then the function can be interpolated on the interval [0,1] using a third degree polynomial. This is called cubic interpolation. The formula of this polynomial can be easily derived.

A third degree polynomial and its derivative:

For the green curve:

The values of the polynomial and its derivative at x=0 and x=1:

The four equations above can be rewritten to this:

And there we have our cubic interpolation formula.

Interpolation is often used to interpolate between a list of values. In that case we don't know the derivative of the function. We could simply use derivative 0 at every point, but we obtain smoother curves when we use the slope of a line between the previous and the next point as the derivative at a point. In that case the resulting polynomial is called a Catmull-Rom spline. Suppose you have the values p₀, p₁, p₂ and p₃ at respectively x=-1, x=0, x=1, and x=2. Then we can assign the values of f(0), f(1), f'(0) and f'(1) using the formulas below to interpolate between p₁ and p₂.

Combining the last four formulas and the preceding four, we get:

OPENGL and PYTHON

One of the most time consuming aspects of dealing with PyOpenGL is that OpenGL is a C library and hence, if you code in C++ you share the same basic data types specially things like (void *) pointers, C arrays and the casting operation between types... But Python has its own data types!

1) I'll give you an example: Vertex Buffer Objects need to be passed a C array of GL_FLOAT values in order to specify vertex data. I was managing vertex data but in python lists. I discovered i had two options here: whether i used another dependency library such as Numpy with their immediate conversion between lists and arrays...or i could just use the "array" type. I finally chose this last option.

 from array import array  
 vertex_array = array('f', vertex_list)  
 index_array = array('i', index_list)  

where 'f' stands for float and 'i' for integer.

2) Another big problem i faced is how on earth i could update the vertex data sent to the buffer instead of deleting/creating/sending everything again as if i restarted the app.

 glBindBuffer(GL_ARRAY_BUFFER, self.vboId)  
 c_void_ptr = glMapBuffer(GL_ARRAY_BUFFER, GL_READ_WRITE)  
 c_float_array_ptr = cast(c_void_ptr, POINTER(c_float))  
 # change vertex data    
 for i in range(len(vertex_list)):  
    c_float_array_ptr[i] = vertex_list[i]  
 glUnmapBuffer(GL_ARRAY_BUFFER)  

I discovered the buffer in video memory could be mapped to a chunk in RAM so that when changing one, it immediately applies to GPU. This is using "glMapBuffer/glUnmapBuffer".

But this function returns a "C void pointer" which in python terms is just an integer refering to some memory address.

We need a way to cast this void pointer to a float pointer (float array). That is the raison d'être of the next line. Needless to say i needed to import the ctypes module.

Then we can access finally the c_float_array_ptr as an iterable assigning float values from the python vertex_list!

Here is a video snippet of how the app works.

FFMPEG and Multiprocessing

We are at the edge of the end of production here, very few stand still and with them very much of our daily joy because "there is no good or bad company", it's the people that conform and that you have a continuous treat with that count and make the working environment such a great place.

Anyways im gonna talk (as usual) about the last tool i ve had to code at work. Apparently, there s been a mismatching of shots between the two studios involved and from a production point of view they needed to have the whole movie in playblast sequentially so that they had the screen split in two, on top of it the last anim playblast and at the bottom the last refine so that they could compare and make sure the outer studio was getting the last version for lighting etc....

My first thought was to have a look at Adobe Premiere SDK and see if by any chance it had any python API i could play with. 

After a bit of research i found that there was no way to import an xml file with shots and duration and automatically convert it to a final video. Also the only thing you can do with premiere is plugin development with C++ at the "filters level" which means it is not as tweakable as Maya by any far. It was too much overkill for my  needs.

Then somehow i started to look for tools in linux and i bumped into ffmpeg. So surprised and amazed it was not my first choice! Now surely i would recommend it to anyone having to play with video compositing and mixing.

Now i can start to code!

TOOL SKELETON

First iterate through the anim and refine folders. As there is no conflict and no need to share data this could be easily parallelized. Each process would fill a dictionary where the key is like "ACT0X_SQ00XX_SH00XX" and the value the complete filepath of the most recent file.

       
def return_file_dict(root,queue):
    '''
    iterate through each filesystem branch and fill the dictionary, finally put it in the multiprocess safe queue
    '''
    queue.put(root)
    queue.put(file_dict)

def main():
    process_list = []
    queue = Manager().Queue()
    
    for root, dictionary in zip([DST_TMP_LAST_ANIM,DST_TMP_LAST_CROWD_OR_REFINE],[last_anim_dict,last_crowdrefine_dict]):
        p = Process(target=return_file_dict,args=(root,queue,))
        process_list.append(p)
        p.start()
    
    for p in process_list:
        p.join()
    
    '''
    Rescue both dictionaries and merge top/bottom with ffmpeg
    '''
       
 

After these all i needed was to filter both dictionaries and merge the two playblasts corresponding to a given shot/entry in the dict with ffmpeg.

Now these would open a gnome-terminal for each command. So the next thought was to pipe all the commands to a string which then would be executed in a single call to subprocess.call.

But there was a problem: there is a limit in the number of characters you can send as a command to subprocess.call. This was a good idea in the sense that it would only require a call and all would happen in the same terminal/linux process.

The next logical step was to say Ok i can't send all the commands as a string but i can dump the string to a shell script file and execute that shell script from within the subprocess call!!

       
    #
    # compound all the shell script commands into command_element_string
    #

    with open(FFMPEG_COMMANDS, 'w') as f:
        f.write(command_element_string)

    command = 'sh ' + FFMPEG_COMMANDS
    subprocess.call(['gnome-terminal','-x','bash','-c',command],shell=False,env=os.environ.copy())
    
 

FFMPEG SHELL SCRIPT CALLED FROM SUBPROCESS

       
command_element_string +='ffmpeg -y -i ' + last_anim_filepath + ' -i '+ last_crowdrefine_filepath + ' -filter_complex "[0:v]scale=w=999:h=540[v0];[1:v]scale=w=999:h=540[v1];[v0][v1]vstack=inputs=2[v]" -map "[v]" -map $RESULT:a -ac 2 -b:v 4M ' + output_filepath +';\n'
    
 

This would force the resolution to be w=999 h=540 of each of the videos we vertically stack. We Force it because if the resolutions dont match the conversion will fail.

Also another important comand here is the "$RESULT" value which in this case must be 0 or 1 depending on the audio track we choose to be embedded in the output file.

This can vary since there were playblasts from anim as well as from refine that were missing the audio. $RESULT is the result of doing and ffprobe test to each one of the files to ask for audio info.

The only unavoidable case left is when neither of the two files has an audio track, in which case the conversion fails. So far i could take care of this as well but i havent found yet this case so most probably not gonna treat it.

This is the embedded function in the shell script:

       
    command_element_string = 'function ttl_ffprobe()\n'
    command_element_string += '{\n'
    command_element_string += 'RESULT_ANIM="" ;\n'
    command_element_string += 'RESULT_REFINE="" ;\n'
    command_element_string += 'RESULT_ANIM=$(ffprobe -i $1 -show_streams -select_streams a -loglevel error) ;\n'
    command_element_string += 'RESULT_REFINE=$(ffprobe -i $2 -show_streams -select_streams a -loglevel error) ;\n'
    command_element_string += 'CHANNEL_SELECTION=0 ;\n'
    command_element_string += 'if [ -z "$RESULT_ANIM" ]\n'
    command_element_string += 'then\n'
    command_element_string += '\tCHANNEL_SELECTION=1\n'
    command_element_string += 'fi\n'
    command_element_string += 'return $CHANNEL_SELECTION\n'
    command_element_string += '}\n'
    
 

Once we have side by side all the playblasts anim and refine, all that is left is to merge all of them into the final sequence/movie which can be easily done with the "cat" command and properly chosen container. I refer you to the documentation: https://ffmpeg.org/ffmpeg.html

Plugin Loading And Threads

Back to coding: TTL_Cameratronic

 

After some time solving issues regarding the publishing process of the shots i m finally back to coding. This time it's a tool whose purpose is to be able to ease the task of the crowd department. They need to work faster and cook simulations faster so the way the tool will work is: they will load the layout of a sequence of shots, with the corresponding cameras obviously, perform the cache of all the sequence and then cut where necessary.

This will require the tool first to iterate through the file system searching for camera files which come as alembics, read from the last published version the duration of the shot and then reference all the cameras setting them with the corresponding frame offset.

Ideally, the tool should also create a master camera which will switch between all the cameras referenced thus composing the sequence's final camera behaviour. I will do this by "parentconstraining" the master camera to all the shot cameras and then keyframing the weights to 0 or 1. Seems easy to do this way.

The tool also will show list of the camera names (which will have to be renamed after the shot number they represent), the start and end frames and finally the alembic file to be loaded (since there may be several different versions we will take always by default the last one).

Those are images of the look & feel of the tool still in development but so far functional: with splash screen and the final result in a list. It needs some tweaks more like a "delete cameras button" for example and offsetting the start frame. Also it lacks the master camera functionality. But i think all left to do won't be much problematic.

GUI & Threads

Since the search process in the filesystem can take a while i will have to deal with threads. Recall that the main thread is responsible for all the GUI painting so any process i want to do has to be pushed into the background in a secondary thread.

Also there is a progress bar which means in this case  a third thread responsible for actually referencing all the camera.

Both secondary threads are used sequentially, so there is no harm and trouble in things like data sharing or concurrency.

The bigger problem I faced, so to speak, and the reason of this post is that loading the alembic plugin caused me some pain at first. The tool has to check whether the plugin is loaded, and if not, proceed to the load.

Now, in the beginning i tried to do this in the same thread responsible for the alembics referencing just before. The result was Maya crashing....

Then, intuitively ( i hadnt read the documentation at that moment) i decided to move that method to the main thread. Now it wasnt crashing but the cameras werent loaded. But i noticed something: loading a maya plugin seems to take some time, a time where Maya is busy presumably registering all the plugins and furthermore seemingly also updates the GUI. This made me think of the "evalDeferred()" method and its updated, non-deprecated equivalent "executeDeferred()" which according to documentation:

maya.utils

The maya.utils package is where utility routines that are not specific to either the API or Commands are stored. This module will likely expand in future versions.

Currently, the maya.utils package contains three routines relevant to threading (see the previous section for details on executeInMainThreadWithResult).

There are two other routines in maya.utils:

• maya.utils.processIdleEvents(). It is mostly useful for testing: it forces the processing of any queued up idle events.
• maya.utils.executeDeferred().
  (Similar to maya.utils.executeInMainThreadWithResult() except that it does not wait for the return value.) It delays the execution of the given script or function until Maya is idle. This function runs code using the idle event loop. This means that the main thread must become idle before this Python code is executed.
  There are two different ways to call this function. The first is to supply a single string argument which contains the Python code to execute. In that case the code is interpreted. The second way to call this routine is to pass it a callable object. When that is the case, then the remaining regular arguments and keyword arguments are passed to the callable object.

 As said "It delays the execution of the given script or function until Maya is idle."

All i had to do is put the plugin loading method in the main thread and execute the code inside my thread with Maya.utils.executeDeferred().

Additional Comment

Another solution which havent been tested but i believe should work according to the documentation is if you really want the code of loading the plugin in the thread you should use executeInMainThreadWithResult().

Despite restrictions, there are many potential uses for threading in Python within the context of Maya; for example, spawning a thread to watch a socket for input. To make the use of Python threads more practical, we have provided a way for other threads to execute code in the main thread and wait upon the result.

The maya.utils.executeInMainThreadWithResult() function takes either a string containing Python code or a Python callable object such as a function. In the latter case, executeInMainThreadWithResult() also accepts both regular and keyword arguments that are passed on to the callable object when it is run.

 

 
 

Python Idiosyncrasies (I)

I want to post here some Python language characteristics i find different and interesting for anyone coming from more traditional ones such as C. So far i've used them every once and a while in the time i've been coding tools here in Madrid and i intend to do several related posts in the future as i learn new features.

I have to say that this only relates to Python 2.7 which is the version Mayapy 2015 comes with.

Composite conditions

Here we find the two key words: "any" and "all"

Both can receive as input a list of expressions that return a boolean and in turn the result is a boolean

"any" equivalent: "if sentence1 or sentence2 or sentence3.... or sentenceN"

"all" equivalent: "if sentence1 and sentence2 and sentence3... and sentenceN"

This way those complex expressions in the conditional can be reduced in combination with a list comprehension. For example this snippet of code shows how to check if some letters appear in a string:

1:  if any(letter in "somestring" for letter in ['a','b','c','s']):  

would return True whereas:

1:  if all(letter in "somestring" for letter in ['a','b','c','s'])  

would return False. 

Iterating multiple lists (I)

Use here the zip() command as stated here:

1:  for elementA, elementB in zip(listA,listB):  

It is worth noting that the zip command iterates until the last element of the shortest of the lists. Though many times both lists have same length, it might not always be the case. Furthermore, you might encounter the need to iterate until the longest list, returning a predefined value for empty element. There is another command for this from the itertools module:

1:  import itertools  
2:  list1 = ['a1','b1']  
3:  list2 = ['a2','b2','c2']  
4:  for element1, elemen2 in itertools.izip_longest(list1,list2)   

By default empty indices' value are set to 'None'.

What happpens if you want this behaviour but also be able to use the element's index?

There is the "enumerate" command that returns a list's current index and element.

1:  for index,element in enumerate(list1):  

And even more: we can combine iterating through multiple lists with the index of the element. In this case we will have one index and two variables holding each current element like so:

 for same_index, elementA, elementB in enumerate(zip(listA,listB)):  

 

 Iterating multiple lists (II)

What we have seen previously is okay, but there is a better way to do the same things in terms of execution time and memory consumption and it relies again on the itertools module.

Using a context manager i have measured the execution time of both functions hereby:

 def slow():  
          for i, (x,y) in enumerate(zip(range(10000000),range(10000000))):  
              pass  
 def quick():  
          for i, x,y in itertools.izip(itertools.count(), range(10000000),range(10000000)):  
              pass  
 measure(slow)  
 measure(quick)  

 Slow() function took 3.32 seconds whereas quick() took 1.30!! A good reason to take into account the itertools module.

Joining strings

Difference between os.path.join() and str.join() both are similar in behaviour.

os.path.join('path','to','directory')

takes a variable number of string arguments to assemble them all into a string with os.sep (operating system path separator). Note that it won't put an os.sep character at the beginning of the string neither at the end. But, if it's an absolute path you can always set as first argument the os.sep character.

if you have the arguments in a list instead of comma separated you can expand the list into it using the * operator thus this would also work:

os.path.join(*['path','to','directory'] 


 Now the str.join method does take a tuple or list as argument and the 'str' string or character will be the separator thus this:

'-'.join(['a','b','c'])

yields the following string : 'a-b-c'. Note again that there is no '-' at the beginining nor at the end.   

Overriding Python str class setter.. How?

Coding another tool in python as usual i was getting tired of printing several debug messages. The script performs some calls to multiple main methods that in turn call other secondary  methods. I wanted to track the final status of each call so i thought that it would be a good idea to print a message each time a "msg" object changes its value. Thus recurring to the 'set value triggers function call' paradigm.

One solution i found was to create a new class that inherits from "object" and use @property and @<member>.setter with an intermediate string object to store the text.

But i wasn't satisfied enough and coming from C++ i was wondering if it was possible to inherit directly from "str" class and just override its setter.

Since i found no info on the web, i resorted to write to a tech forum. 

Here is my post:


I'm coding a script that performs different complex tasks and i want to output status messages regarding the execution of the script.
Rather than doing a "print" statement after each main function call i 've thought it would be nicer to have a string object that automatically prints something each time the variable's value changes.
So this lead me to write a custom class with a variable and a setter to it like the following:

Code:

class Msg(object):
    def __init__(self):
        self._s = None
    
    @property
    def status(self):
        return self._s
    @status.setter
    def status(self, value):
        self._s = value
        call_custom_function()

Now the question: I've come to the above solution but i'm wondering if it's possible to inherit directly from str class and just override the setter... thus not needing an intermediate variable like _s and be able to do something like

Code:

msg = Msg()
msg = "hello"

instead of the current:

Code:

msg = Msg()
msg.status = "hello"

I've searched the web and i haven't found how the str class works besides the fact that it's a "sequencer" type just like a list...

The same day i received several answers that despite interesting were missing the point of my question, until i got finally the explanation.

Here is the final reply that enlighted me:

Code:

msg = Msg()
msg = "hello"

to anyone who write python code, these lines mean this:

1. Create an instance of the class Msg and assign it to msg
2. msg is now the string "hello"

Your instance of the Msg class has been overwritten. Thats why it makes no sense. Why create an isntance of a class just to over-write it with a string.

Ahhhh this is what i was getting wrong!!! In a language like C++ one can overload the "=" operator and hence the second line would call the overloaded method from the "=" operator from the msg instance of Msg() Class. The type of "msg" doesnt change!!!!

But in Python we dont have such behaviour, so when we assign "hello" we are just changing the type of the variable to be a string!! 

How To Get Rid of PyQt Widgets Correctly

Introduction. The Tool.

In recent weeks i was told it would be nice to have some kind of reference editor outside Maya. Something simple that allowed animators to chose which references they wanted to load in the scene and which ones they didn't want.

What are the advantages for this requirement? The main reason is although we have at the studio powerful workstations in terms of Ram, CPU and Graphics processor some assets like set, props etc are really big, one single prop can take 3 GB!! and depending on the scene you can have almost 400 references. If each prop took that much space.. you can do the math.. it's simply unmanageable. It's not that huge in reality but it remains a big problem also if you take into account the amount of time it takes to load them all and finally open the scene. An animator would normally only want to load the character he/she is about to work with leaving aside all the props and set elements that don't interact with the character. This enables everyone to work faster.

Obviously the external reference editor must be "non-destructive". What i mean for this is it should not delete the reference node in Maya. Why? Obviously this external reference editor is useful for opening a scene file for the first time. Once the scene is loaded in Maya the animator must use the Maya reference editor to load/unload assets. In this case, to load all the necessary assets once the animation is finished, so that everything is in place when the playblast is published. So we need to let the animator the chance to load in Maya the rest of the assets and for this, he needs the reference node of the asset to be present in the scene.

After analyzing the Maya ASCII scene file it was clear what changes to do to the file to unload a specific asset.

Design. The problem.

Here is what i thought it would be a good design: i would use a dynamic list of widgets where each line would be composed of a QCheckBox showing the current state of the reference and the reference node of the asset.

I used the same approach as other times when i needed to code a dynamic list of widgets which consisted mainly in two steps:

A) we have a widget that triggers the fullfillment of the dynamic list. It can be something like a QComboBox to select the file's work area.

B) each time the dynamic list is filled we need to create a "line widget" with its proper layout which contains the QCheckbox and the QLineEdit. Those widgets are created each time which also means they need to be properly deleted, otherwise we will run into memory problems. And that was the origin of the bug i had.

When i first coded a dynamic list like this and wasn't that much versed into python i googled to look for the proper way to delete a widget, and i found this site in stackoverflow to be very useful although somewhat confusing. So many ways to apparently delete QWidgets!!

Digging into the proper solution.

There were three methods that apparently reached the same result:

1) the close() method in the QWidget class

2) the setParent() to None method also in the QWidget class also

3) the deleteLater() also in the QWidget class

I always thought the setParent() to None in each parent widget worked well. So in the method before filling the list i called a cleanup_scrollArea() method which was coded like this:

for i in reversed(range(layout.count())): 
        layout.itemAt(i).widget().setParent(None)

Relying on the fact that in the documentation they say: "the new widget is deleted when its parent is deleted".

I wont explain much. Only tell that this apparently works. Setting the the parent of a widget to None breaks the connection of the PyQt tree and causes all the children to not show anymore.

But there was a big bug. Whenever i tried repeatidly to test the tool with different files the tool crashed within the third or fourth iteration. The dynamic list's behaviour was apparently correct and working well, everything looked alright and i had no error message to give a hint of the problem.

I had the suspicion it had to do with a problem in the deletion of the widgets because the memory increased in each iteration even if the file had less refereneces to show than the previous one!. And obviously it was crashing when you tried to repeatidly use it. It must be a problem in the dynamic list!!

The Solution.

It  took me a short but intense moment  to figure out what was happening. And here my experience with a language such as C/C++ that deals with memory management helped me a lot since PyQt is a bind for Nokia's Qt written in C++.

What was happening?

Setting the  widget's parent to None only breaks the connection in the Qt widgets tree and causes the python reference to be deleted by the garbage collector. But what about the C++ QWidget Object that python was referencing? C++ does not have a garbage collector, so the C++ object's memory has to be deleted manually.

Here is why we have to use deleteLater() 's QWidget method. That's what it does, it frees the memory the C++ object is using..That's what we were missing! Furthermore deleting the C++ object makes the python references invalid therefore we don't need to set anymore to None the parent's widget,

The cleanup_scrollArea() method became:

while aLayout.count() > 0:

    item = aLayout.takeAt(0)

    widget = item.widget()

    if not widget:

        continue

    widget.deleteLater()

Design Improvement Quick Note

Creating and deleting widgets is expensive. It's a very stressful task even for a language like C++ with it's new and delete methods. So it may look like this behaviour for a dynamic list is not the best fit.

In the PyQt documentation they say that for this it may be better to use a QStackedWidget and play with the show() / hide() methods of the widgets which i believe reserves memory for a set of widgets and in the next iteration it reuses the same widgets changing their properties, hiding and adding new widgets on demand as needed. Might want to try this sometime!