FFMPEG and Multiprocessing

We are at the edge of the end of production here, very few stand still and with them very much of our daily joy because "there is no good or bad company", it's the people that conform and that you have a continuous treat with that count and make the working environment such a great place.

Anyways im gonna talk (as usual) about the last tool i ve had to code at work. Apparently, there s been a mismatching of shots between the two studios involved and from a production point of view they needed to have the whole movie in playblast sequentially so that they had the screen split in two, on top of it the last anim playblast and at the bottom the last refine so that they could compare and make sure the outer studio was getting the last version for lighting etc....

My first thought was to have a look at Adobe Premiere SDK and see if by any chance it had any python API i could play with. 

After a bit of research i found that there was no way to import an xml file with shots and duration and automatically convert it to a final video. Also the only thing you can do with premiere is plugin development with C++ at the "filters level" which means it is not as tweakable as Maya by any far. It was too much overkill for my  needs.

Then somehow i started to look for tools in linux and i bumped into ffmpeg. So surprised and amazed it was not my first choice! Now surely i would recommend it to anyone having to play with video compositing and mixing.

Now i can start to code!

TOOL SKELETON

First iterate through the anim and refine folders. As there is no conflict and no need to share data this could be easily parallelized. Each process would fill a dictionary where the key is like "ACT0X_SQ00XX_SH00XX" and the value the complete filepath of the most recent file.

       
def return_file_dict(root,queue):
    '''
    iterate through each filesystem branch and fill the dictionary, finally put it in the multiprocess safe queue
    '''
    queue.put(root)
    queue.put(file_dict)

def main():
    process_list = []
    queue = Manager().Queue()
    
    for root, dictionary in zip([DST_TMP_LAST_ANIM,DST_TMP_LAST_CROWD_OR_REFINE],[last_anim_dict,last_crowdrefine_dict]):
        p = Process(target=return_file_dict,args=(root,queue,))
        process_list.append(p)
        p.start()
    
    for p in process_list:
        p.join()
    
    '''
    Rescue both dictionaries and merge top/bottom with ffmpeg
    '''
       
 

After these all i needed was to filter both dictionaries and merge the two playblasts corresponding to a given shot/entry in the dict with ffmpeg.

Now these would open a gnome-terminal for each command. So the next thought was to pipe all the commands to a string which then would be executed in a single call to subprocess.call.

But there was a problem: there is a limit in the number of characters you can send as a command to subprocess.call. This was a good idea in the sense that it would only require a call and all would happen in the same terminal/linux process.

The next logical step was to say Ok i can't send all the commands as a string but i can dump the string to a shell script file and execute that shell script from within the subprocess call!!

       
    #
    # compound all the shell script commands into command_element_string
    #

    with open(FFMPEG_COMMANDS, 'w') as f:
        f.write(command_element_string)

    command = 'sh ' + FFMPEG_COMMANDS
    subprocess.call(['gnome-terminal','-x','bash','-c',command],shell=False,env=os.environ.copy())
    
 

FFMPEG SHELL SCRIPT CALLED FROM SUBPROCESS

       
command_element_string +='ffmpeg -y -i ' + last_anim_filepath + ' -i '+ last_crowdrefine_filepath + ' -filter_complex "[0:v]scale=w=999:h=540[v0];[1:v]scale=w=999:h=540[v1];[v0][v1]vstack=inputs=2[v]" -map "[v]" -map $RESULT:a -ac 2 -b:v 4M ' + output_filepath +';\n'
    
 

This would force the resolution to be w=999 h=540 of each of the videos we vertically stack. We Force it because if the resolutions dont match the conversion will fail.

Also another important comand here is the "$RESULT" value which in this case must be 0 or 1 depending on the audio track we choose to be embedded in the output file.

This can vary since there were playblasts from anim as well as from refine that were missing the audio. $RESULT is the result of doing and ffprobe test to each one of the files to ask for audio info.

The only unavoidable case left is when neither of the two files has an audio track, in which case the conversion fails. So far i could take care of this as well but i havent found yet this case so most probably not gonna treat it.

This is the embedded function in the shell script:

       
    command_element_string = 'function ttl_ffprobe()\n'
    command_element_string += '{\n'
    command_element_string += 'RESULT_ANIM="" ;\n'
    command_element_string += 'RESULT_REFINE="" ;\n'
    command_element_string += 'RESULT_ANIM=$(ffprobe -i $1 -show_streams -select_streams a -loglevel error) ;\n'
    command_element_string += 'RESULT_REFINE=$(ffprobe -i $2 -show_streams -select_streams a -loglevel error) ;\n'
    command_element_string += 'CHANNEL_SELECTION=0 ;\n'
    command_element_string += 'if [ -z "$RESULT_ANIM" ]\n'
    command_element_string += 'then\n'
    command_element_string += '\tCHANNEL_SELECTION=1\n'
    command_element_string += 'fi\n'
    command_element_string += 'return $CHANNEL_SELECTION\n'
    command_element_string += '}\n'
    
 

Once we have side by side all the playblasts anim and refine, all that is left is to merge all of them into the final sequence/movie which can be easily done with the "cat" command and properly chosen container. I refer you to the documentation: https://ffmpeg.org/ffmpeg.html

Authentication Between Linux Servers (an RSA Introduction)

This week at work has been really problematic. It all started when the filesystem of one of our servers which had fedora installed crashed. That was the beginning of a very long week. As a result i 've had to tinker with the transference of data between different linux servers, basically doing a copy of some critical root folders.

The copy was made throught a script in python and for each copy statement i needed to authenticate to the target server which nullifies the automatization of the script. This leads me to the topic of this post: i needed to authenticate automatically without password between the server sending the data and the one receiving it. After googling a bit, i found that RSA authentication was the way to go. When i was early at college i remember my father  suggesting i read a book called "The Code Book" by Simon Singh. The author revisits the history of criptography in a very comprehensive manner and this book was the reason i took criptography as an optional subject at university (In spain, in order to complete the degree you usually have some "optional" different complementary subjects you get to choose to differentiate your academic itinerary. Anyways i dove into my library and rescued the book from the shelves.

I want this post to be a refresher/reminder of how the RSA works. But i'll start first by recalling how Whitfield, Hellman and Merkle solved the traditional problem of the key distribution which is the first step to the public/private key cypher.

WHITFIELD,HELLMAN & MERKLE: KEY DISTRIBUTION SOLVED

I will skip the personal descriptions the author makes in the book, yet interesting to get to know the men. I will get straight to the point, first with the analogy and then with the simple maths.

The cyclic problem with traditional criptography is that in order to send a cyphered message you need first to share the same key with the recipient, which leads to same problem again: how do i cypher the key with another key to be able the cypher the initial message?

Suppose Alice wants to send a message to Bob without Eve knowing, who in fact is listening. Alice the message written in a letter in a chest with a padlock which she owns the key and sends the chest to Bob.

What does Bob? He cannot unlock the chest because he doesnt own the key. This is the crutial fact in my opinion because traditionally all efforts at this point consisted of Bob trying to get Alice's key. On the other hand, Bob puts on the chest his own padlock and returns the chest back to Alice. Alice this time unlocks her padlock and removes it and sends again the chest. At this point, Bob receives the chest which he can open because he owns the key of the only padlock remaining. 

If you understand the message sent as the "key" that can be used to cypher a message, Bob and Alice just shared the message using both their own different private key!!!!

It is worth noting that this implies that the order in which code must not matter, contrary to traditional cyphering. If one codes with key A and then codes the cyphered message again with key B, you must exactly follow the inverse order when decoding: first start by the last, decode with key B and then with key A. If you try to decode first with key A and then with key B it's pretty obvious you wont get the original message. In the idea explained before: it's Alice who first puts the padlock and then Bob but it's Alice the first to unlock (so to decode) in opposition to traditional coding rules (It should be first Bob to unlock since he was the last to lock up the chest)

Now the difficulty was obviuosly to translate this idea to something machines and computers could understand, and since they only understand 0s and 1s, numbers they needed to find a mathematical function with properties alike.

Mathematical functions are usually "two-way" which means you can revert the calculation, in other words: f(A) = B, then you can find g(x) such that g(B) = A. For example the doubling function has its counterpart when dividing by 2.

Whitfield, Hellman & Merkle concentrated on "one-way" mathematical functions and they found modular arithmetics field to be very rich in "one-way" functions. Take for example: 3 exp(x) [mod 11],  for the firsts values of x=1,2,3,4,5,6... you have the following results: 3,2,6,4,5,1 which is pretty irregular. Let's see how this works:

This is how the problem of the key distribution in criptography was solved. Now it has still an inherent problem: since this solution is based on the communication between Alice and Bob it requires both to be "online". If Alice is in Boston and Bob is in Hawaii and she wants to send him a message immediately, Bob needs to be awake so they can share the same key, then Alice can send the message whenever she wants. But they both need to be awake and online first to set the cypher key. This will be solved by RSA public/private key.

Martin Hellman (left) and Whitfield Diffie (right)

RIVEST, SHAMIR & ADLEMAN: ASYMMETRIC CRYPTOGRAPHY

Like most of the great inventions in history, one breakthrough cannot be understood without the previouses ones in the field. The RSA (Rivest-Shamir-Adleman) public/private key was developped thanks to the idea of asymmetric cryptography invented again by Whitfield Diffie. Although Whitfield didn't have a clear concise idea of such a cypher. This is where those three men come in. After reading Whitfield's paper Rivest Shamir and Adleman started their quest to find such cypher.

But first: What is asymmetric criptography? So far, all the criptography was symmetric which means that both the receiver and the sender own the same information, this is: the same key to code and decode the message. Asymmetric cypher proposes that you use a key for coding the message and a different one for decoding.

Whitfield had imagined a system of pair of keys one public which everyone could find in a book such as the telephone guide used to encode the message for that recipient and another private one that only the recipient owns and that enables him to decypher the message coded with the public key. Whitfiled just didnt have a concrete example. After Whitfield published his paper in 1975 several mathematitians started the search for a special one-way function that could be inverted in restricted conditions, this is, when the recipient has a special information.

Rivest, Shamir and Adleman found what they were looking for. The RSA cypher relies on the difficulty to find the prime factors of a big number N. Alice can choose a number N big enough. how? by choosing first two prime numbers also big enough, then Alice sends this N number (her public key) to whoever wants to communicate with her but keeps p and q secret ( N = p x q). So Bob finds N in the telephone guide and encodes a message to Alice. Since Alice knows p and q (her private key) she can decypher the message... But what happens to Eve who is listening? Can't she deduce p and q knowing N? As stated it's far far very far easier to calculate how much is p x q than given N, get to know p and q, its prime factors.

What follows is an example of how the RSA works:

Rivest,Shamir & Adleman at the RSA time creation.

The strength of the RSA application of Whitfield's idea resides in that nowadays it's computationally  very time-consuming to try to factorize N and hence discover p and q. The bigger the prime numbers, the bigger N and then more time needed to try to invert the one-way function. We are talking here of centuries given the large numbers. Mathematitians claim that discovering another way of factorizing in prime numbers other than the obvious one consisting of dividing N for each prime number p such that p < N is very difficult. They are quite convinced there must be a mathematical rule that states that's the only way of doing it. So until someone discovers a shortcut or computers become powerful enough to break RSA in a reasonable amount of time, RSA will still rule communications worldwide.  

As for Whitfield Diffie and Martin Hellman they ve been awarded this year 2016 with the Turing Award by the Association for Computing Machinery (ACM).

Whitfield (left) and Hellman (right) nowadays.

RSA SERVER AUTHENTICATION

We ve explored how the RSA works and how one can send a private message to a person/institution/company. The recipient must own a pair of keys, one he makes public to everyone he wants to receive messages from and one private so he is the only one able to decypher it.

Well, there is another use of RSA that works just the contrary. Imagine you want to login to a server claiming you are who you claim to be. This time it's the sender, not the receiver who needs to generate the pair of keys. The sender keeps his private key secret and sends his public key to the server. So the sender cyphers the message with its private key and sends it the to the server he wants to authenticate. The receiver, because he owns the sender's public key can decypher the message.

Here is a diagram of a laptop trying to communicate with a server but the same process is valid for interservers communication.

Linux commands, let's say server A wants to authenticate to server B:

1). Logon to server A. Type: ssh-keygen
The utility will prompt you to select a location for the keys that will be generated. By default, the keys will be stored in the ~/.ssh  directory  within your user's home directory. The private key will be called  id_rsa  and the associated public key will be called id_rsa.pub ,

2) Copy the id_rsa.pub  public key to the target server B using:
 cat ~/.ssh/id_rsa.pub | ssh username@remote_host "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"   

3) Now you can ssh from server A to B without being prompted for a password. In my case, i used it to start a script that copies between servers so i can do a :

scp -p /path/to/file username@remote_host:/path/to/ 
 
Sources & links:
- "The Code Book", by Simon Singh
- https://www.digitalocean.com/community/tutorials/how-to-configure-ssh-key-based-authentication-on-a-linux-server
- https://en.wikipedia.org/wiki/Public-key_cryptography
- http://www.nytimes.com/2016/03/02/technology/cryptography-pioneers-to-win-turing-award.html?_r=0

Plugin Loading And Threads

Back to coding: TTL_Cameratronic

 

After some time solving issues regarding the publishing process of the shots i m finally back to coding. This time it's a tool whose purpose is to be able to ease the task of the crowd department. They need to work faster and cook simulations faster so the way the tool will work is: they will load the layout of a sequence of shots, with the corresponding cameras obviously, perform the cache of all the sequence and then cut where necessary.

This will require the tool first to iterate through the file system searching for camera files which come as alembics, read from the last published version the duration of the shot and then reference all the cameras setting them with the corresponding frame offset.

Ideally, the tool should also create a master camera which will switch between all the cameras referenced thus composing the sequence's final camera behaviour. I will do this by "parentconstraining" the master camera to all the shot cameras and then keyframing the weights to 0 or 1. Seems easy to do this way.

The tool also will show list of the camera names (which will have to be renamed after the shot number they represent), the start and end frames and finally the alembic file to be loaded (since there may be several different versions we will take always by default the last one).

Those are images of the look & feel of the tool still in development but so far functional: with splash screen and the final result in a list. It needs some tweaks more like a "delete cameras button" for example and offsetting the start frame. Also it lacks the master camera functionality. But i think all left to do won't be much problematic.

GUI & Threads

Since the search process in the filesystem can take a while i will have to deal with threads. Recall that the main thread is responsible for all the GUI painting so any process i want to do has to be pushed into the background in a secondary thread.

Also there is a progress bar which means in this case  a third thread responsible for actually referencing all the camera.

Both secondary threads are used sequentially, so there is no harm and trouble in things like data sharing or concurrency.

The bigger problem I faced, so to speak, and the reason of this post is that loading the alembic plugin caused me some pain at first. The tool has to check whether the plugin is loaded, and if not, proceed to the load.

Now, in the beginning i tried to do this in the same thread responsible for the alembics referencing just before. The result was Maya crashing....

Then, intuitively ( i hadnt read the documentation at that moment) i decided to move that method to the main thread. Now it wasnt crashing but the cameras werent loaded. But i noticed something: loading a maya plugin seems to take some time, a time where Maya is busy presumably registering all the plugins and furthermore seemingly also updates the GUI. This made me think of the "evalDeferred()" method and its updated, non-deprecated equivalent "executeDeferred()" which according to documentation:

maya.utils

The maya.utils package is where utility routines that are not specific to either the API or Commands are stored. This module will likely expand in future versions.

Currently, the maya.utils package contains three routines relevant to threading (see the previous section for details on executeInMainThreadWithResult).

There are two other routines in maya.utils:

• maya.utils.processIdleEvents(). It is mostly useful for testing: it forces the processing of any queued up idle events.
• maya.utils.executeDeferred().
  (Similar to maya.utils.executeInMainThreadWithResult() except that it does not wait for the return value.) It delays the execution of the given script or function until Maya is idle. This function runs code using the idle event loop. This means that the main thread must become idle before this Python code is executed.
  There are two different ways to call this function. The first is to supply a single string argument which contains the Python code to execute. In that case the code is interpreted. The second way to call this routine is to pass it a callable object. When that is the case, then the remaining regular arguments and keyword arguments are passed to the callable object.

 As said "It delays the execution of the given script or function until Maya is idle."

All i had to do is put the plugin loading method in the main thread and execute the code inside my thread with Maya.utils.executeDeferred().

Additional Comment

Another solution which havent been tested but i believe should work according to the documentation is if you really want the code of loading the plugin in the thread you should use executeInMainThreadWithResult().

Despite restrictions, there are many potential uses for threading in Python within the context of Maya; for example, spawning a thread to watch a socket for input. To make the use of Python threads more practical, we have provided a way for other threads to execute code in the main thread and wait upon the result.

The maya.utils.executeInMainThreadWithResult() function takes either a string containing Python code or a Python callable object such as a function. In the latter case, executeInMainThreadWithResult() also accepts both regular and keyword arguments that are passed on to the callable object when it is run.

 

 
 

Python Idiosyncrasies (I)

I want to post here some Python language characteristics i find different and interesting for anyone coming from more traditional ones such as C. So far i've used them every once and a while in the time i've been coding tools here in Madrid and i intend to do several related posts in the future as i learn new features.

I have to say that this only relates to Python 2.7 which is the version Mayapy 2015 comes with.

Composite conditions

Here we find the two key words: "any" and "all"

Both can receive as input a list of expressions that return a boolean and in turn the result is a boolean

"any" equivalent: "if sentence1 or sentence2 or sentence3.... or sentenceN"

"all" equivalent: "if sentence1 and sentence2 and sentence3... and sentenceN"

This way those complex expressions in the conditional can be reduced in combination with a list comprehension. For example this snippet of code shows how to check if some letters appear in a string:

1:  if any(letter in "somestring" for letter in ['a','b','c','s']):  

would return True whereas:

1:  if all(letter in "somestring" for letter in ['a','b','c','s'])  

would return False. 

Iterating multiple lists (I)

Use here the zip() command as stated here:

1:  for elementA, elementB in zip(listA,listB):  

It is worth noting that the zip command iterates until the last element of the shortest of the lists. Though many times both lists have same length, it might not always be the case. Furthermore, you might encounter the need to iterate until the longest list, returning a predefined value for empty element. There is another command for this from the itertools module:

1:  import itertools  
2:  list1 = ['a1','b1']  
3:  list2 = ['a2','b2','c2']  
4:  for element1, elemen2 in itertools.izip_longest(list1,list2)   

By default empty indices' value are set to 'None'.

What happpens if you want this behaviour but also be able to use the element's index?

There is the "enumerate" command that returns a list's current index and element.

1:  for index,element in enumerate(list1):  

And even more: we can combine iterating through multiple lists with the index of the element. In this case we will have one index and two variables holding each current element like so:

 for same_index, elementA, elementB in enumerate(zip(listA,listB)):  

 

 Iterating multiple lists (II)

What we have seen previously is okay, but there is a better way to do the same things in terms of execution time and memory consumption and it relies again on the itertools module.

Using a context manager i have measured the execution time of both functions hereby:

 def slow():  
          for i, (x,y) in enumerate(zip(range(10000000),range(10000000))):  
              pass  
 def quick():  
          for i, x,y in itertools.izip(itertools.count(), range(10000000),range(10000000)):  
              pass  
 measure(slow)  
 measure(quick)  

 Slow() function took 3.32 seconds whereas quick() took 1.30!! A good reason to take into account the itertools module.

Joining strings

Difference between os.path.join() and str.join() both are similar in behaviour.

os.path.join('path','to','directory')

takes a variable number of string arguments to assemble them all into a string with os.sep (operating system path separator). Note that it won't put an os.sep character at the beginning of the string neither at the end. But, if it's an absolute path you can always set as first argument the os.sep character.

if you have the arguments in a list instead of comma separated you can expand the list into it using the * operator thus this would also work:

os.path.join(*['path','to','directory'] 


 Now the str.join method does take a tuple or list as argument and the 'str' string or character will be the separator thus this:

'-'.join(['a','b','c'])

yields the following string : 'a-b-c'. Note again that there is no '-' at the beginining nor at the end.   

Driving A Vector's Direction By Euler With Maya Node Editor

Introduction

Past thursday one my fellow mates at the office in charge of refining the shots and animating dynamics asked me to help him with a problem he was facing: he needed to map the euler rotation angles from a CTRL transform to the nucleus' gravity direction vector. In short: control with angles the direction of a vector.

He was trying with the "angleBetween" node looking for some way to solve his problem. I have to say that at first he wouldnt explain  to me correctly what he needed but that's because he was striving for it himself.

The next day, this is last friday, while at the bus in the morning to the office i remembered the conversation he had a couple of days earlier asking for this to my fellow riggers. They seemed to fill the expectations of my comrade in need with their answers. It wasnt clear for me but i was too busy with something else and didnt want to intercede.... Until now, where this was something that was puzzling him for quite a few days now.

So i started to suppose what was the problem and started to think how i would solve it with nodes.

Controlling Vectors through Rotation

In fact, once aware of the problem the solution is just applying some simple math. We have a curve/CTRL's transform that represents rotations with Euler angles.

Well, we need to apply the rotation of the control to the vector. In terms of maths, we need to get the matrix representation of the rotation. Maya's node editor has a ComposeMatrix node that generates a matrix with information on translation, rotation, scale, shear depending on what the input is.

Next step is to multiply this matrix by the vector we want to transform. Again, Maya has VectorProduct node for this that can do different operations such as Dot, Cross, VectorMatrix and PointMatrix.

v' = M * v

All that is left is just attach the output of the operation to the nucleus's gravity vector....and voilà!!!

Voilà????? Hold on a second, there is a little problem.

One little problem

The way i approached for the first time the node network i was convinced it had to work from the beginning since these are simple maths. But i was committing one error that stems from the fact that i was a little unaware of how nodes work.

I was using the current gravity vector direction to feed the operation and then the result attached back to the nucleus as input..... And i tried to test with known trigonometric values and i was getting a strange behaviour: the numbers just didn't correspond to a valid result.

My network suffered from a "cycle" that never stopped: i'm feeding the input of the attribute with the output of the same attribute after doing some calculation....

But i found one workaround for this.

Workaround

After trying inefficiently to break that cycle using an intermediate transform where i would store the result and pass it on again to nucleus i decided to write a post in a Maya forum convinced of the fact that this can be made without using a transform or any other supplemental entities.

But i kept thinking after the post, enbraved by the fact that i wasnt getting a quick question and i was stuck. Well, if all i need is the starting value of the nucleus how about putting that same initial value into a transform and use it instead of the nucleus to feed the calculation? That way  i can attach directly the result to the nucleus gravity vector. All i have to make sure is that both vector values: nucleus initial value and transform's value are the same!! It doesnt matter really which channels of the transfrom i use provided they represent the three vector componentes X, Y,Z. I decided to use the translate.

Additional Comments

It's worth noting that since the initial value of the vector is such that the module is 1.0, the results after the matrix multiplication must be also of module 1.0. Since a rotation matrix doesnt change the module of a vector. 

This was helpful to rapidly notice wrong numbers and therefore there was a problem. Also, testing with cosine and sine values of most known angles such as 0,30,45,60,90 degrees.... One is used to see numbers like 0.707 or 0.5 and 0.866 etc....

Also, note that because we are using the rotation the translation of the control/curve doesnt matter.

Overriding Python str class setter.. How?

Coding another tool in python as usual i was getting tired of printing several debug messages. The script performs some calls to multiple main methods that in turn call other secondary  methods. I wanted to track the final status of each call so i thought that it would be a good idea to print a message each time a "msg" object changes its value. Thus recurring to the 'set value triggers function call' paradigm.

One solution i found was to create a new class that inherits from "object" and use @property and @<member>.setter with an intermediate string object to store the text.

But i wasn't satisfied enough and coming from C++ i was wondering if it was possible to inherit directly from "str" class and just override its setter.

Since i found no info on the web, i resorted to write to a tech forum. 

Here is my post:


I'm coding a script that performs different complex tasks and i want to output status messages regarding the execution of the script.
Rather than doing a "print" statement after each main function call i 've thought it would be nicer to have a string object that automatically prints something each time the variable's value changes.
So this lead me to write a custom class with a variable and a setter to it like the following:

Code:

class Msg(object):
    def __init__(self):
        self._s = None
    
    @property
    def status(self):
        return self._s
    @status.setter
    def status(self, value):
        self._s = value
        call_custom_function()

Now the question: I've come to the above solution but i'm wondering if it's possible to inherit directly from str class and just override the setter... thus not needing an intermediate variable like _s and be able to do something like

Code:

msg = Msg()
msg = "hello"

instead of the current:

Code:

msg = Msg()
msg.status = "hello"

I've searched the web and i haven't found how the str class works besides the fact that it's a "sequencer" type just like a list...

The same day i received several answers that despite interesting were missing the point of my question, until i got finally the explanation.

Here is the final reply that enlighted me:

Code:

msg = Msg()
msg = "hello"

to anyone who write python code, these lines mean this:

1. Create an instance of the class Msg and assign it to msg
2. msg is now the string "hello"

Your instance of the Msg class has been overwritten. Thats why it makes no sense. Why create an isntance of a class just to over-write it with a string.

Ahhhh this is what i was getting wrong!!! In a language like C++ one can overload the "=" operator and hence the second line would call the overloaded method from the "=" operator from the msg instance of Msg() Class. The type of "msg" doesnt change!!!!

But in Python we dont have such behaviour, so when we assign "hello" we are just changing the type of the variable to be a string!! 

How To Get Rid of PyQt Widgets Correctly

Introduction. The Tool.

In recent weeks i was told it would be nice to have some kind of reference editor outside Maya. Something simple that allowed animators to chose which references they wanted to load in the scene and which ones they didn't want.

What are the advantages for this requirement? The main reason is although we have at the studio powerful workstations in terms of Ram, CPU and Graphics processor some assets like set, props etc are really big, one single prop can take 3 GB!! and depending on the scene you can have almost 400 references. If each prop took that much space.. you can do the math.. it's simply unmanageable. It's not that huge in reality but it remains a big problem also if you take into account the amount of time it takes to load them all and finally open the scene. An animator would normally only want to load the character he/she is about to work with leaving aside all the props and set elements that don't interact with the character. This enables everyone to work faster.

Obviously the external reference editor must be "non-destructive". What i mean for this is it should not delete the reference node in Maya. Why? Obviously this external reference editor is useful for opening a scene file for the first time. Once the scene is loaded in Maya the animator must use the Maya reference editor to load/unload assets. In this case, to load all the necessary assets once the animation is finished, so that everything is in place when the playblast is published. So we need to let the animator the chance to load in Maya the rest of the assets and for this, he needs the reference node of the asset to be present in the scene.

After analyzing the Maya ASCII scene file it was clear what changes to do to the file to unload a specific asset.

Design. The problem.

Here is what i thought it would be a good design: i would use a dynamic list of widgets where each line would be composed of a QCheckBox showing the current state of the reference and the reference node of the asset.

I used the same approach as other times when i needed to code a dynamic list of widgets which consisted mainly in two steps:

A) we have a widget that triggers the fullfillment of the dynamic list. It can be something like a QComboBox to select the file's work area.

B) each time the dynamic list is filled we need to create a "line widget" with its proper layout which contains the QCheckbox and the QLineEdit. Those widgets are created each time which also means they need to be properly deleted, otherwise we will run into memory problems. And that was the origin of the bug i had.

When i first coded a dynamic list like this and wasn't that much versed into python i googled to look for the proper way to delete a widget, and i found this site in stackoverflow to be very useful although somewhat confusing. So many ways to apparently delete QWidgets!!

Digging into the proper solution.

There were three methods that apparently reached the same result:

1) the close() method in the QWidget class

2) the setParent() to None method also in the QWidget class also

3) the deleteLater() also in the QWidget class

I always thought the setParent() to None in each parent widget worked well. So in the method before filling the list i called a cleanup_scrollArea() method which was coded like this:

for i in reversed(range(layout.count())): 
        layout.itemAt(i).widget().setParent(None)

Relying on the fact that in the documentation they say: "the new widget is deleted when its parent is deleted".

I wont explain much. Only tell that this apparently works. Setting the the parent of a widget to None breaks the connection of the PyQt tree and causes all the children to not show anymore.

But there was a big bug. Whenever i tried repeatidly to test the tool with different files the tool crashed within the third or fourth iteration. The dynamic list's behaviour was apparently correct and working well, everything looked alright and i had no error message to give a hint of the problem.

I had the suspicion it had to do with a problem in the deletion of the widgets because the memory increased in each iteration even if the file had less refereneces to show than the previous one!. And obviously it was crashing when you tried to repeatidly use it. It must be a problem in the dynamic list!!

The Solution.

It  took me a short but intense moment  to figure out what was happening. And here my experience with a language such as C/C++ that deals with memory management helped me a lot since PyQt is a bind for Nokia's Qt written in C++.

What was happening?

Setting the  widget's parent to None only breaks the connection in the Qt widgets tree and causes the python reference to be deleted by the garbage collector. But what about the C++ QWidget Object that python was referencing? C++ does not have a garbage collector, so the C++ object's memory has to be deleted manually.

Here is why we have to use deleteLater() 's QWidget method. That's what it does, it frees the memory the C++ object is using..That's what we were missing! Furthermore deleting the C++ object makes the python references invalid therefore we don't need to set anymore to None the parent's widget,

The cleanup_scrollArea() method became:

while aLayout.count() > 0:

    item = aLayout.takeAt(0)

    widget = item.widget()

    if not widget:

        continue

    widget.deleteLater()

Design Improvement Quick Note

Creating and deleting widgets is expensive. It's a very stressful task even for a language like C++ with it's new and delete methods. So it may look like this behaviour for a dynamic list is not the best fit.

In the PyQt documentation they say that for this it may be better to use a QStackedWidget and play with the show() / hide() methods of the widgets which i believe reserves memory for a set of widgets and in the next iteration it reuses the same widgets changing their properties, hiding and adding new widgets on demand as needed. Might want to try this sometime!