redline to Qmater?

[simonpeterarnold]

can anyone put this process in a manual?

Managed to get it working with a 12core Qmaster setup today and it has improved my render times significantly. I am working with one of my UNIX people to write a script that will automate the process a little as at the minute I have to use RedRushes to Export a UNIX redline script and then manually input by cut and paste in to Qmaster to render.

Will get back when I have something more concrete to show in the way of scripts.In the mean time this is how I have got it working.

I have 3 Mac Pro server nodes each is a 2 x 2.4Ghz dual core so I have effectively 12 nodes for rendering. You need to install Qmaster.app on each server. You get this on the FCP or Compressor disc.

I then set up 1 server to run as a Manager and Service. Setting it up to run using one instance this then spreads rendering of a clip across all 4 cores evenly. The other 2 I set to run just as a service again running as 1 instance on each. So when you sent out to render more than one clip it will render 3 at a time farming each one out to each server. I was getting around 106 frames at HD per minute so over 3 servers that is 318 frames per minute. So thats around 4.7 x realtime at 25fps.

You can run with a setup of 4 instances on each server that allows you to render out 4 clips per server on a 4 core machine but I found it reduces performance significantly although I render out more clips at once I get less frames overall over the same duration.

[Kaku Ito]

wow, you can probably sell that.

I had used Qmaster many times for Compressor transcoding for DVD jobs and it was pretty simple, but using conjunction with REDALERT! is something else.

I had to go through transcoding hell this past few days and more to come, so really looking forward to your script.

[Kaku Ito]

Although my cluster is working with Compressor, the script from REDrush ends up showing "error" in the Compressor. but I'm getting closer.

[simonpeterarnold]

Well my unix guy has come up with the goods so I now have a nice workflow from RedRushes to Qmaster render farm. We have the script running all the time watching a folder that we export the RedRushes script from. It then parses it into another script with Qmaster commands changing all the syntax to work with Qmaster (painfull process). This new script runs automatically and sends a command line script for each clip to Qmaster and that does the rest. Its all automatic apart from the initial redrushes stage.

Make sure you Qmaster is set up properly and that all the servers or location s you store the .R3D and you renders are accessible by the same mounts on all nodes. This is very important. Getting the Qmaster setup to work properly is job in itself but it is worth it.

Put the script on your ingest station/Qmaster manager machine. It can go in any folder but for ease of use we have created a folder in Applications for it called redline.

Here is the script we have called it redline.sh

#!/bin/bash
cd /Applications/redline
timestamp=`date +"%d_%m_%y.%H.%M"`
for file in *.txt
do
# create script from input file
for line in `cat $file| tr -s " " ":"`
do
echo "#"
echo -n "/Applications/Apple\ Qmaster.app/Contents/MacOS/Apple\ Qmaster -clustername pfl_xserve -command \"Shell\" -options "
echo -n "\<command\>"
echo -n $line | tr ':' ' ' | sed -e "s/[^a-zA-Z0-9._]/\\\&/g" | tr -d "\n"
echo "\<\/command\> &"
done > run.sh

# run the script
sh run.sh
mv $file completed/$file.$timestamp
done


Then to run in the background you need to edit crontab to run it at regular intervals.

* * * * * /Applications/redline/redline.sh > /dev/null 2&>1


Hope this helps as it has made my life a lot more easier.

[Kaku Ito]

Gawd, Simon that's enormous.

Can I introduce this in my magazine column? Please let me know.
Let me know all of the proper credits to be mentioned, too.

[Irri]

Thanks for sharing the script. It's an interesting solution. Working fine here.

[simonpeterarnold]

Gawd, Simon that's enormous.

Can I introduce this in my magazine column? Please let me know.
Let me know all of the proper credits to be mentioned, too.

Sure you can, Our Programmer was Andy Bridger I just explained to him what I needed to be done and pointed him towards the commands needed and and he didi the rest, I just helped with some research on the correct redline and Qmaster commands to use. Glad it has been on help.

[Kaku Ito]

Simon, thanks, I'll let you know when I can figure it out to make it work on my side and let you know when it gets covered.

Kaku

[simonpeterarnold]

Simon, thanks, I'll let you know when I can figure it out to make it work on my side and let you know when it gets covered.

Kaku

Ok thanks

[Andrae Palmer]

I'm only getting 1 instance of REDline when exporting a script from REDrushes to Qmaster. I'm on a Mac pro with 8 cores and 8 instances setup. In Qmaster I add a new Shell... then set environment by loading the script file. I then submit it to my cluster. The cluster is connected to 10 rendering nodes. 8 on the Mac Pro and 2 on a G5. Only 1 instance on the Mac Pro are processing the job. Anyone else having a similar problem?

The Mac Pro is the Services and cluster controller. The G5 is services only. The managed check box is checked on both for "Shake and other shell command processing".

This is the script:

REDline -i /Users/Andrae/Desktop/RED\ DEMO\ REEL\ FOOTAGE/A001_06293F.RDM/A001_C001_0629Q9.RDC/A001_C001_0629Q9_001.R3D --outDir /Users/Andrae/Desktop/redline -w 0 -R 1 --useMeta --filter 5 --fit 1 --QTcodec 0 --resizeX 2048 --resizeY 1080 --noClobber