Network Rendering

From The Foundry MODO SDK wiki
Revision as of 03:00, 3 April 2017 by Captain Obvious (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Starting with 301, modo includes built-in network rendering to facilitate rendering single frames across multiple machines. It is also possible to control modo through the use of startup commands, scripts, headless mode and Telnet, thus allowing render controllers to use modo as a slave.

Built-In Network Rendering

This section provides a basic overview of how to use the built-in network rendering in modo. For more information see the Link to modo docs.

modo’s built-in rendering is fairly easy and straight-forward to set up. One machine is designated as the master and the rest as slaves. All machines cooperate to render a single frame at a time.

Setting up Built-In Network Rendering

Prior to modo 601, users needed to designate a common location where scenes and textures could be found by the master and all of its salves. This was set through the 'Network Shared Location in the Preferences. As of 601, modo will automatically send assets to other machines if the shared location cannot be reached or is not set. Either way, you will also need to turn on Use Network Render Nodes on the master.

A single license of modo allows for one master node and up to 50 slaves. If you run modo on a machine that does not have a license, you will be asked if want a get a license or run in slave mode. Select the latter and the machine becomes a slave, and will wait for rendering tasks from the master. A machine with a license can enter a slave mode as well by selecting Enter Slave Mode from the Render menu.

While in slave mode, modo will show a Render Slave dialog. The Set Shared Network button can be used just like the Network Shared Location preference, and slave mode can be quit by clicking the Exit Slave Mode button.

On the master machine, you can see a list of known machines via the LAN Viewport, which is accessible from the Render menu and at the bottom of the Link to modo docs. This shows all machines and their current status.

Rendering on Slaves with Legacy Graphics Hardware

It is not uncommon for slave machines to have relatively primitive graphics hardware. modo requires a minimum level of OpenGL support to run, and will fail to start on unsupported. A work-around is to create a new config file for the slave machines that does not have any OpenGL viewports.

First, make a backup copy of your modo configuration file on a machine that can support OpenGL, and run modo. From the Layout menu, select Clear to remove all viewports. You can now add some non-OpenGL viewports like an Event Log Viewport or a LAN Viewport, although you won’t be able to interact with these while in slave mode. Quit modo to save this configuration. You can copy this file to each of the slave machines and run modo on them without issue. Finally, you can restore your original config on the OpenGL-capable machine.

Network Rendering and Firewalls

modo uses Bonjour to find slaves. Bonjour is Apple’s implementation of an open networking standard known as Zero Config, or multicast DNS. It allows programs to publish the existence of services and the IP, port and machine name running those services. It does not provide any actual communication -- it simply allows for service discovery, sort of like a phone book for programs on the local network. Individual applications then need to know how to connect to those services themselves. Because of this, it only affects network security insomuch as it reduces so-called "security through obscurity." It also does not cross routers or subnets without some involved configuration, meaning that your LAN will not be exposed over the internet. If you need to cross routers and subnets, you should be able to find the information you need via [1].

Bonjour uses UDP port 5353. If you have slaves that are not visible in the LAN viewport, it is likely that this port is being blocked by your firewall, and opening this port should allow machines to be found. This port is normally opened when Bonjour is installed.

As mentioned above, Bonjour is only used to identify services, not to communicate. Older versions of modo used TCP port 59265 to send render commands to specific slaves. modo 601 uses TCP ports 59267 and 59268. If you can see machines in the LAN viewport but the slaves seem to never get any render requests, make sure that these ports are open in your firewall.

Other Networking Issues

There have been reports of problems with Windows machines when their Computer Description (as set in the System Properties Control Panel) is set to anything. Sometimes, this causes modo to fail to properly identify the local machine, which results in a "License In Use" error. Simply clearing the Computer Description so it is empty will fix this.

On OS X, modo may report a similar problem when the Computer Name (as set in the Sharing part of the System Preferences) is empty. Assigning a computer name will fix this problem.

There have also been reports of routers with improperly configured DNS settings. This can cause modo to improperly resolve machines on the LAN as machines on the internet, thus resulting in a similar "License In Use" problem, or else causing slaves to not be found. Disconnecting the router from the network appears to fix the problem. Refer to your router’s documentation and your ISP for information on resolving DNS issues.

Router Configuration Issues

Some routers have other settings which may be covered by the following article on the Apple Support Community site:

Configure Huawei hg655b router for bonjour

If you have other issues, you can ask at the Luxology forums.

Entering Slave Mode On Startup

You can run modo with the -slave command line argument to force it into slave mode. You can programatically set the Network Shared Location and other preferences through the use of startup commands, or just by ensuring the appropriate preference is already in the config. This example shows the startup command method.

<atom type="StartupCommands">
    <list type="Command">!pref.value render.netPath {path}</list>
</atom>

It is important to note that you must use the -slave switch to turn on slave mode at startup. You cannot use render.slave as a startup command, as networking is not initialized until after all startup commands have executed, and as such will fail.

Fire-and-Forget Rendering via Startup Commands

The startup command mechanism can be used to launch modo, render a frame or an entire animation, and then quit. This can be used by external applications to render a scene. Startup commands are intentionally processed before license checking, thus allowing you to control modo in a non- interactive state.

The most basic example opens a scene, renders an animation, and tells modo to quit.

<atom type="StartupCommands">
    <list type="Command">!scene.open {c:\test.lxo} normal</list>
    <list type="Command">!render.animation {*}</list>
    <list type="Command">app.quit</list>
</atom>

The first thing this does is load up the scene at c:\test.lxo. The leading exclamation point suppresses any dialogs that may open.

The Render.animation command actually renders the animation using the frame range set in the render item. This also uses a leading exclamation point to suppress dialogs. The asterisk passed as the second argument is a special case that causes the command to use the output items to decide what file formats to use, what paths to save to and what kinds of outputs to save.

The final command causes modo to quit. If you omit this on an unlicensed slave, the slave will open the "license not found" dialog after rendering is complete.

You can use the first and last channels of the render item to set the frame range you want rendered. As of modo 601, there is only one render item in the scene, with an item type of polyRender. The Select.itemType command selects all items of a particular type, so we can use it to select the single render item, and then use Item.channel to change the first and last frame. This example limits the rendering to only frame 60.

<list type="Command">select.itemType polyRender</list>
<list type="Command">item.channel first 60</list>
<list type="Command">item.channel last 60</list>

Advanced Render Control

This section is more for developers who want to control modo through another application.

Scripts as Startup Commands

Startup commands can also be used to execute scripts, since they are treated just like any other command. This script could start up its own event loop, listening for communications from another application on the local machine or the network, loading up scenes, changing scene settings and rendering frames.

Scripts can do fairly powerful things. For example, they can change the render outputs, file formats, output paths, show or hide objects, enable or disable materials, and so on. This allows for some very fine control by the render controller.

The only thing you won’t be able to do is easily abort a render in progress or get feedback while a frame is rendering. Your only option for aborting a render is to kill the modo process.

To avoid writing a config file from your program, you can also use the -cmd switch to pass a single command to modo. This command executes after all other startup commands have executed. This can be more convenient than startup commands, as you can pass arguments to your script when you execute modo itself.

Telnet

modo 501 and later supports an implementation of the Telnet protocol. This can be used to allow other programs to control modo with standard commands. Note that because telnet runs concurrently to modo, you will either need to run modo on a licensed machine or in headless mode.

Headless Mode

Eternal programs can control modo through headless mode. This can be done by using pipes to redirect stdin and stdout, or through modo's telnet implementation.

Not Saving the User Config on Quit

Executing modo with -dbon:noconfig will cause modo to read the user config on startup, but it will not save it on shutdown. This is useful if you plan on making a number of changes to the preferences or other state through a script, but you don't want those changes written to the user config when the application quits. This can be especially important if the machine you're running on is a user's machine and is using their user config, and you don't want to permanently change their settings.

Networking baking via render farm controller

Overview

Since bake items store everything needed to bake in the scene itself without relying on any temporary state such as item selections, it is feasible to use a render farm to bake. This works much like rendering an animation, but instead of rendering a sequence of frames you render a sequence of bakes. Each bake item can result in one of more "bake processes," each of which correspond to a single number.

In order to bake using your render farm controller, you may need to edit the render farm controller software's render scripts so that the modo rendering scripts fire the right commands.

How to use

Baking happens via the bakeitem.bake command, which takes an optional sequence argument. The sequence argument tells it to bake a single bake item in the scene. In order to render the 5th bake in a scene, simply fire the command bakeItem.bake sequence:4 (note that counting starts at zero). This can be done from command-line MODO, sent via Telnet, etcetera. Once the baking is complete, we need to make sure images are saved via !clip.saveAll. That's it!

In order to find out how many bakes there are in a scene, run the command bakeItem.count — this will open a dialogue with the number of bakes in the scene. If that number is 10, that means you want to bake the sequence numbers from 0 to 9. This command can also be queried: bakeItem.count ?

Example code

Stand-alone Python rendering script

A very simple render script in Python could look something like this. A more sophisticated version would communicate via Telnet to keep track of multiple machines.

import subprocess
 
# Path to your modo_cl executable
modocl = "/Applications/MODO 10.0v1.app/Contents/MacOS/modo_cl" 
 
def BakeSegment(scene, start, end):
    p = subprocess.Popen(modocl,
                        stdin   = subprocess.PIPE,
                        stdout  = subprocess.PIPE,
                        stderr  = subprocess.PIPE,
                        universal_newlines=True,
                        shell   = False)
 
    # Load the scene    
    p.stdin.write("scene.open {%s}\n" % scene)
    for seq in range(start, end+1):
        # Bake the sequence number
        p.stdin.write("bakeItem.bake sequence:%s\n" % seq)
    # Save all images when done
    p.stdin.write("!clip.saveAll\n")
 
    # Quit MODO
    p.stdin.write("!app.quit\n")
 
# Call the function with a path to a scene, as well as a sequence start/end
BakeSegment("/Users/Shared/Shared Documents/SimpleBake/Scenes/simplebaketest.lxo", 0, 3)

Customizing render farm controllers

  • Deadline.
  • Royal Render: Customization / Rendering / Custom tile render -- follow these instrucitons, but instead of duplicating the SoftImage render application, duplicate the modo one, and replace the render commands with baking commands as per the script above.


Extended Render Progress Information

As of modo 601, rendering can provide extended progress information to external applications through a rolling log system. This is primarily useful for when other applications such as render controllers are controlling modo through headless mode or telnet.

To enable extended output, execute the following command to output rolling logs to the console output:

log.toConsoleRolling true

Here is example output from modo rendering an empty scene from headless mode with logging to the console enabled.

@start  modo_cl [12345]  Luxology LLC
> log.toConsoleRolling true
+ ok
> render
! (renderProgress) Initalizing Render | @:Sat Mar 31 11:25:01 2012
! (renderProgress) Info | i:initTableau [Initializing tableau...]| @:Sat Mar 31 11:25:01 2012
! (renderProgress) Info | i:initGeometryCache [Initializing geometry cache...]| @:Sat Mar 31 11:25:01 2012
! (renderProgress) Info | i:initLightCache [Initializing light cache...]| @:Sat Mar 31 11:25:01 2012
! (renderProgress) Info | i:initVolumeCache [Initializing volume cache...]| @:Sat Mar 31 11:25:01 2012
! (renderProgress) Info | i:initPhotoMap [Initializing photon map...]| @:Sat Mar 31 11:25:01 2012
! (renderProgress) Render Begin | renderAs:frame | firstFrame:0 | lastFrame:0 | frameStep:1 | scenePath: | @:Sat Mar 31 11:25:01 2012
! (renderProgress)     Frame Begin | frame:0 | @:Sat Mar 31 11:25:01 2012
! (renderProgress)         RenderPass Begin | renderPass:1/1 | eye:mono | renderPassName: | @:Sat Mar 31 11:25:01 2012
! (renderProgress)             FramePass Begin | framePass:1/1 | @:Sat Mar 31 11:25:01 2012
! (renderProgress)                 Info | i:rendering [Rendering...]| @:Sat Mar 31 11:25:01 2012
! (renderProgress)                 Rendering | sceneProgress: 2.96% | frame:0,  2.96% | renderPass:1/1,  2.96% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:02 2012
! (renderProgress)                 Rendering | sceneProgress: 9.19% | frame:0,  9.19% | renderPass:1/1,  9.19% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:02 2012
! (renderProgress)                 Rendering | sceneProgress:15.70% | frame:0, 15.70% | renderPass:1/1, 15.70% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:02 2012
! (renderProgress)                 Rendering | sceneProgress:21.93% | frame:0, 21.93% | renderPass:1/1, 21.93% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:02 2012
! (renderProgress)                 Rendering | sceneProgress:28.15% | frame:0, 28.15% | renderPass:1/1, 28.15% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:02 2012
! (renderProgress)                 Rendering | sceneProgress:34.18% | frame:0, 34.18% | renderPass:1/1, 34.18% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:02 2012
! (renderProgress)                 Rendering | sceneProgress:40.59% | frame:0, 40.59% | renderPass:1/1, 40.59% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:02 2012
! (renderProgress)                 Rendering | sceneProgress:46.88% | frame:0, 46.88% | renderPass:1/1, 46.88% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:02 2012
! (renderProgress)                 Rendering | sceneProgress:53.04% | frame:0, 53.04% | renderPass:1/1, 53.04% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:03 2012
! (renderProgress)                 Rendering | sceneProgress:59.26% | frame:0, 59.26% | renderPass:1/1, 59.26% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:03 2012
! (renderProgress)                 Rendering | sceneProgress:65.57% | frame:0, 65.57% | renderPass:1/1, 65.57% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:03 2012
! (renderProgress)                 Rendering | sceneProgress:71.70% | frame:0, 71.70% | renderPass:1/1, 71.70% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:03 2012
! (renderProgress)                 Rendering | sceneProgress:77.90% | frame:0, 77.90% | renderPass:1/1, 77.90% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:03 2012
! (renderProgress)                 Rendering | sceneProgress:84.15% | frame:0, 84.15% | renderPass:1/1, 84.15% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:03 2012
! (renderProgress)                 Rendering | sceneProgress:90.28% | frame:0, 90.28% | renderPass:1/1, 90.28% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:03 2012
! (renderProgress)                 Rendering | sceneProgress:96.51% | frame:0, 96.51% | renderPass:1/1, 96.51% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:04 2012
! (renderProgress)                 Rendering | sceneProgress:100.00% | frame:0, 100.00% | renderPass:1/1, 100.00% | framePass:1/1 | eye:mono | @:Sat Mar 31 11:25:04 2012
! (renderProgress)             FramePass Complete. | @:Sat Mar 31 11:25:04 2012
! (renderProgress)         RenderPass complete. | renderPassRenderTime:2.434 [2.4s] | @:Sat Mar 31 11:25:04 2012
! (renderProgress)           Stat | res:720x480
! (renderProgress)           Stat | numThreads:8
! (renderProgress)           Stat | aaSamples:8
! (renderProgress)           Stat | numSurfaces:0
! (renderProgress)           Stat | numSegments:0
! (renderProgress)           Stat | numVertices:0
! (renderProgress)           Stat | numPolygons:0
! (renderProgress)           Stat | numNodes:1
! (renderProgress)           Stat | memGeometryCache:144  [0.14 KB]
! (renderProgress)           Stat | memLightCache:104 [0.10 KB]
! (renderProgress)           Stat | memIrradianceCache:0 [0.00 KB]
! (renderProgress)           Stat | memBucket:16884352 [16.1 MB]
! (renderProgress)           Stat | memFrame:5529600 [5.27 MB]
! (renderProgress)           Stat | numBuckets:345 (15x23)
! (renderProgress)     Frame complete. | frameRenderTime:2.434 [2.4s] |@:Sat Mar 31 11:25:04 2012
! (renderProgress) Render complete | sceneRenderTime:2.434 [2.4s] | @:Sat Mar 31 11:25:04 2012
! (renderProgress) Info | i:cleaningUp [Cleaning up...]| @:Sat Mar 31 11:25:04 2012
+ ok
>

See the headless mode article for more information about using headless mode and an explanation of the prefixes at the beginning of each line.

All lines share a standard format. For example:

! (renderProgress)           Stat | memGeometryCache:144  [0.14 KB]
  • The leading exclamation point indicates that this is a logged event, with (renderProgress) being the name of the logged system.
  • Stat is the data header. Single line entries will just be identified with the one word, while longer operations that have sub-sections will be followed with "Begin" and "Complete".
  • A vertical bar separates the data header from the actual data. The exact data is different for each row, but pieces of data always take the form of "name:value [user]", where name is a machine-readable name of the data and value is the value of the data. The square braces are optional, and if present will contain a human-readable version of the value.
  • All lines except for statistics end with a timestamp prefixed with @:. These timestamps are generated with the standard C asctime() function.

Here are some of the common lines that are output to the log:

  • Info: An information message. The 'i' data is the internal name of the value, while the square braces contain a localized string suitable for display in a user interface.
  • Stat: A single statistic about the render. These are the only values that do not have timestamps.
  • Rendering: Updated frequently, these lines provide real-time progress about the render. The various pieces of data are statistics that can be used to provide feedback in a user interface.
  • Begin/Complete: These indicate when a render, frame, render pass, or frame pass is beginning or has completed. The lines inside a Begin/Complete pair will be indented for readability.

Using log.toConsoleRolling will also enable the messageDialogs output, which contains the human-readable output of various dialogs that would open in the full version of modo, but which are suppressed in headless mode.

Render Controller Helper Scripts

The Render Controller Helper Scripts category includes some scripts demonstrating common rendering tasks.

More Information