Nuke Tips – Masking in Nuke

nuke_masking_header

The Matte in the Nuker Mask

Before I start, I could not remember who and where did I get the advice to avoid relying the mask input in Nuke.

Just a nagging feeling when I’m new to Nuke and in the end, abuse the mask function a lot where applicable.

So the idea of masking in Nuke is similar to the Mask/Track Matte function in After Effects. If you understand how alpha works, masking should be straight forward.

Quick checklist for Masking in Nuke

  1. Masking uses the same method like how alpha works. White is opaque while black is transparent.
  2. Nuke allows you to choose your desire channel when applying a mask. You can use existing channel from upstream or if the node supports it, the Mask input.
  3. Be warned that masking is not recommended for some effects which I’ll explained more below.

How masking works

nuke_masking_demo1

^The above GIF shows a straightforward process of using the mask input in Nuke. If you see a tiny arrowhead at the side of a node, 99.99% that it will be for mask input.

Choosing your channels

nuke_masking_custom_channellayer

If you planning to reuse a set of masks frequently in your comp, I recommend creating a set of channels/layers and copy it to the node stream.

When not to use mask

nuke_masking_defocus_demo

As shown in the GIF, be very careful when you think of masking an effect like Defocus.

Soft edges in the mask will not blend the effect properly as it combines the the Defocus effect and the original plate. So instead of a proper transition of strong to softer blurring, we see the part of the blur effect being composite back onto the original plate.

If the node doesn’t support channel that drives the amount of effect (e.g. Defocus, Blur), look for alternative node instead.

Here’s the sample script use for this demo

set cut_paste_input [stack 0]
version 9.0 v5b43
StickyNote {
 inputs 0
 name StickyNote1
 label "You can use this as\na form of garbage\nmatte through this method.\n\nJust make sure to label\nproperly if you work in a\nteam environment."
 note_font "Verdana Italic"
 note_font_size 16
 selected true
 xpos 258
 ypos 97
}
add_layer {taukeke taukeke.mask none taukeke.triangle}
Roto {
 inputs 0
 output {taukeke.mask none -taukeke.triangle}
 curves {{{v x3f99999a}
  {f 0}
  {n
   {layer Root
    {f 2097152}
    {t x44700000 x44070000}
    {a pt1x 0 pt1y 0 pt2x 0 pt2y 0 pt3x 0 pt3y 0 pt4x 0 pt4y 0 ptex00 0 ptex01 0 ptex02 0 ptex03 0 ptex10 0 ptex11 0 ptex12 0 ptex13 0 ptex20 0 ptex21 0 ptex22 0 ptex23 0 ptex30 0 ptex31 0 ptex32 0 ptex33 0 ptof1x 0 ptof1y 0 ptof2x 0 ptof2y 0 ptof3x 0 ptof3y 0 ptof4x 0 ptof4y 0 pterr 0 ptrefset 0 ptmot x40800000 ptref 0}
    {curvegroup Ellipse1 512 bezier
     {{cc
       {f 8192}
       {px x42140000
        {xc1ee9640 0}
        {x44450000 x43620000}
        {x41ee9640 0}
        {0 xc20256d4}
        {x44528000 x438e8000}
        {0 x420256d8}
        {x41ee9640 0}
        {x44450000 x43ac0000}
        {xc1ee9640 0}
        {0 x420256d8}
        {x44378000 x438e8000}
        {0 xc20256d4}}}
      {cc
       {f 8192}
       {p
        {{{x42140000 xc1ee9640}}
      {{x42140000 0}}}
        {{{x42140000 0}}
      {{x42140000 0}}}
        {{{x42140000 x41ee9640}}
      {{x42140000 0}}}
        {{{x42140000 0}}
      {{x42140000 xc20256d4}}}
        {{{x42140000 0}}
      {{x42140000 0}}}
        {{{x42140000 0}}
      {{x42140000 x420256d8}}}
        {{{x42140000 x41ee9640}}
      {{x42140000 0}}}
        {{{x42140000 0}}
      {{x42140000 0}}}
        {{{x42140000 xc1ee9640}}
      {{x42140000 0}}}
        {{{x42140000 0}}
      {{x42140000 x420256d8}}}
        {{{x42140000 0}}
      {{x42140000 0}}}
        {{{x42140000 0}}
      {{x42140000 xc20256d4}}}}}}
     {tx x42140000 x44450000 x438e8000}
     {a bm x41400000 osw x41200000 osf 0 str 1 spx x44700000 spy x44070000 sb 1 ltn x42140000 ltm x42140000 tt x40e00000}}
    {curvegroup Bezier4 512 bezier
     {{cc
       {f 8192}
       {px x42140000
        {0 0}
        {x44a90000 x43ee0000}
        {0 0}
        {0 0}
        {x44b54000 x43ee8000}
        {0 0}
        {0 0}
        {x44b54000 x43530000}
        {0 0}
        {0 0}
        {x44c16000 x439d0000}
        {0 0}
        {0 0}
        {x44cb8000 x439d8000}
        {0 0}
        {0 0}
        {x44bd8000 x432c0000}
        {0 0}
        {0 0}
        {x44c78000 x42800000}
        {0 0}
        {0 0}
        {x44bdc000 x42240000}
        {0 0}
        {0 0}
        {x44b62000 x43010000}
        {0 0}
        {0 0}
        {x44b5e000 x42240000}
        {0 0}
        {0 0}
        {x44a84ccd x421d9998}
        {0 0}}}     idem}
     {tx x42140000 x44b94e8c x435245d1}
     {a osw x41200000 osf 0 str 1 spx x44700000 spy x44070000 sb 1 ltn x42140000 ltm x42140000 tt x40800000}}
    {curvegroup Bezier3 512 bezier
     {{cc
       {f 8192}
       {px x42140000
        {0 0}
        {x44930000 x43ba0000}
        {0 0}
        {0 0}
        {x449ac000 x43ba0000}
        {0 0}
        {x42ac0000 x40800000}
        {x448c0000 x43e60000}
        {xc2ac0000 xc0800000}
        {xc1b00000 x42480000}
        {x44768000 x43a50000}
        {x41b00000 xc2480000}
        {xb9000000 x4257ffe8}
        {x448ec000 x435e0000}
        {0 xc2dc0000}
        {x41f00000 xc2840000}
        {x44834000 x43120000}
        {xc1f80000 1}
        {0 0}
        {x44748000 x43140000}
        {0 0}
        {xc2ac0000 xc0000000}
        {x4486c000 x42780000}
        {x42ac0000 x40000000}
        {xc1600000 xc2400000}
        {x44964000 x43280000}
        {x41600000 x42400000}
        {x42e80000 xc1e00000}
        {x448a0000 x439e0000}
        {xc2e80000 x41e00000}
        {xc1c15480 x38000000}
        {x448a0000 x43ca0000}
        {x42900000 0}}}     idem}
     {tx x42140000 x448a62e9 x43885d17}
     {a osw x41200000 osf 0 str 1 spx x44700000 spy x44070000 sb 1 ltn x42140000 ltm x42140000 tt x40800000}}
    {curvegroup Bezier2 512 bezier
     {{cc
       {f 8192}
       {px x42140000
        {0 0}
        {x44558000 x43c30000}
        {0 0}
        {0 0}
        {x44508000 x43e10000}
        {0 0}
        {0 0}
        {x44668000 x43e30000}
        {0 0}
        {0 0}
        {x446a0000 x431c0000}
        {0 0}
        {0 0}
        {x445b0000 x431a0000}
        {0 0}
        {0 0}
        {x44578000 x434c0000}
        {0 0}
        {x42780000 xc1c00000}
        {x44348000 x43260000}
        {xc2780000 x41c00000}
        {xc2c40000 xc1a00000}
        {x442c0000 x43c70000}
        {x42c40000 x41a00000}}}     idem}
     {tx x42140000 x44513000 x43944000}
     {a osw x41200000 osf 0 str 1 spx x44700000 spy x44070000 sb 1 ltn x42140000 ltm x42140000 tt x40800000}}
    {curvegroup Bezier1 512 bezier
     {{cc
       {f 8192}
       {px x42140000
        {0 0}
        {x430e0000 x43a20000}
        {0 0}
        {0 0}
        {x43580000 x43ac0000}
        {0 0}
        {0 0}
        {x43740000 x438b0000}
        {0 0}
        {xc2a80004 x427e6660}
        {x43c56667 x43a13334}
        {x421cccc8 x42ef3330}
        {xc1900000 x41c00000}
        {x44048000 x43b30000}
        {x41900000 xc1c00000}
        {0 0}
        {x44180000 x432a0000}
        {0 0}
        {0 x40800000}
        {x440c8000 x43200000}
        {0 xc0800000}
        {x41e00000 x42b80000}
        {x43dd0000 x43880000}
        {xc0c00000 xc0000000}
        {0 0}
        {x43d90000 x436e0000}
        {0 0}
        {0 0}
        {x43be0000 x43660000}
        {0 0}
        {x42080000 x41000000}
        {x43a2cccb x43853334}
        {xc2080000 xc1000000}
        {xc1c00000 x42600000}
        {x43a90002 x42e1999c}
        {xc22b3330 xc1a00000}
        {0 0}
        {x437b333a x428d999b}
        {0 0}}}     idem}
     {tx x42140000 x43bb1b91 x437656a5}
     {a osw x41200000 osf 0 str 1 spx x44700000 spy x44070000 sb 1 ltn x42140000 ltm x42140000 tt x40800000}}}}}}
 toolbox {selectAll {
  { selectAll str 1 ssx 1 ssy 1 sf 1 }
  { createBezier str 1 ssx 1 ssy 1 sf 1 sb 1 tt 4 }
  { createBezierCusped str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { createBSpline str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { createEllipse str 1 ssx 1 ssy 1 sf 1 sb 1 tt 7 }
  { createRectangle str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { createRectangleCusped str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { brush str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { eraser src 2 str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { clone src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { reveal src 3 str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { dodge src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { burn src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { blur src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { sharpen src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { smear src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
} }
 toolbar_brush_hardness 0.200000003
 toolbar_source_transform_scale {1 1}
 toolbar_source_transform_center {960 540}
 colorOverlay {0 0 0 0}
 blending_mode minus
 lifetime_type "all frames"
 lifetime_start 37
 lifetime_end 37
 motionblur_shutter_offset_type centred
 source_black_outside true
 name Roto2
 selected true
 xpos 280
 ypos 327
}
set N82445c00 [stack 0]
Dot {
 name Dot14
 label "For this example, I created a\nnew channel called \"taukeke\"\nwhich I need to copy its layer\ninto the stream using the\nCOPY node for use later\nin the downstream"
 note_font "Verdana Bold Bold Bold Bold"
 note_font_size 16
 note_font_color 0xffffffff
 selected true
 xpos 314
 ypos 463
}
Dot {
 name Dot8
 selected true
 xpos 314
 ypos 574
}
push $N82445c00
Roto {
 inputs 0
 output alpha
 curves {{{v x3f99999a}
  {f 0}
  {n
   {layer Root
    {f 2097152}
    {t x44700000 x44070000}
    {a pt1x 0 pt1y 0 pt2x 0 pt2y 0 pt3x 0 pt3y 0 pt4x 0 pt4y 0 ptex00 0 ptex01 0 ptex02 0 ptex03 0 ptex10 0 ptex11 0 ptex12 0 ptex13 0 ptex20 0 ptex21 0 ptex22 0 ptex23 0 ptex30 0 ptex31 0 ptex32 0 ptex33 0 ptof1x 0 ptof1y 0 ptof2x 0 ptof2y 0 ptof3x 0 ptof3y 0 ptof4x 0 ptof4y 0 pterr 0 ptrefset 0 ptmot x40800000 ptref 0}
    {curvegroup Bezier5 512 bezier
     {{cc
       {f 8192}
       {px 1
        {0 0}
        {x44576000 x44214000}
        {0 0}
        {0 0}
        {x4461a000 x43ecc000}
        {0 0}
        {0 0}
        {x44562000 x43ea8000}
        {0 0}}}     idem}
     {tx 1 x445a6000 x44044aab}
     {a bm x41400000 osw x41200000 osf 0 str 1 spx x44700000 spy x44070000 sb 1 tt x40800000}}
    {curvegroup Bezier4 512 bezier
     {{cc
       {f 8192}
       {px 1
        {0 0}
        {x44b2c000 x44638000}
        {0 0}
        {0 0}
        {x44be0000 x44670000}
        {0 0}
        {0 0}
        {x44c40000 x44180000}
        {0 0}
        {0 0}
        {x44d18000 x44668000}
        {0 0}
        {0 0}
        {x44de0000 x446c0000}
        {0 0}
        {0 0}
        {x44cdc000 x44100000}
        {0 0}
        {0 0}
        {x44e50000 x43880000}
        {0 0}
        {0 0}
        {x44d50000 x43880000}
        {0 0}
        {0 0}
        {x44c50000 x44040000}
        {0 0}
        {0 0}
        {x44c64000 x436c0000}
        {0 0}
        {0 0}
        {x44bb0000 x436c0000}
        {0 0}}}     idem}
     {tx 1 x44ca62e9 x44121746}
     {a osw x41200000 osf 0 str 1 spx x44700000 spy x44070000 sb 1 tt x40800000}}
    {curvegroup Bezier3 512 bezier
     {{cc
       {f 8192}
       {px 1
        {0 0}
        {x4498c000 x44348000}
        {0 0}
        {0 0}
        {x44a4c000 x44390000}
        {0 0}
        {x42f00000 x41c00000}
        {x448c8000 x44540000}
        {xc2f00000 xc1c00000}
        {xc2f80000 x40800000}
        {x44878000 x44080000}
        {x42f80000 xc0800000}
        {xc1f00000 x42a80000}
        {x44a04000 x43e60000}
        {x41f00000 xc2a80000}
        {0 0}
        {x448a4ccd x43903333}
        {0 0}
        {0 0}
        {x448dcccd x43686666}
        {0 0}
        {x42280000 xc2e40000}
        {x44ac4000 x43e50000}
        {xc2280000 x42e40000}
        {x42f00000 xc1900000}
        {x448c8000 x441a0000}
        {xc2f00000 x41900000}
        {xc2180000 x40c00000}
        {x448ec000 x443f0000}
        {x42180000 xc0c00000}}}     idem}
     {tx 1 x44961333 x440ee666}
     {a osw x41200000 osf 0 str 1 spx x44700000 spy x44070000 sb 1 tt x40800000}}
    {curvegroup Bezier2 512 bezier
     {{cc
       {f 8192}
       {px 1
        {0 0}
        {x44488000 x44478000}
        {0 0}
        {0 0}
        {x443a8000 x43400000}
        {0 0}
        {0 0}
        {x44488000 x43440000}
        {0 0}
        {0 0}
        {x44550000 x43ca0000}
        {0 0}
        {0 0}
        {x44668000 x43d00000}
        {0 0}
        {0 0}
        {x44748000 x43460000}
        {0 0}
        {0 0}
        {x4483c000 x43520000}
        {0 0}
        {0 0}
        {x445b8000 x44498000}
        {0 0}}}     idem}
     {tx 1 x445bd000 x43c94000}
     {a osw x41200000 osf 0 str 1 spx x44700000 spy x44070000 sb 1 tt x40800000}}
    {curvegroup Bezier1 512 bezier
     {{cc
       {f 8192}
       {px 1
        {0 0}
        {x43280000 x44510000}
        {0 0}
        {0 0}
        {x43830000 x44568000}
        {0 0}
        {0 0}
        {x43ca0000 x44180000}
        {0 0}
        {0 0}
        {x43db0000 x445e0000}
        {0 0}
        {0 0}
        {x441c8000 x44600000}
        {0 0}
        {0 0}
        {x44210000 x430a0000}
        {0 0}
        {0 0}
        {x44050000 x430e0000}
        {0 0}
        {0 0}
        {x44050000 x43ff0000}
        {0 0}
        {0 0}
        {x43dd0000 x43c90000}
        {0 0}
        {0 0}
        {x43b20000 x44030000}
        {0 0}
        {0 0}
        {x439d0000 x432c0000}
        {0 0}
        {0 0}
        {x43420000 x43340000}
        {0 0}}}     idem}
     {tx 1 x43ccaaab x44003555}
     {a osw x41200000 osf 0 str 1 spx x44700000 spy x44070000 sb 1 tt x40800000}}}}}}
 toolbox {selectAll {
  { selectAll str 1 ssx 1 ssy 1 sf 1 }
  { createBezier str 1 ssx 1 ssy 1 sf 1 sb 1 tt 4 }
  { createBezierCusped str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { createBSpline str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { createEllipse str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { createRectangle str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { createRectangleCusped str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { brush str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { eraser src 2 str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { clone src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { reveal src 3 str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { dodge src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { burn src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { blur src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { sharpen src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
  { smear src 1 str 1 ssx 1 ssy 1 sf 1 sb 1 }
} }
 toolbar_brush_hardness 0.200000003
 toolbar_source_transform_scale {1 1}
 toolbar_source_transform_center {960 540}
 colorOverlay {0 0 0 0}
 blending_mode minus
 lifetime_type "all frames"
 motionblur_shutter_offset_type centred
 source_black_outside true
 name Roto1
 selected true
 xpos 305
 ypos -128
}
Dot {
 name Dot12
 label "The MASK\nroto shape output\nas alpha"
 note_font "Verdana Bold Bold Bold"
 note_font_size 16
 note_font_color 0xffffffff
 selected true
 xpos 339
 ypos -54
}
Dot {
 name Dot11
 selected true
 xpos 339
 ypos 30
}
Constant {
 inputs 0
 channels alpha
 color 1
 name Constant1
 selected true
 xpos -367
 ypos -262
}
ColorBars {
 inputs 0
 PAL true
 name ColorBars1
 selected true
 xpos -221
 ypos -384
}
Copy {
 inputs 2
 from0 rgba.alpha
 to0 rgba.alpha
 name Copy1
 selected true
 xpos -221
 ypos -238
}
Dot {
 name Dot9
 label "I copy a white alpha\nchannel into this stream\nsince the ColorBars don't\ngenerate it's own Alpha\nchannel"
 note_font "Verdana Bold"
 note_font_size 16
 note_font_color 0xffffffff
 selected true
 xpos -187
 ypos -125
}
Dot {
 name Dot1
 selected true
 xpos -187
 ypos 30
}
push $cut_paste_input
Ramp {
 output {rgba.red rgba.green rgba.blue -rgba.alpha}
 replace true
 p0 {100 -12}
 p1 {100 908}
 type plinear
 color {1 0.417224 0 1}
 color_panelDropped true
 name Ramp1
 selected true
 xpos 65
 ypos -214
}
Dot {
 name Dot10
 label "Just a regular\nramp aka gradient"
 note_font "Verdana Bold Bold"
 note_font_size 16
 note_font_color 0xffffffff
 selected true
 xpos 99
 ypos -129
}
Merge2 {
 inputs 2+1
 name Merge1
 selected true
 xpos 65
 ypos 26
}
Dot {
 name Dot13
 label "The Merge node\nis masked using\nthe mask input\nconnected to the\nMASK roto"
 note_font "Verdana Bold Bold Bold"
 note_font_size 16
 note_font_color 0xffffffff
 selected true
 xpos 99
 ypos 127
}
Grade {
 inputs 1+1
 white {0 0.556425 1 1}
 white_panelDropped true
 gamma 5
 maskChannelMask taukeke.mask
 name Grade2
 selected true
 xpos 65
 ypos 333
}
Copy {
 inputs 2
 from0 -taukeke.mask
 to0 -taukeke.mask
 channels taukeke
 name Copy2
 selected true
 xpos 65
 ypos 557
}
Dot {
 name Dot7
 selected true
 xpos 99
 ypos 659
}
ColorWheel {
 inputs 0
 gamma 0.45
 name ColorWheel1
 selected true
 xpos 862
 ypos -225
}
Dot {
 name Dot15
 label "ColorWheel is a great\nexample to demo the\nRED GREEN BLUE\nchannel as mask"
 note_font "Verdana Bold Bold Bold Bold"
 note_font_size 16
 note_font_color 0xffffffff
 selected true
 xpos 896
 ypos -66
}
Dot {
 name Dot2
 selected true
 xpos 896
 ypos 55
}
set N10566800 [stack 0]
Dot {
 name Dot4
 selected true
 xpos 1013
 ypos 55
}
Dot {
 name Dot16
 label "The RED channel is\nused here as the\nmask for Grade1"
 note_font "Verdana Bold Bold Bold Bold Bold"
 note_font_size 16
 note_font_color 0xffffffff
 selected true
 xpos 1013
 ypos 124
}
Dot {
 name Dot3
 selected true
 xpos 1013
 ypos 196
}
set N10566400 [stack 0]
Dot {
 name Dot17
 label "The BLUE channel is\nused here as the\nmask for ColorCorrect1"
 note_font "Verdana Bold Bold Bold Bold Bold Bold"
 note_font_size 16
 note_font_color 0xffffffff
 selected true
 xpos 1013
 ypos 270
}
Dot {
 name Dot5
 selected true
 xpos 1013
 ypos 360
}
set N9e5df800 [stack 0]
Dot {
 name Dot18
 label "The GREEN channel is\nused here as the\nmask for HueShift1\n\nSo you noticed the copy\npasta I mean paste\ndescription. :)"
 note_font "Verdana Bold Bold Bold Bold Bold Bold"
 note_font_size 16
 note_font_color 0xffffffff
 selected true
 xpos 1013
 ypos 429
}
Dot {
 name Dot6
 selected true
 xpos 1013
 ypos 517
}
push $N9e5df800
push $N10566400
push $N10566800
Grade {
 inputs 1+1
 multiply {{curve x1 0 x50 1 x100 0}}
 maskChannelMask rgba.red
 name Grade1
 selected true
 xpos 862
 ypos 192
}
ColorCorrect {
 inputs 1+1
 saturation {{curve x1 1 x50 0 x100 1}}
 maskChannelMask rgba.green
 name ColorCorrect1
 selected true
 xpos 862
 ypos 356
}
HueShift {
 inputs 1+1
 hue_rotation {{frame*2}}
 maskChannelMask rgba.blue
 name HueShift1
 selected true
 xpos 862
 ypos 513
}
Merge2 {
 inputs 2
 also_merge taukeke
 name Merge2
 selected true
 xpos 862
 ypos 655
}
Dot {
 name Dot19
 label "The Defocus node uses the\ntaukeke.mask channel as the\nmask input."
 note_font "Verdana Bold Bold Bold Bold Bold Bold Bold"
 note_font_size 16
 note_font_color 0xffffffff
 selected true
 xpos 897
 ypos 754
}
Defocus {
 channels rgb
 defocus 30
 ratio 2.65
 maskChannelInput taukeke.mask
 name Defocus1
 selected true
 xpos 862
 ypos 851
}

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.