Serializing State

Serializing State

Serializing Program State Between Steps

A running Pipeline keeps two records in its build directory. program.dat is the serialized CPS program, rewritten at step boundaries. The workflow/ directory holds one XML file per flow node, the graph of steps and blocks executed so far, which the Stage View, the Pipeline Graph View 163 and the console all read. This listing looked inside build 3 of the durable-demo job of Restart Mid-Build while its sh step was running:

The state of a running Pipeline on diskShell

docker exec l2-jenkins sh -c 'cd /var/jenkins_home/jobs/durable-demo/builds/3 &&
  ls -l program.dat && ls workflow | xargs && grep descriptorId workflow/6.xml'

Output

-rw-r--r-- 1 jenkins jenkins 8465 Sep 25 13:48 program.dat
2.xml 3.xml 4.xml 5.xml 6.xml
    <descriptorId>org.jenkinsci.plugins.workflow.steps.durable_task.ShellStep</descriptorId>

Flow node 6 is the sh step. When the build finishes, program.dat is deleted and the nodes are compacted into workflow-completed/flowNodeStore.xml.

Serialization has a consequence you will meet sooner or later: every local variable in scope at a step boundary must be Serializable. A regular-expression Matcher is not:

cps-serial: a Matcher still in scope when the sh step startsGroovy

node('linux') {
  def m = ('booknest-1.4.2' =~ /(\d+)\.(\d+)\.(\d+)/)
  echo "Major version: ${m[0][1]}"
  sh 'echo "a step boundary"'
}

Output

Major version: 1
...
agent-1 seems to be removed or offline (java.io.IOException: cannot find current thread in ...
...
Caused: java.io.NotSerializableException: java.util.regex.Matcher
Finished: FAILURE

The shell command ran; saving the program afterwards failed, and the first symptom was a misleading "agent seems to be removed or offline." Search such logs for NotSerializableException, then set the variable to null before the next step or confine it to a @NonCPS method (The Groovy Sandbox) that returns only a string.

MENU