Last week, Nuxeo released the Nuxeo Platform Fast Track 7.2 which introduced some major enhancements to the platform. One of the highlights of this release is the addition of server-side scripting for the Automation Service. This feature will make it much easier for you create advanced business logic in your applications using the Nuxeo Platform’s powerful Automation Service. In this blog, I will discuss Automation Chains in detail, why we introduced Automation Scripting, and how we implemented them. We will also look at the challenges we faced and how we overcame them. Finally, I will give you a glimpse of what’s coming next!

About Automation and Chains

The initial idea behind Automation in the Nuxeo Platform was to provide simple building blocks (Operations) and then let Nuxeo Studio users assemble these building blocks together to express the business logic they needed. This approach was quite successful and we saw a lot of Studio users who are not Java developers per se but are able to define their own business logic without needing help from a Java developer. The first assertion was that Automation Chains will be mainly used by non-Java developers doing simple things and that people with programming skills would prefer using Java code to design complex processes.

This first assertion was quickly proven wrong! Here’s why:

  • Business users started to address more and more complex business processes because of the success they had with our Automation
  • Even Java developers quickly learned that expressing their custom logic using Automation Chain was very convenient
  • The Nuxeo framework started making extensive use of Automation: the Workflow engine being a good example

Since the Automation Chains System was built keeping the simplicity in mind, the underlying concepts were also very simple: a chain is a simple pipeline of operations and execution is linear. But soon people wanted to express more complex business logic that required conditional statements or loops. They started to bend the Automation Chain to express their business logic and, to be fair, we provided the tools for:

  • Execution of sub chains (RunChain)
  • Execution of sub chain on a list of documents (RunOperationOnList)
  • RunScript Operation
  • Using ternary expressions in RunChain to achieve a conditional statement

Automation Chains

As a result, some people achieved very complex business processes using just Nuxeo Studio. We were very happy with that result! However, on taking a closer look we could clearly see that there was room for improvement. Typically, with some procedural concepts like conditional statements and loops the same logic can be expressed in a much clearer manner. For quite some time we have been thinking about addressing this issue in Automation because your pain and problems are ours to take care of (we make our living by providing you support!).

Automation Scripting

When we were thinking about how we could take Automation Chain functionality to the next level we looked into several options. We provided a YAML representation of chains inside Nuxeo Studio and we also did some experiments with Visual Programming tools like Blockly. Although we did not abandon the “visual programming” approach, based on the feedback we had from users we realized that plain scripting was a better option because:

  • Loops and conditions are easy to handle in script
  • Most people who wouldn’t dare write Java code are confident in writing JavaScript

So, we decided to build Automation Scripting based on JavaScript where Automation Chains would simply be exposed as primitives, pretty much like building a DSL for Automation. JavaScript can be used for handling the control flow and code structure while Automation Operations can provide the primitives. In the plain Automation Chain model, implementing a loop and a condition would imply using 3 chains. With the new model, it can be done in a few lines of script :

var doc_list = (Document.Query(null, {"query":'select * from Document',"currentPageIndex":0,"pageSize":20}));
for (var doc_index in doc_list) {
   var doc = doc_list[doc_index];
   if ((doc.getType()) == 'File') {
     Document.SetProperty(doc,{"xpath":'dc:description',"save":true,"value":'This is a File'});
   }
}

Another interesting aspect of JavaScript is that everything related to expressions in parameters becomes more natural since you can use the JavaScript syntax rather that MVEL!

Technical Implementation and Challenges

Nashorn Integration

Although the very first implementation of Automation Scripting was based on Rhino we quickly migrated to Java 8 and Nashorn because:

  • Mapping between Java and JavaScript objects (ScriptObjectMirror) is handled natively
  • This combination provides better control over execution without having to depend on restricted API

Technically, we exposed a new AutomationScriptingService that allows execution of JavaScript:

AutomationScriptingService scriptingService = Framework.getService(AutomationScriptingService.class);

InputStream stream = this.getClass().getResourceAsStream("/simpleAutomationScript.js");
scriptingService.run(stream, session);

JS Automation Lib and Precompilation

Automation Scripting primitives are the Operations that can be called directly. The Scripting does not use low level Java API but the Automation API to interact with the platform:

var docs = Repository.Query(null, { "query": "select * from Document where dc:nature='even'"});

Automation API is dynamic by nature, new modules can register new Operations or Chains that should then be exposed as new JavaScript functions. To handle that AutomationScriptingService will generate a small JavaScript library wrapping the native Java API. Basically, the idea is that each Operation will become a function attached to a JavaScript Object that is the category of the Operation.

Java definition

 <small>@Operation(id = "Document.SetProperty") public class SetDocumentProperty {

    @Param(name = "xpath")
    protected String xpath;

    @Param(name = "value", required = false)
    protected Serializable value;

    @Param(name = "save", required = false, values = "true")
    protected boolean save = true;

    @OperationMethod(collector = DocumentModelCollector.class)
    public DocumentModel run(DocumentModel doc) throws OperationException {
      //...
    }</small>

JavaScript definition as exposed by the JavaScript wrapper

Document.SetProperty = function(doc, params) { ... }
where params = { "xpath" : x , "value" : y, "save" : z };

This JavaScript wrapper is generated based on the actual registered Automation Operations. To avoid a costly compilation each time, the JS Wrapper is compiled once and then reused leveraging the Nashorn persistent-code-cache that was introduced in jdk8u25.

Isolation

In order to run several scripts in parallel without having context information bleeding between one context to another, the current implementation uses:

  • Per-thread pooled Scripting Engine
  • Creation of new ScriptingContext for each execution

Coupled with the precompilation, we have a very fast yet safe execution of Automation Scripting in the context of the JVM.

JavaScript based Automation Operation

The base building block of Automation is the Operation, the high level API that you can manipulate inside Nuxeo Studio and call from the outside using the REST API. Starting with Nuxeo Platform 7.2, we now have 3 ways of implementing an Operation:

  • Java Operation: java class with the @Operation annotation
  • Automation Chain: xml definition of a pipe of operations
  • Automation Scripting: JavaScript code implementing a function

This new flavor of Automation Operation is registered using the operation extension point of the new AutomationScriptingComponent:

<scriptedOperation id="Scripting.HelloWorld">
   <inputType>string</inputType>
   <outputType>string</outputType>
   <category>Scripting</category>
   <param name="lang" type="string"/>

   <script>
     function run(ctx, input, params) {
       if (params.lang === "fr") {
         return "Bonjour " + input;
       } else if (params.lang == "en") {
         return "Hello " + input;
       }
     }
   </script>
</scriptedOperation>

This means that in addition to providing a direct API to execute Automation Scripting from your Java code, you can also use Automation Scripting to build new Operations that you can use inside your chains, your listeners and your Workflows from Nuxeo Studio. Since a JavaScript-based operation is a first class citizen operating exactly like a Java-based Operation or a Chain, it can also be used inside Automation Scripting itself.

About Migration

The Challenge

Having a better tool for the future is good, however having a way to use it for existing chains would be even better. So, our goal is to provide a smooth migration path for users interested in simplifying some complex Automation Chains by leveraging Automation Scripting:

  • From the point of view of the rest of the application it will be invisible
  • From the point of view of the developers it would be a huge relief in terms of maintenance and complexity

The problem is that the paradigms are not the same (and that was actually the point), so there is no straight forward migration path. Even if we define some patterns, there will always be some code blocks that require reorganization.

Code blocks, did I say Code Blocks ?

The Return of Blockly

That is where Blockly comes back into play! Blockly is very good at providing a view on code blocks that can be plugged together: that’s pretty much one of the goals of visual programming. This means that even if the advantage of designing a complete Automation process using Blockly was not demonstrated (my kids liked it, but they are not the target!), for refactoring some code block in the middle of a paradigm migration it does make sense.

Nuxeo Automation Scripting: Blocky Editor

What’s Coming Next

Better Security Isolation

Nuxeo Platform 7.2 was aligned on Java 8 partially for Automation Scripting requirement of Nashorn. With the recently released JDK 8 update 40 there are several Nashorn related optimizations but more importantly this update introduces a Nashorn Class Filter that allows fine-grained control over access to Java classes from JavaScript code. This will allow making Automation Scripting safer. The idea would be to restrict access to Automation API and to a very small subset of Nuxeo API. This should be available in 7.3 thanks to NXP-16480!

Debugger and Step-by-Step Execution

One difficulty related to complex automation chains is that they may be complex to debug. Nuxeo Platform already provides extended logging capabilities and you can use Java Server-side debugging. However, one of the goals of Automation is to open the platform to “non-Java developers”. So, we are thinking about adding specific debugging features that do not rely on the Java debugger.

Some of the options we are exploring are:

Migration Tool

Work has already started on this and this tool is currently integrated as part of the Nuxeo Automation Scripting Blockly Editor.

Integrate Community Feedback

Of course, what we integrate regarding Automation Scripting in the next releases will be heavily influenced by the feedback we get from the Nuxeo community. So, don’t be shy! Download the Nuxeo Platform and test Automation Scripting using Nuxeo Studio integrate editor (with code completion). Test it and then, please, give us your feedback!

Try It Yourself!

Automation Scripting is part of Nuxeo Platform 7.2, so check it out! Here’s how you can get started:

Blog Tag: Insights Blog Tag: Product & Development