[Nuxeo Tech Report] News from the Developer Front #5


Thu 04 April 2013 By Laurent Doguin

Here's a brand new tech report taken straight from the developer meetings.


High Volume Infrastructure

Redis and Workmanager

Persistent WorkManager


Ben started using Redis to manage work queues.

The goal is to integrate Redis and the WorkManager so that:


  • async jobs can be shared between cluster nodes

  • work queues can be mono-threaded, even in a cluster environment

  • work queues can be persisted (and not lost at server shutdown)


For this to work for real we probably need to write a custom Redis WorkManager. You can follow the status of this task with NXROADMAP-137 - Provide a distributed workers infrastructure .

Nuxeo and Redis


The support for a persistent WorkManager should be an option: It will depend on Redis (that does not run 'easily' under Windows). When it is not activated, jobs are only defined in memory. To make it safer, we can at least warn before restarting the server: 'your server is currently processing xxx jobs and cannot be shut down right now...'.

We want to be able to use Redis for other use cases (like locks), so we need to share a nuxeo-redis-service that at least provides the configuration for connecting to Redis from Nuxeo.

NB: Ben will create the required Jira issues and associate to NXROADMAP-137

WorkManager and batch mode


In addition, recent benchmarks show that with the current WorkManager implementation we have an issue with mass import. It was working before the latest changes on threadpools, but currently the bench fails. Ben is investigating this point: first goal being to know if this is a test setup issue or a real code issue.

VCS / Soft Delete


Florent started to implement the Soft delete support in VCS. It means doing the deletes in 2 steps:


  • mark the deleted objects

  • delete marked objects in batch


This approach that avoids "on delete cascade" seems to be better for performance. This will also avoid consistency issues when docs are deleted by synchronous listeners but used by async listeners.

Florent started implementing this and added a new config in the repository descriptor. The code is almost ready for a first test. Once we have validated the principle, we can address the remaining tasks:

  • batch delete support
  • GC system

Publishing, Relations, Tags, Renditions, Versions


The publishing feature can be improved to address a few issues:


  • publish may duplicate some relations

  • tags are relations, but on versions, so they are not translated the right way

  • rendition publishing overrides the proxy (and then removes relations)


The issue is that we need to work on a specification about:


  • what should Nuxeo do

  • what is a missing feature

  • what is a bug

  • what is an evolution


Keeping in mind that this may all be a question of use case.


  • some people may want to keep/translate the tags when publishing (ex: using tag as a meta-data)

  • some other may want to reset the tags when publishing (ex: using tag as a validation flag)


Feedback and use cases are very welcome :)

Connect/Studio

Next Generation Connect UI


We are working on a new interface for the Nuxeo Online Services portal. An AngularJS based web UI for Connect "self-care" is starting to be usable. The associated roadmap task is NXROADMAP-148

Studio Release Schedule

2.10.2


Connect 1.14.1 and Studio 2.10.2 are now up.

2.11 / DAM:


The main focus is to add more DAM features (target is May).

2.12:


This is, at least for now, the Nuxeo 5.7 scope:


  • work on Workflows

  • work on Action + Filters

  • work on Directory Widget

  • fix technical debt


Automation


We have started organizing the work on automation. This work include several aspects:


We will also try to ensure convergence of other existing automation clients:


  • Python client used by Drive will be managed by Antoine

  • Objective C iOS client: see with Arnaud, Ineat and may be Smart&Soft

  • Android Client (there is some code we should reintegrate)


Vlad will be the lead developer on the automation topic, but he will get help from several people, especially:


  • Julien: on backward compat testing

  • Stephane: on debugging

  • Olivier: on marshaling

  • Damien: on REST binding


The roadmap and NXP Jiras have been updated accordingly: see NXROADMAP-139

Automation debug and error management


There are actually three different use cases we would like to address:


  • allow some exception bubbles to the client: 'get more than Operation XXX failed'

  • provide a way to manage exceptions at the chain level: 'try to execute this operation and otherwise execute an other one...'

  • provide a way to debug an automation chain: 'see what was inside the context when operation X was called...'


Even if these three topics are related, technically the will require separated tasks.

See NXP-11281

Bubble exceptions to client


The idea is that someexceptions should be sent to the client without being wrapped because they contain a message that can be meaningful to the client. We can have two approaches for that:


  • use the RecoverableClientException that was introduced for Quota system

  • use explicit exception declaration in operation method signature


cf NXP-11283

Automation chain debug


Stephane started some work for support about capturing automation context to help debugging.

Jira task was updated: NXP-10081

Automation and REST binding


cf NXP-11284

Why is automation not RESTful ?


Automation is more a RPC system than a Rest binding. It does actually use JAXRS, but the Rest resource is the operation itself:


  • calling GET will give you the definition

  • calling POST will execute it

  • PUT would create an operation

  • DELETE would remove an operation from the server

Why is it an issue ?


Having a simple RESTful CRUD API would help:


  • simple testing

  • integration with Javascript MVC model

Build a CRUD RESTful API on document and adapter


Here we are talking about reverting the automation paradigm:


  • the resource is the document and not the operation

  • the 'Operation' is the Http Verb and is limited to CRUD


We could build that as an extension of the existing content automation API. This basically means that automation exposes three endpoints:


  • Operation RPC endpoint (/site/automation/${operationId})

  • Operation RPC 'batch endpoint' (/site/automation/batch)

  • CRUD (/site/automation/CRUD/${resource})


This new CRUD API should focus on CRUD on a DocumentModel or it's adapter. Contrary to operations, CRUD API cannot be combined inside an automation chain and doesn't need to appear inside Studio.

NB: This could also be a good opportunity to add the Blob Download API inside this Rest binding

Metrics/Graphite

Status


Integration is merged in 5.7-SNAPSHOT. Cleanup still needs to be done:


  • remove Simon related stuff

  • remove Probes (should be replaced by Metrics Health Check)

Documentations and configuration


A draft documentation has been added. Need to add setup documentation NXP-11164. Next bench will show us if we need to add more stuff, either inside Nuxeo (counters) or inside Graphite.

Infrastructure work

Tomcat DS / Hibernate upgrade / Seam patch / H2 upgrade + MVCC branch

Reminder


This branch was created a few weeks ago by Stephane to fix an issue with some DB connections being used in auto-commit mode. Starting from this problem, we discovered linked issues that forced us to do several changes. This new branch contains:


  • usage of Tomcat native pools

  • an upgrade on Hibernate / Seam

  • an upgrade of H2 database

  • a switch to H2 MVCC mode

Testing this new branch


On this new branch there are a lot of failed tests: both unit and integration. When looking at the logs it is not clear:


  • what error are actually test setup errors

  • what are "normal errors"

  • what are real errors that actually cause the test to fail


To address that Stephane started a cleanup of the unit tests:


  • align all tests on feature runner


    • even JUnit3 tests based on RepositoryTestCase can actually use the FeatureRunner



  • allow usage of a feature to capture the log


    • make test capture the log to check for errors

    • make test fail if there are unprocessed / unexpected errors



  • fix tests setup / tear down steps

  • fix tests one by one


So far this work is not finished since there are still issues with some addons.

Tomcat 7 upgrade

Reminder


The work was started in December in a branch associated with NXP-10071. The branch was not merged because of impact inside addons regarding server-api changes...

How we finally merge


Julien will:


  • restart a build

  • identify the errors

  • create associated Jira tickets


and then we can dispatch the tickets so that we can finally do this upgrade!

JavaScript in Nuxeo

Context


JS is (was?) hype and we're a little late joining the battle. Let's get to it.

Javascript issues


Javascript comes with a set of issues:


  • poor maintainability


    • JS can be cryptic, (and jQuery does not help)

    • refactoring is a challenge



  • no clear dev model


    • prototype base model + several OOP libs

    • JS let's you do the same things in 3 different ways

    • JS has some built-in flows (scoping, bad closure...)



  • no dependency management

  • poor testability (at least for now)


All of these points are even stronger when put in the context of a platform like Nuxeo.

Thomas found a site that references best practices on Javascript: http://superherojs.com/.

NB: As you will see, with JS, even coding rules can become hype!

Test drive and Connect


In the context of Next Generation Connect, Damien started to test some technology solutions:


  • use CoffeeScript instead of vanilla Javascript

  • use tools like YeoMan


    • Yo: template / scriptlet system

    • Bower simple dependencies management

    • Grunt: makefile



  • do TDD in JS/Html5


Based on this first experience, we must define the blueprints of what the future Nuxeo JS Framework should be. So far we have mainly discussed the opportunity to use CoffeeScript vs plain JavaScript:

Pros:


  • CoffeeScript is prettier and shorter than JavaScript

  • CoffreeScript hides some of the quirk of Javascript

  • the generate JS is clean

  • we can maintain a bijection between JS <=> CoffeeScript

  • there seem to be some traction on CoffeeScript


Cons:


  • debugging needs to be done in pure JS (although a CoffeeScript source map now seems to be available)

  • CoffeeScript does not solve / hide all the JavaScript quirks and pitfalls

  • future of CoffeeScript is not clear whereas ECMAScript sustainability is good


We would really like to have your feedback on this. Please tell us what you think of Javascript in a Nuxeo Platform context.


Category: Product & Development
Tagged: Insight, Nuxeo Automation, Nuxeo Studio