For DevelopersHere comes the first tech report since Nuxeo World and the release of our latest LTS Nuxeo Platform 5.8.

Focus for 5.9.1


Here are the main tasks we'll be working on for the few next weeks.

nuxeo.io

API

Introspection

We have a REST API but so far we don't have a way for a remote client to easily introspect the structures available on the server side:


  • Schemas

  • Document Types

  • Widgets

  • Layouts


Having such an API endpoint does make sense:


  • Mobile SDK needs require it to fetch document types and layouts to generate display

  • The Mule connector requires it to fetch document types to make data mapping easier

  • Studio would need it to build the target platform definition from a live distribution

  • CMIS has this (at least for schemas/types)


We already have:


  • A widget/layout endpoint that was initially done for Studio and Android SDK

  • A schema/doc endpoint that was added for the Mule Connector


Globally what we would like is a consistent API:

GET /nuxeo/api/v1/config/types
GET /nuxeo/api/v1/config/widgets
GET /nuxeo/api/v1/config/layouts

In addition to the pure REST binding, we probably want:


  • A simple WebEngine UI (Like what we have for Automation?)

  • A support in platform explorer to:


    • Persist definitions associated to distribution

    • Allow simple browsing




NXP-13282 was created as an umbrella task.

REST Bindings Pluggability

We have the first level of pluggability for WebEngine via the Fragments system. Unfortunately, we cannot at the same time contribute a WebEngine fragments to an extension point.

For now, we have to use two different modules, but:


  • It does not make sense

  • It can be tricky


The task to fix this problem is probably for Stephane: NXP-13283

REST Binding TCK

The current TCK page we have is not a 'real' TCK, meaning you cannot run the tests from it. And it is not really documentation either. But it can be useful for someone that wants to understand how Nuxeo HTTP APIs work. We can probably do better.

Damien suggested to use fitnesse to have readable tests. So far we did not find a way to use it for that purpose (validating a client), but any volunteers or ideas are welcome.

iOS SDK


Arnaud is working on the iOS connector with our friends from Smart&Soft.

API Bindings

We have an initial binding in Objective-C with:


  • REST Document endpoint

  • Operation API


The binding no longer uses RESTKit, but a bare HTTP lib (ASI Http). The API returns Document object wrapping of bare JSON objects.

SDK Packaging

Nuxeo SDK is exposed via CocoaPods for dependencies management: https://github.com/nuxeo/cocoapods-specs.

Build and Tests

The connector is build by Jenkins: tests include calling an API on a locally deployed Nuxeo server.

Ideally:


  • The existing test should be aligned with the TCK

  • iOS client should be added to the TCK page

Next Steps

Next, we'll work on making the connector able to manage cache for offline usage.

Adding the Cache

ASI Request allows us to easily hook the HTTP request/response to add caching. But, this is still to be done:


  • Hook caching in HTTP layer

  • Manage storage (SQLLite for structure + Blob for response)

Tree Cache

Some apps will manage a tree of documents. This tree needs to be accessible offline. We have 2 options:


  • Retrieve the tree using n recursive 'getChildren' requests


    • We can use the standard request level caching



  • Retrieve the tree in one query


    • We need to introduce a dedicated API and a higher level of cache




NB: we should be able to compare with the Android logic that uses a "per list caching logic".

API Playground


We should work on building a nice API playground, something like a hybrid between:


  • Nuxeo Explorer

  • The Automation Doc

  • The REST API doc


We'll also have to define what is the target server for the playground:


  • For nuxeo.io users/customers, the target is clear: their instances

  • For other users that just want to take a look


    • Use a central site like demo.nuxeo.com?

    • Use Docker and a Core Server to provision on demand short lived instances?

    • Provide an AMI that people can use with a free AWS account?



Nuxeo IDE

Nuxeo IDE 1.2


Vlad is working on a new version of Nuxeo IDE.

IDE and Studio Integration


Anahide added a new registry on the Studio side so that the IDE can export operation without overriding existing configuration NXS-1242. For the multi-bundles use case, this is something we should probably handle on the IDE side (merge operation from all bundles before sending to Studio).

Make Nuxeo Less Stateful


Current Java API and WebFramework have inherited a stateful nature (EJB3 SFSB + JSF). With 5.9, it is time to start cleanup and lighten our framework.

Better Core Session Management


The stateful nature of the repository API creates concurrency issues. Rather that focusing on shielding problems of session sharing, we should work on making it almost stateless. It will be a much better solution.

The solution is to disconnect the DocumentModel object and make the reconnect dynamic. This implies:


  • Remove usages of sid

  • Remove usages of CoreInstance.getInstance


Florent started the work NXP-13148

We also need to discuss how we can make DocumentModel ThreadSafe. While doing this cleanup work, it will logically push our code to rely heavily on transaction demarcation. As a result, we may have to migrate all tests in Transactional mode:

Reduce Seam Beans Scopes


If we better manage CoreSession state, we should be able to make all our bean short lived:


  • This would reduce the memory footprint

  • We could remove part of the code to manage invalidations


We'll try to address that in 2 steps:


  • 1: Reduce the Seam beans scope


    • Remove invalidation code

    • Simplify navigation logic



  • 2: Disconnect CoreSession and DocumentModels


    • Leverage LayoutDemo use case




Umbrella task is NXP-13163

CI and QA Status


Having a good CI chain continues to be a critical and never ending task since:


  • We continue extending the platform (hopefully)

  • We support more deployment targets

  • We continue adding new commiters

  • We do releases more often

Build and Path Length


In the context of Jenkins duty, Florent did some renaming to shorten the path that made the Windows build failed: NXP-8659

NB: because, yes, even in 2013, Windows command line still has path length restriction.

Maven3 Migration


The main work is about distribution tools. Done so far:


  • Added some unit tests on previous assembly (needed to validate migration)

  • Validated that we can run a dependency tree inside a custom assembly plugin


To be done:


  • Port the existing commands of distribution tools to the new Maven 3 model


Considering that:


  • We need to migrate to maven3 for a long time

  • This should solve some bugs

  • This should improve the performances

Ongoing Tasks and Issues

Nuxeo Cluster Management


We want a way to define and deploy a set of Nuxeo Marketplace packages on all nodes of a cluster. Typical use cases include:


  • Install an addon on all nodes

  • Install a hoffix on all nodes

  • Update a studio package on all nodes


This requirement exists:


  • For nuxeo.io

  • For all customers using a cluster


We created NXP-13272 as an umbrella task, see the task description for more info.

Mule Connector


The Mule Connector is almost completed:


  • Automation Client is embedded

  • Mule Connector can now listen to Nuxeo events

  • Mule Connector can introspect Nuxeo types


Alain started writing nice documentation on using Mule's format.

VCS Repository

VCS ACL System Optimizations


Ben played a lot recently with the ReadACL system. He also implemented a new Simpler ReadACL system that allows you to skip one indirection in the security checking process.

It should be faster, but the trade-off is that you cannot have negative permissions: only positive permission and block all. So far, the tests have not shown a direct improvement, but still:


  • We will add this mode for PGSQL even if we don't activate it by default

  • We will add global switch (nuxeo.properties) to activate simple ACL mode

Query and Big Sorting


We have some PageProvider queries that use sort on a nullable field. It looks like in this case, PGSQL cannot efficiently use the index. It was fixed by making the PageProvider implementation automatically add the not null clause in the NXQL Query.

Full text/TS_Vector issue


Since 5.7, in PGSQL, we store the full text in clear (not 'vertorized') inside the database: this is required to allow "phrase search". Unfortunately, not storing directly the TS Vector makes the search query slower as the number of rows grows.

Options are:


  • Store only the TS_vector


    • Drop phrase search



  • Store TS_Vector and plain text


    • Make storage bigger




We need to:


  • Document the options


    • Explain the alternatives



  • Provide some migration tools


    • At least SQL scripts




NB: the real solution is to get the full text out of the SQL DB and use Elastic Search, but that's for later.

Automation Server

Automation Marshaling Issue


The ResponseHelper used by default in Automation needs to be reviewed so that we avoid hard-coding marshaling logic. As long has we have this hardcoded class: adding a JAX-RS ResponseWriter also means updating this class.

Exception Handler is Different Between WebEngine and Automation

Automation Exception management has been improved, but we still have some specific processing that should not be here.


  • Returning 404 when a document is not found does not make sense in the context of an Operation

  • 404 should mean resource not found => operation or chain is not found

  • We sometimes return 403 instead of 401


This means we need to review how we manage errors and associated HTTP status.

500: generic server; error 404: resource not found (operation or chain); 403: forbidden (not enough rights); 401: not authenticated (need to be logged); 409: conflict (dirty update)

This work also includes some changes on the PageProvider.

Existing task (NXP-12493) to be completed.

Nuxeo Tech Talk


The topic of the next Nuxeo Tech Talk is AngularJS. We'll welcome Julien Boucquillon who will talk about AngularJS in mobile apps. Damien Metzler will give an overview of the work we've done so far around AngularJS.