[Nuxeo Tech Report] News from the Developer Front #12
- Nuxeo IDE
- Make Nuxeo Less Stateful
- CI and QA Status
- Nuxeo Cluster Management
- Mule Connector
- VCS Repository
- Automation Server
- Nuxeo Tech Talks
Focus for 5.9.1
Here are the main tasks we'll be working on for the few next weeks.
We have a REST API but so far we don't have a way for a remote client to easily introspect the structures available on the server side:
- Document Types
Having such an API endpoint does make sense:
- Mobile SDK needs require it to fetch document types and layouts to generate display
- The Mule connector requires it to fetch document types to make data mapping easier
- Studio would need it to build the target platform definition from a live distribution
- CMIS has this (at least for schemas/types)
We already have:
- A widget/layout endpoint that was initially done for Studio and Android SDK
- A schema/doc endpoint that was added for the Mule Connector
Globally what we would like is a consistent API:
In addition to the pure REST binding, we probably want:
- A simple WebEngine UI (Like what we have for Automation?)
- A support in platform explorer to:
- Persist definitions associated to distribution
- Allow simple browsing
NXP-13282 was created as an umbrella task.
REST Bindings Pluggability
We have the first level of pluggability for WebEngine via the Fragments system. Unfortunately, we cannot at the same time contribute a WebEngine fragments to an extension point.
For now, we have to use two different modules, but:
- It does not make sense
- It can be tricky
The task to fix this problem is probably for Stephane: NXP-13283
REST Binding TCK
The current TCK page we have is not a 'real' TCK, meaning you cannot run the tests from it. And it is not really documentation either. But it can be useful for someone that wants to understand how Nuxeo HTTP APIs work. We can probably do better.
Damien suggested to use fitnesse to have readable tests. So far we did not find a way to use it for that purpose (validating a client), but any volunteers or ideas are welcome.
Arnaud is working on the iOS connector with our friends from Smart&Soft.
We have an initial binding in Objective-C with:
- REST Document endpoint
- Operation API
The binding no longer uses RESTKit, but a bare HTTP lib (ASI Http). The API returns Document object wrapping of bare JSON objects.
Nuxeo SDK is exposed via CocoaPods for dependencies management: https://github.com/nuxeo/cocoapods-specs.
Build and Tests
The connector is build by Jenkins: tests include calling an API on a locally deployed Nuxeo server.
Next, we'll work on making the connector able to manage cache for offline usage.
Adding the Cache
ASI Request allows us to easily hook the HTTP request/response to add caching. But, this is still to be done:
- Hook caching in HTTP layer
- Manage storage (SQLLite for structure + Blob for response)
Some apps will manage a tree of documents. This tree needs to be accessible offline. We have 2 options:
- Retrieve the tree using n recursive
- We can use the standard request level caching
- Retrieve the tree in one query
- We need to introduce a dedicated API and a higher level of cache
NB: we should be able to compare with the Android logic that uses a "per list caching logic".
We should work on building a nice API playground, something like a hybrid between:
- Nuxeo Explorer
- The Automation Doc
- The REST API doc
We'll also have to define what is the target server for the playground:
- For nuxeo.io users/customers, the target is clear: their instances
- For other users that just want to take a look
- Use a central site like demo.nuxeo.com?
- Use Docker and a Core Server to provision on demand short lived instances?
- Provide an AMI that people can use with a free AWS account?
Nuxeo IDE 1.2
Vlad is working on a new version of Nuxeo IDE.
IDE and Studio Integration
Anahide added a new registry on the Studio side so that the IDE can export operation without overriding existing configuration NXS-1242. For the multi-bundles use case, this is something we should probably handle on the IDE side (merge operation from all bundles before sending to Studio).
Make Nuxeo Less Stateful
Current Java API and WebFramework have inherited a stateful nature (EJB3 SFSB + JSF). With 5.9, it is time to start cleanup and lighten our framework.
Better Core Session Management
The stateful nature of the repository API creates concurrency issues. Rather that focusing on shielding problems of session sharing, we should work on making it almost stateless. It will be a much better solution.
The solution is to disconnect the DocumentModel object and make the reconnect dynamic. This implies:
- Remove usages of
- Remove usages of
Florent started the work NXP-13148
We also need to discuss how we can make DocumentModel ThreadSafe. While doing this cleanup work, it will logically push our code to rely heavily on transaction demarcation. As a result, we may have to migrate all tests in Transactional mode:
Reduce Seam Beans Scopes
If we better manage CoreSession state, we should be able to make all our bean short lived:
- This would reduce the memory footprint
- We could remove part of the code to manage invalidations
We'll try to address that in 2 steps:
- 1: Reduce the Seam beans scope
- Remove invalidation code
- Simplify navigation logic
- 2: Disconnect CoreSession and DocumentModels
- Leverage LayoutDemo use case
Umbrella task is NXP-13163
CI and QA Status
Having a good CI chain continues to be a critical and never ending task since:
- We continue extending the platform (hopefully)
- We support more deployment targets
- We continue adding new commiters
- We do releases more often
Build and Path Length
In the context of Jenkins duty, Florent did some renaming to shorten the path that made the Windows build failed: NXP-8659
NB: because, yes, even in 2013, Windows command line still has path length restriction.
The main work is about distribution tools. Done so far:
- Added some unit tests on previous assembly (needed to validate migration)
- Validated that we can run a dependency tree inside a custom assembly plugin
To be done:
- Port the existing commands of distribution tools to the new Maven 3 model
- We need to migrate to maven3 for a long time
- This should solve some bugs
- This should improve the performances
Ongoing Tasks and Issues
Nuxeo Cluster Management
We want a way to define and deploy a set of Nuxeo Marketplace packages on all nodes of a cluster. Typical use cases include:
- Install an addon on all nodes
- Install a hoffix on all nodes
- Update a studio package on all nodes
This requirement exists:
- For nuxeo.io
- For all customers using a cluster
We created NXP-13272 as an umbrella task, see the task description for more info.
The Mule Connector is almost completed:
- Automation Client is embedded
- Mule Connector can now listen to Nuxeo events
- Mule Connector can introspect Nuxeo types
Alain started writing nice documentation on using Mule's format.
VCS ACL System Optimizations
Ben played a lot recently with the ReadACL system. He also implemented a new Simpler ReadACL system that allows you to skip one indirection in the security checking process.
It should be faster, but the trade-off is that you cannot have negative permissions: only positive permission and block all. So far, the tests have not shown a direct improvement, but still:
- We will add this mode for PGSQL even if we don't activate it by default
- We will add global switch (nuxeo.properties) to activate simple ACL mode
Query and Big Sorting
We have some PageProvider queries that use sort on a nullable field. It looks like in this case, PGSQL cannot efficiently use the index. It was fixed by making the
PageProviderimplementation automatically add the
not nullclause in the NXQL Query.
Full text/TS_Vector issue
Since 5.7, in PGSQL, we store the full text in clear (not 'vertorized') inside the database: this is required to allow "phrase search". Unfortunately, not storing directly the TS Vector makes the search query slower as the number of rows grows.
- Store only the TS_vector
- Drop phrase search
- Store TS_Vector and plain text
- Make storage bigger
We need to:
- Document the options
- Explain the alternatives
- Provide some migration tools
- At least SQL scripts
NB: the real solution is to get the full text out of the SQL DB and use Elastic Search, but that's for later.
Automation Marshaling Issue
ResponseHelperused by default in Automation needs to be reviewed so that we avoid hard-coding marshaling logic. As long has we have this hardcoded class: adding a JAX-RS
ResponseWriteralso means updating this class.
Exception Handler is Different Between WebEngine and Automation
Automation Exception management has been improved, but we still have some specific processing that should not be here.
- Returning 404 when a document is not found does not make sense in the context of an Operation
- 404 should mean resource not found => operation or chain is not found
- We sometimes return 403 instead of 401
This means we need to review how we manage errors and associated HTTP status.
500: generic server; error 404: resource not found (operation or chain); 403: forbidden (not enough rights); 401: not authenticated (need to be logged); 409: conflict (dirty update)
This work also includes some changes on the PageProvider.
Existing task (NXP-12493) to be completed.
Nuxeo Tech Talk
The topic of the next Nuxeo Tech Talk is AngularJS. We'll welcome Julien Boucquillon who will talk about AngularJS in mobile apps. Damien Metzler will give an overview of the work we've done so far around AngularJS.
Category: Product & Development