[Nuxeo Tech Report] News from the Developer Front #11


Mon 16 September 2013 By Laurent Doguin

For Developers
Here's the latest tech report taken straight from the developers techlead meetings. We have news about the upcoming 5.8, DAM, Drive, JavaScript, infrastructure, mobile, Content Automation and more.


 

Platform Improvements


This chapter is about ongoing or near future platform improvements.

 

Select2 Widgets


CAP/DM Integration


Select2 widget is now integrated as a replacement for the old selection/chain selection/suggestion widgets in many screens:

  • Edit forms

  • Users/Groups


In addition we now have functional testing of these new widgets via WebDriver. To be able to maximize usage, and add small features that have been missing for a long time, we still have some work to do:

  • Allow simple filter toggle: to replace the Document search in the relation screen

  • Manage obsolete directory entries in a consistent way (work in progress)

  • Full AJAX integration (done)

  • Allow to completely replace the RichSuggestion JSF widget

    • Define ordering

    • Some CSS classes

    • Improve default properties on widget types




To completely finish the work we should:

  • Finish the Layout Demo integration

  • Add a WebDriver tests to test Select2 in LayoutDemo


Vocabularies


We currently have several schema for vocabularies:

  • vocabulary

  • xvocabulary

  • l10nvocabulary

  • l10nxvocabulary


For now, Select2 supports them all. Ideally, we want to push usage of l10n vocabularies since it is better for:

  • Search/suggestion

  • Usability to add/edit entries on-the-fly


But before integrating this inside Studio, for consistency reasons, we need to fix VirtualNavigation so that it supports l10nvocabularies. Guillaume will take a couple of days to fix this so we can do the Studio alignment.

Chain Select


For now, we don't have an exact replacement for the chain select, but Select2 provides a tree mode that can be used for the same use cases. We know we could build a real chain select with Select2, but:

  • Not sure if this is really needed:

    • Tree Select2 should be ok for most use cases

    • Tree Select2 is easier to manage




Studio Impacts


Once the polish work is done on Select2 and Vocabularies we can:

  • Integrate Select2 inside Nuxeo Studio

  • Remove the Studio specific chain selects

    • Move the old widgets inside Nuxeo for compatibility



  • Upgrade Vocabularies editors (l10n support)

    • Deprecate the non i10n vocabulary



  • Manage directories on Studio, and not only Vocabularies

  • Automatically migrate existing Select/Chained Select to Select2


Studio tasks will be created this week.

 

Redis/Workmanager


With NXP-12239 Florent started to work on integrating Redis with Nuxeo to provide persistent Jobs management. This will open the way for:

  • Avoiding loosing jobs when a node is restarted

  • Dispatching work across nodes

  • Have dedicated nodes for specific processing


This work follows the work initiated by Ben on Redis, and the next steps should be:

  • Use Redis for Cluster invalidations

  • Use Redis for lock management on documents


 

Seam/JSF Infrastructure Improvement


The work on this subject has been started, especially via the fix in ViewState management, but there are other fixes we want to do.

NXRuntime Injection


The branch on Seam that supports NXRuntime injection (NXP-12207) has just been merged. The current code should allow:

  • Injecting a NuxeoRuntime Service by it's interface

  • Referencing a NuxeoRuntime Service via it's shortName in the xhtml


Please report any strange injection issue!

Remove Seam Transaction Failed Message


Now that we have forked Seam, we can easily remove the ugly "Transaction failed" message: NXP-12483.

Improve Concurrent Conversation Call Management


The way Seam deals with Concurrent calls to a conversation isn't efficient. We can surely do better. See NXP-12487

 

Importer


Based on the work done for Adulact, we (Benjamin) are working on improving the default importer. The idea is basically to merge:

  • platorm-importer: that provides the infrastructure

  • scan-importer: that provides the scheduling logic

  • nuxeo-adulact: that provides the plugability via MVEL and XPath expressions


Benjamin has starting the work on this, and we should soon have a more powerful importer API that can be used in most cases "without having to write code"

 

Monitoring/Debugging


With NXP-11698 worked on adding traces that allows to generate sequence diagram of listeners execution.

This seems to be a great tool:

  • We need to write some documentation

  • We need to provide a sample diagram


 

DAM


Functional Testing


Thomas added some WebDriver tests to have functional testing of the DAM ui.

Box Layout in Nuxeo DM


The DM/CAP big icon view is still not migrated to the new box layout. In the same scope, we are still using the Picture Book:

  • That is not based on Layouts (which makes it not configurable)

  • That does not allow to configure views size


Associated task is NXP-11046
Assets Meta-data Management

For now, we are still using mistral to extract the meta-data on images. The current system has some drawbacks

  • Some meta-data are badly extracted

  • In some cases the meta-data are too big for the fields and break the transaction! NXP-8964

  • Memory leaks NXP-12160

  • We cannot do write back


We have some existing work on that:

  • Laurent did some work for blog posts:

    • Integrated EXIF Tool

    • XMP extraction



  • Nelson contributed some fixes NXP-11582


This basically means that we only need to do some cleanup work to finish this.

 

Deployment-Fragments


Reminder: getting rid of the deployment-fragment has been part of the roadmap for a long time.

Motivations include:

  • Avoid deployment pre-processor requirement

  • Remove hard dependency on application server

  • Fix reloading issues with IDE regarding resources contributed via deployment-fragments

    • I18n translations

    • Navigation cases



  • Make web contribution more consistent with the rest of the framework

  • Difficulty to deploy in a solid war


This is related to NXROADMAP-37

Tasks


Build service to contribute servlets and filters NXP-12437

This is probably the first infrastructure task. We can leverage the existing work that is part of Apricot. Stephane can probably bootstrap the work since he knows the associated code.

Review JSF/Seam reload architecture NXP-12438

Normally, JSF/Seam should already provide what is needed to avoid requiring deployment-fragment. Anahide and Stephane already did some work on it. Anyway, now that we have hooked Richfaces and Seam code: we can probably fix what is needed if possible.

Provide a translation service for Nuxeo NXP-12439

It would be very easy to replace the current message-bundle system with a Nuxeo service + extension point (especially because we can easily hook Seam now). However, it would be good to leverage this task to really improve how translations are managed:

  • Ensure hot reload

  • Plug on an existing translation system


We'll start with a quick Spec, and since Laurent spent some time on it, he'll be the winner!

 

Other Deployment Changes


JBoss Binding


For, 5.8, re-enabling the JBoss support would imply some significant work:

  • Choose what JBoss version to align on

  • Update packaging

  • Manage issues between Seam/JSF/Hibernate versions without breaking EAP system


Given the ratio between work and benefit, we'll drop JBoss support for 5.8. We will reconsider that when we have:

  • A real requirement


and/or

  • A potential visibility on what we could gain from JBoss support.


SOAP/CXF


Switch from Metro to CXF is done.

We now have to:

  • Update the documentation

  • List what we have gained for CXF

  • Decide if we want (in a near future) to use CXF for JAX-RS too


 

Nuxeo Drive


In terms of features, Nuxeo Drive has not evolved much, but the overall stability has been improved a lot:

  • Many bugs fixed

  • Management of big files has been improved

  • Improvements on the OSX build

  • Improved Stability of server side mapping extensions

  • More tests


But we still have work to do, short term roadmap must include:

  • Adding support for HTTP proxy, code is almost done

  • Doing the alignment on PyQT (required to avoid crash on the client side)

  • Auto-update system

  • FS notification system to avoid event loop


 

Issues and Problems



 

SQL Server


Support and Performances


So far most clients that used SQL Server with Nuxeo ended up with troubles:

  • Performance issues

  • Deadlocks


Based on the work we did and the feedback from customers, our understanding is that our support of SQL Server has progressed and the situation has improved a lot:

Some Customers have:
- Changed isolation level
- Added a custom custered index
- Implemented Soft delete
- Reported fragmentation issues
- Changed lock escalation policy

This means that with a SQL server database that is created by a 5.7 and with a DBA that applies our recommendations, we should be 'almost ok'. Last investigations showed that we must still:

  • Add some more index in default config NXP-12378

  • May be change one of the stored procedure NXP-12379


We (Ben) will update the public documentation NXP-12445:

  • List specific SQL Server SQL tricks: especially for people migrating from 5.6

  • List configuration recommendations


In the same idea, Ben is working on having a procedure for reporting problems like what we have for Oracle/PGSQL.

Read ACL Issue


Issue with ReadACL computations have been reported. This ended up being a side effect of 2 bugs:

  • Java bug: some code was modifying the NuxeoPrincipal.getAllGroups() lists by side effect (i.e. bad code)

  • TSQL issue: the ReadACL computations are not resilient enough when principals groups contain duplicated groups


The java bug is corrected, the T-SQL issue may be related to a TX issue. We are investigating.

Infrastructure Work


We know that under load we are very likely to encounter deadlocks issues (again). To address this mid term issue, we have several possiblities:
Tables Ordering

We could change some VCS internals to limit the randomness of tables inside the queries: this should reduce the cases of Deadlocks. See NXP-12444
Concurrency Exception

We should define a high level concurrency Exception that should be raised:

  • When VCS detects a dirty update (Stephane had started work on that at some point)

  • When the RDBMS (ex: SQL Server) throws a deadlock exception


Having this explicit exception should allow:

  • To automatically restart/retry in case of non interactive tasks

  • To display a dedicated message "same player shoot again' to the interactive user

  • To report an HTTP 409 in Automation calls


Stephane already worked on this a long time ago, see NXP-4914.

We should extend this task to:

  • Handle Exception that can be raised by the DB server

  • Raise a RecoverableClientException so that we can

    • Display an appropriate message to users

    • Rerun Workers




Complex Types Management

An other improvement we could do is change the way we manage complex properties update:

  • For now: DELETE + INSERT

  • Target: UPDATE


The goal is to limit the number of DELETE statements that can create locking issues (especially in SQL Server, but not only). However, this modification:

  • Is far from trivial

  • Will make the java part much more complicated (needs to compute a diff on complex properties)


 

Integration


Windows Hosting


Let's be honest: as a first experience, Nuxeo under Windows is not exactly great. We must improve our support for Windows:

  • Better Service wrapper

  • Better integration of the logger

  • Better integration of the monitoring


Unless we have a volunteer for that, we will try to hire someone that will focus on Windows integration. (Yes reader, it could be you! Wanna join Nuxeo?)

LDAP Server Issues


We have several customers that reported issues when using Nuxeo with LDAP. In most cases, the issue is actually not really a Nuxeo issue:

  • LDAP server too busy

  • Network errors


We already have a retry parameter, but:

  • It only handles a specific LDAP error code

  • Tt does not covers all code using LDAP


The result is that after retrying several times, the overloaded server probably ends up closing sockets: then Nuxeo receive Socket Error. We'll try to be more resilient:

  • Retry on more exceptions

  • Add try/catch in getEntry too

  • Log big errors!


Thierry leads that effort, but it does not means he will have to do all the work!

LibreOffice


We continue having issues with OpenOffice. The last problem was related to an encoding issue that does work ok in LibreOffice and does not with OpenOffice. This means that we will soon have to choose between the 2 projects and probably only support one of the two. The other point is that it becomes more and more painful to maintain JODConverter and manage OOo process in Nuxeo and tests.

There are several alternatives as explained in NXP-12447.

 

Small Bugs


Opensocial and Multi-Repository


As reported by a customer, the OpenSocial DashBoard does not properly work in multi-repository mode. This is something we should fix:

  • Container and gadgets should be initialized with the name of the target RepositoryName

    • Passing it as a Gadget default preference as we did for other parameters



  • Automation JS client integrated in the gadgets should handle the correct header


NXP-12464

EventAdmin/JMX


We recently changed the way JMX endpoints are published: looks like we lost some of them in the battle. Stephane will take a look at where the Event Monitoring endpoint is gone: NXP-12488

 

Ongoing Work


I know we could lists a lot of things here... but, let's focus on the most recent points!

 

Action System


We (Anahide in fact!) have done a lot of work on the Action System:

  • Defining actions types

  • Use properties for Actions


Even if the infrastructure work is done, the default integration in CAP/DM is not fully completed. As a result a part of the actions inside CAP/DM cannot be configured with this model: for example, we have still several action categories that will render actions without taking care of their type and properties.

In addition, we currently have no simple way to do JavaScript based actions: only the context menu uses JS actions and it is still done using some custom code.

 

5.8 Cleanup


Before releasing a 5.8, some cleanup should be done:

Deprecated QueryModel


In 5.7.2, we still have warning on startup about deprecated QueryModel that will be removed in 5.6! Even if we are very conservative with deprecation messages: it is probably more than time to clean that up.

See NXP-12435

JBPM


Remove the JBPM addon from "default addons": NXP-12448

Websites + blogs


Remove Websites and Blogs and bundle them as addons: NXP-11728

 

Nuxeo IDE


According to NXIDE-283, it looks like the Seam component deployment is broken from Nuxeo IDE. We'll investigate with Anahide and Stephane: this is something we should fix for 5.8.

 

Mobile Bindings


Android:


Nicolas is moving forward on Fragments NXMOB-48. For now we have upgraded the Nuxeo Online Servicesor to depend on Android SDK 11 to use the Fragment model.

Nicolas is migrating all the existing activities to fragments. Using the Fragments allows:

  • To have better granularity on UI items in the SDK

  • To dynamically adapt UI layout depending on the device (phone vs tablet)


This migration should be over in a few days. Next step will be to address the client and maven integration.

iOS


Arnaud started implementing an iOS connector using:

  • RestKit

  • Automation REST API (i.e. not the original RPC API)


With this approach, we can expect to quickly have something that manages:

  • Data retrieval and display from Nuxeo

  • Offline access (thanks to the embedded caching system)


 

API


Automation RPC


We have done most of the changes we wanted to make:

There are probably more improvements to be made, but let's wait for some "real life feedback".

Automation REST


We have a working document based REST API that includes:

  • Document bindings

  • Adapters

  • Configurable fragment of JSON associated to documents (relations, acls, urls...)


In addition, Damien, Thomas and Vlad are adding more end points to expose users, directories, audit,...

This work on the API will continues over the next weeks:

  • Real life use cases may require adjustments

  • We will have new ideas


By itself, this is not an issue, but we have to be careful about what we include in 5.8 release. Since it will be 'maintained API' (we may use addon package for the others until ready).

nuxeo.js


The goal is to have a standalone module that contains:

  • The JS client for Automation

    • Automation RPC

    • Automation REST Wrapper

    • Blob uploader/downloader



  • Some unit tests

  • Bindings for different JS framework

    • Browser JS

    • angular.js

    • node.js




Part of the work has been started and for now we have a working client with some unit tests and a shell. Remaining work includes:

  • node.js binding (xhr is different!)

  • angular.js binding (merge/align with what Damien has already done)

  • Packaging

  • Integration inside Nuxeo CAP

    • Select2

    • Drag&Drop




Category: Product & Development
Tagged: Features, Nuxeo Platform