[Nuxeo Tech Report] News from the Developer Front #11
Mon 16 September 2013 By Laurent Doguin
This chapter is about ongoing or near future platform improvements.
- Select2 Widgets
- Seam/JSF Infrastructure Improvement
- Other Deployment Changes
- Nuxeo Drive
Select2 widget is now integrated as a replacement for the old selection/chain selection/suggestion widgets in many screens:
- Edit forms
In addition we now have functional testing of these new widgets via WebDriver. To be able to maximize usage, and add small features that have been missing for a long time, we still have some work to do:
- Allow simple filter toggle: to replace the Document search in the relation screen
- Manage obsolete directory entries in a consistent way (work in progress)
- Full AJAX integration (done)
- Allow to completely replace the RichSuggestion JSF widget
- Define ordering
- Some CSS classes
- Improve default properties on widget types
To completely finish the work we should:
- Finish the Layout Demo integration
- Add a WebDriver tests to test Select2 in LayoutDemo
We currently have several schema for vocabularies:
For now, Select2 supports them all. Ideally, we want to push usage of l10n vocabularies since it is better for:
- Usability to add/edit entries on-the-fly
But before integrating this inside Studio, for consistency reasons, we need to fix VirtualNavigation so that it supports l10nvocabularies. Guillaume will take a couple of days to fix this so we can do the Studio alignment.
For now, we don't have an exact replacement for the chain select, but Select2 provides a tree mode that can be used for the same use cases. We know we could build a real chain select with Select2, but:
- Not sure if this is really needed:
- Tree Select2 should be ok for most use cases
- Tree Select2 is easier to manage
Once the polish work is done on Select2 and Vocabularies we can:
- Integrate Select2 inside Nuxeo Studio
- Remove the Studio specific chain selects
- Move the old widgets inside Nuxeo for compatibility
- Upgrade Vocabularies editors (l10n support)
- Deprecate the non i10n vocabulary
- Manage directories on Studio, and not only Vocabularies
- Automatically migrate existing Select/Chained Select to Select2
Studio tasks will be created this week.
With NXP-12239 Florent started to work on integrating Redis with Nuxeo to provide persistent Jobs management. This will open the way for:
- Avoiding loosing jobs when a node is restarted
- Dispatching work across nodes
- Have dedicated nodes for specific processing
This work follows the work initiated by Ben on Redis, and the next steps should be:
- Use Redis for Cluster invalidations
- Use Redis for lock management on documents
Seam/JSF Infrastructure Improvement
The work on this subject has been started, especially via the fix in ViewState management, but there are other fixes we want to do.
The branch on Seam that supports NXRuntime injection (NXP-12207) has just been merged. The current code should allow:
- Injecting a NuxeoRuntime Service by it's interface
- Referencing a NuxeoRuntime Service via it's shortName in the xhtml
Please report any strange injection issue!
Remove Seam Transaction Failed Message
Now that we have forked Seam, we can easily remove the ugly "Transaction failed" message: NXP-12483.
Improve Concurrent Conversation Call Management
The way Seam deals with Concurrent calls to a conversation isn't efficient. We can surely do better. See NXP-12487
Based on the work done for Adulact, we (Benjamin) are working on improving the default importer. The idea is basically to merge:
platorm-importer: that provides the infrastructure
scan-importer: that provides the scheduling logic
nuxeo-adulact: that provides the plugability via MVEL and XPath expressions
Benjamin has starting the work on this, and we should soon have a more powerful importer API that can be used in most cases "without having to write code"
With NXP-11698 worked on adding traces that allows to generate sequence diagram of listeners execution.
This seems to be a great tool:
- We need to write some documentation
- We need to provide a sample diagram
Thomas added some WebDriver tests to have functional testing of the DAM ui.
Box Layout in Nuxeo DM
The DM/CAP big icon view is still not migrated to the new box layout. In the same scope, we are still using the Picture Book:
- That is not based on Layouts (which makes it not configurable)
- That does not allow to configure views size
Associated task is NXP-11046
Assets Meta-data Management
For now, we are still using mistral to extract the meta-data on images. The current system has some drawbacks
- Some meta-data are badly extracted
- In some cases the meta-data are too big for the fields and break the transaction! NXP-8964
- Memory leaks NXP-12160
- We cannot do write back
We have some existing work on that:
- Laurent did some work for blog posts:
- Integrated EXIF Tool
- XMP extraction
- Nelson contributed some fixes NXP-11582
This basically means that we only need to do some cleanup work to finish this.
Reminder: getting rid of the deployment-fragment has been part of the roadmap for a long time.
- Avoid deployment pre-processor requirement
- Remove hard dependency on application server
- Fix reloading issues with IDE regarding resources contributed via deployment-fragments
- I18n translations
- Navigation cases
- Make web contribution more consistent with the rest of the framework
- Difficulty to deploy in a solid war
This is related to NXROADMAP-37
Build service to contribute servlets and filters NXP-12437
This is probably the first infrastructure task. We can leverage the existing work that is part of Apricot. Stephane can probably bootstrap the work since he knows the associated code.
Review JSF/Seam reload architecture NXP-12438
Normally, JSF/Seam should already provide what is needed to avoid requiring deployment-fragment. Anahide and Stephane already did some work on it. Anyway, now that we have hooked Richfaces and Seam code: we can probably fix what is needed if possible.
Provide a translation service for Nuxeo NXP-12439
It would be very easy to replace the current message-bundle system with a Nuxeo service + extension point (especially because we can easily hook Seam now). However, it would be good to leverage this task to really improve how translations are managed:
- Ensure hot reload
- Plug on an existing translation system
We'll start with a quick Spec, and since Laurent spent some time on it, he'll be the winner!
Other Deployment Changes
For, 5.8, re-enabling the JBoss support would imply some significant work:
- Choose what JBoss version to align on
- Update packaging
- Manage issues between Seam/JSF/Hibernate versions without breaking EAP system
Given the ratio between work and benefit, we'll drop JBoss support for 5.8. We will reconsider that when we have:
- A real requirement
- A potential visibility on what we could gain from JBoss support.
Switch from Metro to CXF is done.
We now have to:
- Update the documentation
- List what we have gained for CXF
- Decide if we want (in a near future) to use CXF for JAX-RS too
In terms of features, Nuxeo Drive has not evolved much, but the overall stability has been improved a lot:
- Many bugs fixed
- Management of big files has been improved
- Improvements on the OSX build
- Improved Stability of server side mapping extensions
- More tests
But we still have work to do, short term roadmap must include:
- Adding support for HTTP proxy, code is almost done
- Doing the alignment on PyQT (required to avoid crash on the client side)
- Auto-update system
- FS notification system to avoid event loop
Issues and Problems
Support and Performances
So far most clients that used SQL Server with Nuxeo ended up with troubles:
- Performance issues
Based on the work we did and the feedback from customers, our understanding is that our support of SQL Server has progressed and the situation has improved a lot:
Some Customers have:
- Changed isolation level
- Added a custom custered index
- Implemented Soft delete
- Reported fragmentation issues
- Changed lock escalation policy
This means that with a SQL server database that is created by a 5.7 and with a DBA that applies our recommendations, we should be 'almost ok'. Last investigations showed that we must still:
We (Ben) will update the public documentation NXP-12445:
- List specific SQL Server SQL tricks: especially for people migrating from 5.6
- List configuration recommendations
In the same idea, Ben is working on having a procedure for reporting problems like what we have for Oracle/PGSQL.
Read ACL Issue
Issue with ReadACL computations have been reported. This ended up being a side effect of 2 bugs:
- Java bug: some code was modifying the
NuxeoPrincipal.getAllGroups()lists by side effect (i.e. bad code)
- TSQL issue: the ReadACL computations are not resilient enough when principals groups contain duplicated groups
The java bug is corrected, the T-SQL issue may be related to a TX issue. We are investigating.
We know that under load we are very likely to encounter deadlocks issues (again). To address this mid term issue, we have several possiblities:
We could change some VCS internals to limit the randomness of tables inside the queries: this should reduce the cases of Deadlocks. See NXP-12444
We should define a high level concurrency Exception that should be raised:
- When VCS detects a dirty update (Stephane had started work on that at some point)
- When the RDBMS (ex: SQL Server) throws a deadlock exception
Having this explicit exception should allow:
- To automatically restart/retry in case of non interactive tasks
- To display a dedicated message "same player shoot again' to the interactive user
- To report an HTTP 409 in Automation calls
Stephane already worked on this a long time ago, see NXP-4914.
We should extend this task to:
- Handle Exception that can be raised by the DB server
- Raise a
RecoverableClientExceptionso that we can
- Display an appropriate message to users
- Rerun Workers
Complex Types Management
An other improvement we could do is change the way we manage complex properties update:
- For now: DELETE + INSERT
- Target: UPDATE
The goal is to limit the number of DELETE statements that can create locking issues (especially in SQL Server, but not only). However, this modification:
- Is far from trivial
- Will make the java part much more complicated (needs to compute a diff on complex properties)
Let's be honest: as a first experience, Nuxeo under Windows is not exactly great. We must improve our support for Windows:
- Better Service wrapper
- Better integration of the logger
- Better integration of the monitoring
Unless we have a volunteer for that, we will try to hire someone that will focus on Windows integration. (Yes reader, it could be you! Wanna join Nuxeo?)
LDAP Server Issues
We have several customers that reported issues when using Nuxeo with LDAP. In most cases, the issue is actually not really a Nuxeo issue:
- LDAP server too busy
- Network errors
We already have a retry parameter, but:
- It only handles a specific LDAP error code
- Tt does not covers all code using LDAP
The result is that after retrying several times, the overloaded server probably ends up closing sockets: then Nuxeo receive Socket Error. We'll try to be more resilient:
- Retry on more exceptions
- Add try/catch in
- Log big errors!
Thierry leads that effort, but it does not means he will have to do all the work!
We continue having issues with OpenOffice. The last problem was related to an encoding issue that does work ok in LibreOffice and does not with OpenOffice. This means that we will soon have to choose between the 2 projects and probably only support one of the two. The other point is that it becomes more and more painful to maintain JODConverter and manage OOo process in Nuxeo and tests.
There are several alternatives as explained in NXP-12447.
Opensocial and Multi-Repository
As reported by a customer, the OpenSocial DashBoard does not properly work in multi-repository mode. This is something we should fix:
- Container and gadgets should be initialized with the name of the target RepositoryName
- Passing it as a Gadget default preference as we did for other parameters
- Automation JS client integrated in the gadgets should handle the correct header
We recently changed the way JMX endpoints are published: looks like we lost some of them in the battle. Stephane will take a look at where the Event Monitoring endpoint is gone: NXP-12488
I know we could lists a lot of things here... but, let's focus on the most recent points!
We (Anahide in fact!) have done a lot of work on the Action System:
- Defining actions types
- Use properties for Actions
Even if the infrastructure work is done, the default integration in CAP/DM is not fully completed. As a result a part of the actions inside CAP/DM cannot be configured with this model: for example, we have still several action categories that will render actions without taking care of their type and properties.
Before releasing a 5.8, some cleanup should be done:
In 5.7.2, we still have warning on startup about deprecated QueryModel that will be removed in 5.6! Even if we are very conservative with deprecation messages: it is probably more than time to clean that up.
Remove the JBPM addon from "default addons": NXP-12448
Websites + blogs
Remove Websites and Blogs and bundle them as addons: NXP-11728
According to NXIDE-283, it looks like the Seam component deployment is broken from Nuxeo IDE. We'll investigate with Anahide and Stephane: this is something we should fix for 5.8.
Nicolas is moving forward on Fragments NXMOB-48. For now we have upgraded the Nuxeo Online Servicesor to depend on Android SDK 11 to use the Fragment model.
Nicolas is migrating all the existing activities to fragments. Using the Fragments allows:
- To have better granularity on UI items in the SDK
- To dynamically adapt UI layout depending on the device (phone vs tablet)
This migration should be over in a few days. Next step will be to address the client and maven integration.
Arnaud started implementing an iOS connector using:
- Automation REST API (i.e. not the original RPC API)
With this approach, we can expect to quickly have something that manages:
- Data retrieval and display from Nuxeo
- Offline access (thanks to the embedded caching system)
We have done most of the changes we wanted to make:
- Fixed a lot of limitations NXP-10078
- Align Chain and Operation models NXP-11096
- Debugging and exceptions NXP-11281
- Custom marshaling
There are probably more improvements to be made, but let's wait for some "real life feedback".
We have a working document based REST API that includes:
- Document bindings
- Configurable fragment of JSON associated to documents (relations, acls, urls...)
In addition, Damien, Thomas and Vlad are adding more end points to expose users, directories, audit,...
This work on the API will continues over the next weeks:
- Real life use cases may require adjustments
- We will have new ideas
By itself, this is not an issue, but we have to be careful about what we include in 5.8 release. Since it will be 'maintained API' (we may use addon package for the others until ready).
The goal is to have a standalone module that contains:
- The JS client for Automation
- Automation RPC
- Automation REST Wrapper
- Blob uploader/downloader
- Some unit tests
- Bindings for different JS framework
- Browser JS
Part of the work has been started and for now we have a working client with some unit tests and a shell. Remaining work includes:
- node.js binding (xhr is different!)
- angular.js binding (merge/align with what Damien has already done)
- Integration inside Nuxeo CAP
Category: Product & Development