[Nuxeo Tech Report] News from the Developer Front #4


Wed 06 March 2013 By Laurent Doguin

Hello everyone! The latest tech report has landed! There is lots of work on DAM right now :)


Core/VCS Optimizations


Benoit, Stephane and Florent are working on several changes in the Core. Lots of them are already merged onto the main development branch.

JDBC Pool and DataSources Management


DataSources have recently been:

  • internalized (declared via a Nuxeo Extension Point system rather than at container level)
  • aligned on Tomcat pool (to avoid the leaks issue in Apache DBCP)

There are still some issues; this requires more testing.

Hibernate, Audit, XA and Seam


After a great fight, Stephane is now reaching the goal :

  • fixed DataSource management
  • updated Hibernate libs
  • recompiled Seam with new dependencies (to match hibernate)
  • changed H2 config (to have true MVCC via Server mode)
  • fixed Fulltext extraction workers

Some cleanup was also started inside the tests, so that we don't have anymore errors caused H2 concurrency issue. Stephane did the required fixes in most Nuxeo repositories except on AddOns. The goal is to finish the cleanup on nuxeo-features and merge, then :

  • do an quick explanation to all dev team
  • dispatch the addons tests among developers

    UUID Formatting


    UUID format change is done for PGSQL (using PGSQL native support for UUID). To go further, we are now switching from UUID to database-generated Long (i.e. sequence). This will have an impact on UI (and potentially security): docs ID will be 0001, 0002, 0003 ...

But on the other hand, the DB should be much faster:

  • UUID size reduced ( 36 Bytes in String => 16 Bytes in UUID => 8 Bytes in Long )
  • processing on Long is better that on String

Anyway, the "Long format" won't be the default. It may come in handy for very large repositories.
NB: In the near future, we may use Redis to manage ID generation and then migrate Cluster stuff in Redis too.

Ben did the benchmarks for UUID vs VARCHAR.

Reader Benchmark
Writer Benchmark

The results are almost the same when using Sequence instead of native UUID :

+10% / 15% improvement in throughput
response time are 15 to 30% faster

Tag


Florent added a way to query on tags using NXQL.

The documentation now describes how it can be used.

Examples from that doc:

  • SELECT * FROM Document WHERE ecm:tag = 'tag1'
  • SELECT * FROM Document WHERE ecm:tag IN ('tag1', 'tag2')

-- documents with either tag

  • SELECT FROM Document WHERE ecm:tag/ = 'tag1' AND ecm:tag/* = 'tag2'

-- documents with both tags with queryAndFetch:

  • SELECT ecm:tag FROM Document WHERE dc:title = 'something'
  • SELECT ecm:tag FROM Document WHERE ecm:tag LIKE 'abc%'
  • SELECT ecm:tag/1 FROM Document WHERE ecm:tag/1 LIKE 'abc%' AND ecm:tag/*2 = 'tag1'

-- simpler version of above

  • SELECT ecm:tag FROM Document WHERE ecm:tag LIKE 'abc%' AND ecm:tag/* = 'tag1'

    Versioning


    Based on customer feedback, the versioning policy has been changed when documents are created/updated with the FileManager. Versioning policy can now be configured in the FileManager and it is deactivated by default. These changes impact:

  • Import action

  • Drag&Drop action

However, this creates a consistency issue with other create/update actions:

  • create/update via the Forms
  • create/update via Drive
  • update file via D&D
  • create/update via automation

The proposed solution could be:

  • activate FileManager versioning by default in 5.7
  • add create version action (check-in minor and check-in major) in the summary tab
  • keep the version increment option in edit form
  • make update Blob via D&D consistent with FileManager versioning policy

Drive has planned a more clever policy: A version is created if..

  • the author has changed
  • last update was a long time ago

We need to work on a global spec so that we can be consistent. Any feedback is welcome :)

Monitoring Nuxeo

Current Situation


Today, we have a Counter/Probe system based on Java Simon. Our current solution lacks several aspects:

  • JMX/REST API isn't very friendly (ex: no simple getter via JMX)
  • we don't have any infrastructure to aggregate and correlate data
  • we don't have a set of "smart" counters for monitoring a "standard Nuxeo server"
  • we don't have a monitoring console

Ben and Mathieu started looking at a global solution. Based on their feedback, the couple Metrics+Graphite was selected.

Metrics


Metrics library provides all the primitive features we need:

  • counters
  • gauges
  • meters
  • health checks

Ben started plugging some of the existing Nuxeo Counters inside Metrics to be able to test them. See NXP-11079.

Graphite


Graphite is in charge of:

  • storing the data history (with configurable archiving policies)
  • providing operations on the collected data (derivative, aggregation, ...)
  • providing graph features

The data for the graph is fetched:

  • from Metrics for Nuxeo specific data
  • from Diamond for system info
  • from Diamond for DB info

    Testing!


    Ben and Mathieu have setup a Graphite server and defined a configuration to monitor Nuxeo's server during a benchmark. The goal is to validate:

  • the architecture and configuration

  • that Metrics does not consume too many resources
  • that we indeed have all valuable information exposed

The result of this is:

  • a benchmark with all monitoring data
  • a pre-version of the documentation about "monitoring Nuxeo"

Ben integrated a Graphite Dashboard on CI. See sample dashboard.

Next steps


Ben will start the merge/cleanup :

  • remove Simon
  • remove nuxeo-runtime-management (or at least everything that is not useful anymore)
  • write a documentation

ESB/SOA

Talend


In the context of EasySOA, a Nuxeo-Talend connector was started: https://github.com/tiry/nuxeo-talend-components
We're starting to have a basic connector.

Mule


While Tiry was at it, he tried another OpenSource ESB, Mule.
The goal is the same as with Talend, a simple connector. The result can be seen here: https://github.com/tiry/nuxeo-mule-connector

If you wonder what Mule can be used for, take a look at its list of connectors: SalesForce, GitHub, Jira, GoogleDoc, Magento, Jenkins, Redis, SAP, SugarCRM ... they are all here!

OAuth 2


From the beginning, OAuth provides mechanisms to manage authentication between two applications, or one user and two applications. But with OAuth1, the system was really not easy to use in a SOA Context.

With OAuth2, the situation is different: Protocol was updated to better fit with the SOA requirements. As a result, OAuth2 is used more and more in SOA and is by default integrated inside Mule and the Connector Framework.

Content Automation and SOA


Content Automation and Code Generation

With both Talend and Mule, we had the same issue: Having a list of operations that is dynamic is not compliant with ESB standard logic. Most systems seem to assert that the list of operation is known. This seems indeed to be true for most systems, but not for content automation, which is one of its main strengths.

In the context of Mule, it is very easy to expose automation operation via UI or even via SOAP, if I have the java signature. Based on our current model, the idea would be to generate a class that contains all the java signatures of accessible operations, and then let Mule DevKit generate all the plumbing code.

Work on Complex Types


All work done is inside a branch. Next steps:

  • validate non-regression with Ben and merge
  • finish processing of Complex Types

    Automation Schema Settings


    In current automation implementation, the choice of schemas/properties that must be fetched on documents before marshaling is defined by the Http Header X-NXDocumentProperties. While it does work, there is no global setting to define default configuration in the Java Automation client. Since this would be useful for the Mule connector, this will be added.

Automation API


As you know, content automation is a corner stone of the Nuxeo Platform:

  • a lot of clients are using Nuxeo remotely via automation
  • Automation is the power of Studio
  • REST API rocks

    Swagger like API Doc


    I've started playing with REST API Documentation solutions like Swagger and Apiary; you can see a preview here.

The goal is to have:

  • a nice Rest API documentation
  • a JS based console to be able to easily test API

We'll see if we can leverage Swagger/Apiary code + some Lise CSS wizardry to make our automation API look nicer and provide a real JS based console.

NXP-11095 was created to track this work on API showroom.

Exception Handler/Debug


In the current model, it can be very complicated to handle exceptions inside an automation chain. The same pain applies when you want to understand why a chain is crashing. This is clearly something we must improve, because with workflow, we have more and more complex chains.

Exception Management


When running an automation chain, errors can occur (ex: mail server not available, doc. not found ...). We must provide a way for Studio users to be able to define how exceptions must be managed. Based on what we saw with ESB, a solution is to define a chain dedicated to exception management. You can take a look at Mule documentation for an explanation of this model.

Trace


Stéphane started some work for support about capturing automation context to help debugging: see NXP-10081. It is basically about stacking context, and input and output for each operation that is executed, so that in case of error we can track what happened.

This unfinished work is, for now, in a branch on automation-core. This tracing system should be activated:

  • when dev mode is activated (or may be a dedicated flag as suggested by Florent)
  • when an exception management chain is defined

    Complex Types and Branch Merge


    The refactoring on marshaling is still waiting in his branch NXP-10080.

Before finishing and merging this work, we should check that we won't break the protocol: see NXP-11093.

Tiry started doing http capture/replay, but Julien may have a solution to run tests with nuxeo-automation-client V(n-1) against nuxeo-automation-server V(n).

Automation Chain Parameters


Currently we have no easy/clear way to define what the parameters of a chain are. In fact, currently a chain can only retrieve values from the automation context. See NXP-11096 for details.

Contributions

OAuth 2 and OpenID


Nelson contributed some code to support OAuth2 and OpenID. This code was partially merged in a branch:

  • for adding OAuth2 service providers
  • for adding support of OpenID

This is just the first step since a lot of things should be added to have a real OAuth2 support:

  • Nuxeo should be accessible as an OAuth2 provider (i.e. call Nuxeo via OAuth2)
  • Nuxeo Opensocial should be able to use OAuth2

But before going further, we need to solve some issues:

  • Google OAuth2 Libs depends on Guava
  • Shindig and GWT depends on google-collections

Guava is actually a new version of google-collection, but the compatibility does suck.
As a result:

  • with Guava, Nuxeo does not start since Shindig startup failed
  • with Google Collection, OAuth2 startup fails

Upgrading to Guava implies upgrading Shindig ...

This explains why:

  • the OAuth2 partial support is not committed in master
  • I needed a to use a homebrew Google Collection to be able to test OpenID. See NXP-10907.

UX

Log-in Screen


For the log-in screen, we have several aspects we would like to make configurable:

  • some of the CSS properties
  • define visibility of some page blocks (like for the right banner, header, footer ... )
  • add OpenID providers icons/links

Doing this should solve at least two problems:

  • Studio customization of the log-in screen
  • OpenID integration

Basically, the approach will be to integrate inside the authentication service a new extension point that provides plugability for OpenID and Studio requirements; see NXP-10918.

Safe Edit


Safe Edit mode NXP-10258 is almost ready, and will be available in the next days:

  • add a WebDriver test
  • adjust SafeEdit/Page leave detection
  • activate it in the trunk

    Access Keys/Ajax


    We already have the "infrastructure" to associate key binding to actions. See NXP-9383.
    We still need to figure out:

  • What are the actions that should be associated with shortcuts

  • How we advertise shortcuts

    Admin Message


    Lise will move the "admin message" to bottom of the page.

    WAR Resources Cleanup


    Lise wants to start a cleanup work on the icons. There are several goals:

  • remove icons that are not used

  • change the names that don't make sense
  • switch to PNG when needed

Before we start this work, we should define the scope of the changes and review compatibility impacts. The idea is that Lise will start a webapp-resources-compat that will hold resources that have been renamed or deleted from the trunk version.

The bundle could also hold other deprecated web resources: xhtml, js... We surely already have some of those...

Misc

Web Mobile


The mobile webapp has been aligned on Cordova 2.2, but since the project is moving fast we are still late! Anyway, the build and packaging has been automated for both iOS and Android and is now part of our CI chain (using a MacOS Jenkins slave!). The CI chain produces the binaries:

  • APK can be installed on any Android device that is configured to accept dev applications
  • iOS app can be installed on devices that have been explicitly listed in the build config file

Automating build and deployment on an Android emulator is not an issue, but WebDriver and Cordova both instrument the Android WebView so WebDriver and cannot be run against a Cordova enabled WebView ...

Current features include:

  • browse content
  • download files
  • import file
  • file exchange between Nuxeo App and other apps on the device

    Quota


    Mariana has taken over the dev of the quota add-on. What has been done:

  • better UI (including JS charts)

  • better enforcement of rules (check quota consistency over the tree)
  • better management of versions

What she's working on:

  • probably some UI changes (based on Lise's feedback)
  • admin center part
  • reporting on quota usage (simple content view)
  • manage max quota on personal workspace
  • initial computation: add support for versions

For version management, we may want to add a configuration to know how we count versions:

  • count size of version Blobs (what is done currently)
  • only count Blobs that are different from live doc (much more complex because of the possibilities to delete/restore versions)

DAM - Layouts/Widgets/Actions


Some of the work on DAM is the opportunity to:

  • make DAM completely configurable via Studio
  • improve our layout/widget infrastructure

NXROADMAP-131
NXROADMAP-127

What Was Recently Done

FancyBox


Fancybox based actions now support forms inside their included popup, see NXP-9444. That makes it possible to change the 'new', 'import file' actions into real actions ([NXP-9433], still needs a bit of styling) as well as the 'new asset' action in DAM.

This makes the model clean and this will allow us to use it inside Studio. This work will be re-used to call an automation chain via a form (also inside workflows).

PageProvider/Refresh and Events


DAM may require some changes in the PageProvider caching/invalidation system.
There is no hurry since it works for now, but:

  • it means that we should improve the framework
  • it would be cool that content view refresh does work in DAM

Changes have been integrated into master (NXP-10851).

NB: implied to do changes in most of the PageProviders

URL Codec


DAM requires some changes in URL Codec, basically because the URL now stores information about the content view (which page ...). These changes, as well as the URL client side update could be pushed to CAP/DM. Thomas and Anahide will review that. This could open the way to switch CAP/DM in Ajax too.

The goal is to be able to generate a URL for a content view that can be bookmarked + use Html5 history.push (see https://jira.nuxeo.com/browse/NXP-10978).

In Progress

Thumbs View


The thumbs view (default view) in DAM was rebuilt using a thumbnail adapter and a layout(NXP-10825).

The DM icon listing view should be aligned on the same infrastructure. In addition, we must ensure that Studio has the ability to generate such a layout. For that, we need to build a box-like listing layout template.

Bulk Edit


Arnaud and Thomas are working on NXP-11054 to reenable Bulk Edit in DAM and at the same time integrate it inside DM. For now the scope is a "Simple" bulk edit using a pre-configured layout (damBulkEdit in DAM, bulkEdit in DM). The layout will be able to be overridden through Studio.

NB: Based on what was done with diff, Antoine may have ideas for dynamically computing a "shared layout"

Saved Search


Vlad is working on NXP-11056 to build a content view save feature. Code will be based on the faceted search save code.

restDocumentLink


Make restDocumentLink generate URL with a repositoryName instead of a Document: NXP-11045

What's in Progress

Content View Configuration in Studio


Make content view slots more configurable, including from Studio:

  • extract selection related actions in a separate form
  • add configurable page size range

    ReRender - Ajax Panel Identification


    DAM uses Ajax; this means the widgets should reference the zones that should be refreshed. In the future we may want to extend the ID generation system to be able to handle the Ajax refresh use case as well as the cross validation issues.

But for now, we'll look at something simpler and faster. The idea is to add to the container widget an attribute to be able to force the ID associated to the nested a4j:panel element. For DAM, this should be enough; we only need three zones and this would make DAM configuration simpler.

Define container widget holding an outputPanel and allowing to set the ID (NXP-11029).

As decided, DAM defines fixed/hardcoded refresh zones. Configuring Ajax behavior involves actions and widgets:

  • the action defines:
  • if Ajax is supported
  • what are the rerendered zones

  • the widget that renders the actions defines:

  • if Ajax is used
  • an optional rerender zone

NB: adding more Ajax in the pages may force us to sync all Ajax queue to an unique queue.

Widgets that Render Actions that Define Widgets...


In the current model, the action categories are handled by different templates that are aggregated in the global template used by the widget. To make this completely pluggable, we could use WidgetTypes to define rendering of Action Category(NXP-11026).

Documentation!!!


The action/widget model was greatly improved to support new use cases. As a side effect, the model is more complex: more options, more opportunities to make mistakes...

Tasks for updating documentation are already created. But because the model (and vocabulary) is complex, we should leverage layoutDemo to demonstrate use cases for Ajax refresh and widgets/actions composition.

Actions EL Rationalization


We should use JSF EL for action filters. The current action filter implementation relies on a bad JEXL integration. Aligning on JSF EL will:

  • make context and syntax more consistent between actions and widgets
  • fix bugs in the current action filter system

Since we are using more and more actions and are now also mixing them with widgets, it really makes sense to address this point.

About Actions


The asset browser view (as the summary view in DM) is based on a grid layout. This grid layout contains three zones, each containing an action widget. In each of the three zones, the action widget will render subwidgets defined by actions. Using the action system rather that simply nesting subwidgets has several advantages:

  • it's easy to manage visibility via filter
  • it's easy to manage ordering
  • it's easy to contribute a new subwidget from an add-on

This is good, but the sad point is that this is not how Studio Layout Editor works. Indeed Studio editor only defines nested widgets via a "WYSIWYG" editor.

As a side effect, when changing the summary tab of DM:

  • you have to rebuild it from the ground up
  • you potentially lose all the widgets contributed by add-ons

This is not ideal, and this is probably not what we want for DAM. Basically the real problem is that:

  • Studio's editor manages widgets in a static way
  • Studio can not easily do widget rendering based on filters

A possible option would be to have an hybrid model:

  • add action attributes at the widget level
  • so widgets can be placed in Studio normally
  • widgets have extended properties (controller) such as filter, category...
  • dynamically generate the required actions at widget deployment time
  • so that Studio does not have to be aware of actions

Thomas and Anahide will work on this in the next few weeks and this may create new features inside Studio:

  • View editor (not a layout, not a tab, but a view!)
  • Global widget definition (to allow reuse)

Logically, this brings back the question about what is a target platform and how we will define the default config for DAM actions/widgets.


Category: Product & Development
Tagged: Features, Nuxeo Platform