Issue :



We recently realized on a CPS instance while plugging an external catalog
(ZSQLCatalog actually) that a lot of reindexations (something like 10)
occurred at creation time. Same behavior for content modifications and still
the same with workflow transition executions.



Usually, the problem is not visible but as soon as you get big catalog (for
instance 600 K entries) then the creation or workflow transition execution
had a tendency to be slow down.



First step : object repository objects



Historically, the object repository objects within CPS3 were indexed.
It wasn't necessarily anymore since CPS branch 3.1.x and the new rearch
mechanism that basically aggregates the repository object attributes on the
proxy itself with the help of dedicated proxy indexes. (Ben did a great work
at this time and the search speed of CPS grew up big time after this).



Thus, the first step was to avoid repository object indexations.



Direct benefits :

  • Half of the indexations avoided
  • Lighter catalog and thus more efficient ones when they become really
    big.




Second step : getting only one reindexation per-object and
per-transaction no matter what appends to the objects during the
transaction.



First, I've been trying to figure out where to call my
reindexObject() method within the CPS architecture. It's definitely
not mangeable especially when you got during the same transitions : object
reindexation, object security reindexation and workflow transition executed
with the object involved. The CMF is definitely not optimized at all on this
point.



Then, we started to discuss with Florent about a possible post-publishing
hook on the Zope publisher. No existing hooks were available for this and no
way to patch properly the framework for our extensions.



We discussed about the problem on zope-dev and zodb-dev (thanks to the
people over there for the answers)



The problem we were faced to was that the ZODB-3.2 (the one Zope-2.7.x is
using) doesn't provide good ways to hook just before the first commit
phase of the transaction.



There's a way to register a hook executed at the beginning of the first
commit phase implementing this interface :


http://cvs.nuxeo.org/cgi-bin/viewcvs.cgi/CPS3/CPSCore/interfaces/IZODBTransactionHook.py?rev=1.1&view=auto


and registering the object implementing this interface like this  :




>>> get_transaction().register(hookInstance)



The problem is that this hook is called too late in the commit process
(during phase1) and the objects are already frozen at this time. It's thus
not possible to deal with Persistent Objects and thus no way to cope with
our Indexation problems.



We discovered that Zope-3.3.x would have provide a way to do the job with
the Synchronizers and the beforeCompletions() API :



See the discussion there :



http://mail.zope.org/pipermail/zope-dev/2005-April/024566.html




I finally decided to extend the ZODB-3.2 transactions to make them support
pre-commit hooks with a subscriber mecanism that I may use for an Indexation
manager subscriber.



A pre transaction commit hook has been added to the ZODB Transaction
supporting subscribers registration. Each subscriber has a commit() method
called just before the
first commit phase that contains the actual code to execute and an abort()
method called at the beginning of the abort() method of the transaction
class. :


Pre-commit hook definition with subscribers :

http://cvs.nuxeo.org/cgi-bin/viewcvs.cgi/CPS3/CPSCore/PatchZODB.py

Subscriber definition :

http://cvs.nuxeo.org/cgi-bin/viewcvs.cgi/CPS3/CPSCore/TransactionCommitSubscribers.py?rev=1.1&view=auto

Subscriber interface :

http://cvs.nuxeo.org/cgi-bin/viewcvs.cgi/CPS3/CPSCore/interfaces/ITransactionCommitSubscriber.py?rev=1.1&view=auto

You may register your own subscribers by implementing the interface

above and register them to the list of pre-commit transaction subscribers.



This is checked in and available in the HEAD of CPSCore.



Note that the extensions currently defined within CPSCore will
disapear.



The threads are still running talking about an implementation for ZODB-3.4
of a simple before transaction commit hook :


(Post originally written by Julien Anguenot on the old Nuxeo blogs.)