Thursday, December 8, 2011

nginx buildout failing at openssl on Mac OS X Lion

The error message was:
make[1]: *** [objs/src/event/ngx_event_openssl.o] Error 1
make: *** [build] Error 2
nginx-build: Command failed with exit code 2: make
Running the steps suggested here solved the problem: http://code.google.com/p/phusion-passenger/issues/detail?id=650#c2

Tuesday, November 15, 2011

How to disable comments on comments

The threaded view of plone.app.discussion's comment item tends to look a bit confusing when there are many threads. Ons possible solution would be disallowing comments on comments, so you'd just get a flat list of items. This is similar to how many other commenting systems work (like Facebook and Twitter, for example).

Comments on comments can be disabled in the ZMI: go to 'portal_types' -> 'Discussion Item', uncheck "Allow Discussion".

Thursday, October 27, 2011

Goodbye GW20e, hello Freitag

After almost four years at Goldmund, Wyldebeast and Wunderliebe, the time has come to move on.

In december i will start a new job at the weekly magazine Der Freitag in Berlin. I look forward to working with our small team of in-house web developers. It's a good place to put some of my skills to work, and i expect to learn a lot of new things too. And of course i look forward to Berlin.

Life without GW20e will not be the same. It's been a truly amazing time, and i feel lucky to have been part of it. A big thank-you is due for all the things i've learned, and for all the good times we've had.

I'll still be posting my (possibly) interesting (to some) Plone stuff here.

Thursday, June 9, 2011

Migrate files from Compound-/ArrayField to File objects in folder

A migration script. The use case is a site which currently allows users to add a list of files to custom objects. I want to make these objects folderish, extract the files from field and store them as regular Plone files, so we can migrate to Plone 4 and use blobstorage.

import transaction
from zope.component import queryUtility
from Acquisition import aq_parent

from plone.i18n.normalizer.interfaces import IIDNormalizer
from Products.CMFCore.utils import getToolByName
from Products.CMFPlone.utils import _createObjectByType
from Products.CMFPlone.utils import safe_unicode
from Products.Five.browser import BrowserView
from Products.Five.browser.pagetemplatefile import ViewPageTemplateFile

class MyContentTypeMigrationView(BrowserView):
""" Migrate from non-folderish objects with files in CompoundField/ArrayField
to folderish (containing ATFile objects)."""

__call__ = ViewPageTemplateFile("templates/migrate_files.pt")

def __init__(self, *args, **kwargs):
super(MyContentTypeMigrationView, self).__init__(*args, **kwargs)
self._catalog = getToolByName(self.context, 'portal_catalog')
self._file_counter = 0
self._mycustomtype_counter = 0
self._normalizer = queryUtility(IIDNormalizer)

def rename(self, obj):
"""Rename object: strip trailing '.pdf', at least from id. """
stripped_extensions = ['.pdf',]
for extension in stripped_extensions:
# No need to set a Title yet, not sure if we have to at all.
# obj.setTitle(obj.Title().rstrip(extension))
parent = aq_parent(obj)
oid = obj.getId()
if oid.endswith(extension):
parent.manage_renameObject(oid, oid.rstrip(extension))
changed = True

def create_file_objects(self, obj):
"""Read files from 'documents' CompoundField, create ATFile objects and
delete files from 'documents' CompoundField.

"""
files = obj.getDocuments()
for field_file in files:
# file from CompoundField may be empty
if field_file is not None:
filename = field_file.filename
if not filename:
# filename may be empty, in that case take it from Plone
# object url
filename = field_file.absolute_url().split('/')[-2]
file_field_id = field_file.id()
try:
# filename must be unicode
filename_safe_uni = safe_unicode(filename)
new_id = self._normalizer.normalize(filename_safe_uni)
self.context.plone_log("Creating file %s (%s)" %
(filename, file_field_id) )
if hasattr(obj, new_id):
self.context.plone_log("Object with id %s exists, skipping" % new_id)
else:
_createObjectByType('File', obj, id=new_id, title=filename)
self.context.plone_log("File object created")
file_object = obj.get(new_id)
file_object.setFile(field_file)
file_object.reindexObject()
self._file_counter += 1
self.context.plone_log("File content set (#%d)" % self._file_counter)
# "Delete" file from CompoundField
delattr(obj, file_field_id)
self.context.plone_log("File deleted from CompoundField")
except:
self.context.plone_log("ERROR creating / setting / deleting file")
import pdb; pdb.set_trace()

def html(self):
"""Render something"""
html = ''
brains = self._catalog(
portal_type='MyCustomType',
)
for brain in brains:
obj = brain.getObject()
self.rename(obj)
self.create_file_objects(obj)
obj.reindexObject()
# commit subtransaction
transaction.get().commit(True)
self._mycustomtype_counter += 1
self.context.plone_log("MyContentType object nr %d)" % self._mycustomtype_counter)
html += 'migrated <a href="%s">%s</a><br />' % (
obj.absolute_url(), obj.getId() )
return html

How to plug into Plone's id (for url) generating mechanism

Plone can convert titles to id's, which are used in the URL. To use this conversion in your own code, use the IIDNormalizer utility.

[edit: removed broken link to Plone's developer documentation, added code snippet]


from zope.component import queryUtility
from plone.i18n.normalizer.interfaces import IIDNormalizer

id = queryUtility(IIDNormalizer).normalize(title)

Wednesday, June 1, 2011

Plone 3 on Natty: Python 2.4 with PIL

Installing PIL in a (hand-compiled) python2.4 yielded "python "ZLIB (PNG/ZIP) support not available", although i did have zlib1g-dev installed.

Found the answer in http://ubuntuforums.org/showthread.php?t=1751455#post10810486: Natty puts some libs in /usr/lib/x86_64-linux-gnu/, where PIL can't find them. Symlinking helps.

Monday, May 16, 2011

Managing virtualenvs with mkvirtualenv

Read this, thanks Doug!

Thursday, May 5, 2011

Commands to replace xdv namespaces with diazo

The plone.app.theming page is clear enough about migrating products from collective.xdv to plone.app.theming. Here are some commands to make the necessary namespace substitutions in xml files:


perl -p -i -e 's/xmlns=\"http:\/\/namespaces.plone.org\/xdv\"/xmlns=\"http:\/\/namespaces.plone.org\/diazo\"/' *.xml
perl -p -i -e 's/xmlns:css=\"http:\/\/namespaces.plone.org\/xdv\+css\"/xmlns:css=\"http:\/\/namespaces.plone.org\/diazo\/css\"/' *.xml

Wednesday, May 4, 2011

Creating Plone content when installing / upgrading a product

[update 2012-05-23: link to GS import/upgrade at http://collective-docs.readthedocs.org/en/latest/components/genericsetup.html#custom-installer-code-setuphandlers-py]
[update 2013-03-23: underscore added to function call]

Following up on my previous post, i've extended the method to create objects a bit.

Short reminder: this method is intended for use when Generic Setup (profiles/default/structure) can't create content in the way you want, for example when you have custom content types, or want to change "exclude_from_navigation" settings or workflow state. This article assumes you will call your setuphandler.py's methods from a Generic Setup import or upgrade step.

## setuphandlers.py

from Products.CMFPlone.utils import _createObjectByType
from Products.CMFCore.utils import getToolByName
from Products.CMFCore.WorkflowCore import WorkflowException

def setupVarious(context):

    # Ordinarily, GenericSetup handlers check for the existence of XML files.
    # Here, we are not parsing an XML file, but we use this text file as a
    # flag to check that we actually meant for this import step to be run.
    # The file is found in profiles/default.

    if context.readDataFile('My.Product_various.txt') is None:
        return

    # Add additional setup code here

def _my_structure():
    return [
        {   'id': 'new-folder', 
            'title': 'New Folder',
            'description': 'Folder for authenticated users',
            'type': 'Folder',
            'workflow_transition': 'retract',
            'exclude_from_nav': True,
            },
        ]

def _createObjects(parent, children):
    """This will create new objects, or modify existing ones if id's and type
    match.

    This takes two arguments: the parent to create the content in, and the
    children to create.

    Children is a list of dictionaries defined as follows:

    new_objects = [
        {   'id': 'some-id', 
            'title': 'Some Title',
            'description': 'Some Description',
            'type': 'Folder',
            'layout': 'folder_contents',
            'workflow_transition': 'retract',
            'exclude_from_nav': True,
            'children': profile_children,
            },
        ]
    
    * layout:               optional, it sets a different default layout
    * workflow_transition:  optional, it tries to start that state transition
        after the object is created. (You cannot directly set the workflow to 
        any state, but you must push it through legal state transitions.)
    * exclude_from_nav:     optional, excludes item from navigation
    * children:             optional, is a list of dictionaries (like this one)

    """

    parent.plone_log("Creating %s in %s" % (children, parent))

    workflowTool = getToolByName(parent, "portal_workflow")

    existing = parent.objectIds()
    for new_object in children:
        if new_object['id'] in existing:
            parent.plone_log("%s exists, skipping" % new_object['id'])
        else:
            _createObjectByType(new_object['type'], parent, \
                id=new_object['id'], title=new_object['title'], \
                description=new_object['description'])
        parent.plone_log("Now to modify the new_object...")
        obj = parent.get(new_object['id'], None)
        if obj is None:
            parent.plone_log("can't get new_object %s to modify it!" % new_object['id'])
        else:
            if obj.Type() != new_object['type']:
                parent.plone_log("types don't match!")
            else:   
                if new_object.has_key('layout'): 
                    obj.setLayout(new_object['layout'])
                if new_object.has_key('workflow_transition'): 
                    try:
                        workflowTool.doActionFor(obj, 
                            new_object['workflow_transition'])
                    except WorkflowException:
                        parent.plone_log(
                            "WARNING: couldn't do workflow transition")
                if new_object.has_key('exclude_from_nav'):
                    obj.setExcludeFromNav(new_object['exclude_from_nav'])
                obj.reindexObject()
                children = new_object.get('children',[])
                if len(children) > 0:
                    _createObjects(obj, children)

def createContent(context):
    portal = getToolByName(context, 'portal_url').getPortalObject()
    _createObjects(portal, _my_structure())

Tuesday, March 15, 2011

Building Varnish on Ubuntu: libpcre

While trying to use plone.recipe.varnish, i ran into an error:
SystemError: ('Failed', './configure --prefix=/home/kees/project_buildouts/wsr/parts/varnish-build')


Going into the directory in /tmp and trying to run ./configure there, i get:
configure: error: Package requirements (libpcre) were not met:

No package 'libpcre' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables PCRE_CFLAGS
and PCRE_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.


The answer is to simply install the package libpcre3-dev.

On Debian, you might also have to do:

export PCRE_LIBS="-L/usr/lib -lpcre"
export PCRE_CFLAGS=-I/usr/include/pcre

Monday, February 7, 2011

Upgrading my Samsung Apollo Galaxy to Froyo

Finally got round to upgrading my phone (Samsung i5801) to Android 2.2. It was relatively easy, following the how-to at xda-developers. The result is stunning, the phone is much more responsive now.

Update 2011-02-08: After updating a second time (to the latest JPF firmware) with the SIM and memory cards in the phone, all my apps and settings were gone, this did not happen the first time (JPB firmware with SIM and memory removed).

Some minor roadblocks:

  • Having to use Windows (for Odin)

  • Having to install a driver in Windows to acces the phone, this step was missing from the how-to. Admittedly i used an ancient Vista.

  • Seeing Odin report failure. Luckily, the phone was accessible right away and reported running Froyo.



Nice things:

  • 3G, wifi, apps worked right away (it's probably a good idea to remove your SIM and memory card)

  • Froyo supports USB tethering by default

  • Has better UI for Gmail.

  • Speech input for web searches.

  • Boots faster.

  • Everything is faster.

Tuesday, January 25, 2011

Buildout error: Products.CMFPlone 4.1a1 requires 'Zope2>2.13.0'.

When running my Plone 4.0.2 buildout, i got this error:

We already have: Zope2 2.12.13
but Products.CMFPlone 4.1a1 requires 'Zope2>2.13.0'.


But where did i say i wanted to use Plone 4.1a1? It seems packages which depend on Products.CMFPlone (like plone.app.registry) by default (when running buildout without the -N switch) get the latest version, which is 4.1a1. And this, in turn, tries to pull in the rest of Plone 4.1.

This doesn't happen with Plone 4.0.3, as http://dist.plone.org/release/4.0.3/versions.cfg does the pinning, so upgrading to 4.0.3 is the best option.

Wednesday, January 19, 2011

Why is my Data.fs so big?

If your Data.fs is still big after packing, here are some tricks you might try to find out why it is big.

1. Write a custom script (example see below) that does a catalog query and returns each brain's getObjSize(). This will only find catalog'ed objects, but these are probably the ones that make your Data.fs big.

2. Check portal_historiesstorage to see how many old versions of objects are stored, and how big these objects are. Maybe there's this one news item with a 4Mb image of which there are 365 versions.

3. Check portal_purgepolicy's setting: by default it will keep an infinite number of versions (-1).

Thanks to Huub for these tips.


# example script to get a list of large content objects
from Products.CMFCore.utils import getToolByName

catalog = getToolByName(context, 'portal_catalog')
total = 0
for r in catalog():
size_str = r.getObjSize
(num, pow) = size_str.split(' ')
num = float(num)
if pow == 'kB':
size = num*1
if pow == 'MB':
size = num*1000
size = int(size)
if size > 100:
print size, r.getURL()
total = total + size

print "Total %s kB" % total
return printed

Wednesday, January 12, 2011

Scrape a Picasa web album

A friend shared a Picasa album with me, but i couldn't download it. Picasa Webalbums Assistant to the rescue, thanks Bradley. Works on Ubuntu too, does just what i wanted.

Monday, January 3, 2011

Making cron send mail to an external address

So, you've filled in "me@example.com" as your address in your crontab's MAILTO field, but nothing happens. Probably you're getting a message like "Mailing to remote domains not supported".

There's an easy way to send mail to an external address from a cron job: set up
http://untroubled.org/nullmailer/. It asks you for the address of an SMTP server and uses that to send your precious update notifications of that remote server you never log in on anymore.

If you have accidentally installed another mail program (like mailutils), you can dpkg-reconfigure nullmailer to make it work again.