Tuesday, September 3, 2013

Python 2.4 bootstrapping


If you still have Plone 3 sites, most likely you will encounter issues when running bootstrap. At Four Digits, Ralph tweaked the bootstrap.py a bit to make it work.
If '/usr/bin/python2.4 python2.4 bootstrap.py' yields things like:
pkg_resources.VersionConflict: (setuptools 0.6c11 (/tmp/tmpSXDs7g/setuptools-0.6c11-py2.4.egg), Requirement.parse('setuptools>=0.7'))
and then you use the new buildout 2 bootstrap and you get a syntax error, and you fix that and then you get other errors, this alternative bootstrap.py might help:

##############################################################################
#
# Copyright (c) 2006 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Bootstrap a buildout-based project

Simply run this script in a directory containing a buildout.cfg.
The script accepts buildout command-line options, so you can
use the -c option to specify an alternate configuration file.
"""

import os, shutil, sys, tempfile, urllib, urllib2, subprocess
from optparse import OptionParser

if sys.platform == 'win32':
    def quote(c):
        if ' ' in c:
            return '"%s"' % c  # work around spawn lamosity on windows
        else:
            return c
else:
    quote = str

# See zc.buildout.easy_install._has_broken_dash_S for motivation and comments.
stdout, stderr = subprocess.Popen(
    [sys.executable, '-Sc',
     'try:\n'
     '    import ConfigParser\n'
     'except ImportError:\n'
     '    print 1\n'
     'else:\n'
     '    print 0\n'],
    stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
has_broken_dash_S = bool(int(stdout.strip()))

# In order to be more robust in the face of system Pythons, we want to
# run without site-packages loaded.  This is somewhat tricky, in
# particular because Python 2.6's distutils imports site, so starting
# with the -S flag is not sufficient.  However, we'll start with that:
if not has_broken_dash_S and 'site' in sys.modules:
    # We will restart with python -S.
    args = sys.argv[:]
    args[0:0] = [sys.executable, '-S']
    args = map(quote, args)
    os.execv(sys.executable, args)
# Now we are running with -S.  We'll get the clean sys.path, import site
# because distutils will do it later, and then reset the path and clean
# out any namespace packages from site-packages that might have been
# loaded by .pth files.
clean_path = sys.path[:]
import site  # imported because of its side effects
sys.path[:] = clean_path
for k, v in sys.modules.items():
    if k in ('setuptools', 'pkg_resources') or (
        hasattr(v, '__path__') and
        len(v.__path__) == 1 and
        not os.path.exists(os.path.join(v.__path__[0], '__init__.py'))):
        # This is a namespace package.  Remove it.
        sys.modules.pop(k)

is_jython = sys.platform.startswith('java')

setuptools_source = 'http://peak.telecommunity.com/dist/ez_setup.py'
distribute_source = 'http://python-distribute.org/distribute_setup.py'


# parsing arguments
def normalize_to_url(option, opt_str, value, parser):
    if value:
        if '://' not in value:  # It doesn't smell like a URL.
            value = 'file://%s' % (
                urllib.pathname2url(
                    os.path.abspath(os.path.expanduser(value))),)
        if opt_str == '--download-base' and not value.endswith('/'):
            # Download base needs a trailing slash to make the world happy.
            value += '/'
    else:
        value = None
    name = opt_str[2:].replace('-', '_')
    setattr(parser.values, name, value)

usage = '''\
[DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]

Bootstraps a buildout-based project.

Simply run this script in a directory containing a buildout.cfg, using the
Python that you want bin/buildout to use.

Note that by using --setup-source and --download-base to point to
local resources, you can keep this script from going over the network.
'''

parser = OptionParser(usage=usage)
parser.add_option("-v", "--version", dest="version",
                          help="use a specific zc.buildout version")
parser.add_option("-d", "--distribute",
                   action="store_true", dest="use_distribute", default=False,
                   help="Use Distribute rather than Setuptools.")
parser.add_option("--setup-source", action="callback", dest="setup_source",
                  callback=normalize_to_url, nargs=1, type="string",
                  help=("Specify a URL or file location for the setup file. "
                        "If you use Setuptools, this will default to " +
                        setuptools_source + "; if you use Distribute, this "
                        "will default to " + distribute_source + "."))
parser.add_option("--download-base", action="callback", dest="download_base",
                  callback=normalize_to_url, nargs=1, type="string",
                  help=("Specify a URL or directory for downloading "
                        "zc.buildout and either Setuptools or Distribute. "
                        "Defaults to PyPI."))
parser.add_option("--eggs",
                  help=("Specify a directory for storing eggs.  Defaults to "
                        "a temporary directory that is deleted when the "
                        "bootstrap script completes."))
parser.add_option("-t", "--accept-buildout-test-releases",
                  dest='accept_buildout_test_releases',
                  action="store_true", default=False,
                  help=("Normally, if you do not specify a --version, the "
                        "bootstrap script and buildout gets the newest "
                        "*final* versions of zc.buildout and its recipes and "
                        "extensions for you.  If you use this flag, "
                        "bootstrap and buildout will get the newest releases "
                        "even if they are alphas or betas."))
parser.add_option("-c", None, action="store", dest="config_file",
                   help=("Specify the path to the buildout configuration "
                         "file to be used."))

options, args = parser.parse_args()

if options.eggs:
    eggs_dir = os.path.abspath(os.path.expanduser(options.eggs))
else:
    eggs_dir = tempfile.mkdtemp()

if options.setup_source is None:
    if options.use_distribute:
        options.setup_source = distribute_source
    else:
        options.setup_source = setuptools_source

if options.accept_buildout_test_releases:
    args.insert(0, 'buildout:accept-buildout-test-releases=true')

try:
    import pkg_resources
    import setuptools  # A flag.  Sometimes pkg_resources is installed alone.
    if not hasattr(pkg_resources, '_distribute'):
        raise ImportError
except ImportError:
    ez_code = urllib2.urlopen(
        options.setup_source).read().replace('\r\n', '\n')
    ez = {}
    exec ez_code in ez
    setup_args = dict(to_dir=eggs_dir, download_delay=0)
    if options.download_base:
        setup_args['download_base'] = options.download_base
    if options.use_distribute:
        setup_args['no_fake'] = True
        if sys.version_info[:2] == (2, 4):
            setup_args['version'] = '0.6.32'
    ez['use_setuptools'](**setup_args)
    if 'pkg_resources' in sys.modules:
        reload(sys.modules['pkg_resources'])
    import pkg_resources
    # This does not (always?) update the default working set.  We will
    # do it.
    for path in sys.path:
        if path not in pkg_resources.working_set.entries:
            pkg_resources.working_set.add_entry(path)

cmd = [quote(sys.executable),
       '-c',
       quote('from setuptools.command.easy_install import main; main()'),
       '-mqNxd',
       quote(eggs_dir)]

if not has_broken_dash_S:
    cmd.insert(1, '-S')

find_links = options.download_base
if not find_links:
    find_links = os.environ.get('bootstrap-testing-find-links')
if not find_links and options.accept_buildout_test_releases:
    find_links = 'http://downloads.buildout.org/'
if find_links:
    cmd.extend(['-f', quote(find_links)])

if options.use_distribute:
    setup_requirement = 'distribute'
else:
    setup_requirement = 'setuptools'
ws = pkg_resources.working_set
setup_requirement_path = ws.find(
    pkg_resources.Requirement.parse(setup_requirement)).location
env = dict(
    os.environ,
    PYTHONPATH=setup_requirement_path)

requirement = 'zc.buildout'
version = options.version
if version is None and not options.accept_buildout_test_releases:
    # Figure out the most recent final version of zc.buildout.
    import setuptools.package_index
    _final_parts = '*final-', '*final'

    def _final_version(parsed_version):
        for part in parsed_version:
            if (part[:1] == '*') and (part not in _final_parts):
                return False
        return True
    index = setuptools.package_index.PackageIndex(
        search_path=[setup_requirement_path])
    if find_links:
        index.add_find_links((find_links,))
    req = pkg_resources.Requirement.parse(requirement)
    if index.obtain(req) is not None:
        best = []
        bestv = None
        for dist in index[req.project_name]:
            distv = dist.parsed_version
            if distv >= pkg_resources.parse_version('2dev'):
                continue
            if _final_version(distv):
                if bestv is None or distv > bestv:
                    best = [dist]
                    bestv = distv
                elif distv == bestv:
                    best.append(dist)
        if best:
            best.sort()
            version = best[-1].version

if version:
    requirement += '=='+version
else:
    requirement += '<2dev -c="" 0:="" a="" above="" add="" already="" any="" apparently="" args.append="" args="" back="" bootstrap="" br="" buildout="" by="" c="" clean="" cmd.append="" cmd="" command="" directory="" easy_install.="" egg="" eggs_dir="" else:="" env="" error="" errors="" exitcode="" for="" function="" if="" import="" in="" install="" into="" is="" is_jython:="" isn="" it="" main="" message="" n="" none:="" not="" occurred="" ook="" options.config_file="" options.eggs:="" otherwise="" output="" prefer="" prefers="" print="" provided="" push="" requirement="" s="" shutil.rmtree="" subprocess="" sys.executable="" sys.exit="" sys.stderr.flush="" sys.stdout.flush="" t="" temporary="" that="" the="" there="" this="" to="" trying="" up="" was="" we="" were="" when="" windows="" would="" ws.add_entry="" ws.require="" zc.buildout.="" zc.buildout.buildout.main="" zc.buildout.buildout="">
Credits for this go to Ralph.

Wednesday, July 17, 2013

Reusing code between views and viewlets

You may wonder, as i did today, how you can re-use code from your View inside a Viewlet. Just to help out the occasional googler, i'm recording my solution here.

Any Viewlet has its View as an attribute (see ViewletBase's __init__ method). So in your Viewlet class, you can do something like:
Class MyViewlet(ViewletBase):
    ...
    def reused_method(self):
            return self.view.original_method()

Thursday, July 11, 2013

try-except HTTPError and URLError


urllib2's HTTPError subclasses URLError, so the bit of code below would never get to the second except statement, because that error would aready be caught in the first except. To get this code to work, reverse the order.



            try:
                response = urlopen(url)
            except URLError:
                # This happens if the server can't be reached.
                # Most likely the service is down, in this case we don't want
                # to do anything.
                # Another possibility is that we have the API url wrong.
                self.url_errors.append(value)
                return True
            except HTTPError:
                # as Flattr API does return an error code (404 not found) when
                # looking for users that do not exist, urllib2 treats that as
                # an error, so no need to investigate or differentiate between
                # that and a normal HTTP error. Both are enough to not validate
                # a Flattr user.
                self.http_errors.append(value)
                return False


Wednesday, June 19, 2013

Update your robots.txt, and nginx.conf

[update: not sure if robots.txt is respected]
We recently had an interesting afternoon when our servers were all but brought down by a crawler called 80legs. This crawler showed no mercy on our poor site and bore down with great vengeance and furious anger on articles so old that the cache didn't have them.
We get a lot of crawlers, if you've spent hours gazing at your instance-Z2.logs like i have (beats a washing machine!), you've seen a lot of them. What was surprising was the amount of traffic it generated: it was more like a DDoS attack than a crawling service.
The thing with 80legs is: Crawls are initiated by their users. Any user. There is no check to see if the user is affiliated with the site to be crawled. You can create an account with a fake mailaddress. If they have, as they boast, 50,000 servers across the world getting 10,000 pages (the maximum amount on a free account) in no time, i can imagine other sites also having trouble with this. 
So after the initial quick fix (blocking the user agent '008' in nginx), i contacted the company to ask what measures they take to keep unthinking (or even malicious) users from bringing down sites. The answer was swift, simple and sucky: nothing. They apologized, and added our site to a list.
80legs says they obey robots.txt, and i think it's wise that you update yours. But the wikipedia page has some examples of cases where robots.txt was not respected, and a Twitter search indicates there are still problems. So the best approach seems to be not to rely on robots.txt, and set up your webserver / cache to ignore requests from this user agent.

Monday, April 8, 2013

Testing CSS in Plone with Robot Framework

[update 09-04: Modified as per Asko's comment]

You can use RF to make sure your CSS rules work as you want them to. This will give you confidence to move around large parts of you CSS codebase and be certain that everything still works. To do so, you must define your own library. 

Defining a custom library

In your robot test file, link the library like so:
Library  der.freitag.tests.robot.libraries.FreitagRobotSelenium2Keywords
The class FreitagRobotSelenium2Keywords in tests/robot/libraries.py holds the relevant methods to use Selenium: 
# -*- coding: utf-8 -*-
from robot.libraries.BuiltIn import BuiltIn

from selenium.webdriver.support.color import Color


class FreitagRobotSelenium2Keywords(object):

    @property
    def s2l(self):
        return BuiltIn().get_library_instance('Selenium2Library')

    def value_of_css_property(self, property_name, locator):
        """Returns value of property found by locator
        """
        s2l = self.s2l
        element = s2l._element_find(locator, True, True)
        prop_value = element.value_of_css_property(property_name)
        s2l._info('value of css property {0} : {1}'.format(property_name,
                                                           prop_value))
        return prop_value

    def _color_checker(self, color_value, locator, property):
        color_value = color_value.lower().replace('hex', '', 3)
        color_expected = Color.from_string(color_value).rgba
        color_found = self.value_of_css_property(property, locator)
        if color_expected != color_found:
            error_msg = 'Expected color {0}, found {1} instead.'
            raise AssertionError(error_msg.format(color_expected, color_found))

    def check_color(self, color_value, locator, property='color'):
        """Checks if element found by locator has correct value
        """
        self._color_checker(color_value, locator, property)

The methods to get and check the color were taken from https://github.com/bettertest-org/RobotSelenium2Keywords.

Using the library

The rest of our robot_colors.txt looks like this:
...
*** Variables ***

${X_AUTHOR_NAMES} =  //article//span[@class]


*** Test cases ***

Scenario: Author names are pink by default, no hover needed
  Given i am on the homepage
   Then author names are shown in pink


*** Keywords ***

Suite Setup
    Open browser  ${PLONE_URL}  browser=${BROWSER}  remote_url=${REMOTE_URL}  desired_capabilities=${DESIRED_CAPABILITIES}

Suite Teardown
    Close all browsers

Given i am on the homepage
    Location should be  ${PLONE_URL}

Then author names are shown in pink
    Check color  \#bd014d  ${X_AUTHOR_NAMES}


More info

Check out plone.app.robotframework.

Thursday, March 7, 2013

Robot Framework testing hints

Updates:
2013-03-15 Remove external content loaded by javascripts
2013-03-14 Be careful with Element Should Not Be Visible
2013-03-13 Using Chrome driver in RF tests
2013-03-12 Notes about Plone-specificity

Some things that come in handy for Plone developers working on Robot Framework tests:

Tags

RF supports tags. Add a line [Tags] tag1 tag2:
*** Test cases ***

Scenario: Clicking the submit button hides it
  Given i am logged in
    and i am on an article
   When i simulate clicking the comment submit button
   Then the submit button has class disabled

Scenario: Submitting a comment displays it in the page
[Tags] working_on_it
  Given i am logged in
    and i am on an article
   When i type something in the comment box
    and i click the comment submit button
   Then the page shows the comment
You can now run only the latter test: ./bin/test -m der.freitag -t working_on_it (This is Plone-specific. See Asko's comment below.)

"Typing" in TinyMCE

TinyMCE is in an iframe, get to it with Select Frame:
i type something in the comment box
    Select frame  form-widgets-comment-text_ifr
    Input text  tinymce  ${COMMENT}
    Unselect Frame
More useful commands and selectors at http://rtomac.github.com/robotframework-selenium2library/doc/Selenium2Library.html

Using Chrome driver

In order to also test in Google Chrome, get the chromedriver binary from http://code.google.com/p/chromedriver/downloads/list, install it as per http://code.google.com/p/selenium/wiki/ChromeDrive (short version: put the driver on your $PATH and chmod +x), and run your (Plone) tests (on Linux) with:
$ ROBOT_BROWSER=Chrome ./bin/test -m der.freitag -t wip 

Be careful with Element Should Not Be Visible

"Element should not be visible" will also return True (ie. not fail) if the element is not in the page... so probably you'll first want to check if it's there at all.

Remove external content loaded by javascripts

If your site loads external javascripts like ads, these may inject invalid HTML, or open popups, or just make the page load slower. How they behave is unpredictable, which can make creating tests all but impossible. You'll probably want to disable everything that depends on something from the outside.
To do this in Plone, you may create a Generic Setup profile for RF tests that disables the viewlets that load these external scripts. In our testing.py, we now have:
class DerFreitagAcceptanceLayer(PloneSandboxLayer):

    def setUpPloneSite(self, portal):
        applyProfile(portal, 'der.freitag:default')
        applyProfile(portal, 'der.freitag.tests:testing')
        applyProfile(portal, 'der.freitag.tests:robot')
The der.freitag.tests:robot profile lives in tests/robot/profile, it just has a viewlets.xml.
The tests layer DerFreitagAcceptanceLayer is a separate layer created specifically for RF tests. Actually DerFreitagRobotLayer would have been a better name.

Wednesday, February 27, 2013

Using a spare router to extend wifi coverage (and an upstart init script to control routing)

Or, "How i learned to route eth0 through wlan0 and script it".

First off, a disclaimer

This is not about using my spare router as a signal amplifier. I didn't find a way to do that with my Sweex LW150. Also note that, as the title unfortunately omits, this story also needs a spare laptop. Or something else that has wifi and an ethernet port, and can run iptables.

The house is not too big, the signal is too weak.

The use case: Wireless router is at one end of the house, near the stables. Large parts of the house, including the salon and the servants' quarters, are out of reach. And halfway, in the music room, there's this small laptop that's always on when i'm at home, and only runs Spotify and Guayadeque (on Lubuntu 12.10).

Iptables, and scripting it

So i hooked up the spare router to the laptop (which by now runs a dhcp server). To make traffic pass from the spare router (eth0) to the internet connection (wlan0), there's some iptables stuff to do. To make sure the routing restarts when the laptop has gone down, i created an upstart configuration. This also enables me to turn routing off and on quickly. Here's how:
  1. Create /etc/init/route-eth0-wlan0.conf:
    # route-eth0-wlan0 - Route traffic from eth0 to internet on wlan0
    #
    # We have internet on our machine's wlan0 interface, an unrelated wireless
    # router in eth0, and want to route that router's traffic.
    # The router on eth0 will then be an extra access point for our internet,
    # extending its range.

    description "Route eth0 through wlan0"

    # Make sure we start before an interface receives traffic
    start on (starting network-interface
              or starting network-manager
              or starting networking)

    stop on runlevel [!023456]

    console output

    pre-start script
        # Set up IP FORWARDing and Masquerading (NAT)
        iptables --table nat --append POSTROUTING --out-interface wlan0 -j MASQUERADE
        iptables --append FORWARD --in-interface eth0 -j ACCEPT
        # enable IP forwarding
        echo 1 > /proc/sys/net/ipv4/ip_forward
    end script

    post-stop script
        # disable IP forwarding
        echo 0 > /proc/sys/net/ipv4/ip_forward
        # delete old configuration, if any
        # Flush all the rules in filter and nat tables
        iptables --flush               
        iptables --table nat --flush
        # delete all chains that are not in default filter and nat table, if any
        iptables --delete-chain    
        iptables --table nat --delete-chain
    end script
  2. Symlink /etc/init.d/route-eth0-wlan0 -> /lib/init/upstart-job
Now i can start and stop the routing by running sudo service route-eth0-wlan0 (start|stop).

Update 2013-03-27: The motto "because we can" applies to all of the above. After, i applied "use the right tool for the job" and bought a wireless repeater from Amazon (Anker Kompakt, 26 Euros, works great).

Monday, February 11, 2013

My Barcelona Testing Sprint



[edit 14-2: see bottom of article for information that came after the sprint]

The Barcelona Sprint was about Plone's testing. Most of Plone's code is covered by tests, and these tests are automatically run by a Jenkins server, so we know when stuff starts not working anymore.




One focus of the sprint was improving the Jenkins setup, so a commit or pull request that breaks the build triggers an email, irc notification or otherwise to the author. This code now goes by the name of mr.roboto [1].

Another focus area was to get more Robot Framework tests into Plone. RF fires up a real browser, where Javascript also works. This way we can test things like overlays and folder re-ordering.

To show everyone how to use this goodness, we wrote documentation and modified the template for creating new Plone products so the boilerplate is already there.

To go more into detail, the sprint went like this for me:


I spent the first day familiarizing myself with Robot Framework tests, using these links: [2-9]. I set up a basic add-on package that tested if an overlay popped up.

The second day i paired up with Carles Bruguera of UPC University (upc.edu), and we started working on Products.CMFPlone. The great thing about RF is that you can write stuff like:
Scenario: Login overlay:
 Given I am not logged in
  When I click the login button
  Then I should get an overlay with login form

Statements like "I am not logged in" are called keywords, you define these somewhere else in your test. It makes for easy reading, you could (theoretically) even show it to your customer and ask "Is this what it is supposed to do?"


 

 The third day we continued this work. All the while, Laura Perez Mayos (of Iskra, iskra.cat) and Víctor Fernández de Alba (also UPC) were working on the other tests in CMFPlone. By the end of the day we were done [10], and there was beer. Have a look and see what it looks like [11].

Some more information on the sprint will probably seep through coming week. The project page is [12], the twitter hashtag #bcnsprint. The last link you should definitely see is [13].

[1] https://github.com/plone/mr.roboto
[2] http://code.google.com/p/robotframework/wiki/HowToWriteGoodTestCases
[3] http://robotframework.googlecode.com/hg/doc/userguide/RobotFrameworkUserGuide.html?r=2.7.6
[4] http://robotframework.googlecode.com/hg/doc/libraries/BuiltIn.html?r=2.7.6
[5] http://rtomac.github.com/robotframework-selenium2library/doc/Selenium2Library.html
[6] Reference documentation: http://code.google.com/p/robotframework/
[7] https://github.com/plone/plone.act/blob/master/src/plone/act/keywords.txt
[8] http://datakurre.pandala.org/2012/09/getting-started-with-robotframework-and.html
[9] http://datakurre.pandala.org/2012/09/speed-up-your-plone-add-on-tests-on.html
[10] For a given value of done. As the testrunner will try out different browsers, we might find some failing tests. So far we only tried Firefox and a little of Chrome.
[11] https://github.com/plone/Products.CMFPlone/tree/master/Products/CMFPlone/tests/robot
[12] http://www.coactivate.org/projects/barcelona-sprint/project-home
[13] https://www.youtube.com/watch?v=3cShYbLkhBc

information that came after the sprint:
http://plone.293351.n2.nabble.com/Introducing-mr-roboto-td7563163.html