Agendaless Blog

Pyramid Traversal Tutorial, updated

Interested in learning about Pyramid's traversal approach for sites with hierarchies? For my Pyramid tutorial in Brazil last October, I wrote a "quick" Traversal Tutorial. I didn't talk too much about it, as it had some typos. I just fixed the wording and also confirmed that it runs under Python 3.4.

This tutorial tries to introduce traversal in some bite-sized chunks with working code:

  • A Bootstrap-layout that places the elements used in the tutorial
  • A basic root object with a view at /hello....with no routes
  • Views that get their information and security retrieved and passed in by the framework
  • A nested hierarchy of resources
  • Ditto, but with resources of different types, with type-specific views...and no stupid /sometype/1/anothertype/2 in the resource location (URL)
  • Adding any kind of thing inside any kind of thing
  • Persistence, first with ZODB then with SQLAlchemy (using adjacency list relationships and polymorphic tables)

posted: 2014-07-30 09:54  by Paul Everitt   |   permalink

Traversal in AngularJS using ui-router

Summary: Traversal is an alternative to routes for matching URL patterns to web application request handling. I implemented traversal in AngularJS using ui-router.

Web applications accept HTTP requests at a URL, run them through some logic, and return responses. Matching the data in the incoming request to the responsible logic is the job of the application server.

Nearly all modern application servers use the routing pattern for this job. With routes, you write a little pattern for what the URL would look like and what code should be called. If the URL matches the pattern, the matching code is called.

Most people don’t realize that there are alternatives, even alternatives that predate routing. My experience on this predates CGI, and we didn’t even have zeros and ones back then. In 1996 we published work on “object publishing” which became the system known as “traversal”.

In a nutshell, traversal assumes a URL is an object path in your application. “/departments/marketing/launchteam/doc1/delete” is then mapped to a series of nested Python dictionaries, ending with the “delete” operation. The app server takes the URL, breaks it into paths to objects, and uses the information in those objects to dispatch to your code. It is an object-oriented routing system.

AngularJS is a popular JS MVC framework. It, of course, presumes that you will write little URL patterns to specify routes to turn into views. I had an application that didn’t quite fit that mold…particular containers could hold a heterogenous set of things. Putting the object type in the URL? “/section/1/column/2/group/1/report/7/edit”? Kill me now.

I went on a little journey to implement traversal in AngularJS. Kill me now.

ui-router and State-Based Views

AngularJS unbundled their ngRoute package in the 1.2 series as people were writing interesting and more powerful alternatives. ui-router is by far the most popular and introduces some powerful, mind-altering concepts. In a nutshell, you think of your application not as URLs with requests and responses, but states in a state machines. Interacting with your web app changes the state, causing the UI to update.

Along for the ride you get a number of features that deeply change how you think about the design of your application: nested states and named views, in particular. It will take you several passes through writing your app before you achieve Zen (I am still on my way) but I can say that this is something new/fresh/clever/original.

But crap if they don’t start with declarative URL patterns as the specifier for routes and state changes. Dammit. However, they provide a completely imperative set of hooks to do whatever the damn well you please. I hijacked that and did traversal using ui-router.

Traversal with ui-router

Here’s the approach:

Fetch the graph

I had an application where I could reasonably fetch a small amount of metadata for the entire application, enough to make my tree/graph. Some applications have 500,000 objects and this wouldn’t work. I’m not convinced that I can/can’t accomplish traversal with lazy loading.

Once the data comes in, you have to walk everything and assign an _parent reference and, for efficiency sake, assign a full resource path.

The everything pattern

I have a URL pattern that matches every URL. Every time the URL changes, it pumps through my state change logic.

Get the context and view name from URL

I look at the URL and, using the traversal algorithm, convert:

/departments/marketing/launchteam/doc1/delete

…to:

parents: [departments, marketing, launchteam]
parent: launchteam
context: doc1
viewName: delete

Meaning, I use the URL to discern the path in the local object graph.

Find the resourcetype-viewname state

Based on that path, I get the context type. Oh, it is an ExpenseReport? That means I want the expensereport-delete state.

Transition with data

I then tell ui-router to transition to the expensereport-delete state, passing in the data we figured out (context, parents, etc.)

Extras

You have to handle default views as index, not found, errors, broadcasting an AngularJS custom event so directives can update themselves, etc.

How Well It Worked

I spent a lot of time on the Z-shaped Angular learning curve. I also didn’t realize the subtle Zen of ui-router. So it was painful getting things in place. But once in place…man, it’s a fun, object-oriented way to write Angular views.

I tried a science experiment to remove ui-router and implement traversal as my own Angular provider. The basics actually worked very quickly. It was all downhill after that. I’m not smart enough to write my own provider. Too many low-level promises to keep and optimizations to make.

Moreover, I know realize that there is another ui-router Zen out there that transcends URL. You still want some of the URL, so that bookmarks and emailing links work. But the composability of state changes is the next deep dive.

We in Pyramid/Zope have struggled to make people care about traversal. It’s like the SQL fight: they have already learned how to think square, so everything to them (including the web) is square. Same for routes. My data isn’t object oriented, so why should my URLs be?

Thus, I don’t plan to ever package this up and in fact, I don’t plan to use it. ui-router, nested states, and named views are the new direction, and AngularJS 2.0 will likely tread new territory, as does ui-router-extras and friends (future states, parallel states, etc.)

posted: 2014-07-11 12:22  by Paul Everitt   |   permalink

Agendaless Summer Consulting Special

Agendaless Consulting is running a summer consulting special for the months of June, July, and August 2014.

  • 2 hours of free offsite pre-consulting work.
  • $1250 for 10 hours of offsite consulting work.
  • $150/hr for every hour of consulting work thereafter (offsite or onsite) in the three months this summer. This is a 25% savings from our standard consulting rate.

Here's some examples of what you might use this special for:

  • New project kickoff.
  • Code review.
  • Small feature implementations.
  • Migration of codebases to Python 3 from Python 2.

Agendaless Consulting is the company behind the Pyramid web framework and the Supervisor process control system. We are Python experts, with many years of experience in web apps. We also have skill in Zope, JavaScript, and mobile phone applications. We have been in business since 2006. References are available upon request.

Contact us via email at chrism@agendaless.com to get some help.

posted: 2014-06-04 22:07  by Chris McDonough   |   permalink

PyCon 2014 Recap

PyCon 2014 in Montréal was a great conference! Check out the Agendaless PyCon 2014 talks:

Paul Everitt presented a half-day tutorial, "Python 3/2 Web Development with Pyramid ":

Tres Seaver gave a talk, "By Your Bootstraps: Porting Your Application to Python3":

We had lots of good conversations with folks, both in the "hallway track" and at the Pyramid booth. Great job by the Montréal Pythoneers organizing and running such a terrific conference! And for those who didn't get one there, the "Hacker" T-shirt is available online:

posted: 2014-04-21 15:26  by Tres Seaver   |   permalink

Pyramid for Plone Developers: Training at Plone Symposium MW 2014

We are pleased to be offering a two day training session at the 2014 Plone Symposium Midwest this year. The two-day course will cover Pyramid development topics, aimed at Plone developers.

For details, please see the training page.

posted: 2014-04-17 19:44  by Tres Seaver   |   permalink

PyTennessee 2014 Talks

Chris and I have given our talks at PyTennesse 2014. Mine was a two-hour tutorial, "Pyramid Development in a Nutshell", which I cut down from Paul's Quick Pyramid Tutorial. (See my slides here; examples are on github)

Chris' talk was his "Substance D: Build Civilized Web Applications" talk. See a video of the same talk from the RuPy 2013 conference).

posted: 2014-02-23 20:41  by Tres Seaver   |   permalink

SubstanceD Screencast #6: Adding Editing Objects inside a Folder

This screencast is the sixth in a series of short tutorials on installing SubstanceD and using it to build a simple intranet application. It starts from where Episode 5 leaves off.

This episode covers adding objects to a folder, and editing those objects' properties.

posted: 2013-12-20 22:30  by Tres Seaver   |   permalink

SubstanceD Screencast #5: Editing Object Properties / Undo

This screencast is the fifth in a series of short tutorials on installing SubstanceD and using it to build a simple intranet application. It starts from where Episode 4 leaves off.

This episode covers updating object properties, and undoing those changes..

posted: 2013-12-20 22:00  by Tres Seaver   |   permalink

SubstanceD Screencast #4: SubstanceD Managment Interface Overview

This screencast is the fourth in a series of short tutorials on installing SubstanceD and using it to build a simple intranet application. It starts from where Episode 3 leaves off.

This episode provides a quick tour of the SubstanceD management interface (SDI), including navigating through the "tree" of objects in the database.

posted: 2013-12-20 21:30  by Tres Seaver   |   permalink

SubstanceD Screencast #3: Starting the Server, Viewing the Admin Interface

This screencast is the third in a series of short tutorials on installing SubstanceD and using it to build a simple intranet application. It starts from where Episode 2 leaves off.

This episode demonstrates starting the WSGI server in development mode, viewing the default splash page in the browser, and logging into the the SubstanceD admin interface.

Annotated Commands

For those who'd like to read rather than listen, this is the set of commands outlined:

  1. Start the server using the development configuration:
    $ ../../bin/pserve development.ini
    
  2. Grep the generated, random password from the config file:
    $ cd src/myintranet
    $ grep passwword development.ini
    
  3. View the default splash page at http://localhost:6543/.
  4. Log into the SubstanceD management interface at: http://localhost:6543/manage.

posted: 2013-12-13 23:00  by Tres Seaver   |   permalink

SubstanceD Screencast #2: Generating the Initial 'myintranet' Project

This screencast is the second in a series of short tutorials on installing SubstanceD and using it to build a simple intranet application. It starts from where Episode 1 leaves off.

This episode demonstrates generating the initial version of the myintranet project, checking it into Git, and installing it and its added dependencies into the virtual environment.

Annotated Commands

For those who'd like to read rather than listen, this is the set of commands outlined:

  1. List the templates available for generating a project:
    $ bin/pcreate --list-templates
    
  2. Generate the myintranet project:
    $ cd src
    $ ../bin/pcreate --template=substanced myintranet
    $ cd myintranet
    
  3. Check the initially generated code into a local Git repository:
    $ git init .
    $ cat > .gitignore
    *.pyc
    __pycache__
    *.egg-info
    *.fs*
    blobs
    tmp
    ^D
    $ git add .
    $ git commit -m "Initially-generated project."
    
  4. Install the project and its extra dependencies into the environment:
    $ ../../bin/python setup.py develop
    
  5. Verify that myintranet is importable from our checkout:
    $ cd ../..
    $ bin/python
    ...
    >>> import myintranet
    >>> print(myintranet.__file__)
    

posted: 2013-12-13 22:00  by Tres Seaver   |   permalink

SubstanceD Screencast #1: Setting Up the Environment

This screencast is the first in a series of short tutorials on installing SubstanceD and using it to build a simple intranet application.

This episode demonstrates creating a Python virtual environment and installing the SubstanceD software into it.

Annotated Commands

For those who'd like to read rather than listen, this is the set of commands outlined:

  1. Create a Python 3.3 virtual environment:
    $ /opt/Python-3.3.3/bin/pyvenv intranet
    $ cd intranet
    
  2. Install and update setuptools:
    $ wget https://bitbucket.org/pypa/setuptools/downloads/ez_setup.py
    $ bin/python3 ez_setup.py
    $ bin/easy_install -U setuptools
    
  3. Check out SubstanceD from Github:
    $ mkdir src
    $ cd src
    $ git clone git@github.com:Pylons/substanced
    $ cd substanced
    
  4. Run SubstanceD's setup.py to install it in the environment:
    $ ../../bin/python3 setup.py develop
    
    Note that this step also installs SubstanceD's dependencies.
  5. Verify that SubstanceD is importable from our checkout:
    $ cd ../..
    $ bin/python
    ...
    >>> import substanced
    >>> print(substanced.__file__)
    

posted: 2013-12-13 22:00  by Tres Seaver   |   permalink

SubstanceD Screencasts

We're starting a series of short (3 - 5 minute) screencasts, demonstrating the SubstanceD application development platform. The series is built around the tasks needed to set up a simple intranet site.

For an overview of SubstanceD as a development platform, start with the talk given by Chris McDonough at the RuPy 2013 confernce: "SubstanceD: Building Civilized Web Applications".

posted: 2013-12-13 21:30  by Tres Seaver   |   permalink

Rupy 2013 Python Sprint

Recently we held a 3-day-long Python sprint in Budapest before the RuPy conference. Here's what got done.

Substance D (web application server)

Participating: Chris McDonough, Balázs Reé, Godefroid Chapelle

  • Allow management views to be categorized.
  • Allow deletion of objects that cannot successfully be unpickled.
  • Add an API for get_editable_adapter.
  • Allow schemas that inherit from substanced.schema.Schema to be used successfully in non-FormView code (no longer necessary to call .bind() on these to run things through deserialization).
  • Grid error messages caused by connectivity problems now disappear if the client notices that the server is back up via SSE.
  • Attempted to figure out how to more better show "global" views (like the auditing tab, the database tab, and the undo tab).
  • Various small UI improvements.

Deform (form library)

Participating: Domen Kožar, Balázs Reé, Chris McDonough

  • Experimented with using angular.js for client-side validation (fairly successfully).
  • Refactored and improved resource registry.

Pyramid (web framework)

Participating: Fabian Neumann

  • Audit docs for places we could better use .. deprecated:: and other Sphinx/RST directives.

posted: 2013-10-11 23:03  by Chris McDonough   |   permalink

Mozilla Persona and Substance D

We're running this agendaless.com website on Substance D, the content framework for Pyramid that Chris came up with. We're dogfooding a lot of the good work done by Substance D folks, plus related work like Domen's Deform2.

It's also a testing ground for some of our own stuff. Tres has it acting as a WebDAV server, which is how he does his authoring. He did this as a Substance D add-on, which helps prove how well Pyramid's add-on system functions.

We were talking about logins and thinking about re-using the Twitter-OAuth-based login I did on my Substance D site for a local sports league. Instead, Tres looked at Mozilla Persona and whipped up support for that. I can now use a Google account and get transparent login when authoring blog posts here.

It's worked, well, just about perfectly.

posted: 2013-10-11 16:12  by Paul Everitt   |   permalink

Categories:   python   python3   zope   pyramid   plone   opensource   substanced   screencast   consulting