Recently we held a 3-day-long Python sprint in Budapest before the RuPy conference. Here's what got done.
Substance D (web application server)
Participating: Chris McDonough, Balázs Reé, Godefroid Chapelle
- Allow management views to be categorized.
- Allow deletion of objects that cannot successfully be unpickled.
- Add an API for get_editable_adapter.
- Allow schemas that inherit from substanced.schema.Schema to be used successfully in non-FormView code (no longer necessary to call .bind() on these to run things through deserialization).
- Grid error messages caused by connectivity problems now disappear if the client notices that the server is back up via SSE.
- Attempted to figure out how to more better show "global" views (like the auditing tab, the database tab, and the undo tab).
- Various small UI improvements.
Deform (form library)
Participating: Domen Kožar, Balázs Reé, Chris McDonough
- Experimented with using angular.js for client-side validation (fairly successfully).
- Refactored and improved resource registry.
Pyramid (web framework)
Participating: Fabian Neumann
- Audit docs for places we could better use .. deprecated:: and other Sphinx/RST directives.
We're running this agendaless.com website on Substance D, the content framework for Pyramid that Chris came up with. We're dogfooding a lot of the good work done by Substance D folks, plus related work like Domen's Deform2.
It's also a testing ground for some of our own stuff. Tres has it acting as a WebDAV server, which is how he does his authoring. He did this as a Substance D add-on, which helps prove how well Pyramid's add-on system functions.
We were talking about logins and thinking about re-using the Twitter-OAuth-based login I did on my Substance D site for a local sports league. Instead, Tres looked at Mozilla Persona and whipped up support for that. I can now use a Google account and get transparent login when authoring blog posts here.
It's worked, well, just about perfectly.
Something I knew about, but didn't look at until now...there's a thriving Pyramid Meetup group for London. They've had 4 meetings so far and have one tonight. Rach Belaid and James Cook are organizers.
Already a good number of people signed up, some familiar names and some fascinating names. Pretty interesting comments on the signups as well.
Months ago I thought it would be fun to interview some people, and I always knew who I wanted first. Liz Leddy: Plone core developer, former Plone Foundation board member, speaker, and rabble-rouser of the group-hug variety. Below is a transcript, with an audio version available.
1) Tell us a little about yourself, the stuff you're currently doing, and the awesomest thing you've done in the last year.
I'm Elizabeth Leddy, I've been working with Plone since I was 19 years old, and I'm 30 years old this year, can you believe it! I got into Plone when I was working in college as part of a student internship where I was learning this new thing called CSS (can you believe it? tech comes a long way.) It's amazing to me that I'm still doing it this year.
I've done quite a few things with Plone: government intelligence, medical startups, internal implementations, and I've been working as a consultant for the last four years based out of the Bay Area. Currently working with Eric Steele and Ryan Foster on a startup doing an athlete management platform on top of Plone. Turns out a lot of the problems in elite athelete management can be solved with proper content management and communications. In fact we just got back from training the Cleveland Cavaliers (professional basketball team.) Work a lot with the US ski team and some other big names coming through, so we're really excited.
It has been interesting to see how, once you get outside the standard intranet domain, how many of the standard patterns solved by Plone can apply, just with some different wording.
Me: I've been thinking about that lately. We all come from a background where we solve a certain class of problems about how people work together. Other systems and approaches don't even try to solve those problems.
Liz: Yes, I see this throughout other platforms and systems outside of Plone, I call this the shiny object syndrome, where decision makers are really drawn to pretty graphs etc., which for example IBM are really amazing at doing. But then you get down to the real problems, like communication, knowledge management where you get people out of emails and into systems. That's where the real problem is and if we can have that conversation to get decision makers beyond shiny object syndrome, we can really get change both internally and when communicating with their customers.
2) What's one of the awesomest things you've done in the last year?
I do a lot of shenanigans, but the thing I'm most proud of is where we've taken sponsorship. The donations we had coming in were these really great companies. When I started on the Plone Foundation board this year, I really wanted to get our sprint funding together, with some brazen comments like "If we spend $25,000 we can raise $25,000." Of course I then had to back up a statement like that, and we did, we're back on track to reach our high-water mark from 2009 levels of sponsorship.
We've already raised enough money to continue the sprint funding next year, which is the main thing, taking it year-by-year. We have big Plone 5 push, and getting the right people to the right places at the right time and making sure they're productive, we'll see a really big release come.
Furthermore, the excitement that comes behind knowing that sponsors still care enough about Plone that they just need to be asked in the right way is just awesome. I love that.
3) How did you find Plone, what kind of stood out and made 19-year-old Liz want to do it?
Actually I came across Plone the way a lot of people do: somebody paid me to do it. I was getting a blazing $8/hour, but I was 19 at the time and trying to get my computer science degree, it was a good thing. At the time I wasn't technical, and most people's introduction to Plone is, you're handed the system and you can customize via the browser and do all these things. You don't need to be technical. As I learned computer science and object orientation, then I could dig more and more into Plone and do technical things.
When I graduated from college, I didn't expect to do Plone. After a couple of post-college jobs, I got hired at BAE Systems to do Java JSR-168 templates, really portlet standard back then. Luckily a guy said "I heard you know a little bit about Python and Plone, do you think you want to do a Plone install?" Of course I said yes, save me! From there it has just continued, I've never had issues finding work and interesting projects.
4) What piques your interest in Plone these days?
Interestingly I do use other systems for smaller projects and keep an open mind, but each time on bigger projects and clients that last multiple years, every time it turns out, "All of this is solved by Plone already." You can just get up and go.
I specialize in startup types with a lot of one-off prototypes to get funding. Even if Plone isn't the end game, it's there for a few years, get them going quickly, let them flesh out their thinking, change directions, etc.
People think startups are, have an idea and eight months later, sell to Facebook. But the reality is, it's a four year process to even get going. It's a moving target that they have follow constantly, thus it doesn't make sense to invest in a platform until you've really figured out what you're trying to sell, who are your investors, and how much they care. It's a different reality than "The Social Network" would like us to believe.
5) Let's change gears away from technology. I'm more of a Plone observer these days. Just watching, it seems like the world of Plone has gotten a second wind.
It definitely has. A lot of factors going into it. First, we have new energy on the development side, people like Rok Garbas and all these young people from countries such as Brazil. These people that haven't become curmudgeonly old developers yet, taking ownership, filling places where people have left. There's a natural cycle where, if you've worked in a project for ten years, you can get cynical after a while, just like any relationship, you definitely get some kind of itch.
We've had people naturally fade away: their businesses went a different direction, or they had twins. Then we have young people who come in and think, "Well, nobody's going to do it so I'm going to do it, and I'm going to do it my way." We've needed that kind of fresh blood for some time. They're very exciting, they get other people excited.
Also, as soon as we said "Seriously, we are going to do Plone 5", people start getting their changes lined up, saying "Yeh, let's do it!" This is combined with a cash influx from the Plone Foundation for sprints. We then go to symposiums and talk with Plone companies who are excited about Plone 5. All we had to do was ask for sponsorship.
6) On a related question, there's Plone-the-Software and Plone-the-Community which makes the software. But there's this other thing, Plone-the-Foundation. How is it doing?
The motto of the Foundation is "Protect and Promote Plone". This is my first year on the board. Managing sprints and sponsorship was one of my main things coming in.
They (the PF) does everything from handling GPL issues, what licenses go in which new packages, trademark violation stuff that we handle on a regular basis, the contributor agreement, marketing and sending people to events. This year we've expanded the scope to be more aggressive at sponsorship to help make sprints happen. That's part of protecting Plone, to make sure it sticks around, and that's why people want to donate. I think we've been doing a pretty good job this year.
Me: I think May of next year will be around the ten year anniversary of the wheels getting in motion for the Plone Foundation. The Plone Foundation is a pretty good success story as it changed hands from the founders and first generation and now getting a good second wind. That's a neat story that not every community has.
Liz: Ten years, wow! It's actually a story I've told a couple of times this year, for example at OSCON. Not all software makes it to twelve years. What does that kind of durability mean and what are the implications? The younger open source people aren't aware of what happened in the late 90's with software getting acquired and the legal battles, so I spend time talking to new sofware projects about why a foundation is a good idea if they plan on sticking around.
It (the Plone Foundation's durability) is something we should be proud of.
7) Zoom out a little: what kinds of things are on Plone's plate for the next two years?
The next major release is the biggest thing, being the new plan for Plone 5. We have a lot of things to get rid of by updating our code base, get rid of a lot of legacy stuff. That's a hard job that usually nobody wants to do but we've discovered is more important.
We're at a point where we can modernize our front-end tools, get rid of legacy code, and do things that make it a better development platform, not just for developers but more friendly for people that don't want to write any Python. I'm really excited to see what happens with Plone 5.
Also, moving on to CSS and other frameworks that aren't custom to Plone, borrowing tools instead of building our own, to kind of move on and learn our lesson to integrate other tools. We have this bad rep from around the Plone 3 timeframe. Now we're reaching a generation that has either never heard of us or have forgotten that, so that's an awesome opportunity to kind of rebrand and say: "You've never heard of Plone, but it's pretty legit."
We have plone.com as well, a three-year process at figuring out marketing when you don't have a "BDFL" company to lead the marketing. Instead we have a group of companies and we've yet to figure that out, because there aren't a lot of those kinds of success stories to pull from. We know if we had better marketing we could do more. That's our biggest challenge in the next year, figuring out how that works and what are the politics behind it, just try some things and amend them if they don't work.
Me: It's interesting that you tied Plone 5 and the marketing push together. With Plone 5 as kind of a relaunch, you have a chance to sharpen up the story and get some consensus on the storytelling. Not just refactoring the software and tossing out cruft, but maybe there's cruft in the story?
Liz: Definitely, and I hope it's something I hope we figure out. Something I think is pretty cool, Heidi Reinke from the Universtiy of Wisconsin Oshkosh is having her marketing students to a marketing plan for Plone. They'll do interviews and make proposals, but it's nice to have a third-party bring a new perspective where we learn from them. I'm fascinated to see how this little experiment works.
8) The RV that you rented for PyCon 2013 was absolutely fabulous. You've set the bar so high now. What other shenanigans do you have up your sleeves?
We've constantly bugged the Plone Foundation to buy a hot air balloon. Carol Ganz at SixFeetUp is two hours away from getting her hot air balloon pilots license so stay tuned.
9) How many times have you been on the road this year? When is the next time people can get their dose of the fabulous Liz Leddy experience?
So far this year, 3 sprints, 2 non-Plone conferences talking about Plone, two conferences, on the road two times a month the whole year. I'll have a some down time. I'd really like to make the Arnhem sprint in November. Some people are sprinting a whole month there! If not, a sprint at Bodega Bay in February. I talked ostensibly forever about a sprint in Hawaii, so I'm going to make that happen next year. And of course, PyCon.
Next Thursday, Oct 17, I'm doing a webinar with the good folks doing PyCharm at JetBrains. I'm use PyCharm pretty heavy, and recently I'm doing nearly everything (including my Pyramid work) using Python 3 in PyCharm.
Drop by the webinar, see what PyCharm looks like, and ask me some pointed questions, like that time in Vienna when I....oh right, that's a secret.
Last month we unveiled the Quick Tour of Pyramid, a high-level presentation of the basic features in Pyramid. The tour got positive feedback and has been updated and refined as work on Pyramid 1.5 progresses.
We now have something to accompany the Quick Tour: the Pyramid Quick Tutorial. 21 steps through the basic features, with working code, analysis, and linking to reference material. The tutorial is the result of a serious, sustained effort including several for-fee engagements, resulting in an open-content, commercial-quality tutorial that is part of the core Pyramid docs.
We're quite happy to have this available for evaluators and new developers. Over time, we'd like Pyramid to be a fantastic starting point for new Python web developers, with lots of evaluator-friendly resources.
Here's a bit of a look inside.
How It's Made
A couple of years ago I taught a Pyramid tutorial, for-fee, at a Plone Conference. I chose a different approach to tutorials: break the work into independent sections, one concept per section, and make it so even a room full of students would be successful doing it themselves. That formula has been tweaked quite a bit. Here's how the Pyramid Quick Tutorial is produced.
Each of these section is done as Sphinx documents with a common structure:
- Explain the background and give the objectives of that section
- Provide a series of steps to produce a working example in that topic
- The steps link to a subdirectory with a working Python package and link in the code
- Following the steps, some analysis of what was done
- For people that finish faster than others, some extra credit questions
- Lots of deeper linking to the official docs
I then let them go off and do the work for the step, then we reconvene and talk about it. Some students cut-and-paste, some type it all. If they get too far behind, they can just get the package from GitHub and teleport themselves into the present. All the material is available online in the official Pyramid docs, so they can come back later and freshen up.
The sections also have a modest amount of test code, to make sure what is being taught continues to work in the future.
Have tutorial, will travel
Have an in-house team of developers interested in Python 3 for web development? Running a conference with tutorials and up for a high-quality, open-content half/full/two-day tutorial? I know somebody that might be interested. [wink]
Lately I've given my Python 3 Web Development With Pyramid tutorial in some cool venues (Cambridge UK, Brasilia) and now this material is a part of the core Pyramid docs. Still, for each venue, I produce some new sections relevant to the particular needs of that organization or context.
I'm hoping to do more events and more material (next up: a traversal tutorial and an add-on development tutorial for more advanced usage.) Give me a shout, @paulweveritt on Twitter.
tl;dr Training and talks, meeting old friends, Plone getting a new burst of momentum.
Last week I was in Brasilia for the joined-up Plone Conference 2013 and Python Conference Brasil. Below is a trip report.
At first I was surprised. Giuseppe Romagnoli invited me to give a keynote at the conference and I had some reluctance. I've been out of the loop for a while. Would I have anything useful to say? Well, that's never stopped me before, so I said yes.
I then had, of course, a visa issue. Last time I went, in 2002, you just strolled up to the embassy and got your visa. Since then, the good ol' US of A has apparently "enhanced" its treatment of Brazilians. Turnabout's fair play, so in return, getting a visa there was a ten-business-day process involving your passport. Thanks to Ana Maria Amorim, a wonderful woman whose husband was recently the ambassador to the US, I dodged a bullet.
I also signed up for a two-day Pyramid training course and a "State of Pyramid" talk. Since I was giving the training course two weeks earlier in Cambridge UK, I thought I was in good shape. Alas, I couldn't resist the urge to tinker, so I was frantically creating material all the way up to the last moment.
The flight was uneventful and actually, fun…I got on the Atlanta->Brasilia hop and heard Chrissy Wainwright from Six Feet Up call out my name. Yay! Hanging out with Chrissy was one of the treats of the conference. She's a superb traveler.
The hotel for the speakers was directly on the Brasilia equivalent of the "Mall" in Washington…a long strip of open space with government buildings lining each side. Though the training venue required a bus ride, the conference venue was only a sub-kilometer walk.
The first night before the training I got to have dinner with Chrissy, Vitória Vasconcelos, and my longtime dear friend Luciano Ramalho.
Over the years I've developed a certain approach for training. I break a course into a series of bite-sized chunks, each on a specific topic that students want/need to learn. I make a LOT of these sections. For two days, over 30 in my "Python 3 Web Development with Pyramid."
More later on how I do the tutorials, but for now: it was a lot of fun. I enjoy getting feedback and improving the material. I also enjoy doing my little part to push Python 3, as it is the first-class citizen for the tutorials.
Our hotel had a nice restaurant, fancy outside deck, nice handsome staff, modern music, and a nice menu.
Around the corner was a parking lot where, for the last 18 years, a local entrepreneur brought a portable grill and served kabobs, a rice dish, and beers. You would sit on little plastic stools underneath a string of lights powered by an electric cord going across the street. Each item was around $2, paid on the honor system.
We went there every night.
This was a real highlight of the conference. Authentic food, fantastic conversation. Under mango trees that the gentleman planted himself many years ago.
It's been a while since I hung around with the Plone folks for a sustained period. Two things really jumped out at me:
- The new generation of Plone leadership is really, REALLY good. From all over the world, a refreshing new viewpoint on what things to do and how, and a tangible feeling of joy and purpose.
- Plone itself seems to have a new dose of momentum. Certainly having the President of Brazil launch a new Plone-powered website for the Brazilian government is a notable sign. But there's more to it. The Plonistas seem to be making some serious steps forward.
There were four simultaneous tracks as well as two keynotes per day in a ginormous (2,500 person) auditorium. The conference sold 500 tickets but I suspect the high-water mark on attendance was 400 on a single day. Probably 3/4 of the talks were in Portuguese, which seemed like a very good idea. On both the Python side and Plone side, new generations of Brazilian free software developers were being minted quickly (as the entrance fee was kept very low.)
Lots of good talks on the Plone side, plus a mini-track for Pyramid. I alas didn't see too many talks. The first day was spent tweaking (i.e. writing) my State of Pyramid talk. I also decided, late on the first day, to rewrite my keynote. Sigh. Bad Paulie.
In particular, I liked Mikko's talk. He helped poke at some questions and thinking that I wanted to get my brain wrapped around, and let's face it, he is a very hilarious and quite charismatic speaker.
Érico Andrei was quite a force at the conference, both from a Brazilian side and a Plone side. He has a lot of leadership qualities. It's fun to see him in action. Ditto for Paul Roeland, who several people noted to me in private has a real gift for bringing the various forces together in the world of Plone.
Getting to see Alan Runyan, Sidnei da Silva, and Matt Hamilton again after quite some time was worth the trip, just for that. To think how long we've all known each other and everything we've gotten to work on together…it's quite neat.
State of Pyramid
On the afternoon of the first day, I usurped Chris McDonough's throne and gave a past/present/future talk on Pyramid. Typical Paulie presentation, all sauce, no steak. Lot of fun, though.
I got a lot out of it, I must say. I listened to some of the feedback to learn more about how people see Pyramid in the continuum of Python web frameworks. Our marketing message of "The Start Fast, Finish Big, Stay Finished Framework" seems to match what people are looking for, as well as were they have slotted Pyramid in their heads. I also gave a demo of Substance D (demo.substanced.net), Chris's content framework atop Pyramid.
What Pyramid means for the world of Plone was an undercurrent of the conference. I tried to keep my mouth shut and my ears open, so this was a useful aspect to attending.
The second day I gave a keynote that was a series of funny stories masquerading as lessons learned. It was intended to be, largely, the same talk I gave at PyCon DE in Leipzig, 2011. However, the afternoon before the talk I got a bug up my ass and decided, well hell, why not just rewrite it. Which was risky due to the stage layout: I wound up with 82 slides and no presenter mode to see the next slide (to time the jokes), and had to ask Luciano to be on stage and press "next".
Which meant I had to memorize all 82 slides. I'm no Randy Pausch, Paul Graham, Eben Upton, Paul Hildebrandt, or any of the other extremely smart keynoters with important things to say. Sitting in their keynotes is always a humbling experience…even if I tried really hard, I know I couldn't match them.
Lately, though, I've tossed out my "just wing it" approach and actually prepared and rehearsed. It's not fair to far-travelling audiences to put in a weak effort. Also, for the topic of this talk, I think there are some experiences that we in the Python/Zope/Plone world went through that are worth imparting.
I wouldn't mind giving the talk again. I already know some changes to the material I'd like to make and some work I should do on the delivery. Anybody need an above-average-funny, below-average-profundity speechifier?
Paul Hildebrandt's Keynote
This was my first time meeting Paul, a senior software engineer at Walt Disney Animation Studios. We hung out together at the meat-on-a-stick soiree the first night. What a wonderful person, very smart but quite human and with a heart most certainly in the right place. (More below.)
He gave a deeply enjoyable keynote called "Under the Hat" which, alas, wasn't recorded. My 7th grade son would have flipped out if he could have seen it. Not just a talk on how technology can enable creativity. Not just a talk on the pervasiveness of our little child Python in the digital entertainment business. But also, the human side of building organizations that actually value people.
The evening of the second day saw an official party on the lake at the Navy base outside Brasilia. Well done, conference folks! Time flew by as people ate, drank, and even danced until the wee hours. 60 people from probably 20 countries.
At the risk of leaving out so many other memorable moments, one seemed to capture the real spirit of this event.
During my talk I made reference to the legendary Castle Sprint (must…write…blog post…) and, as analysis-in-hindsight, wondered what prompted underpaid people from 30 countries to travel to Austria and work for free on software given away to others. Sure the beer was good, but that's quite a thing to do.
In the talk I concluded that there was a social element going on that I hadn't understood. Most of these developers were probably the smartest person in their little town. Too smart for the others to really appreciate. They got out of school and maybe got a decent job at some soul-sucking enterprise.
Then, they go online, peek their head in the Plone/Python world, and discover…a whole bunch of people just like them, from wonderful places across the globe. Not only that, but these people valued them, understood them, and in fact, wanted them to come join the fun. Go to Austria, soak in some high-octane thinking, hang around for once with people who get me? Sure, sign me up!
After that talk a young man from Brazil came up and told me that was indeed his story, and here he was at a Python event in Brazil, getting to be a part for the first time of something big. But even more…he always wanted to be a digital animator, and unbelievably, the conference had The Disney Guy! He tweeted Paul Hildebrandt to say thank you for coming.
What did Paul do? Not only did he reply, he memorized the young man's face, found him in the crowd, and introduced himself. Then proceeded to tell him the steps he should take to start his journey possibly to Disney Animation Studios. Paul didn't have to do that: it's just who he is, and he did it, and now that young man's journey is so much different than if he didn't stick his head in the door of free software and look around.
tl;dr Well-polished introductory material for Pyramid
Pyramid has always had great docs. In fact, the docs were released before the software! The docs are comprehensive and always-updated on each of the (over 100) Pyramid (and its predecessor, BFG) releases.
However, we've known for a long time that the docs needed a casual introduction. The docs are aimed at those looking for a deep treatment. We've needed a shallow-but-wide overview that linked into the deeper explanations.
Enter the Quick Tour of Pyramid. A casual introduction to 19 topics that Python web developers want in a web framework, each with links into further material, including a deeper treatment in the new Quick Tutorial (the topic of a later post.)
We've been thinking a lot lately about introductory material for Pyramid. The Quick Tour is the product of this and has undergone several refinements since its first appearance. For example, all the code snippets are included from known-to-work Python modules on disk. We also greatly expanded the use of Intersphinx, to ensure our links to other Pyramid/Python code don't break without generating an error.
Lately we've come to view Pyramid as the "Start Small, Finish Big, Stay Finished" web framework. The software excels at this. For the "Start Small" part, we want the documentation to as well. Give it a look, and if you find a mistake or have some feedback, let us know.
Chris McDonough, Chris Rossi and I are competing in the 2013 WSGI Wrestle contest (originally "WSGI War", but changed to protect some queasy stomachs) this weekend. We will be building our application using Pyramid and SubstanceD.
Watch the Definitely Not Built By Aliens team in real time as we bash together an app in 72 hours!
Some upcoming travel for Agendaless:
Python Brazil / Plone Conference 2013
PyCon IE 2013
- Chris then flies from Budapest to Dublin to speak at PyCon Ireland 2013 on SubstanceD <http://substanced.net> on Sunday, 2013-10-13.
- Chris will also be leading the Pyramid sub-sprint in Dublin following the conference, 2013-10-14 - 2013-10-15.
posted: 2013-09-25 21:29 | permalink
This story is about how the practice we call "sprinting" in the Zope / Python ecosphere came to be.
What is a "Sprint"?
As used by Zope and Python folks, a "sprint" is an intensely focused session, usually lasting less than a week, in which a group of developers work together to design and implement features for a library / framework application. Many conferences now host sprinting in the days before or after the conference proper, but there are also lots of sprints which occur outside of any conference.
Where did the term come from?
Back in the fall of 2001, I had my first (and only) performance review at Zope Corporation (then Digital Creations). As part of the review, Jim Fulton asked if there were any changes I would like to see in the way the company developed software. At that point, the official process we used was pretty heavy, with lots of "Big Design Up Front" artifacts.
I had been following Extreme Programming (XP) via the "mother-of-all-wikis" since the mid 1990s. I suggested that it would be good to find a way to experiment with some of the practices of XP, but in a way that wouldn't involve making a big committment to the whole discipline. We were in the early stages of re-writing the Zope application server, and decided that we would hold a short, in-house session (four days), where we would consciously attempt to apply as many of the XP practices as seemed feasible while working on the next-generation Zope.
Because we had only a short time to experiment, but still wanted to achieve a significant delivery milestone, one of the XP values we didn't embrace was the notion of software development as "long-distance running" (no overtime, etc.) We therefore began to refer to the upcoming experiment as a "sprint".
Scrum, a different "agile" methodology from XP, used the term "sprint" to refer to a month-long development increment. I was only lightly aware of Scrum at the time, and wan't (consciously, at least) borrowing the term.
The goal of that initial sprint was to (re-)implement the "Interface" package, which prototyped the PEP 245 interface semantics (but not syntax) for Zope2. The new package we built, zope.interface, is still the core of the component architecture today, and is used by projects such as Twisted and Pyramid
The results of that first trial was terrific: we found that the practices we applied (pair-programming, test-driven development, YAGNI) led to a very productive outcome. More imporant, we found that the four participants left with a much stronger understanding of the design of the software we had built, and were therefore much better able to keep working on the code separately.
Sprints as Evangelism
Reviewing the successes of the first, in-house sprint, we wanted to find ways to repeat the process. In January of 2002, we were trying to design a new, in-core mechanism for translating software-generated text (from templates and Python code, rather than user-entered content). We came up with the notion of invitng the authors of the main Zope2 i18n/l10n products (Stephan Richter's ZBabel and Juan David Polomar's Localizer) to ZC for an "i18n sprint".
As we had hoped, this sprint laid the technical foundations for the i18n support in Zope-related code to this day. Beyond our imagining, however, was the impact the sprint had on the non-ZC sprinters: they were fired up about the new technology, and were able to go back to their own communities and rally support for it. Indeed, one of those sprinters, Stephan Richter, went on to write the first Zope3 book: he is still active in the Zope software world to this date.
From that experience of the social/organizational benefits of sprinting, we began scheduling sprints to take advantage of participants expertise and interest in the new Zope software, and were thus able to expand the committer base to 400+ committers over the next three years.
You have nothing to lose but your chains!
I'm declaring bankruptcy on helping people with issues related to old versions of distribute / pip / setuptools. We are at version 1.0 of setuptools -- it is past time to get yourself re-bootstrapped onto the shiny, new, post-merge setuptools. Here is how:
In a virtualenv where you use 'easy_install':
$ /path/to/virtualenv/bin/easy_install -U setuptools
In a virtualenv using old pip + distribute (via update to dummy distribute 0.7.3):
$ /path/to/virtualenv/bin/pip install -U distribute
In a project using buildout 1.x:
$ wget http://downloads.buildout.org/1/bootstrap.py $ /path/to/Python bootstrap.py $ bin/buildout
In a project using buildout 2.x:
$ wget http://downloads.buildout.org/2/bootstrap.py $ /path/to/Python bootstrap.py $ bin/buildout
If you are stuck using system packages which haven't yet updated to post-merge setuptools (I'm looking at you Debian), you'll need to either:
- Break that habit, and learn to use virtualenv.
- Lobby the distribution to release updated packages.
Embrace the shiny!
Until fairly recently, it was pretty common for a custom software development company to sign a contract to build software with an indemnification clause not unlike this one:
Indemnification. Vendor agrees to indemnify, hold harmless, and at BigCo’s option defend BigCo and its subsidiaries, affiliates, and their respective officers, directors, employees, agents, contractors, successors and assigns (collectively, the “BigCo Indemnitees”), from any and all claims and all related losses, liabilities, damages, costs and expenses (including, without limitation, reasonable legal fees and disbursements and costs of investigation, litigation, settlement, judgment, interest and penalties) (collectively, “Losses”) arising from or related to: (a) any allegation that the Deliverables, Vendor Materials, or Services: (i) violate or in any way infringe or misappropriate any third party’s patent, copyright, trademark, trade secret or other intellectual property right, (ii) violate any applicable law, rule or regulation, or (iii) constitute, or contain material that constitutes, libel, defamation or a violation of the right of privacy or publicity; and (b) any claim that is based, in whole or in part, on any act or omission of Vendor (or any of Vendor’s Personnel). These indemnification provisions shall be conditioned upon Vendor not consenting to any judgment or decree or entering into any settlement of such action without BigCo’s prior written approval.
In the minds of many US corporate lawyers, it would show bad faith for a consulting company to object to a clause like the above. It would indicate that either a) the consulting company was twirling their metaphorical mustaches, planning to lazily inject clearly stolen, patent-tainted intellectual property into software they're instead being paid good money to write from scratch or b) the consulting company was planning to at very least bill for more hours than they worked, reusing existing -- possibly patentable -- code they did not own, left over from other consulting work. Either way, a corporate lawyer might argue, this consulting company is clearly not looking out for BigCo's best interests. If they were, they'd just sign the contract with the indemnification clause intact. After all, the consulting company would clearly be at fault in such a case, and an honest company wouldn't provide patent-infringing code in the first place! Just what kind of scam this Agendaless company trying to pull, anyway?
Even though we risk such "bad faith" claims, Agendaless does not sign contracts with such clauses in them unmodified. If we receive a contract with such language, we ask for the clause to be stricken. If it cannot be stricken, we ask for the clause to contain limiting language such as a upper bound on damages below the value of the contract, or a specific list of patents that we must not infringe, or a limitation specifying willfulness of infringement. And if that doesn't work, to our dismay and to the dismay of the folks at BigCo who want the work done, we'll turn the contract down. This is likely to be true no matter how small or how large the contract is.
You might ask: why would we, at the very birth of a project -- the exquisitely beautiful union of two corporate entities -- want to poison the well like this? It seems like craziness!
Two words: patent trolls. A nicer term for a patent troll is a "Non-Practicing Entity" or "NPE". From http://blog.tridentcap.com/2011/11/ip-alert-time-to-reexamine-patent-indemnification-clauses.html
An NPE is an entity that owns patent rights but has no material business operations of its own. Many NPEs acquire patents for the sole purpose of generating license revenue through litigation or threat of litigation. The NPE has no operations that can give rise to any counterclaims, and therefore there is little to no business pressure the defendant can bring to bear to encourage settlement. Given that NPEs are counter-claim proof, it serves their interests to sue every company with business operations in a given sector, and they have little to no incentive to settle without extracting significant monetary value.
In other words, almost literally, patent trolls really are trolls. But instead of collecting tolls for a physical bridge they don't actually own, they sit underneath a metaphorical bridge waiting for you to infringe upon their "intellectual property". This is their entire business model. They sue whomever they can, casting a shakedown net far and wide against companies that had no prior knowledge that they were violating any patent, betting that many companies will be willing to settle rather than spend money on legal fees to defend against their claims. They don't actually produce anything based on their patents, their patents are only used as a litigation tool.
Patent trolls obtain patents by buying them from other legal entities. Since patents in the US are granted seemingly by people who have absolutely no idea what they're granting, they are comically overbroad. For example, here's an excerpt from a patent granted by the US PTO (Patent and Trademark Office) entitled "Method for making money on the Internet" (http://www.google.com/patents/US8296192) filed in June of 2009:
A computer-implemented system for monetizing internet content. The system includes an internet content provider providing online information on a web page and an online comment section associated with said online information, said online comment section capable of having posted thereto reader comments in a free default format. The system includes computer executable instructions for performing steps of displaying an offer associated with said online information to alter a reader comment from a free default format to a distinctive format for a fee, said fee being dynamically adjustable; receiving by the internet content provider; and displaying the reader comment in the distinctive format on the web page. The system also includes a processor for executing the computer executable instructions, and a memory for storing at least the computer executable instructions.
If your eyes glazed over trying to read that, you are in good company. The language it employs is complete nonsense, and could be interpreted to mean just about anything. The rest of the patent isn't much more informative. It's basically total gibberish, with no actual useful technical information. But it claims to own the idea of making money on the internet, and it is a valid patent that has actually, in reality, in this real world in which we live, been granted to an entity. On Earth.
And who do you think owns that patent currently? Amazon? Google? Microsoft? No. The person that owns that patent is likely indeed, by all available evidence, such an NPE. It's a lawyer named Roddy McKee from Milford OH. That entity apparently has another patent in the works entitled "Universal one-click online payment method and system" http://www.faqs.org/patents/app/20100332337 . I'd wager a guess that Roddy McKee can't fix his Outlook when it breaks, much less design and operate grand systems such as those described by his patents and patents-pending. Delicious, right?
There are thousands of such entities in the US, and tens of thousands of such overbroad patents granted to seemingly anyone who can write enough technojargon to befuddle the USPTO. In 2012, more than half of patent litigation cases filed in the U.S. were filed by NPEs (http://www.law.com/corporatecounsel/PubArticleCC.jsp?id=1202595510252&Majority_of_2012_Patent_Litigation_Filed_by_Patent_Trolls&slreturn=20130706192341). From the same article: "Of the 10 most frequent patent litigation filers, nine were patent monetizers and only one was a company making products, the study revealed."
The indemnification clause above exposes Agendaless to unbounded liability from such patent trolls. If a patent troll were to sue BigCo claiming that code we licensed to them under the terms of an associated contract infringed on one of the patents in their stable, the language in the contract above exposes us to "any and all claims and all related losses, liabilities, damages, costs and expenses (including, without limitation, reasonable legal fees and disbursements and costs of investigation, litigation, settlement, judgment, interest and penalties) (collectively, “Losses”)". Emphasis mine.
In software patent cases, such "losses", however imaginary, by patent trolls can add up to millions of dollars. Without taking damages into account, even if we were to "win" such a case, the legal costs alone, and time spent not billing on other projects would put Agendaless out of business many times over. According to https://en.wikipedia.org/wiki/Patent_troll "the cost of defending against a patent infringement suit as of 2004 is typically $1 million or more before trial and $2.5 million for a complete defense, even if successful." So, it's no wonder, that many would choose to settle instead. From http://arstechnica.com/tech-policy/2012/07/new-study-same-authors-patent-trolls-cost-economy-29-billion-yearly/: "the median amount spent to pay off a troll suit is ... $230,000 for large companies". This same article, however, indicates that the mean value to pay off a troll lawsuit for large companies is $7.27 million dollars. So win or lose, you lose.
Given the current software patent landscape, blanket indemnification is therefore a risk too great for us to assume. Agendaless is a company with historical gross earnings of fewer than a million dollars per year. We write many thousands of lines of code every year. It would be financially crippling to need to figure out if some line of code, or some integration of many lines of code violated any US patent, especially when most software patents are so ridiculously overbroad. It would take an astromonical amount of time to do due diligence: we'd never actually deliver anything in a reasonable amount of time and every project would cost a prohibitive amount of money. It is not a task that can be accomplished.
But even so, if somehow magically we could always know in some reasonable amount of time when we were violating some patent with some bit of code or some integration, what would we do when we did ? Would we stop development of the system? How would the customer feel about such a situation? "Sorry, we had to stop writing your system because it might infringe on this ridiculous patent." Would the customer just say "oh ok, then, I guess you don't need to deliver the software then." Unlikely. We'd almost certainly be asked to deliver something anyway. No one could actually know how to proceed on such a project. There will almost certainly be rework involved. Who pays for the rework? How long is it going to take? Will it still be on time? It's a minefield of liability, even without getting sued.
As a result, as a custom software development company, we just need to assume that code we write violates some crazy patent somewhere, and we hope for the best. The same is true of any US-based consulting firm. Every line of code written by every programmer on the planet (or at least in the US), every website propped up by every company, is a potential target for frivolous and crippling litigation. And there is virtually no defense, except at very least to avoid blanket-indemnifying other parties, to the same extent that you can avoid getting a black eye by avoiding inviting someone to punch you in the face. So it's not out of bad faith that we choose to not blanket-indemnify. It's purely out of self-preservation. By the way, we'd love to be indemnified ourselves! Our current self-indemnification strategy is to have very little in holdings. Patent trolls don't sue people without money. It's a terrible strategy, but it's the only one we have.
Any small- or medium-sized US-based consulting firm that signs a contract with an unbounded patent indemnification clause in it is taking an extreme risk. They will almost certainly be put out of business at the first whiff of litigation.
Lawmakers are supposedly trying to fix this mess through things like the cutely-named SHIELD Act ("Saving High-Tech Innovators from Egregious Legal Disputes Act"). And reportedly some states such as Vermont are offering high-tech businesses some vague protection against patent troll suits. But as of right now, there's no true solution in sight other than protecting ourselves by limiting our liability.
In the meantime, something makes patent infringement language negotiations truly interesting for Agendaless in particular. We almost never enter into a consulting engagement where the customer doesn't already use at least some of our software, and is therefore already implicitly a licensee of ours on other terms. This is because we create and publish open source software, and -- by and large -- customers usually engage us as consultants as a result of having already put into production some of that software. Our standard open source license, like many others, (http://repoze.org/LICENSE.txt) is very simple, but it does have an explicit non-indemnification clause in it, or at least we interpret it as such (the "Disclaimer"). Typically, if they have our open source software in production, this is the license that customers have agreed to, under which they are already consuming our software.
As a result, we've found ourselves in some pretty comical situations. We've needed to negotiate with lawyers for hours, agonizing over patent infringement clause wording that will end up covering perhaps 500 lines of code as part of a minor customization job that might net us a few thousand dollars. In the meantime, we'll know that the company which the lawyer represents already has, in production, hundreds of thousands of lines of related code that we've produced. The company has been using this body of code under our standard open source license to run some core element of their business. This code will have always been put into production long before any Agendaless people might become in-the-flesh-involved. And using the logic of the contract clause we now must negotiate, we can only assume that the customer is, has been, and will continue to be highly exposed to a patent infringement suit that identifies some functionality of this large body of unindemnified code. But nevermind that! Now that there's an opportunity to have a ceremony involving an actual paper contract, it makes sense that a blanket patent indemnity for some relatively obscure body of code that is orders of magnitude smaller than what already exists unindemnified in production should become critically important. That's sarcasm, by the way.
If you want to get particularly indignant about patent trolling, you can listen to Ira Glass' This American Life episode called When Patents Attack! http://www.thisamericanlife.org/radio-archives/episode/441/when-patents-attack . It's a reasonably easy and "fun" way to learn about the wild world of software patents. Just be sure to keep any sharp objects out of reach while listening.
Who ever said the business side of software consulting was not interesting?
The week of September 9 I am getting in a metal tube, off to Cambridge University to conduct a private 2-day Pyramid training. Really looking forward to it. I enjoy training, specifically Pyramid training, and enjoy getting to work with organizations like this.
I'm in a spate of Sphinx doc writing. Immediately afterwards I will polish up my Pyramid tutorials for two classes in September. I have settled on a process and style for both, described herein.
If I go to a tutorial, I don't want to feel stupid. I already know I'm but a marginal step afore the horde. Paying for the material and the instructor to remind me of this is just cruel. Instead, I want the elephant served one bite-at-a-time. Progress me through the complexity, please, stepwise, and leave me at a point of confidence constantly.
I thus conduct my classes and write my material the way I'd like to be taught: fast, fun, introduce concepts in isolation, and let my students constantly do stuff, but safely. This means a good amount of preparation and a way of doing things that has been successful.
First, I do my training material in Sphinx. It makes me very productive, accelerates my refinement cycle time (typo -> fix -> push -> reload readthedocs), and lets me leave behind the artifacts for others. I'm sure the high capitalists would be agog at the idea of not locking up the intellectual property value. If I can't claim an intellect, I can't value the property. But really, the material is much more valuable as bait for the performance.
Second, I organize the training into a fast series of high-level topics. "View Classes", "Sessions", etc. No big deal there. But each of these is a unit of isolation in Sphinx, in GitHub, with working code. A student can cut-and-paste if they are feeling intimidated or behind, though typing fully is encouraged. Moreover, they can walk away, come back later, and see what they did. Students who get the tutorial source can teleport themselves to the present if they fall behind.
Finally, I sprinkle in some fun. I have "Extra Credit" stuff at the end of each section. This lets people that finish the section quickly have something to challenge them. It also provides a thin excuse for me to tell jokes about unrelated topics.
At PyCon 2013 I gave a Pyramid intro class that presumed….Python 3. It worked in 2.7, but I wrote it in Python 3.3, conducted it in Python 3.3, and pitched it in the docs as "assuming" 3.3. I suspect this was the first time in PyCon that a web tutorial targeted Python 3. It felt good to help the Python 3 charge.