The Future of TurboGears

A bit over a week ago several of us decided to get together in Atlanta
this past weekend and hack on an experimental Pylons/TurboGears
integration project. We wanted to discover if we could re-implement
the TurboGears API on top of paste+pylons in a way that would make
TurboGears better.

The goal was not just to re-implement things, but to see if that
re-implementation actually improved the readability, flexibility, and
ease-of-mantinance of the TurboGears project. In other words, we
wanted to make TurboGears: easier for new developers to work on,
easier for us to maintain, and more flexible so we can take advantages
new python web developments in the future.

If you're interested in the details of the sprint, I blogged about it,
and you can read it at:

http://compoundthinking.com/blog/index.php/2007/06/27

On all counts, I think we can now say that this experiment has been a
huge success. And after lots of good discussion with Alberto, Kevin,
and Ben Bangert from the Pylons project, the way forward is now much
clearer. The results of the sprint will become the basis for
TurboGears 2.0.

To make life easy for people who want to install TurboGears 2.0
alongside the current version we will be creating a new package called
tg for TurboGears 2.0. And Alberto is planing to promote it out of
the pygears branch into trunk today.

The new tg package will implement a very large percentage of the
current TurboGears API, and thus is intended to provide an very easy
upgrade path for current TurboGears users. In particular current
controller code should be very simple to port, and Kid, Genshi,
SQLAlchemy, and SQLObject code will be supported, so most of your
template and model code will require almost no changes, or will be be
usable as is.

There's still work being don on TurboGears 2, and it's hard to predict
a release schedule in open source projects. But if you pressed me I
would say that I expect to see a beta in the next couple of months.
It could be earlier, it could be later... If you're the adventurous
type and don't want to wait until then can check it out of subversion
today, but expect a lot of rapid changed.

At the same time the turbogears development team will continue to
maintain and support TurboGears 1.x for our current users. We want to
accommodate users who create new TG 1.0 projects for a long time, but
we also want to make it easy for those who want the latest and
greatest stuff to get started with TurboGears 2.0 as soon as possible.

There will be a TurboGears 1.1 beta release in the next few weeks,
which will fully support the current API, and will switch the default
templating engine from Kid to Genshi, and the default ORM from
SQLObject to SQLAlchemy. This release will not include an upgrade to
CherryPy 3, because that has required backwards incompatible changes,
and has taken much more work than expected. If someone wants to take
up the standard, there may be a TurboGears 1.2 wiith CherryPy 3
support, but as far as I know nobody had volenterred to do this.

And of course, SQLObject and Kid will remain supported in 1.1, to give
us full backwards compatability with current applications. Catwalk
and Model Designer will remain SQLObject only applications in 1.1.

And that brings us back to TurboGears 2.0. In addition to the current
TurboGears API, the new tg package will support all kinds of new
features that will make our user's life easier. Many of these
features are the direct result of our integration with Pylons, because
we'll be getting access to a lot of great stuff they've already done.
And we'll be sharing almost all of our infrastructure code with
another group of great developers.

The end result is that TurboGears users that have sites that need to
scale up, will have have access to world class caching options
(including memcached support), as well as a number of other
performance enhancing features and options. For the majority of us
who are more interested in rapid development, and deployment
flexibility, than raw performance, TurboGears 2 will include all kinds
of new features. TurboGears 2 will also implement a more flexible
Object Dispatch system, as well as direct Routes integration, so it
will be even easier to get exactly the URL's you want.

There's a lot more going on too like 1.1 TurboGears 2 will make Genshi
and SQLAlchemy the defaults because they provide a better experience
for the majority of web developers. We won't be forcing our opinions
on anybody, and other ORM's and templating engines will be supported,
but we'll be focusing all our documentation effort on those defaults.
We want to make very sure that our documentation tells a clear and
compelling story about how this stack of components makes web
development easier.

At the same time, we'll be working to support people who want to
develop and maintian sub-projects for automatic CRUD (like Catwalk or
the Django interface) outside the TurboGears 2 core. Our hope is that
these tools will be broadly usable by anybody who is working withing
the context of a framework that implements the Web Server Gateware
Interface standard. Ultimately I think Pylons, TurboGears, Paste,
Zope, and all kinds of other python web developers toolkits will be
working together on creating great tools for rapid development.

Thanks to Pylons, Paste, SQLAlchemy, Genshi, Babel, and a whole host
of other contributions from around the web TurboGears development is
getting easier and more fun. I for one, am really looking forward to
all the good things that are in the pipeline!

I'm looking forward to some spirited discussion of the TurboGears 2.0
plan, and the plan for TurboGears 1.1 on the TurboGears trunk mailng
list.

(http://groups.google.com/group/turbogears-trunk)

--
Mark Ramm-Christensen
email: mark at compoundthinking dot com
blog: www.compoundthinking.com/blog

Thanks very much, this kind of info is very useful for anyone just
starting to implement things in tg such as myself :)

KISS all the way!

I'm sorry to be asking the hard questions but
what will people win for using TG2 instead of "plain" pylons?
why will someone that didn't had the joy of coding in TG1 will want to
learn TG2?
which are the core TG2 features, that is what is TG-only and what we
get from pylons? I have half an answer there, dict-dispatch and
@expose and of course all the great stuff in
http://trac.turbogears.org/browser/projects and
http://turbogears.org/cogbin

I think we should write something on why TG+pylons is better then
pylons, I say we as in TGpeople but not me because I honestly don't
know anything about pylons.

that will be a great day. May I suggest we start defining entry points
for all the TG stuff? maybe have turbogears.tools for things like
paginate,scheduler,feed controller,etc. and of course reinstate the
extensions entry point for identity et all.

Actually I'm looking forward to that kind of hard questions! I think
they will help us to clarify the way we work with Pylons, and to
better decide who does what and how we will work together better in
the future.

My understanding of things is that Pylons will be focused on
providing maximum flexibility, and creating lower level framework that
allows all kinds of choices.

TurboGears will stay focused on the thing's it's always been focused
on -- creating a full-stack framework for rapid web development.
We'll give you a stack of components that work well together. We'll
create basic templates that get you going quickly, and we'll focus on
developing a coherent, well documented standard API for our
developers.

We need to make choices about integrating tools, creating sane
defaults, and making the experience of getting started as simple and
rich as possible. There have been requests for a higher level API on
top of pylons for a while, and at least in part TurboGears 2 will
provide that for people.

We'll be providing a layer above some of the raw WSGi internals that
are present in paste. Returning a dictionary is one of the core
pieces of the turbogears API, and we'll be doing that. Object
dispatch is part of the core TurboGears way of doing things and that
will be in TurboGears but not Pylons.

We'll also, hopefully, be creating a wide variety of tools for rapid
web development, like the current FeedController, or a restful CRUD
controller based on the atom publishin protocol.

There are lots and lots of possibilities, and only time will tell
exactly how things will work out. But what will be certain is that we
will provide current TurboGears users with a way to migrate to a
TurboGears like API on top of Pylons, and take advantage of all kinds
of cool new things. And we'll provide a nice, standard, easy-to-use
way to create web applications on top of pylons.

Anybody cares to write a short news item about this on
http://docs.turbogears.org/News/Ticker and link up this post, Mark's
blog and some of the reactions (esp. Kevin's [1])?

Cheers, Chris

[1]
http://www.blueskyonmars.com/2007/06/27/turbogears-2-a-reinvention-and-back-to-its-roots/)

I am an application developer and I have used TG and I am very happy
with it. I find it very usable and flexible enough to meet my needs.
I really liked the fact that you could type "tg-admin quickstart" and
get going. This is in contrast to many Java web frameworks which I had
worked with previously which had a steeper learning curve and less
usability.

There is a great deal of discussion about interacting with other
components but what I would really like is "canned" functionality with
sensible and integrated defaults and components rather than a more
loosely coupled framework. I scarcely have the time or bandwidth to
grok all the features of TG. My focus is on Rapid Application
development rather than assembling the Framework which I am going to
use from disparate libraries. Each time a new library is introduced
into a framework must be learned and managed etc.

I don't see what Routes, Pylons etc are buys us in terms of
application development. I want to work with TG components and use TG
as an Application Development Framework. I don't want to have to re-
invent or re-learn TG skills every 6 months.

I am happy enough with CherryPy dispatching and architecture. What is
the status of CherryPy in all of this ? I have been reading some of
the discussions on the Trunk list and it is not clear to me why
CherryPy is being jettisoned especially when I read:

http://www.cherrypy.org/wiki/CherryPyAndPaste

which is partisan but raises points about application developer ease
of use which I do not see answered elsewhere on the internet and in
discussion groups. I particularly value the illustrations of the
pythonic nature of CherryPy vs Paste/Pylons etc

There may be advantages in terms of ease of deployment to using Pylons
Paste etc but I don't see them.

The only real preference I have is for SQLAlchemy over SQLObject. I
think that the inclusion of SQLAlchemy is a real win but I would like
to see a later version of SQLAlchemy being used in TG. Mark, I see
that you are working on the book. :-)

Thanks for all your efforts on Turbogears.

Tom

Thanks all for the great work done so far. Have thoroughly enjoyed
working with TG.

Unsure if this is the forum for adding some more thoughts for 2.0...
but, wanted to indicate that it may be helpful to come up with a
layered approach to component requirement.

For example, consider the case of deployment of a TG-based app in an
embedded systems environment. This is a perfect example that demands a
multi-tiered framework. Not always, do we have the luxury of an
embedded database. With or without a database, data of various formats
would have to be assembled behind the scenes. In this case, it would
help to "setup" a leaner version of the framework, and provide "shims"
to access/convert disparate types of data. Right now, its a little bit
unclear if one could "seperate" out db-related comps from the main
source base, and yet get it to work in an embedded system that has no
database. Also, in many such systems, space (hd/flash/etc) could be of
a premium, driving the need for a more layered approach. A "ezsetup.py
-no_db" or "tg_admin quickstart -no_db" may be a good option to have.

Other needs for embedded systems could include:
. non-db-based authentication via sessions/cookies
. the ability to get built-in log rotations/movement triggered by
age/size etc.
. builds/setup via cross-compiles (I should admit, cross compiling
TG was
a breeze (minimal changes), as compared to the changes to
python /
setuptools needed to support that.

Just some thoughts (from a narrow embedded systems perspective :-)...

Also, it may be helpful, if there could be a cpl-page doc somewhere
that
could describe (graphically) the role of each of the proposed pieces
of
TG 2.0 - so some of us could understand the architecture/interactions
a
lot better.

Thanks again, for all your efforts.
Regards,
/venkat

+10 :)

[]s
Roger

First, you bring up the question of what the added flexibility of
Pylons integration is going to cost you in terms of learning curve.
I won't pretend that there will be no learning curve involved, we are
making some changes, and there will be a few areas of backwards
incompatability.

But you won't have to learn to write Routes yourself -- we have a
single default route, that will call your root controller, and provide
the same CherryPy object dispatch functionality that you are used to.
You won't have to learn WSGI either, because your controller methods
will return dictionaries, just like they always have.

At the same time for those users who have expressed a desire to map
their application to a legacy URL schema, or who have had to hack up
strange default methods to get what they want, Routes will be
available to them. No cost to most users, extra benefit to a few who
need it.

The same will be true of all the caching, session, and WSGI middleware
stuff that's out there. If you don't need it, it won't change the way
you develop TG applications. And if you do, there will be simple
ready-made solutions available to you.

We still plan to provide an integrated set of libraries to make web
application in TurboGears look pretty much the same as it always has.

As for the CherryPy vs Paste page you point to, I agree with some of
it's comments and not with others. I don't have time to to write up
a full response, but I think it's the wrong comparison for us.

Pylons offers a nice layer of abstraction on top of paste, and
TurboGears 2 is going to offer a layer on top of that -- so the fact
that you don't normally want to mess with all the lower level features
of Paste yourself should be perfectly fine. We'll be giving you
ObjectDispatch, a standard Response Object, turbogears style config
objects, and all of the conveniences you've grown to know and love.
It's just that we'll be borrowing Pylons implementation, or building
our own rather than use CherryPy's.

Of course that leads to your other implied question, why not just use
CherryPy. This is a harder quesiton Pylons and CherryPy offer us a
lot of the same features in different ways. And we tried going both
ways, but in the end it turned out Pylons worked better for us.

There's a lot of good design and engenering work in CherryPy 3, and I
wish that we were able to build version 1.1 on CP3 as we'd originally
planed. But due to some internals in TG 1.0, that was more difficult
than anybody anticipated.

Hopefully that answers most of your questions. If you have more,
please let me know and I'll see what I can do to clarify things even
more.

--Mark Ramm

That sounds good. I hope there is a plan to port/wrap/emulate the new
CP3 dispatcher options too!

Nontheless, I think a page addressing this issue would be a good idea
and would reassure a lot of people who are either new to TG or thinking
of using it and are now wondering whether they will be wasting time
learning the CP way of doing things.

This is probably a stupid question as I don't know WSGI at all, but will
it possible to emulate CP tools? They look really useful in CP3. Perhaps
part of the end goal should be to provide all the CP convenience
features on top of paster too?

Thanks
Iain

Well, there are some ways to do things before every request using
__before__() and __after__() methods on the controller classes which
allow for cross cutting behavior to be added to whole controller
hierarchies, and we've added four hooks that can be applied by adding
decorators to individual exposed controller methods:

* before_validate,
* before_call,
* before_render,
* after_render

So you can write

@expose(...)
@before_validate(function_to_call_before_validation)
def my_method(self, arguments):
pass

This will register function_to_call_before_validation() to the be
called before validation is done. These functions will be able to
read (and mutate) the

If you have a use case which isn't covered by either of these
extension mechanisms, let me know and I'll see what we can do.

And honestly, if you need something cross-cutting and the above hooks
don't work writing middleware isn't that difficult, and we'll do what
we can to make it even easier. (I believe there are some plans in
the works for this already).

--Mark Ramm

[...]

This is already here :)
see:
http://docs.python.org/lib/node413.html
and
http://docs.python.org/lib/node414.html

which are standard handlers you can use by just specifying them in
your log.cfg file !

Isn't life beautiful when using Python and working with TG ? ;)

Cheers,
Florent.

That sounds really handy!

Thanks
Iain

Sure, this is as good a place as any.

Well, pylons has a minimal quckstart template equivelent, and an ORM
is not really required for either TG or Pylons. TG2 isn't going to
require a database connection or configuration to run, or
automatically import the DB for you, those imports will be in code you
control.

So it should be a lot easier to build the minimum set of packages you
need to run, and keep your memory footprint small in TG2 as compared
to the original TurboGears.

And if somebody contributes, and helps to maintain one, I'm sure we'd
be very happy to have a TG Minimal template.

This would be cool, but it might be more of a seprate project on top
of pylons. The way I see it Pylons will be the uberflexible
framework that you can build all kinds of different web-app-frameworks
on, and TurboGears will be a particular layer on top of Pylons which
gives you sane defaults, and focuses on creating a compelling
experience for web develoers. I think you could say that TurboGears
is to Pylons as Ubuntu is to Debian.

Yea, I'll put that on the To-Do list. I'm sure it will help a lot of
people to understand what's going on, and it might even encourage
people to get involved with TG2 development. I will say however that
we've got a LOT going on so this may not happen right away.

I second you, has these tools made me choose TG over other frameworks. I
really want to be able to use them in futur releases of TG!

--
Frédéric

http://www.gbiloba.org

Your wish has already been answered

http://code.google.com/p/tgcaptcha/

Cheers
Chris

Thanks for your reply. I have been reading further and I am beginning
to grasp
the Paste/Pylons/Routes stack. I am glad to hear that sensible
defaults will be provided
but that more powerful mechanisms will be available to those projects
who need
them.

I have been looking around at Genshi and it seem like an evolutionary
and logical
replacement or development for Kid which is great as the learning
curve should not be that
steep during the transition.

Can you give us some information on how you guys intend to integrate
with
SQL Alchemy ? Which versions of SQLAlchemy do you intend to support
in
TG 1.0, TG1.1 and TG 2.0 ?

Once again thanks for the answers and all the background information

Tom

> ...
>
> ead more »

That's very much what we want too. We want TurboGears to work
the way it does now, and provide the same function it does now --
creating an easy-to-use full stack framework. Pylons will continue
to provide maximum flexibility, and power.

I think I've said this in another thread, but i think of TurboGears 2
as like Ubuntu Linux and Pylons as Debian. We're focused on user
experience, and creating a novice friendly environment. But we
couldn't do that without the power and flexibilty of the underlying
core. And we don't intend to hide that power and flexibility from
advanced users.

Yea, the developers of Genshi pretty much took the Kid API and
reimplemented it from the ground up with a few nice enhancements.

Well, we already support SA in 1.0 (.3x) and we'll probably require
.4 in TG2. 1.1 will likely require the latest stable release that is
out at that time.

I hope that info helps.

I'm a little afraid by the announced changes, but I'm sure you'll make
TG 2.0 better than TG 1.x this way.
In the other side I was a little afraid to see pylon become a better
basis
for web development than TG, and I'm happy that future TG release will
be Pylon powered :-)

I used some RAD in the past and lot of frameworks and found always
the
limit of their own design. This is the first time I didn't find these
limit with
a framework. I'm speaking of TG of course.
For example I know today why the "tg.flash" is so powerfull and so
user unfriendly (no class of messages, colors and all theses stuff).
This is because these stuff should be provided by another TG
components.
Probably lot of them will appears in the future and any developper
will be
able to chose the one he need.

Please continue to provide stuff of this quality.

May I captivate your attention on two point no well handled yet by TG

1. Form Validation: provide the form validation facilities inside the
controller itself.
2. Keeping context/session information from all method of a
controller.

By (1) I mean be able to raise validators.Invalid from anywhere in the
controller,
and get the form displayed again.
Because some operation are atomic and cannot be split between the
validation
schema and the controller.
For example a lock mechanism, using a 'filename.lock' somewhere on the
file system.
You can test if the file already exist in the validation schema doing
a
> os.path.exist('filename.lock')
but this is only in the controller, when doing a
> os.open('filename.lock', os.O_CREAT|os.O_EXCL)
That you will know if the resource is available or not !
This is problem for record creation in SQL table.

FYI the command above raise an exception if the file already exist
>Traceback (most recent call last):
> File "<stdin>", line 1, in ?
>OSError: [Errno 17] File exists: 'filename.lock'

About (2). We (Web developers) know how crazy it is to store session
info at the server side,
because the user can use the back button at any time and resend old
Form data
and expect them to be correctly updated. This mean some session info
must be stored
into the Forms themself.
Probably some of you already typed these lines :

<input type="hidden" name="tg_paginate_limit" value="$
{tg.paginate.limit}"/>
<input type="hidden" name="tg_paginate_no" value="$
{tg.paginate.current_page}"/>
<input type="hidden" name="tg_paginate_order" value="$
{tg.paginate.order}"/>

To keep pagination info from page to page :-)

TG should help people to do the job transparently by adding all these
context
data in all Form automatically.

Here is how I did it myself , briefly!
I have added a "context_params" attribute into the controller object
containing
the name of all the variables I want to keep from page to page inside
the controller.

class Mailbox(controllers.Controller, identity.SecureResource):

require=identity.in_any_group("manager", "domain-admin")
context_params=[ 'email_addr', 'addr_filter', 'tg_paginate_limit',
'tg_paginate_no', 'tg_paginate_order' ]

Before my controller return a dictionary, I check if any of these
field are set.
If not I take the original value from cherrypy.request.params.
I rewrote my own widget.Form class to verify that all these fields are
set in the form and if not
I add them in hidden fields.

Regards