Posts tagged with “google developer day”…

Last week's Google Developer Day was a huge amount of fun. Particularly interesting was Mano Marks' codelab session on Google App Engine.

I'd played about with App Engine before, but it was only when going off on a tangent and exploring the codelab example code that I started to get a feel for some of the application growth / scalability limitations that the framework poses at the moment. (In the course of the event I managed to find two bugs in App Engine, and then a third bug in Google Chrome when trying to submit the issue.)

Updating domain models through the Django-esque framework can be fraught, lovely though it is. Removing or modifying properties can be done silently, which is fair enough, but records are only updated at write-time. That is to say, if a property is removed, each object must be read and then persisted before the change will be propagated across all data. Changing a property type is even more fraught, as a new temporary property must be added, filled with data from the old, the old removed, and then the data cast and moved back. Google recommend that you don't fetch records several hundred at a time when running these processes, as the app engine may injudiciously kill lengthy requests, but more importantly, because their fetch offset method is inefficient and unreliable with an offset over 1,000. Principly, this is because the fetch method queries the dataset when run, and then only returns the limited data specified in the parameters.

Simply put, it's expensive to page through all data without querying specifically for each chunk, as the underlying mechanism to return data doesn't distinguish between domain and relational data. Having built an ORM framework in the past, we have some opinions on the matter. Google's solution (when it arrives, see the PDF on their thoughts on scaling) will differ, not least as they have some very different considerations for App Engine. Here's a snippet from the FAQ for our LPC framework software, which describes our own solution to the problem when we encountered it in 2000:

Differentiation between domain and relational data

The LothianProductionsCommon (LPC) framework provides highly scalable search and retrieval functionality. Key to this is the underlying differentiation between domain object data representing Cacheable objects and domain object relational data representing their relationships.

Need for flexible primary keys

Cacheable objects must have a simple primary unique key (Id), and the framework assumes that this can be represented by a long integer in most cases (domains where objects cannot be keyed by a numeric ID can be supported through use of the MathHelper's BaseDecToBaseN methods -- if implemented at factory level this allows for IDs to use a 37-byte alphanumeric base, though this approach is highly undesirable).

Partial and complete cache-assisted domain object retrieval

Where an object's primary key is known, it can be fetched simply by using the appropriate broker's Get or Gets method (users should note that polymorphism is not exploited by the broker templates as it cannot be modelled in WSDL when exposing brokers to SOAP requests). This will retrieve the objects required by primary key, and can pass through the cache on an object-by-object basis, optimising requests for groups of objects that are only partially cached.

Get and Gets calls the only way to retrieve objects

This Get or Gets broker call is the only way to retrieve domain objects, and it will reference the underlying application configuration, bringing out the data manipulation language (DML). When using SQL, for instance, the DML may read as:

SELECT id, name, description modified FROM test WHERE {id}

Scheme-based optimisation of Gets calls

The factories will generate valid DML from this template string. In the case of SQL queries, there are various layers of optimisation determined by the ConnectionScheme being used. Firstly, when using Gets to retrieve multiple domain objects from a database, the scheme will decide whether using OR or IN statements is most effective for the given database type.

SELECT id, name, colour FROM test WHERE id = 1 or id = 2 SELECT id, name, colour FROM test WHERE id IN (1, 2)

The scheme will also split large Gets calls where required by the underlying data provider. For instance, if a Gets call is querying for 40,000 objects and the data provider only permits 15,000 bound variables per statement, the scheme will transparently split the query into four and merge the results. Additionally, if the data provider has a query length limit -- for example, of 2,000 characters per query -- the scheme can also transparently split and remerge the query.

Why separate relationship resolution?

Consider the following queries run sequentially on a static dataset:

SELECT id, name, colour FROM table WHERE colour IN ('red', 'blue') SELECT id, name, colour FROM table WHERE id IN ('red')

Running and caching results of both could lead to an unnecessary connection and query being made, as well as the unnecessary transfer of data from data provider to application server, and the unnecessary use of cache space for what are duplicate results.

By splitting out the relational information in the queries, the LPC has an opportunity to cache data more effectively with minimal overhead. For instance, the following query will return only the primary key of the data to be retrieved.

SELECT id FROM table WHERE colour IN ('red', 'blue') = 1, 2, 3, 4, 5

These IDs can then be loaded if not cached, and cached appropriately.

SELECT id, name, colour FROM test WHERE id IN (1, 2, 3, 4, 5)

It is likely the next query will contain duplicate IDs.

SELECT id FROM table WHERE colour IN ('red') = 1, 2, 3

And in this case it will turn out that we have already retrieved and cached objects #1, #2 and #3. In this particular case we can see that the obviated Gets query does not look that costly -- but what if 99% of Gets would be looking for object #2, and what if each object has many large fields?

Managing relationships

Having established that domain objects can only be retrieved by a Get or Gets call, and that they can only be retrieved by a unique primary key, the next step is to define a search framework to resolve the relationship between terms as above like "colour IN ('red', 'blue')" and a set of results as seen, like "1, 2, 3, 4, 5". In addition to this, it is clear that as well as caching domain objects, being able to cache relationship resolution queries and their answers would be beneficial.

SearchCriteria objects contain collections of SearchTerms in the LPC, allowing users to build simple queries as above, where "colour IS red OR blue", and more complex queries with multiple sort orders, fuzzy matching and layers of Boolean logic. SearchCriterias are also sensibly used to key the cache of resolved relationship queries.

When handed to the GetRelationship call, the criteria will be dynamically transformed and interpreted to suit the data provider being used. This allows for dynamic changes in data provider type, and portability -- the LPC can be moved from a MySQL back end to an Oracle one despite substantial SQL format differences, or even backed onto a set of XML files with a SQL to XQL transform. GetRelationship will return a Relationship object, which provides a primitive ordered list of IDs and accompanying lists of relevance scores (used when the underlying query engine can provide such information) and modification timestamps, as well as the SearchCriteria used to generate the relationship. The modification timestamps are used by the caching mechanism to assist with object expiry. If we can see in the relationship query that the object has been modified since it was last cached, we know the cache can be invalidated for that particular object.

Start to finish, retrieving domain objects from a query

1. SearchCriteria is created and populated with SearchTerms.
1. colour IN ('red', 'blue')
2. SearchCriteria is handed to GetRelationship call in broker.
1. RelationshipCache is searched for existing cached Relationship.
2. DML is retrieved from application configuration ("lpc/dml/[Factory's name]:Criteria[Criteria.s name]"): SELECT id, modified FROM test WHERE {criteria}
3. Criteria phrase is translated appropriately: SELECT id, modified FROM test WHERE colour IN ('red', 'blue')
3. Broker returns Relationship object.
1. 1, 2, 3, 4, 5 (etc.)
2. Relationship is cached in RelationshipCache.
4. Relationship's IDs can be passed into broker's Get or Gets call to retrieve domain objects.
1. Cache is searched for existing cached domain objects.
2. DML is retrieved from application configuration ("lpc/dml/[Factory's name]:Gets"): SELECT id, name, colour FROM test WHERE id IN (1, 2, 3, 4, 5)
3. Domain objects are cached in Cache.
5. Cacheable objects are returned.

This post has 0 comments »

The Technology of Tomorrow 2008 conference page has been taken down (now, was). We had been wondering about attending later this month, but after getting no response from emails to their customer care team, and now after calling their ticket line and talking to staff who have no idea what is going on, who knows?

Hopefully the rather publicity hungry Martin Warner will clear things up.

September is a great month for industry events, most notably with the Google Developer Day and TechCrunch / SeedCamp party coming up.

This post has 1 comments »