Support Map, List and Bool type #44

craftzdog · 2014-10-17T13:37:08Z

DynamoDB now supports to store entire JSON-formatted documents as single items.
http://aws.amazon.com/jp/blogs/aws/dynamodb-update-json-and-more/
In this update, following new attribute types have been added:

Map
List
Boolean
Null

This pull request adds a feature to support Map, List and Boolean type of attribute.
In Map and List attribute, it currently does not validate its content format.

craftzdog · 2014-10-27T04:51:22Z

Map

Map can be declared for any attributes, except for hash and range keys.
It can store an object in the attribute.

var Photo = vogels.define('Photo', function (schema) {
  schema.String('userid', {hashKey: true});
  schema.Map('tags');
});

Photo.create({userid:'john', tags: {emily: {x:10, y:8}}}, console.log);

Map can recursively define its schema:

var Photo = vogels.define('Photo', function (schema) {
  schema.String('userid', {hashKey: true});
  schema.Map('location', function(schema) {
    schema.String('name');
    schema.Number('latitude');
    schema.Number('longitude');
  });
});

List

List can be declared for any attributes, except for hash and range keys.
It can store an array in the attribute.

var Photo = vogels.define('Photo', function (schema) {
  schema.String('userid', {hashKey: true});
  schema.List('likes');
});

Photo.create({userid:'john', likes: ['emily']}, console.log);

List can define schema of its element:

var Photo = vogels.define('Photo', function (schema) {
  schema.String('userid', {hashKey: true});
  schema.List('likes', function(schema) {
    schema.String('userid');
    schema.Date('likedAt');
  });
});

jus101 · 2014-12-08T01:25:02Z

Any chance this pull request will make it in to master?

ryanfitz · 2014-12-08T18:21:15Z

I haven't had a chance to work much on this lib, but am planning on getting full document support working over the holidays. I have partial support working in a separate branch, which isn't completely fleshed out yet.

Amazon has released a number of new features to DynamoDB in the past couple of months and unfortunately the current api exposed with vogels makes it impossible to fully utilize these new features. Part of the document support will include a modified way of defining schema to get access to all the new features as well as hopefully better support new DynamoDB features in the future.

tl;dr - No, this will not make it into master, but will offer full document support in the near future

ryanfitz · 2014-12-08T18:54:16Z

I haven't had a chance to work much on this lib, but am planning on getting full document support working over the holidays. I have partial support working in a separate branch, which isn't completely fleshed out yet.

Amazon has released a number of new features to DynamoDB in the past couple of months and unfortunately the current api exposed with vogels makes it impossible to fully utilize these new features. Part of the document support will include a modified way of defining schema to get access to all the new features as well as hopefully better support new DynamoDB features in the future.

tl;dr - No, this will not make it into master, but will offer full document support in the near future

rvdwijngaard · 2014-12-08T18:55:51Z

Hi, that is great news. Looking forward to these changes!

ghost · 2014-12-20T06:05:44Z

Looking forward to this as well!

ghost · 2014-12-29T20:56:45Z

Ryan,

Any ETA on document support? Trying to decide whether to consider this module for inclusion in a Q2 2015 project.

ryanfitz · 2014-12-29T21:23:36Z

@keithchilders I'm looking to get a beta version ready by end of this week

ghost · 2014-12-29T22:50:46Z

@ryanfitz You are the man, thanks! Can't wait to see the new schema API.

ryanfitz · 2015-01-04T22:51:51Z

@keithchilders @winduptoy If you want to take a look, I've pushed up the new json document support branch. I'm still working on updating the README. Most of the examples have been updates, but I still have a few that needed updating to the new api.

The only change for a user is the new way models get defined, all other apis remain the same. Takes no more than a minute to migrate from the old schema define api to the new configuration based api. There are still a few outstanding issues (mostly related to updating nested items), but let me know of any feedback you have.

ghost · 2015-01-05T19:32:59Z

@ryanfitz I will take a look today. Thanks!

ghost · 2015-01-05T22:10:49Z

Ryan,

As you iterate on this would you mind publishing it to npm as 3.0-betaX? It makes pulling in dependencies a lot easier.

ghost · 2015-01-05T22:33:09Z

@ryanfitz Also I like what you're doing with using the Joi module to handle schema validation, default values, etc. I especially like that it uses a chainable API. But, since you're also using your own vogels.types, it would be great if you could maybe wrap the Joi types in a matching vogels.types item so I don't need to remember which type goes with which type library. You could just throw errors on all the Joi methods you don't expose for your own types?

I enjoyed being able to define my own model pseudo-dialect in pure JSON then pass that into some middleware for transforming it to the appropriate vogels constructs and spitting out a vogels model. That's a lot harder to do now that the types aren't described in object keys anymore but in chained method calls.

ryanfitz · 2015-01-05T22:59:42Z

@keithchilders Im planning on getting an rc1 build out later on tonight or tomorrow. I'll let you know once its out.

As for custom types and wrapping joi types, if you look at the code its actually 100% Joi under the hood, its simply some syntax sugar to declare Sets and UUIDs a bit more easily. With the old schema api, I was wrapping joi types and this lead to a number of major pain points. The joi module moves very quickly and keeping up with their changes its extremely difficult. One of my main goals with this new api is to allow users to update their joi module independently of vogels. I do agree that its a bit odd to expose a few custom types and not expose the basic types. I don't know the best solution for this, but I am opposed to wrapping Joi.

The new define api is now 100% JSON, the schema param is actually completely optional and it even supports unknown keys (if you configure it to accept unknown). I don't know how your current pseudo dialect works, but I would think it should actually be much easier to implement with the new api.

ghost · 2015-01-05T23:08:06Z

Tell me more about how to define schema without chaining Joi methods together! That will make me happy :)

Here's some example psuedo dialect based on your previous API:

key: {type: 'String', required: true},
name: {type: 'String', required: true},
version: {type: 'Number', default:0},
date: {type: 'Date', default: new Date()}

All I had to do before was pass my string 'type' as your schema[type] method and invoke it. Now I have to know ahead of time whether stringSet() is a Joi type or or vogels.types type, and there are two sets of supported attributes per-type (default, required, etc.) I also don't love having to require Joi from outside my model; I'd much rather use it at the dependency level you specify in the vogels version I'm using, plus if the models are vogels models it seems like the vogels package should contain everything they need to instantiate them, so at least something like vogels.Joi as a reference to your dependency that I could bring in rather than a separate require of Joi at a version that may or may not be compatible would be preferable from my perspective.

I have moved this feature request into your 3.0 release notes issue, which I figured would be more visible to all your users than this rejected pull request thread :)

New Features * Full JSON document support, including support for List, Map, and BOOL datatypes. Closes #16, #44 * New timestamps config option to automatically add the attributes createdAt and updatedAt when defining a model. These attributes will get automatically get set when you create and update a record. * Flexible schema configuration to allow unknown, dynamic attributes both on top level and nested records * Ability to fully configure global and local secondary indexes. Allows to configure index names, attribute projection settings and index throughput. Closes #43 , #48 * adding deleteTable api to remove the table from DynamoDB. Closes #10 * 100% code coverage with new integration test suite and integration with travis-ci. Bug Fixes * CreateTables checks if error occurs while attempting to create a table. Fixes #41 * Fixed error handling when streaming query, scan and parallel scan requests. * Fixed retry handling when running query and scans. Fixes #45 Updated dependencies * Joi to v5.1.0 * aws-sdk to v2.1.5 - Closes #49 * async to v0.9.0 * lodash to v2.4.1

New Features * Full JSON document support, including support for List, Map, and BOOL datatypes. Closes #16 and Closes #44 * New timestamps config option to automatically add the attributes createdAt and updatedAt when defining a model. These attributes will get automatically get set when you create and update a record. * Flexible schema configuration to allow unknown, dynamic attributes both on top level and nested records * Ability to fully configure global and local secondary indexes. Allows to configure index names, attribute projection settings and index throughput. Closes #43 , #48 * adding deleteTable api to remove the table from DynamoDB. Closes #10 * 100% code coverage with new integration test suite and integration with travis-ci. Bug Fixes * CreateTables checks if error occurs while attempting to create a table. Fixes #41 * Fixed error handling when streaming query, scan and parallel scan requests. * Fixed retry handling when running query and scans. Fixes #45 Updated dependencies * Joi to v5.1.0 * aws-sdk to v2.1.5 - Closes #49 * async to v0.9.0 * lodash to v2.4.1

New Features * Full JSON document support, including support for List, Map, and BOOL datatypes. Closes #16, #44 * New timestamps config option to automatically add the attributes createdAt and updatedAt when defining a model. These attributes will get automatically get set when you create and update a record. * Flexible schema configuration to allow unknown, dynamic attributes both on top level and nested records * Ability to fully configure global and local secondary indexes. Allows to configure index names, attribute projection settings and index throughput. Closes #43 , #48 * adding deleteTable api to remove the table from DynamoDB. Closes #10 * 100% code coverage with new integration test suite and integration with travis-ci. Bug Fixes * CreateTables checks if error occurs while attempting to create a table. Fixes #41 * Fixed error handling when streaming query, scan and parallel scan requests. * Fixed retry handling when running query and scans. Fixes #45 Updated dependencies * Joi to v5.1.0 * aws-sdk to v2.1.5 - Closes #49 * async to v0.9.0 * lodash to v2.4.1

ryanfitz · 2015-02-21T18:34:13Z

Commit 6d5e70e adds full document support

craftzdog added 7 commits October 17, 2014 22:00

Support Map, List and BOOL type

d48f402

Add description about List and Map

6f48655

Add test

33bfee7

Add deserialization test

f1f5a4f

Deep copy data before serializing

8c12444

Support recursive schema for Map and List types

53add54

Add instruction for use of Map and List

bde099b

Fix example

7f860c6

ryanfitz mentioned this pull request Jan 4, 2015

2.0.0 Release Notes #50

Closed

ryanfitz closed this Feb 21, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Map, List and Bool type #44

Support Map, List and Bool type #44

craftzdog commented Oct 17, 2014

craftzdog commented Oct 27, 2014

jus101 commented Dec 8, 2014

ryanfitz commented Dec 8, 2014

ryanfitz commented Dec 8, 2014

rvdwijngaard commented Dec 8, 2014

ghost commented Dec 20, 2014

ghost commented Dec 29, 2014

ryanfitz commented Dec 29, 2014

ghost commented Dec 29, 2014

ryanfitz commented Jan 4, 2015

ghost commented Jan 5, 2015

ghost commented Jan 5, 2015

ghost commented Jan 5, 2015

ryanfitz commented Jan 5, 2015

ghost commented Jan 5, 2015

ryanfitz commented Feb 21, 2015

Support Map, List and Bool type #44

Support Map, List and Bool type #44

Conversation

craftzdog commented Oct 17, 2014

craftzdog commented Oct 27, 2014

Map

List

jus101 commented Dec 8, 2014

ryanfitz commented Dec 8, 2014

ryanfitz commented Dec 8, 2014

rvdwijngaard commented Dec 8, 2014

ghost commented Dec 20, 2014

ghost commented Dec 29, 2014

ryanfitz commented Dec 29, 2014

ghost commented Dec 29, 2014

ryanfitz commented Jan 4, 2015

ghost commented Jan 5, 2015

ghost commented Jan 5, 2015

ghost commented Jan 5, 2015

ryanfitz commented Jan 5, 2015

ghost commented Jan 5, 2015

ryanfitz commented Feb 21, 2015