Getting started with KiokuDB http://www.iinteractive.com/kiokudb/ Jonathan Rockway http://jrock.us/ ---- What is KiokuDB? ---- What is KiokuDB? Object Database ---- Object database? Natively store object graphs (Relational database stores tuples) ---- Object database? Give an object a name Get it back later, by name (That's it) ---- Using KiokuDB Example class 1: package Example; use Moose; # not required has 'foo' => ( is => 'ro', isa => 'Str', ); ---- Connecting to a KiokuDB use KiokuDB; my $DSN = 'hash'; # more later my $kioku = KiokuDB->connect( $DSN ); ---- Storing data my $scope = $kioku->new_scope; my $id = $kioku->store( Example->new( foo => 42 ), ); say "Stored $id"; # Stored A4AE4E5A-E404-11DD-8EB6-4FDF4766D24C ---- Storing data (with a name) my $scope = $kioku->new_scope; # just do this once my $id = $kioku->store( example => Example->new( foo => 'example' ), ); say "Stored $id"; # Stored example ---- Retreiving data my $scope = $kioku->new_scope; my $example = $kioku->lookup( 'example' ); say "Got '", $example->foo, "'"; ---- Store @objects -> @ids Object added to "Root Set" ---- Lookup @ids -> @objects Objects are tied to the most recent scope ---- Scope Make sure that mutally weak refs are held by *something* ---- More ways of adding or updating: store - insert or update deeply insert - dies if object exists update - updates first level deep_update - updates deeply delete - remove object from database ---- More complex example package LinkedList; use Moose; has 'value' => (is => 'ro', required => 1); has 'next' => ( is => 'rw', isa => __PACKAGE__, ); has 'prev' => ( is => 'rw', isa => __PACKAGE__, weak_ref => 1, ); ---- sub append { my ($self, $next) = @_; $self->next( $next ); $next->prev( $self ); return $self; } ---- Even cyclical structures store my $list = LinkedList->new( value => 1 ); $list->append(LinkedList->new( value => 2 )); $kioku->store( list => $list ); undef $list; ---- Cyclical structures lookup $list = $kioku->lookup( 'list' ); say $list->value, $list->next->value, $list->next->prev->value; # 1, 2, 1 ---- Object -> Id say $kioku->object_to_id( $list ); # list say $kioku->object_to_id( $list->next ); # A732B774-E418-11DD-9295-DD8987137C8F ---- Relationships in an OO world Important feature of RDBMSs Easy to do in OO world ---- Relationship example Blog post has author Author has many blog posts ---- package Author; use Moose; use KiokuDB::Util qw(set); has 'name' => ( is => 'ro', isa => 'Str', required => 1, ); has 'posts' => ( is => 'ro', isa => 'KiokuDB::Set', default => sub { set() }, ); ---- package Post; use Moose; has [qw/title content/] => ( is => 'ro', isa => 'Str' ); has 'author' => ( is => 'ro', isa => 'Author', weak_ref => 1, required => 1, ); ---- sub Author::new_post { my ($self, @args) = @_; my $post = Post->new( author => $self, @args ); $self->posts->insert( $post ); return $post; } ---- Using these classes my $jrockway = Author->new( name => 'jrockway' ); $jrockway->new_post( title => 'Hello, world', content => 'OH HAI', ); my $id = $kioku->store( $jrockway ); ---- $jrockway->new_post( title => 'Another post', content => 'This is fun.', ); my $id2 = $kioku->store( $jrockway ); # $id eq $id2 ---- undef $jrockway; # *vanishes in a puff of logic* use YAML::XS qw(Dump); print Dump [$kioku->lookup($id)->posts->members]; ---- Structure retained: - &1 !!perl/hash:Post author: &2 !!perl/hash:Author name: jrockway posts: - *1 - *2 content: OH HAI title: Hello, world - &3 !!perl/hash:Post author: *2 content: This is fun. title: Another post ---- Transactions $kioku->txn_do( sub { my $data = $kioku->lookup( $some_id ); $data->mutate; $kioku->store( $data ); }); ---- Other backends BerkeleyDB (the best) DBI Directory::Transactional CouchDB Amazon SimpleDB ---- Using other backends my $kioku = KiokuDB->new( backend => KiokuDB::Backend::->new( ..., ), ); ---- Simple cases: KiokuDB->connect('hash'); # in-memory KiokuDB->connect('bdb:dir=/tmp/foo', create => 1); KiokuDB->connect('files:dir=/tmp/foo'); KiokuDB->connect('dbi:SQLite:/tmp/foo'); ---- Using KiokuDB in apps ---- KiokuX::Model ---- Create Model package MyApp::Model use Moose; extends 'KiokuX::Model'; ---- Connect my $k = MyApp::Model->new( dsn => 'hash', ); ---- Use my $s = $k->new_scope; $k->insert(Object->new); ---- Gradually add functionality ---- package MyApp::Model; method new_post(AuthorName $author, Str $title){ my $author = $self->lookup("author:$author") || Author->new( name => $author ); my $post = Post->new( title => $title ); $author->add_post($post); $self->store($post); return $post; } ---- my $post = $k->new_post('jrockway', 'Hello'); $post->add_tag('hello'); $k->store($post); ---- Techniques Classes handle relationships Model provides convenience Primary keys ---- What's the object ID? User wants "jrockway" object. How to get it without searching? ---- package Author; use Moose; with 'KiokuDB::Role::ID'; has 'name' ...; sub kiokudb_object_id { return 'author:'. $self->name; } ---- my $j = Author->new( name => 'jrockway' ); $k->store( $j ); # author:jrockway ---- my $j = $k->lookup('author:jrockway'); ---- One other thing I do package Author; sub get_id_for { my ($class, $name) = @_; return "author:$name"; } sub kiokudb_object_id { return $self->get_id_for($self->name); } ---- Then: $k->lookup( Author->get_id_for('jrockway'), ); ---- KiokuDB in apps Build class structure (Test that without DB!) Add KiokuX::Model (Add convenience there, perhaps test with patch) Use! ---- Questions? Ask on #kiokudb ---- Bonus Slides ---- Indexing Explicit indexing via OO structure Implicit indexing via Search::GIN ---- Explicit Indexing You have a set, 'foo'. Create a new object, add to set. Retrieve 'foo'. Retrieve members. ---- Implicit Indexing Search::GIN ---- Implicit Indexing # use GIN backend (DBI also OK) my $kioku = KiokuDB->new( backend => KiokuDB::Backend::BDB::GIN->new( extract => Search::GIN::Extract::Class->new, ... ), ); ---- # add data as normal $kioku->store( Post->new ( ... ), Author->new( ... ), ); ---- # create query my $query = Search::GIN::Query::Class->new( isa => 'Author', ); # get results my $results = $kioku->search($query); ---- Results are Data::Stream::Bulk while( my $chunk = $results->next ){ for my $author (@$chunk){ ... } } ---- Also possible to custom-index my $dir = KiokuDB->new( backend => KiokuDB::Backend::BDB::GIN->new( extract => Search::GIN::Extract::Callback->new( extract => $coderef, ))); ---- my $coderef = sub { my ($object, $extractor, @args) = @_; if($object->isa('Author')){ return +{ name => $object->name, }}}; ---- my $query = Search::GIN::Query::Maual->new( values => { name => 'Jonathan Rockway', }); # execute like before ---- KiokuDB Architecture Layers: Processing Link/Collapse (TypeMap) Live Object Manager Indexer Storage ---- TypeMap Control how non-Moose classes work (DateTime, Path::Class, URI, ...) Moose classes can have roles to control behavior ---- Moose classes Immutable Intrinsic ID::Content Attribute::Lazy ---- Compat with non-Moose classes Typemap Direct serialization code Intrinsic/Non-Intrinsic Connect with: KiokuDB->connect( "bdb:dir=foo", allow_class_builders => 1, ); ---- Tools Command-line utility: "kioku" ---- Dump kioku dump --dsn bdb:dir=/path/to/db --- !!perl/hash:KiokuDB::Entry class: Post data: author: !!perl/hash:KiokuDB::Reference id: 25E94570-F53E-11DD-8BE2-FCCC690E2E7F --- ... ---- Load kioku load --dsn whatever < dump ---- Edit Edit data in your text editor kioku edit --dsn whatever \ --ids 25E94570-F53E-11DD-8BE2-FCCC690E2E7F Wrapped in txn ---- Cleanup kioku gc --dsn whatever kioku fsck --dsn whatever ---- Object Databases Less code Not slow ---- KiokuDB Lots of options DBI (DBAs love it) BerkeleyDB (Scalable) Amazon SimpleDB (Big data center)