diff --git a/pom.xml b/pom.xml index 3abd9e8c6..da0be7d2f 100644 --- a/pom.xml +++ b/pom.xml @@ -170,6 +170,11 @@ 4.4 runtime + javax.media jai-core @@ -183,7 +188,9 @@ ${project.basedir}/src/docbkx/resources/xsl/fopdf.xsl - css/html.css + ${project.basedir}/src/docbkx/resources/css/html.css + ${project.basedir}/src/docbkx/resources/images/ + ${project.basedir}/src/docbkx/resources/images/ false ${project.basedir}/src/docbkx/resources/xsl/html.xsl diff --git a/src/docbkx/index.xml b/src/docbkx/index.xml index fd405e649..3828ec44a 100644 --- a/src/docbkx/index.xml +++ b/src/docbkx/index.xml @@ -37,7 +37,7 @@ - + About this book @@ -69,14 +69,21 @@ - + + - + + + + + + + diff --git a/src/docbkx/reference/samples.xml b/src/docbkx/reference/samples.xml index ee8afc299..6e656c767 100644 --- a/src/docbkx/reference/samples.xml +++ b/src/docbkx/reference/samples.xml @@ -24,7 +24,7 @@ Executing the application creates the following graph in the Graph Database: - + @@ -41,11 +41,15 @@ See the readme file for instruction on how to compile and run the application. An excerpt of the data stored in the Graph Database after executing the application: - - - - - + + + + + + + + + + + + + + + + + + In Java code this looked like. Pretty straightforward. + + + + cast; + } + + class Actor { + int id; + String name; + Set filmography; + Role playedIn(Movie movie, String role); + } + class Role { + Movie movie; + Actor actor; + String role; + } + class User { + String login; + String name; + String password; + Set ratings; + Set friends; + Rating rate(Movie movie, int stars, String comment); + void befriend(User user); + } + class Rating { + User user; + Movie movie; + int stars; + String comment; + } +]]> + + + I wrote some basic tests to assure that the basic plumbing worked. Check. - -
- Role - - ... - -
-
- Actor - - ... - -
-
- User - - ... - -
-
- Rating - - ... - -
diff --git a/src/docbkx/tutorial/import.xml b/src/docbkx/tutorial/import.xml new file mode 100644 index 000000000..6e5ad61fa --- /dev/null +++ b/src/docbkx/tutorial/import.xml @@ -0,0 +1,23 @@ + + + + The dusty archives - Importing Data + + Now it was time to pull the data from themoviedb.org. Registering there and getting an API key was simple, using the API on the commandline with curl too. + Looking at the JSON returned for movies and people I decided to pimp my domain model and add some more fields so that the representation in the UI was worth + the effort. + + For the import process I created a separate importer that used HttpClient and JSON to fetch and parse the data and then some transactional methods to actually + insert it as movies, roles and actors. User data was not available so I created an anonymous user called 'Cineast' that I attributed all the ratings and comments + to. I also created a version of the importer that read the json files from local disk, so that I didn't have to strain the remote API that much and that often. + + + + + + + diff --git a/src/docbkx/tutorial/indexing.xml b/src/docbkx/tutorial/indexing.xml new file mode 100644 index 000000000..540f3e8a0 --- /dev/null +++ b/src/docbkx/tutorial/indexing.xml @@ -0,0 +1,37 @@ + + + + Do I know you? - Indexing + + Then there was an @Indexed annotation for fields. I wanted to try this too. That would guide the next test. I added an @Indexed to the id field of the movie. + This field is intended to represent the external id that will be used in URIs and will stable over database imports and updates. This time I went with the + Finder to retrieve my indexed movie. + + + + movieFinder = finderFactory.createNodeEntityFinder(Movie.class); + Movie retrievedMovie = movieFinder.getByPropertyValue(id); + assertEqual("retrieved movie matches persisted one",forestGump,retrievedMovie); + assertEqual("retrieved movie title matches","Forest Gump",retrievedMovie.getTitle()); + } + +]]> + + + TODO This failed with an exception about not being in a transaction. Oh, I forgot to add the @Transactional. So I added it to the test. + + diff --git a/src/docbkx/tutorial/neo4j.xml b/src/docbkx/tutorial/neo4j.xml new file mode 100644 index 000000000..988817a9b --- /dev/null +++ b/src/docbkx/tutorial/neo4j.xml @@ -0,0 +1,37 @@ + + + + Graphs ahead - Learning Neo4j + + Then came the unknown - how to put these domain objects into the graph. First I read up about graph databases, especially Neo4j. Their datamodel consists + of nodes and relationships all of which can have properties. Relationships as first class citizens - I liked that. Then there was the possibility to index + both by field, value pairs to quickly get hold of them as starting points for further processing. Other useful operations were manual traversal of relationships + and a powerful traversal based on a query like Traversal Description. That all seemed pretty easy. + + I also learned that Neo4j was transactional and provided the known ACID guarantees for my data. This was unsual for a NoSQL database but easier for me to get + my head around than non-transactional eventual persistence. That also meant that I had to manage transactions somehow. Keep that in mind. + + + + diff --git a/src/docbkx/tutorial/recommendations.xml b/src/docbkx/tutorial/recommendations.xml index dd72cf4f4..24eccbf0a 100644 --- a/src/docbkx/tutorial/recommendations.xml +++ b/src/docbkx/tutorial/recommendations.xml @@ -1,8 +1,27 @@ - Adding recommendations + Movies! Friends! Bargains! - Recommendations - + In the last part of this exercise I wanted to add some recommendation algorithms to my app. One was the recommendation of movies that my friends liked very much + (and their friends in descending importance). The second was recommendations for new friends that also liked the movies that I liked most. + + Doing this kind of ranking algorithms is the real fun with graph databases. They are applied to the graph by traversing it in a certain order, collecting information + on the go and deciding which paths to follow and what to include in the results. + + Lets say I'm only interested in the top 10 recommendations each. + + 5) return EXCLUDE_AND_STOP; + Relationship rating = path.lastRelationship(); + if (rating.getType().equals(RATED)) { + rating.getProperty() + return INCLUDE_AND_STOP; + } + return INCLUDE_AND_CONTINUE; + }) +]]> diff --git a/src/docbkx/tutorial/relationships.xml b/src/docbkx/tutorial/relationships.xml new file mode 100644 index 000000000..7b1e9645b --- /dev/null +++ b/src/docbkx/tutorial/relationships.xml @@ -0,0 +1,107 @@ + + + + A convincing act - Relationships +
+ Value in Relationships - Creating them + + Next were relationships. Direct relationships didn't require any annotation. Unfortunately I had none of those. So I went for the Role relationship +between Movie and Actor. It had to be annotated with @RelationshipEntity and the @StartNode and @EndNode had to be marked. +So my Role looked like this: + + + + + When writing a test for that I tried to create the relationship entity with new, but got an exception saying that this was not allowed. Some weird restriction +about having only correctly constructed RelationshipEntities. So I remembered a relateTo method from the list of introduced methods on the NodeEntities. After +quickly checking it turned out to be exactly what I needed. I added the method for connecting movies and actors to the actor - seemed more natural. + + + + +
+
+ Who's there ? - Accessing related entities + + What was left - accessing those relationships. I already had the appropriate fields in both classes. Time to annotate them correctly. For the fields providing +access to the entities on the other side of the relationship this was straightforward. Providing the target type again (thanks to Java's type erasure) and the +relationship type (that I learned from the Neo4j lesson before) there was only the direction left. Which defaults to OUTGOING so only for the movie I had to +specify it. + + + cast; +} + +@NodeEntity +class Actor { + @Indexed + int id; + String name; + @RelatedTo(elementClass = Movie.class, type = "ACTS_IN") + Set cast; + + public Role playedIn(Movie movie, String roleName) { + Role role = relateTo(movie, Role.class, "ACTS_IN"); + role.setRole(roleName); + return role; + } +} + ]]> + +
+
+ May I introduce ? - Accessing Relationships themselves + + While reading about those relationship-sets I learned that they are handled by managed collections of spring data graph. So whenever I add something to the + set or remove it, it automatically reflects that in the underlying relationships. Neat. But this also meant I mustn't initialize the fields. Something I will + certainly forget not to do in the future, so watch out for it. + + I didn't forget to add test for those. So I could assure that the collections worked as advertised (and also ran into the intialization problem above). + + But I still couldn't access the Role relationships. There was more to read about this. For accessing the relationship in between the nodes there was a separate + annotation @RelatedToVia. And I had to declare the field as readonly Iterable<Role>. That should make sure that I never tried to add Roles (which I couldn't create on my own anyway) + to this field. Otherwise the annotation attributes were similar to those used for @RelatedTo. So off I went, creating my first real relationship (just kidding). + + cast; + + @RelatedToVia(elementClass = Role.class, type = "ACTS_IN", direction = Direction.INCOMING) + Iterable roles; + } + ]]> + + After the tests proved that those relationship fields really mirrored the underlying relationships in the graph and instantly reflected additions and removals I was + satisfied with my domain so far and went for some coffee and chocolate. + +
+ diff --git a/src/docbkx/tutorial/repository.xml b/src/docbkx/tutorial/repository.xml new file mode 100644 index 000000000..708fc4bc6 --- /dev/null +++ b/src/docbkx/tutorial/repository.xml @@ -0,0 +1,26 @@ + + + + Serving a good cause - Repository + + That was the first method to add to the repository. So I created a repository for my application, annotated + it with @Repository and @Transactional. + + + movieFinder; + @Autowired + public CineastsRepostory(FinderFactory finderFactory) { + this.finderFactory = finderFactory; + this.movieFinder = finderFactory.createNodeEntityFinder(Movie.class); + } + public Movie getMovie(int id) { + return movieFinder.getById(id); + } +} + ]]> + + diff --git a/src/docbkx/tutorial/running.xml b/src/docbkx/tutorial/running.xml new file mode 100644 index 000000000..e1438db4d --- /dev/null +++ b/src/docbkx/tutorial/running.xml @@ -0,0 +1,55 @@ + + + + Curtains Up! - Get it running +
+ Requisites - Populating the database + + Time to put this on display. But I needed some test data first. So I wrote a small class for populating the database which could be called from my controller. + To make it safe to call it several times I added index lookups to check for existing entries. A simple /populate endpoint for the controller that called it would + be enough for now. + + + + +
+
+ Behind the scenes - Peeking at the Datastore + +
+ Eye candy - Neoclipse visualization + + After filling the database I wanted to see what the graph looked like. So I checked out two tools that are available for inspecting the graph. First Neoclipse, an + eclipse RCP application or plugin that connects to existing graph stores and visualizes their content. After getting an exception about concurrent access, I learned + that I have to use Neoclipse in readonly mode when my webapp had an active connection to the store. Good to know. + + TODO neoclipse image + +
+
+ Hardcore "Hacking" - Neo4j Shell + + Besides my movies and actors connected by ACTS_IN relationships there were some other nodes. The reference node which is kind of a root node in Neo4j and can be used + to anchor subgraphs for easier access. And Spring Data Graph also represented the type hierarchy of my entities in the graph. Obviously for some internal housekeeping + and type checking. + + For us console junkies there is also a shell that can reach into a running neo4j store (if that one was started with enableRemoteShell) or provide readonly access + to a graph store directory. + + + + + It uses some shell metaphors like cd and ls to navigate the graph. There are also more advanced commands like using indexes and traversals. I tried to play around with them + in this shell sesson. + + + +
+
+ diff --git a/src/docbkx/tutorial/security.xml b/src/docbkx/tutorial/security.xml index cef8f2da7..7470f8669 100644 --- a/src/docbkx/tutorial/security.xml +++ b/src/docbkx/tutorial/security.xml @@ -1,8 +1,98 @@ - Adding security + Protecting Assets - Adding Security + To use the user in the webapp I had to put it in the session and add login and registration pages. Of course the pages that only worked with a valid user +account had to be secured as well. + +I used Spring Security to that, writing a simple UserDetailsService that used my repository for looking up the users and validating their credentials. The config is located +in a separate applicationContext-security.xml. + + + + + + + + + + + + + + + + + + + + + + + + ]]> + userFinder; + + @Override + public void afterPropertiesSet() throws Exception { + userFinder = finderFactory.createNodeEntityFinder(User.class); + } + + @Override + public UserDetails loadUserByUsername(String login) throws UsernameNotFoundException, DataAccessException { + final User user = findUser(login); + if (user==null) throw new UsernameNotFoundException("Username not found",login); + return new CineastsUserDetails(user); + } + + public User findUser(String login) { + return userFinder.findByPropertyValue("users","login",login); + } +} + +public class CineastsUserDetails implements UserDetails { + private final User user; + + public CineastsUserDetails(User user) { + this.user = user; + } + + @Override + public Collection getAuthorities() { + User.Roles[] roles = user.getRoles(); + if (roles ==null) return Collections.emptyList(); + return Arrays.asList(roles); + } + + @Override + public String getPassword() { + return user.getPassword(); + } + + @Override + public String getUsername() { + return user.getLogin(); + } + +.... + public User getUser() { + return user; + } +} + + ]]> + + + After that a logged in user was available in the session and could so be used for all the social interactions. Most of the work done next was adding controller methods +and JSPs for the views. diff --git a/src/docbkx/tutorial/setup.xml b/src/docbkx/tutorial/setup.xml index f0fd0b077..6c54f47ff 100644 --- a/src/docbkx/tutorial/setup.xml +++ b/src/docbkx/tutorial/setup.xml @@ -1,44 +1,46 @@ - Project setup + Scope: Spring - ... + Being a Spring Developer, I would, of course, choose components of the Spring Framework to do most of the work. I'd already come up with the ideas - + that should be enough. + + What database would fit both the complex network of cineasts, movies, actors, roles, ratings and friends? And also be able to support the + recommendation algorithms that I thought of? I had no idea. + + But, wait, there was the new Spring Data project that started in 2010 bringing + the convenience of the Spring programming model to NoSQL databases. That should fit my experience and help me getting started. I looked + at the list of projects supporting the different NoSQL databases. Only one mentioned the kind of social network I was thinking of - + Spring Data Graph for Neo4j, a graph database. Neo4j's pitch of "value in relationships" and the accompanying docs looked like what I needed. + I decided to give it a try.
- Spring Data Graph + Preparations - Required Setup - ... + To setup the project I created a public github account and began setting up the infrastructure for a spring web project using maven as build + system. So I added the dependencies for the springframework libraries, put the web.xml for the DispatcherServlet and the applicationContext.xml + in the webapp directory. + + + + + + + With this setup I was ready for the first spike: creating a simple MovieController showing a static view. Check. Next was the setup for Spring Data Graph. + I looked at the README at github and then checked it with the manual. Quite a lot of maven setup for aspectj but otherwise not so much to add. + I added just a few lines to my spring configuration. + + + + + + + I spun up jetty to see if there were any obvious issues with the config. Check. -
- Maven configuration - - ... - -
-
- Spring configuration - - ... - -
-
-
- Spring MVC - - ... - -
- Maven configuration - - ... - -
-
- Spring configuration - - ... - -
diff --git a/src/docbkx/tutorial/social.xml b/src/docbkx/tutorial/social.xml index 2ce723e61..88b696293 100644 --- a/src/docbkx/tutorial/social.xml +++ b/src/docbkx/tutorial/social.xml @@ -1,8 +1,79 @@ - Adding social + Movies 2.0 - Adding social - ... + But this was just a plain old movie database (POMD). My idea of socializing this business was not realized. +
+ See, mom a Cineast! - Users + + So I took the User class that I already coded up before and made it a full fledged Spring Data Graph member. + + + ratings; + + @RelatedTo(elementClass=User.class, type="FRIEND") + Set friends; + + public Rating rate(Movie movie, int stars, String comment) { + return relateTo(movie, Rating.class, "RATED").rate(stars, comment); + } + public void befriend(User user) { + this.friends.add(user); + } +} +class Rating { + @StartNode User user; + @EndNode Movie movie; + int stars; + String comment; + public Rating rate(int stars, String comment) { + this.stars=stars; this.comment = comment; + return this; + } +} + ]]> + + +
+
+ Beware, Critics - Rating + + I also put a ratings field into the movie to be able to show its ratings. And a method to average the stars it got. + + + ratings; + + public int getStars() { + int stars, int count; + for (Rating rating : ratings) { + stars += rating.getStars(); count++; + } + return count == 0 ? 0 : stars / count; + } +} + ]]> + + + Fortunately my tests showed my the division by zero error when calculating the stars for a movie without ratings. I also added a few user and ratings to the +database population code. And three methods to rate movies, lookup users and add friends to the repository. + + + + +
diff --git a/src/docbkx/tutorial/spring-data-graph.xml b/src/docbkx/tutorial/spring-data-graph.xml new file mode 100644 index 000000000..bceab5e2c --- /dev/null +++ b/src/docbkx/tutorial/spring-data-graph.xml @@ -0,0 +1,10 @@ + + + + Conjuring Magic - Spring Data Graph + + But that was the pure graph database. Using this in my domain would pollute my classes with lots of graph database details. I didn't want that. Spring Data Graph + promised to do the heavy lifting for me. So I checked that next. Obviously it heavily depended on aspectj magic. So there would be certain behavour that was + just observable without being visible in my code. But I was going to give it a try. + + diff --git a/src/docbkx/tutorial/user-experience.xml b/src/docbkx/tutorial/user-experience.xml new file mode 100644 index 000000000..dc2b62953 --- /dev/null +++ b/src/docbkx/tutorial/user-experience.xml @@ -0,0 +1,9 @@ + + + + Oh the Glamour - More UI + + TODO screenshots + + + diff --git a/src/docbkx/tutorial/webapp.xml b/src/docbkx/tutorial/webapp.xml new file mode 100644 index 000000000..251e5d904 --- /dev/null +++ b/src/docbkx/tutorial/webapp.xml @@ -0,0 +1,49 @@ + + + + Showing off - Web views + + After I had the means to put some data in the graph database, I also wanted to show it. So adding the controller method to show a single movie with its attributes +and cast in a jsp was straightforward. Actually just using the repository to look the movie up and add it to the model. Then forward to the /movies/show view and voilá. + + TODO screenshot of movie display + + +
+ What was his name? - Searching + + The next thing was to allow users to search for some movies. So I needed some fulltext-search capabilities. As the index provider implementation of Neo4j builds on +lucene I was delighted to see that fulltext indexes are supported out of the box. + + + So I happily annotated the title field of my Movie class with @Index(fulltext=true) and was told with an exception that I have to specify a separate index name for that. +So it became @Indexed(fulltext = true, indexName = "search"). The corresponding finder method is called findAllByQuery. So there was my second repository method for +searching movies. To restrict the size of the returned set I just added a limit for now that cuts the result after that many entries. + + + searchForMovie(String query, int count) { + List movies=new ArrayList(count); + for (Movie movie : movieFinder.findAllByQuery("title", query)) { + movies.add(movie); + if (count-- == 0) break; + } + return movies; +} +]]> + +
+
+ Look what i've found - Listing Results + + + I then used this result in the controller to render a list of movies driven by a search box. The movie properties and the cast was accessed by the getters in the +domain classes. + + + + +
+