Gordon Myers
Articles on Life, Truth, Love, Computers, and Music
A tale of two Sauls
Prior to the election, I heard a number of conservative commentators and religious radio programs comparing candidate Donald Trump to Saul from the New Testament. For those of you not familiar, Saul was an infamous and merciless persecutor of Christians, who went through a dramatic transformation that eventually led to him become perhaps the most ardent supporter of the Christian faith, outside of Christ himself. God completely reformed Saul's character, so much so that he chose to change his name to Paul out of shame over his former self. And St. Paul went on to write nearly half of the books in the New Testament, and established lasting Christian churches where some of the other apostles had failed.
The point of making this comparison was to say that God can use anyone, no matter how morally-questionable they may be, to accomplish greatness. And as Donald Trump has appropriated the word "greatness" for himself, it's not surprising that his supporters eventually drew this comparison. However, as someone who's actually read the Bible, I have to say that this comparison is utter garbage, and anyone who makes it might just have scales on their eyes, blinding them to the truth.
The reasons why this comparison fails are manifold. First, even before Saul of the New Testament went through his remarkable transformation, he was already what you might call a Biblical scholar. Of course the Bible as we think of it hadn't yet been written, but there were the Jewish laws of the Torah, the Psalms, and the other Old Testament stories of King David, Elijah, and all the other prophets - all of which Saul knew like the back of his hand. Saul was a Pharisee, or what they called a "doctor of the law," so he had carefully studied and examined these texts since boyhood. His religion and intellect informed his every decision, and while immoral, nothing he did was irrational. In other words, he had tremendous intellectual prowess, and basically had his PhD in Hebrew and Old Testament Studies. On top of this, he was a powerful and persuasive speaker, motivated by the same youthful zeal that has led revolutions to success. Powerful and persuasive public speaker, historian and scholar, and highly respected religious authority... do any of these sound like monikers you would ascribe to Mr. Trump? The answer is obvious.
But okay, even though their backgrounds are very different, isn't the message of God calling the most unlikely sinner to greatness still a valid argument, with regard to Mr. Trump? There are two reasons why I say, resoundingly: no. First, to witness the reformation and transformation of a sinner into God's chosen requires that the sinner, y'know, actually reforms. Donald Trump has been utterly unrepentant, as a symptom of his own vanity. Baptist Pastor John Piper penned a piece titled How to Live Under an Unqualified President, in which he enumerates a list of reasons why Trump is immoral. (Side note: I was a little annoyed at how Pastor Piper quietly dismissed Hillary Clinton as equally unqualified without any real discussion, because while she definitely had issues with integrity, he is tacitly promoting a very large false equivalence there).
Similarly, Jesuit Priest James Martin has a piece titled I was a stranger and you did not welcome me in which he explains that the policy ideas which Trump has put forth are, in a very real sense, the anti-Christ. And Pope Francis famously declared that Trump's attitude demostrates that he is not a real Christian. Of course none of these criticisms would matter if they meant this was all leading up to a great transformation of character, that will help usher in God's kingdom. But that's not what the Trump supporters on the radio were talking about, and certainly not what Trump is interested in doing. The message of Paul's transformation has somehow been changed from genuine, heartfelt repentance... into brushing aside any lapse in moral judgment, so long as the offender is wearing the right team's colors. They voted for the immoral Saul, with no serious demand to ever see him changed into Paul.
Secondly, and even more importantly, is that Saul of the New Testament was not transformed by a vote of the people; he was transformed by the Holy Spirit. We don't get to dictate how/when the Holy Spirit works, or campaign for God's plan to occur only within our predetermined framework. "His ways are higher than our ways," as Isaiah says. Saul of the New Testament already had the approval of human institutions when he was authorized to round up and slaughter members of the Christian sect. It was not man's will, or the electoral college, that changed him. It was the genuine power of Christ. We cannot be so vain to imagine that a human popularity contest has the same power to compel a real change of heart, least of all from someone who's spent their career craving attention. While I do believe that God is changing and molding the characters of all of us, all the time, He does so on His schedule, not on our election cycle.
So to any of my Christian friends who may have been enticed, during the campaign, by this comparison of New Testament Saul to Trump, I ask you: can you imagine that God could have used someone like Hillary Clinton to do His will? "Have we not all one Father?"
Having categorically brushed aside this comparison between Trump and NT Saul, I will concede that there may still be a valid Biblical comparison to be made. If I had to pick between the two, I'd say Trump is a lot more like Saul of the Old Testament. Old Testament Saul, aka King Saul, is not nearly as well known as New Testament Saul. But his story is fascinating in its own right. It begins by explaining that OT Saul was the son of an incredibly wealthy and influential father - in other words, he came from money and grew up with a silver spoon. The story continues by explaining how he spent his youth essentially chasing tail (or "seeking asses," as the Bible puts it), with no strong sense of purpose in life. Then, through a series of unexpected events, Saul of the Old Testament becomes king of all Israel, to the complete surprise of everyone - including himself! And even after having accepted this prestigious appointment, he shows a reluctance to lead, and instead returns to continue to run his family business, tending to his field.
When King Saul eventually does take up his mantle, he displays utter disregard for even his own most-trusted advisors. And his brutal foreign policy, while at first appearing successful, eventually leads the kingdom into division and disarray, embroiling them in conflict with the Amalekites for generations. Moreover, in the face of an actual terrorist threat, OT Saul proved to be utterly unqualified and helpless. When the Philistines besieged them, Saul could do nothing, and it was his eventual-successor David who had to clean up the mess for him. David became a war hero, and while at first Saul congratulated him for this, he quickly grew jealous of David's renown. He went on to insult, betray, and persecute the war hero unjustly. And famously, King Saul struggled with irrational and erratic outbursts that became increasingly difficult to manage. He refused to let go of perceived slights and pursued his offenders relentlessly, blinding himself to reality.
The story of Saul of the Old Testament quickly starts to resemble watching a train run off the tracks, in slow motion. As his reign continued, Saul became increasingly unhinged. His mind started to unravel. And this happened at the alarm of his closest aides, who proved to be incompentent themselves and unable to deal with their troubled leader. Eventually, Saul did himself in, committing suicide when he was forced to come face-to-face with his own failures. I sincerely hope President Trump is never inspired to commit suicide. But outside of that final point, which of the two comparisons do you think is more fitting?
Am I being a little bit unfair by making this second comparison? Perhaps. (I certainly employed a few clever plays on words.) But just as I believe King Saul was doing his best, but ultimately was not the right man for the job, I can concede that Mr. Trump has what he believes are good intentions. But if the history of King Saul taught us anything, it's that unhinged leaders are unreliable and eventually and inevitably bring about their own demise.
Is God a person?
This is a question that many people have asked throughout the ages, and one that I've heard a lot myself. And as a Christian Scientist, it's sometimes been a point of contention with some of my Christian brothers and sisters, whether I mention anything or not. Recently, after an otherwise-excellent first date ended with the unprompted comment, "I'm sorry Gordon, your views on the Trinity are just too different," it's been something on my mind.
For those of you unfamiliar, the Trinity, or Godhead, is the popular Christian concept that God consists of three-persons-in-one: the Father, the Son, and the Holy Ghost. This same concept is represented more succinctly by the slogan, "Jesus is God." This is a commonly accepted belief in many Christian churches, now considered orthodox, but it was not always as widespread as it is today.
I've been reading a fascinating history book called When Jesus Became God by Richard Rubenstein. The book covers the history of the "Arian Controversy," which includes the famous Council of Nicaea, at the end of the 4th century. This was the point in history when various Christian Bishops came together to flesh out official church doctrine, and this was when they formally consented to the idea that God and Jesus are homoousios, a Greek word borrowed from pagan philosophy, which means "the same stuff." I find it fascinating that any meaningful sense of consensus was largely absent from this council. The Christian community was essentially split down the middle on this idea, and quite a lot of politics, human posturing, and personal amibition went into it. History also shows that this one simple decision quickly reverberated with a lot of unnecessary and ironically un-Christian violence.
Now I don't mean to reignite a centuries-old turf war. But regardless of what church councils "decided," or what's considered popular today, I'd like to examine the question of who God is, and how Jesus fits into that picture, from the perspective of what the Bible actually says and what the teachings of Christian Science reveal. As Isaiah says, Come now, and let us reason together.
Hear, O Israel: The Lord our God is one Lord.
The First Commandment of Judaism and Christianity states that there is one God. What does this mean? Clearly this is opposed to atheism, which says there is no God. And it's similarly opposed to pantheism, which says there are many gods. It's generally accepted, also, that God has the following four properties:
- God is omnipotent, meaning He's all powerful / there's nothing He can't do
- God is omniscient, meaning He knows everything
- God is omnipresent, meaning He fills all space
- God is omnibenevolent, which means He is all good
Those "omni-" words don't appear directly in the Bible, but are they supported by it? Does the Bible imply that God is omnipotent? Well, I can tell you that no less than four times, Jesus said, "all things are possible to God." Check! How about omniscient? In 1st John it says that "God is greater than our heart, and knoweth all things." Check! Omnipresent? Look no further than the poetry of the Psalms: #139 essentially says yes. Lastly, is God good? Well, if he wasn't, it'd be pretty scary praying to Him! But if you're looking for Biblical support, Jesus said, "Why do you call me 'good'? There is none good except God alone."
Now any freshman philosophy major can explain to you that there's no being who could possibly exist in our physical universe that truly has all four of those properties at the same time (and they'd be right), because of something called the problem of evil. But today's post is not to try and dissect that paradox -- that's a whole other can of worms! So for the sake of argument, we'll just take it on faith that God is indeed all-powerful, all-knowing, omnipresent good. Question: can an all-good, all-powerful, all-knowing, omnipresent being be a person?
Again, let me reiterate that lots and lots of people -- many of them smarter than me -- have already thought through this same question over the ages. In fact, one example of someone else who's thought about this question a little bit is Carl Sagan, the famous astrophysicist. Here's a quote from him:
The idea that God is an oversized white male with a flowing beard who sits in the sky and tallies the fall of every sparrow is ludicrous. But if by God one means the set of physical laws that govern the universe, then clearly there is such a God. This God is emotionally unsatisfying. It does not make much sense to pray to the law of gravity.
--Carl Sagan
We seem to be at a crossroads.
On the one hand, the orthodox Christian belief of Jesus as God provides a lot of comfort. To think that there's an infinitely powerful personal friend, ready to help you out of trouble and save you from despair in this physical life, is understandably appealing. But under the microscope of rational thought, inconsistencies start to appear. And if there's some additional qualifications that come with this (i.e. do/say the right thing or you're damned), one starts to question the "omnibenevolent" part. Dig deep enough and things might even seem "ludicrous," as Mr. Sagan put it.
On the other hand, what if God is simply the name for all the principles and laws governing the physical universe? Well then we can see that these are ever-present, and all-powerful by definition, but it just seems like such a cop-out. Those laws don't seem very intelligent, and certainly not very good. Instead it all seems chaotic, unknowable, and feels so cold. It certainly doesn't meet our criteria of the four "omni"s.
But what if I told you that there was a simple mistake with both of these approaches, and indeed, there's a third answer? On the one hand, if you start by saying, "I see, feel, hear, etc. with my material senses, and my material body is who I am," and then you remember reading in the Bible that "God created mankind in His image and likeness," you might naturally think, "well that means God must be just like me -- a material body. And hey, Jesus had one of those! Therefore Jesus must be God." Or on the other hand, if you start by saying, "I see, feel, hear, etc. with my material senses, and my material body is who I am," and then you study the scientific method, you might say, "everything is made out of matter just like me, and there certainly seem to be patterns to what I can observe in the physical world, maybe these patterns are God."
Both of these approaches reason out from the evidence of the material senses. Both start with the idea that I am material, therefore practically everything is material. You try to make God in the image and likeness of yourself. Matter is what we observe with our senses, but the Bible says God is Spirit, and so I think we need to start there. But that's hard, because spiritual things are un-quantifiable. What's the average mass of a mother's love? What's the velocity of it? Its existence is obvious, yet it extends far beyond chemicals and neurons. It can be difficult for people, at first, to reason about things spiritually, because we're not always used to it. But if you don't immediately accept the idea that you are made out of matter, and consider for a moment that instead you are spiritual, then the reasoning goes a little differently. Here's what Mary Baker Eddy, the Founder of Christian Science, had to say:
Human philosophy has made God manlike. Christian Science makes man Godlike. The first is error; the latter is truth. Metaphysics is above physics, and matter does not into metaphysical premises or conclusions.
Christian Science strongly emphasizes the thought that God is not corporeal, but incorporeal -- that is, bodiless. Mortals are corporeal, but God is incorporeal.
As the words person and personal are commonly and ignorantly employed, they often lead, when applied to Deity, to confused and erroneous conceptions of divinity and its dinstinctions from humanity. If the term personality, as applied to God, means infinite personality, then God is infinite Person, -- in the sense of inifinite personality, but not in the lower sense. An infinite Mind in a finite form is an absolute impossibility.
These are just some of the ideas that I'm going to be sharing during a short church meeting this Wednesday, August 24th, here in Madison. I'll be presenting further readings from the Bible as well as more of Eddy's exposition on it, on this same topic, "Is God a person?" If you're interested in finding out a little more, please join me at 7:30 this Wednesday evening.
My Pokemon Go Wishlist
Like many, I've been sucked into the smartphone craze that is Pokémon Go. The nostalgia of reliving a game many of us played 20 years ago, combined with the pride of collecting as many little critters as you can, combined with the inspiration of wandering around your city and discovering hidden gems makes for a potent formula for success. Still, despite the game's immense and immediate popularity, I can't help but noticing that it's not exactly feature-rich. It's definitely an underdeveloped app, which isn't helped by the server-side struggles they've had to keep up with demand. Today I will present my wishlist of the top 3 features I'd love to see rolled out in future iterations of Pokémon Go.
1. Attacking before Capturing
One of the fundamental game mechanics that you learn early on in every other Pokémon game (and there are quite a lot!) is the importance of weakening new prospects by attacking them first before trying to capture them in poké balls. That mechanic is sadly absent from Pokémon Go. It's somewhat made up for with the use of Razzberries, but only somewhat as they often prove ineffective. I'd love to see future iterations of the app allow your existing critters to attack the wild Pokémon before attempting to throw a poké ball at them, with some sort of HP gauge on screen. That would be more true to the original. They could even introduce different consequences depending on the (predetermined) disposition of the Pokémon: perhaps certain animals would stay and fight while others might immediately try to flee when attacked. There are lots of possibilities, but right now it feels like a big missed opportunity.
2. Leveling up at gyms
In the real world, when you go to the gym to workout, you leave feeling stronger. In Pokémon Go, gyms don't exactly work that way. Presently the only way to strengthen your Pokémon is outside of the gym, by feeding them candy. This message seems incongruent with reality. I think the longer a Pokémon remains in a gym the stronger it should be. Or at the least, upon every successful defense of a gym, it should gain some additional CP. Right now gyms are little more than spectacles of pride, but the game would be much improved if they were more, y'know, gym-like.
3. Logging in
One feature that they should consider including in future iterations of Pokémon Go is the ability to log in. I really feel it was a big miss on Niantic's part by not including this feature right off the bat. It can be very counterproductive when you want to play Pokémon Go but you're unsuccessful logging in and thereby are prevented from playing Pokémon Go. It's really a strange game mechanic in which you're not allowed to play the game in the first place, and seems like a pretty glaring oversight on part of the developers. Hopefully, in future releases of the app, there will actually be a functioning app to play. That would seem like a great feature to include.
So that's my wishlist for Pokémon Go. What do you think? Do you have ideas of your own for features you'd like to see? Leave a comment below!
It's the Star Wars movie we need right now, but not the Star Wars movie we deserve
This morning I saw Star Wars: The Force Awakens in theaters. It was everything everyone expected. It lived up to the hype. However I'm not writing this post to join the chorus of accolades; there will be enough of that and indeed there already is. This post assumes you enjoyed the film and are ready to move on to a healthy level of scrutiny. So keep in mind that as I talk about the flaws of the film, I do so earnestly from the perspective of someone who genuinely enjoyed the movie and will probably be back to the theaters to see it again. With that said, there were a couple of things about this film that bothered me.
Star Wars is something that a lot of people have very strong and passionate feelings about. For years it was carefully controlled and guided by George Lucas, another thing altogether that people have strong and passionate feelings about. Since Lucas sold the rights away and the mantle was picked up by J.J. Abrams, it's fair to say that people were nervous he might add too much lens flare, or otherwise not get things right. I'm simultaneously pleased and conflicted to say that J.J. did get it right. Conflicted only because he didn't actually make a new movie at all. The Force Awakens is a play-by-play, carbon copy of A New Hope, down to the smallest detail. Some have even gone so far as to call it plagiarism. I simply call it "playing it safe."
My roommate, who has not yet seen the film, just asked me as I was writing this if I thought the new film was better than Episode Four. That's a question I can't really answer, because it is Episode Four. This film had about as much new material as The Hangover 2 did. It's an exact replica of the original film, just with younger characters and bigger objects.
But that isn't really what bothers me. The new film was exactly what the masses needed most after the disaster that was the prequel trilogy. They had felt betrayed by their beloved director. How could the same man who gave the world Luke Skywalker, Han Solo, and Darth Vader be the one who made Anakin Skywalker, Jar Jar Binks, and midi-chlorians? Alas, as the years went by, Lucas spiraled further into delusion and denial like the hapless CG-junkie that's he's revealed himself to be. And the fans felt the sting of betrayal. The galaxy far, far away had been forever tainted. A Force Awakens calls back to the earlier days of greatness. It pays homage to its roots because it is its roots, with little more than a new face.
However, just as Kylo Ren couldn't shake his Dark Side heritage, the film couldn't shake some of the series' darker memories itself. I'm talking about Hayden Christensen's acting, newly manifest in Daisy Ridley. I know that people are going to praise her performance, and while I do believe she is leagues apart from Mr. Christensen, I still saw sparks of the dark side in her. And I don't mean the force.
In fact, I'm sure that both she and Abrams will be praised: Ridley for her acting, and J.J. for including a strong female lead. I don't buy it though. The first felt forced at times and the latter felt like affirmative action. If feminism in the 21st century has taught us anything, it's that a surefire formula for accolades in the film industry is to have a fiercly independent female lead. People think it's refreshing to have a damsel who's not in distress, who doesn't need a man to save her, and who proclaims her independence at every turn. I don't have a problem with a strong female lead, even though I think it's already become an overplayed trope years ago. But I do have a problem when the only motivation for that choice is pandering, and when the so-called strength of that lead wanders into overcompensation territory, as if it's apologizing for years of other films.
Case in point: Rey's insistence that she doesn't need Fin to hold her hand on Jakku. It was meant to highlight her independence, but it felt robotic, unrealistic, and needlessly rude. That's not how people interact in the real world. And her change of heart toward Fin, later in the film, comes too fast. It makes the character seem flaky and inconsistent. And when Fin is trying to do all the things a conventional male hero is supposed to do, she loves him for it. Now, granted, an argument could be made that Abrams was trying to reincarnate the same cocky, push-and-pull chemistry that Leia and Han perfected years ago into a new generation. But if the prequel trilogy taught us anything, it's that you can't force romance. No matter how many times Anakin said he loved Padmé, we never believed it, because tell-not-show storytelling is never believable.
There are other examples, too, that are a little more glaring, other glimpses where you see her channeling her inner Christensen. Watch Rey's expression as she becries Solo's death. The scene is parallel to when Luke is upset over Obi-wan's death, with only one little thing different: she's only known Han for a matter of hours, whereas Uncle Ben was family to Luke. Yes it sucks, but she's clearly over-acting.
But all of these minor quibbles are still dwarfed by my real complaint with the new film. My main complaint is not really about the film, not anything in it. I had this complaint long before ever seeing the film. My major gripe is about the fact that film exists in the first place. Because its mere existence invalidates the Prophecy of the Chosen One. No, invalidate is too soft a term. It utterly and completely destroys it. In layman's terms, it removes the need for any of the last six films and renders them all pointless.
The Prophecy of the Chosen One was an ancient Jedi prophecy that foretold the coming of a being who would forever bring balance to the force. This being was revealed (and confirmed) to be Anakin Skywalker, who "forever" destroyed the Sith and eliminated the influence of the Dark Side when he took out the evil Emperor Palpatine and sacrified himself as well. This prophecy was the entire point of the original three films, and further reinforced by their prequels.
The trouble is that if evil has been destroyed for good, and the prophey has been fulfilled, that doesn't leave room for any future villains or really any room for future films or literature. Peace is boring. Peace isn't dramatic. Star Wars is popular because the wars make things interesting. I'm ignoring the hoardes of expanded universe novels when I say that Return of the Jedi was meant to be the end. But I have yet to hear any satisfactory explanation as to why the Prophecy of the Chosen One isn't proved to be complete garbage by the re-introduction of the dark side after the fact.
Just to wade through the pedantry a bit, no, the phrase "bring balance to the force" does not simply mean to equalize the number of Jedi with the number of Sith. Lucas himself has confirmed that it meant the elimination of the Sith, and the dialogue in the prequel films supports this. The Jedi believe that the only truly "balanced" state of the Force is when the Dark Side is totally absent. The other main argument, or should I say rationalization, is that balance is inherently temporary. Well, what was the point, then? Why bother prophesying anything if it's all going to be meaningless 30 years later?
All these retconned delusions serve to do is spit in the face of the original story. The Force Awakens is more than just an innocuous reboot -- and it is a reboot in the truest sense of the word, not a sequel. With all its careful orchestration, tribute, and nostalgia, it repents of the the prequel trilogy, but only by selling the soul of the beloved original trilogy.
It had to do that, and I understand why. Although the heritage of the Jedi, explored thorougly in the Knights of the Old Republic, is rich and lasting, it isn't as marketable as Luke Skywalker or Han Solo. It had to be a sequel, not another prequel. What people needed was redemption from the Hollywood disasters that were the prequel films. And that's exactly what they got. But, unfortunately, underneath it all, no one seemed to notice that this redemption came with a price. The original trilogy was now all in vain, as we start over with fresh faces.
And the reason that's a problem is because, at its core, the original story (prequels included!) was good. If you don't believe me, watch any of the What if Star Wars Episode X was good? videos on YouTube. The heart of the story was always pure, even if it did end up with a fat body and an ugly face. And although this new story is great (how can it not be when it's a copy?), I am disappointed that The Force Awakens threw the baby out with the bathwater. The good story-telling that Lucas intended was the Star Wars that we deserved. But Star Wars: A New Hope, Mark 2 was what the people needed.
My Mario Maker Wishlist
I bought my copy of Mario Maker bundled with my brand new Wii U console the day the former came out. I have always been a fan of Mario, but an even bigger fan of puzzle games. And the fact that Mario Maker was a way of blending those two concepts made it too hard to resist. And there are some wonderfully creative levels out there. With that said, although Mario Maker gives level designers a fairly broad palette, overall the game still seems a little premature to me. So without further ado, I present my wishlist of the top three things I'd like to see added to Mario Maker.
#1. Better Level Searching
The interface for finding levels seems like little more than an after-thought in the mind of the developers. Understanding how looming deadlines and high demands for features probably pushed the level choosing interface to the back-burner, I can sympathize. But now that the game's out, I'd really like to see (preferably in the form of a free update) some kind of revision that will make searching for courses and course creators easier. This doesn't seem like a tall-order to me. Really, we just need to two things:
- Search for courses by name
- Search for authors by name
That would make a world of difference. Right now you need to know the obscure 16-digit course IDs in order to find anything, which means you really have to want it. Nintendo, please make this easier for all of us.
#2. Conditional Power-up Blocks
I'm really surprised that Nintendo chose to do power-ups the way that they did with Mario Maker, because it broke all past conventions. In every Mario game ever, when Mario's small, a power-up block would yield a mushroom. Once he's big, it might yield a different power-up like a fire flower or a feather. Or it might not. But in any case, being big was a prerequisite to getting the more "advanced" power-up. I'd like to see this element brought back because I think it adds some challenge to the game. Instant fire flowers cheapens things a bit.
#3. Vertical Levels
Nintendo gives you quite a lot of space to work with horizontally, but only two screens maximum of vertical space. I really wish they had included some sort of toggle for horizontal and vertical mode in much the same way that Microsoft Word lets you choose between portrait and landscape mode. Some folks have already tried to emulate this by using doors and warp pipes to make tessellated faux-vertical levels, but it's really not the same. Think auto-scrolling. How great would it be to have a level that makes you climb with the risk of being swallowed up by the auto-scrolling bottom?
Of course this raises a different concern, namely that people would want vertical levels that you could descend. And if you have two possible positions for the start and end, doesn't that mean you'd also need to allow for horizontal levels that move left? Well, yes! I think that would be a delightful side effect. Especially since some folks are already doing this in their sub-levels anyway.
What do you think about my wishlist? What things you would add? I purposely left off a lot of items that are, well, items, because I wouldn't be surprised to see some of that coming in the form of DLC. But let me know in the comments what you think.
Spirited Away
Today's blog post is going to be a movie review. This isn't a current movie by any means, either. In fact, it's 14 years old. It's also a film that I would expect many of you have already seen. But recently I felt an urge to rewatch it, and discovered a few things in the process. First, I realized that there are still many who haven't seen this film, or haven't even heard of it, and that's a shame. But more importantly, I realized some of the great lessons this film has to offer. I am talking of course about the film Spirited Away by Hayao Miyazaki.
For those not familiar, Miyazaki is a Japanese animator who makes kids' movies. And his films are kind of the gold standard in Japan. They're basically Japan's equivalent of Disney movies. In fact, if I'm not mistaken, Disney actually bought the English version rights to all of his films. Spirited Away in particular is one that I consider to be the crème de la crème of his work, and I'm not alone. It is the most successful Japanese film to date and won an Academy Award.
The story of the film is fundamentally one about a young girl overcoming her own fears and limitations in order to free her parents from the spell of an evil witch. It's also a story about friendship and love. And it's a story that draws heavily on Japanese mythology, featuring dragons, witches, talking frogs, giant babies, and everything from river to radish spirits. The artwork is spectacular, the story is unique, and the messages of the film are splendid. But as I was rewatching this film the other day for the upteenth time, one point in particular struck me as to what makes this film so incredible: there are no villains.
Right away people who have seen the film may dispute this fact. They'll remind me that the evil witch, Yubaba, is the story's central antagonist. And to that I'd say: no, she isn't really. Not really, anyway. At its core, the world of Spirited Away doesn't really have any villains. For me, that's what makes it truly so remarkable. But I need to explain and defend my position of why there really aren't any villains in this film. Please note that spoilers will follow, but this film is good enough that they won't detract at all from watching the film, so please keep reading.
I'll start by debunking the example already named: Yubaba is not a villain. What is it that makes her evil? Well, within the first 10 minutes of the film, she turns the main character's parents into giant pigs. Except... she doesn't actually do that; the parents do that to themselves. They overzealously decide to help themselves to plates full of food left out at what appears to be an abandoned theme park, but is in fact actually an enchanted spirit village. Yubaba isn't present or even aware of the humans at that point; the food is simply not meant for humans with the unfortunate side effects is that it automatically turns non-spirit diners into pigs. Yubaba didn't take any action at all here; it was the parents' own greed that resulted in the curse.
But Yubaba does decide to keep the parents as pigs so that she can raise them for slaughter. And this is the central plot point which the main character, Chihiro, is working to prevent. Yes, this seems evil, but I'm hesitant to really label it as being downright evil, and instead call it a callous business practice, not unlike many of the seemingly callous business practices that we see in the real world. It's not a malovelent, calculated plan; it's simply a matter of convenience. In the eyes of Yubaba, human beings are an entirely different species. They are like animals to her. And what meat company wouldn't freely take advantage of unclaimed livestock that wandered onto their premises? It's not always pleasant to talk about it, but it doesn't make her a fundamentally evil person.
So rather than examining Yubaba's inactions, let's look at her actions. Within the first 30 minutes of the movie, she hires Chihiro to work at the bath house. Chihiro asks her for a job, and she agrees to give her one. In fact, Yubaba has sworn an oath that if anyone, regardless of skill, circumstance, or ability, asks her for a job, she will always grant their request. And not merely on a probationary or provisional basis either; she offers permanent employment to anyone who asks. Does that sound evil to you? You'd be seriously hard-pressed to find that same kind of generosity from even some the best companies!
During Chihiro's employment, at several points, Yubaba compliments Chihiro when she does her job well. Moreover, Yubaba offers a path for Chihiro to follow that will ensure the safe release of her parents. And she keeps her promise. Again, what's evil about any of that? By the end of the film, Chihiro embraces her employer with a hug of gratitude, sending home the message that there aren't villains.
Another character to consider is a spirit called "No Face." Initially a meek, speechless spirit that is denied entrance to the bath house in which all the other spirits partake, this character takes notice of Chihiro and follows her in through a back door. It's shown just how lonely this character is, and how it will do anything to appease others and win their praise and affection. Ultimately it's revealed that this character's people-pleasing tactics are fundementally selfish, as it only acts out the behaviors it thinks others are desiring in order to fill a void. And when it still doesn't feel satisfied, it starts eating the other characters alive, which in turn triggers an increasingly insatiable hunger.
In short, No Face is a character that represents loneliness and lust. It starts out innocent, but due to its hunger for affection, it grows into a disgusting monster that consumes everything and everyone in its path. It does so by preying on the weaknesses of others, but only through illusion and manipulation. In fact, at the zenith of its lust, there is a scene where Chihiro confronts No Face and it is revealed just how desperate the creature is for Chihiro's validation. After having assaulted and consumed half the staff at the bath house, he nearly does the same to Chihiro. That's pretty evil, right?
But let's examine Chihiro's response to this situation: she sits patiently, quietly in front of him, completely unafraid of his condition. She has absolutely no fear. While every other character on screen is either frantically running away or trying to lock the doors, she enters the same room and calmly sits down beside him to chat. When he charges at her signaling he might try to eat her, her first instinct is to help him. She says, "before you eat me, eat this," and then hands him some magical medicine. The pill then triggers a violent reaction in which everything bad in his system is flushed out.
During this reaction, she runs away from him, which makes for a great chase scene. But as soon as that is over, No Face returns to his original form, and starts to follow Chihiro once more. At this point, she stops running away and invites him to sit next to her on the train as a friend. One of the supporting characters is initially shocked that she would let him anywhere near her, but Chihiro responds simply by saying, "I think the bath house makes him crazy. He needed to get out of there."
This, to me, is actually the most telling line of the whole film. This confirms why the message is so powerful, that there are no villains. Even after having been nearly assaulted by this character, she calmly recognizes that it is not the character, but the circumstance, that was the real problem. Chihiro not only helps him out of the unfortunate circumstance, but immediately befriends him and helps draw the best out of him, later helping him to get a job as a personal attendant of another character.
There are more examples, too. Zeniba could be seen as villanious, at least temporarily, for having nearly killed one of the other characters, Haku. But she quickly becomes a fast friend and supporter of both Chihiro and Haku. The giant baby could be seen as villanious, but after being forced out of his spoiled environment, he becomes an ardent defender of Chihiro as well. Even many of the spirit workers could be seen as evil, or at least as dangerous, for their xenophobia of humans, especially at the beginning of the film. But that too is overcome as they work beside Chihiro and begin to appreciate her.
The only villains in Spirited Away are intangible. Qualities like greed, lust, hatred, fear, envy, and theft are the villains of the film. These qualities are acted out, for a time, by various characters. Even Chihiro herself shows a lot of fear at the start of the film. But all of these qualities are overcome, either through the characters' individual growth, or through the help and support of friends. This is a truly powerful message. There are no evil characters; everyone can be redeemed.
Contrast this with even some of the most treasured Disney films. In Lion King there are clearly good characters and bad characters. There is no redemption for Scar. At the end of the film, he is eaten alive by his own hyenas. In Aladdin, there is no hope for Jafar. He is forever banished to a tiny prison. This is actually a pretty common message with most Western Disney movies: there are good people, and bad people, and the bad people should either be killed or exiled.
In Spirited Away, there are just people. (Well, sort of. If you count frogs and radishes as people.) These people sometimes act out bad qualities, and sometimes it can seem pretty scary, but even this can be redeemed by not giving into fear and persistently loving and supporting them. In the world of Spirited Away, gratitude is given even to those who were once seen as enemies. There are no villains. I think we can learn a lot from messages like these. And it sure would do some good to have more movies like that.
Purpose: Letting your brilliance shine through
Today I watched an hour-long video on YouTube. I know this seems like an impossible feat since the attention span of most of the YouTube audience (myself not excluded) is like that of a goldfish. But I was feeling hungry for something substantive tonight, and I don't just mean long.
For a few years now, there have been full, hour-long Christian Science lectures posted on YouTube. If you've never seen one of these before, I really recommend them. They cover a variety of topics and don't let the word "lecture" deter you or imply that they're dry. These are kind of like TED talks, but better. And there are some shorter ones posted, too, to give a sampling of what these are like without having to commit a full hour.
Tonight I watched one titled Purpose: Letting your brilliance shine through by Tom McElroy. The great thing about Christian Science lectures though (in my opinion) is that regardless of what the topic purports to be about, there is something there for you. There is some insight that only you will uniquely glean from listening to the lecture. For instance I wasn't particularly focused on finding my own sense of "purpose" this evening. I actually was stood up for a date so my focus was very different. Even so, I found some valuable, comforting ideas that spoke to me.
Most of all, I think Tom in particular gives a great introduction to Christian Science if you don't know much about it. So if you can spare an hour, I'd really recommend watcing this video. At least add it to your "Watch Later" list. Here's a short excerpt from the video:
To say that there's a science to it [to call it a Christian Science] is to say that if there was ever any truth in that love [if there was ever anything real, anything that actually happened there, anything that ever really took place in the healing, in the love, in the vision; if that was based on anything,] then whatever the truth was behind it [the science of it, the reality of it, the truth of it] still has to be true today. It's timeless. It has to be true for all people under all circumstances. [It has to apply to all people equally.] It's not something we earn or work our way up to and it's not something we can mess up.
Check it out for yourself! I promise you you'll learn something new.
Hey, I made a thing
I just published my very first Chrome Extension to the Google Play Store! It's actually not hard at all, and I'm going to come back here and beef up this post with more of the details when I have some more time. But in a nutshell, if you use Atlassian's Bitbucket service, you might be familiar with "pull requests," otherwise known as code reviews. These are immensely helpful, but sorely lacking one critical feature: there's no way to easily expand/collapse whole sections of a pull request. So, I added that feature! But I did so in a really lazy way. Rather than taking the time to figure out how to hook into the loaded events, I just a gigantic button on top that says "Active Toggle." Click that button once all the files have been loaded up, and it will then add individual "Toggle" buttons to each section. Enjoy!
https://chrome.google.com/webstore/detail/bitbucket-pull-request-to/hfebajohpclnfhfnlhgndbmcdnlchjjd
How to do Joins in MongoDB
If you've come here looking how to perform a JOIN operation on two or more collections in MongoDB, you've come to the wrong place. In fact, that is exactly what I'm about to show you how to do, but trust me, you shouldn't be doing it. If you're trying to do that, that's a clear indication that you have relational data. And if you have relational data, that means you've made the wrong choice for your database. Don't use Mongo; use a relational database instead. I know it's cool and shiny and new, but that is not a good rationale to use it. SQL is your best bet. I'd recommend you read Why You Should Never Use MongoDB by Sarah Mei. Now, with that disclaimer out of the way, I'll dive right into it.
We've been using MongoDB for some of our data storage at work. The choice to use Mongo was a decision made well before I arrived, and incidentally, we might be changing that out at some point in the future. But nevertheless, as it's the system currently in place, I've had to work within that system.
There were a number of pre-established Mongo queries in the codebase I've been working on, and I'm sorry to say many of them were really quite slow. So over the course of a weekend, I tried out a couple of ideas that seemed intuitive enough and managed to speed up some of these common queries by an order of magnitude. The queries I'm talking about grabbed data from multiple collections simultaneously, hence why they were initially so slow, and hence the title of this blog post. In this post I'm going to dissect a number of the techniques I used to speed up these Mongo queries, with plenty of code examples along the way.
Let's say you've got a collection in Mongo called Transactions. This table has a variety of fields on each row, including one field called userId, which is just the ObjectID (aka foreign key, for you SQL folks) of a document in the separate Users collection. You might want to retrieve a list of transactions in a given time period, and show some information on the screen, like the date, the total amount, and the first and last name of that user. But for this first part, let's hold off on any attempts at JOINs, and just look at accessing the Transactions collection alone.
I ran some benchmarks with the following code on my local machine, which was also running a local instance of MongoDB.
MongoClient conn = new MongoClient(new ServerAddress("127.0.0.1", 27017));
DB db = conn.getDB("MyDatabase");
DBCollection transactions = db.getCollection("Transactions");
SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);
Cursor cursor = transactions.find(match);
List<Map> rows = new ArrayList<>();
while ( cursor.hasNext() ) {
Map<String, Object> row = new HashMap<>();
DBObject dbObject = cursor.next();
row.put("total", dbObject.get("total"));
row.put("approved", dbObject.get("approved"));
row.put("canceled", dbObject.get("total"));
row.put("location", dbObject.get("canceled"));
row.put("items", dbObject.get("items"));
row.put("coupons", dbObject.get("coupons"));
row.put("created", dbObject.get("created"));
row.put("updated", dbObject.get("updated"));
row.put("deleted", dbObject.get("deleted"));
row.put("userId", dbObject.get("userId"));
rows.add(row);
}$conn = new Mongo('mongodb://localhost:27017');
$db = $conn->selectDB('MyDatabase');
$transactions = $db->selectCollection('Transactions');
$match = array(
'created' => array('$gt' =>
new MongoDate(strtotime('2014-06-01'))
));
$cursor = $transactions->find($match);
$rows = array();
while ( $cursor->hasNext() ) {
$dbObject = $cursor->getNext();
$rows[] = array(
'total' => $dbObject['total'],
'approved' => $dbObject['approved'],
'canceled' => $dbObject['canceled'],
'location' => $dbObject['location'],
'items' => $dbObject['items'],
'coupons' => $dbObject['coupons'],
'created' => $dbObject['created'],
'updated' => $dbObject['updated'],
'deleted' => $dbObject['deleted'],
'userId' => $dbObject['userId']
);
}This code is obviously sanitized a bit here to highlight what I'm doing, but you might extend this to do any number of things. You might have some sort of POJO that corresponds to a single document in the collection, and instantiate a new one within each loop. Then you would call dbObject.get() to retrieve each of the properties of that row. I iterated this simple test hundreds of times on my local machine, and found, on average, it took 0.663 seconds to complete. And in case you're curious, the date range I've given here corresponds to roughly 30,000 documents. So that's not so bad.
But this pared-down example was not my use case, and my use case was performing poorly. So I thought, well, I don't need all those pieces of data in the dbObject. I really only needed two. So I formulated a hypothesis. My hypothesis was simple: if I only grab the data I need, and none of the data I don't, the query would perform better. This is akin to avoiding SELECT * in SQL (which is something you should always avoid). So to start, I made the most basic modification to the code possible, which you can see here:
SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);
Cursor cursor = transactions.find(match);
List<Map> rows = new ArrayList<>();
while ( cursor.hasNext() ) {
Map<String, Object> row = new HashMap<>();
DBObject dbObject = cursor.next();
row.put("created", dbObject.get("created"));
row.put("total", dbObject.get("total"));
row.put("userId", dbObject.get("userId"));
rows.add(row);
}$match = array(
'created' => array('$gt' =>
new MongoDate(strtotime('2014-06-01'))
));
$cursor = $transactions->find($match);
$rows = array();
while ( $cursor->hasNext() ) {
$dbObject = $cursor->getNext();
$rows[] = array(
'created' => $dbObject['created'],
'total' => $dbObject['total'],
'userId' => $dbObject['userId']
);
}All I've done here is remove the .get() call on the properties I didn't need. In fact, at this point, the database driver is still returning all of those properties; I'm just not accessing them. I wanted to see if that alone would make any difference. And in fact, it did. Hundreds of iterations of this code averaged in at 0.405 seconds. That's a 63% speed improvement. Of course the percentage makes it seem more grandiose than it really is, since that's only a 0.25 second improvement, which is not that big of a gain. But it is still an improvement, and it was consistent. Accessing fewer properties from the cursor results in a speed improvement. But while this sped things up a tiny bit, I knew that we could do better by forcing the database driver to stop returning the extraneous data, a la a project clause:
SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);
DBObject project = new BasicDBObject("total", true);
project.put("userId", true);
project.put("created", true);
Cursor cursor = transactions.find(match, project);
List<Map> rows = new ArrayList<>();
while ( cursor.hasNext() ) {
Map<String, Object> row = new HashMap<>();
DBObject dbObject = cursor.next();
row.put("created", dbObject.get("created"));
row.put("total", dbObject.get("total"));
row.put("userId", dbObject.get("userId"));
rows.add(row);
}$match = array(
'created' => array('$gt' =>
new MongoDate(strtotime('2014-06-01'))
));
$project = array(
'created' => true,
'total' => true,
'userId' => true
);
$cursor = $transactions->find($match, $project);
$rows = array();
while ( $cursor->hasNext() ) {
$dbObject = $cursor->getNext();
$rows[] = array(
'created' => $dbObject['created'],
'total' => $dbObject['total'],
'userId' => $dbObject['userId']
);
}This isn't much different than the last example. I'm still selecting only the data points I care about, but now I've added a project clause to the find() call. This means the database driver is no longer returning all the extraneous properties in the first place. The results? On average, this call took 0.029 seconds. That's a 2,186% speed increase over our original query. And that is worth paying attention to. While my last example wasn't all that telling, this one, on the other hand, confirms my hypothesis. If you only select the data you need, and none of the data you don't need, your queries will perform better. (This is true on any database platform.) The consequence of this is that you can't really use a general-purpose POJO for your collection -- not if you want your code to perform well, that is. Instead, you might have any number of contextual POJOs that access different parts of the same collection. It's a trade-off that may prove worth it for the sheer speed.
And I had one more test, just because I was curious. Up until now I've been using the find() command to grab my data, but Mongo also has another way of retrieving data: the aggregate pipeline. I remember reading somewhere that the AP actually spun up multiple threads, whereas a simple find() call was restricted to one. (Don't ask me for a source on that, I'm vaguely remembering heresay.) So I wanted to see if simply switching out those method calls would have any added bonus. Here's what I tried:
SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);
DBObject project = new BasicDBObject("total", true);
project.put("userId", true);
project.put("created", true);
AggregationOptions aggregationOptions = AggregationOptions.builder()
.batchSize(100)
.outputMode(AggregationOptions.OutputMode.CURSOR)
.allowDiskUse(true)
.build();
List<DBObject> pipeline = Arrays.asList(match, project);
Cursor cursor = transactions.aggregate(pipeline, aggregationOptions);
List<Map> rows = new ArrayList<>();
while ( cursor.hasNext() ) {
Map<String, Object> row = new HashMap<>();
DBObject dbObject = cursor.next();
row.put("created", dbObject.get("created"));
row.put("total", dbObject.get("total"));
row.put("userId", dbObject.get("userId"));
rows.add(row);
}$match = array(
'created' => array('$gt' =>
new MongoDate(strtotime('2014-06-01'))
));
$project = array(
'created' => true,
'total' => true,
'userId' => true
);
$pipeline = array(array('$match' => $match), array('$project' => $project));
$cursor = $transactions->aggregateCursor($pipeline);
$rows = array();
while ( $cursor->hasNext() ) {
$dbObject = $cursor->getNext();
$rows[] = array(
'created' => $dbObject['created'],
'total' => $dbObject['total'],
'userId' => $dbObject['userId']
);
}That test ran, on average, in 0.035 seconds. That's still a 1,794% speed increase over our first test, but it's actually 0.006 seconds slower than then last one. Of course a number that small is a negligible difference. But the fact that there is no difference is worth noting. There is no tangible benefit to using the aggregate pipeline, without a $group clause, versus an ordinary call to find(). So we'd might as well stick with find(), especially considering we weren't aggregating anything, anyway.
But now comes the question of how we go about getting data from other collections. That userId is effectively a foreign key, so we need to do additional queries to get that information. (Side note: you could just duplicate the relevant information instead of, or along with, the foreign key, since that's kind of the Mongo way. But what happens when a person changes their name? This is the problem with non-relational databases.)
The code that I had originally set out to improve did something that I immediately recognized as bad: it looped over the cursor on Transactions, and for each value, ran another query to the Users collection. I refer to these kind of queries as "one-off" queries, since that's kind of what they are. Let me show you some code to better explain what I mean.
SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);
DBObject project = new BasicDBObject("total", true);
project.put("userId", true);
project.put("created", true);
Cursor cursor = transactions.find(match, project);
List<Map> rows = new ArrayList<>();
while ( cursor.hasNext() ) {
Map<String, Object> row = new HashMap<>();
DBObject dbObject = cursor.next();
ObjectId userId = (ObjectId) dbObject.get("userId");
row.put("created", dbObject.get("created"));
row.put("total", dbObject.get("total"));
row.put("userId", userId);
// one-off query to the Users collection
DBObject userProject = new BasicDBObject("firstName", true);
userProject.put("lastName", true);
DBObject user = users.findOne(userId, userProject);
row.put("firstName", dbObject2.get("firstName"));
row.put("lastName", dbObject2.get("lastName"));
rows.add(row);
}$match = array(
'created' => array('$gt' =>
new MongoDate(strtotime('2014-06-01'))
));
$project = array(
'created' => true,
'total' => true,
'userId' => true
);
$cursor = $transactions->find($match, $project);
$rows = array();
while ( $cursor->hasNext() ) {
$dbObject = $cursor->getNext();
$userId = $cursor['userId'];
// one-off query to the Users collection
$userMatch = array('_id' => $userId);
$userProject = array(
'firstName' => true,
'lastName' => true
);
$user = $users->findOne($userMatch, $userProject);
$rows[] = array(
'created' => $dbObject['created'],
'total' => $dbObject['total'],
'userId' => $dbObject['userId'],
'firstName' => $user['firstName'],
'lastName' => $user['lastName']
);
}I can tell you that when I ran this code against my local instance of MongoDB, it took, on average, 0.319 seconds to run. That doesn't seem so bad at all, especially considering all it's doing. Again, this matches about 30,000 documents in the Transactions collection, and clearly we're making just as many calls to the Users collection. But while this seems fine on my local machine, that is not a realistic test. In real world circumstances, you would not have your database on the same server as your codebase. And even if you did, you won't forever. Inevitably you're going to need some code to run on a different server. So I re-ran the same tests using a remote instance of MongoDB. And that made a BIG difference. Suddenly this same little routine took 1709.182 seconds, on average. That's 28-and-a-half minutes. That is ridiculously bad. I will admit, though, that my wifi speed here at home is not the best. I re-ran the same test later, on a better network, and it performed at 829.917 seconds. That's still 14 minutes, which is dreadful.
Why would this simple operation take so long? And what can be done about it? Imagine this: let's say you went into the DMV office and needed to look up a bunch of records. You have a list of the records you need on a handy-dandy clipboard. So you stand in line, and once you're called to the desk, you ask the clerk, one by one, for the records on your clipboard. "Can you give me the details for Record A?" "Can you give me the details for Record B?" That's straight-forward enough, and will be efficient as it can be. But if the clerk you're talking to only has part of the data you need, and tells you you'll need to visit a different office to retrieve the other missing puzzle pieces, then it would be a bit slower.
If you're querying against your own local machine, it would go something like this:
- Ask Cleark A for Record 1
-
Clerk A gives you everything they have about Record 1
- Clerk A tells you to talk to Clerk B for the rest of the information
- Leave Clerk A's line, and stand in Clerk B's line
- Ask Clerk B for the rest of Record 1
- Clerk B gives you the rest of Record 1
- Leave Clerk B's line, and return to Clerk A's line
- Repeat for Record 2...
That doesn't seem very efficient, does it? But that's the trouble with non-relational databases; disparate collections are in different places. And keep in mind that this analogy actually represents the best case scenario, where Clerk A and Clerk B are in the same room. But that isn't realistic. A more realistic illustration would involve Clerk A at the DMV office, and Clerk B located a mile or two down the road, at the Social Security office. So for each record on your clipboard, you drive back and forth from the DMV to the Social Security office. You can see why it'd be so slow. That "drive" back and forth is called network latency.
But we can do better than that. What if, instead of driving back and forth for each record, you simply asked the clerk at the DMV for all the data they had all at once, and then afterwards you compiled a comprehensive list of all the records you'd need from the Social Security office? That way, you'd only have to make the drive over there once, rather than making 30,000 drives back and forth. In Mongo, you'd only be doing two calls to the database: one bulk call to the Transactions collection, and then a subsequent bulk call to the Users collection. Here's some code to illustrate:
SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);
DBObject project = new BasicDBObject("total", true);
project.put("userId", true);
project.put("created", true);
MongoJoinCache join = new MongoJoinCache();
Cursor cursor = transactions.find(match, project);
List<Map> rows = new ArrayList<>();
int i = 0;
while ( cursor.hasNext() ) {
Map<String, Object> row = new HashMap<>();
DBObject dbObject = cursor.next();
Object userId = (ObjectId) dbObject.get("userId");
row.put("created", dbObject.get("created"));
row.put("total", dbObject.get("total"));
row.put("userId", userId);
join.add(userId.toString(), i);
rows.add(row);
i++;
}
DBObject userMatch = join.resolveCache();
DBObject userProject = new BasicDBObject("firstName", true);
userProject.put("lastName", true);
cursor = users.find(userMatch, userProject);
while ( cursor.hasNext() ) {
DBObject dbObject = cursor.next();
Object userId = (ObjectId) dbObject.get("_id");
Set<Integer> indexes = join.get(userId.toString());
for (Integer index : indexes) {
Map<String, Object> row = rows.get(index);
row.put("firstName", dbObject.get("firstName"));
row.put("lastName", dbObject.get("lastName"));
rows.add(index, row);
}
}
public class MongoJoinCache {
private final Set<String> objectIds;
private final Map<String, Set<Integer>> objectToIndexMapping;
private int total = 0;
public MongoJoinCache() {
objectIds = new HashSet<>();
objectToIndexMapping = new HashMap<>();
}
public void add(String objectId, Integer index) {
objectIds.add(objectId);
Set<Integer> indexes;
if (objectToIndexMapping.containsKey(objectId)) {
indexes = objectToIndexMapping.get(objectId);
} else {
indexes = new HashSet<>();
}
indexes.add(index);
objectToIndexMapping.put(objectId, indexes);
total++;
}
public Set<Integer> get(String objectId) {
return objectToIndexMapping.get(objectId);
}
public Integer size() {
return total;
}
public DBObject resolveCache() {
if (size() == 0) {
return null;
}
final BasicDBList ids = new BasicDBList();
for (String id : objectIds) {
ids.add(new ObjectId(id));
}
DBObject match = new BasicDBObject("_id", new BasicDBObject("$in", ids));
return match;
}
}$match = array(
'created' => array('$gt' =>
new MongoDate(strtotime('2014-06-01'))
));
$project = array(
'created' => true,
'total' => true,
'userId' => true
);
$userIds = array();
$cursor = $transactions->find($match, $project);
$rows = array();
$i = 0;
while ( $cursor->hasNext() ) {
$dbObject = $cursor->getNext();
$userId = strval($dbObject['userId']);
$userIds[$userId][] = $i;
$rows[] = array(
'created' => $dbObject['created'],
'total' => $dbObject['total'],
'userId' => $dbObject['userId']
);
$i++;
}
$userMatch = array('_id' => array('$in' => array()));
foreach ( $userIds as $userId => $indexes ) {
$userMatch['_id']['$in'][] = $userId;
}
$userProject = array('firstName' => true, 'lastName' => true);
$cursor = $users->find($userMatch, $userProject);
while ( $cursor->hasNext() ) {
$dbObject = $cursor->getNext();
$userId = strval($dbObject['userId']);
foreach ( $indexes as $userIds[$userId] ) {
$rows[$index]['firstName'] = $dbObject['firstName'];
$rows[$index]['lastName'] = $dbObject['lastName'];
}
}Phew. That's a lot more code! Well, on the Java side anyway. (PHP arrays are awesome; Java people don't even understand.) But the million dollar question is whether there is any tangible benefit to all that extra code I just threw at you. Here are the results: When running against my local machine, this ran for an average of 0.339 seconds. Against the previously-mentioned 0.319 seconds on my local machine, this is slightly slower. So why bother with all this extra code if it's slower? Because that's not really a fair test. The difference of 10 milliseconds is negligible, but more importantly, there is no universe in which you can reliably expect to have zero network latency. There will always be network latency in production environments. So a real test is against a remote instance.
So what happens when I ran this code against a remote MongoDB server? Remember that it took 28 ½ minutes before. With this code shown above, however, on average, it ran in 6.191 seconds. You read that correctly. That is an astronomical speed improvement of 27,508%. And even if your network latency isn't as bad as mine was (and it shouldn't be), you can still see nonetheless that this method will always be faster, by an order of magnitude.
If you think about it, it makes sense. We took a query which had been O(n²) in complexity and reduced it down to O(2n) -- at most. In fact, it's probably less than that in practice, since there are bound to be duplicate foreign keys. I'm guessing that the developers of relational databases do something exactly like this, under-the-hood in their database code. In order to join two tables, they probably first have to collect all the foreign keys from the first table into a set, and then grab the corresponding rows en masse from the second table. The big difference is that with SQL, you don't have to think about any of this. You just type INNER JOIN and bam, it's all magically done for you. But because Mongo doesn't include this as a concept, you have to write your own database wrapper code in order to be able to use it efficiently.
So what's the takeaway from all of this? First, you can write your own wrapper code to effectively perform a join operation in Mongo, and doing so is hugely advantageous in terms of speed, albeit slightly annoying. But it's vital to your success if you're going to use Mongo. Secondly, what I hope is the bigger takeaway is that you shouldn't have to. Like I've been saying all along, if you have to resort to writing what is essentially your own low-level database code, it's a sure sign that you're using the wrong database. If you've read this far, that means you should probably just use SQL.
TL;DR
To conclude, here are the more practical takeaways broken down:
-
Only select the data you absolutely need, and nothing more
- Use a project clause to limit your data
- Don't instantiate general purpose objects with every property
- Don't do one-off queries inside of loops!
- Minimize network latency by grabbing batches
And that is how you do a "Join" in Mongo.
MongoDB or: How I Learned to Stop Worrying and Love the Database
I come from a SQL background. For seven years, the primary database I worked with was Microsoft SQL Server. I did a little bit of work in MySQL occassionally as well, but in either case, I definitely came from the school of thought that SQL is the only way to store data. This is not without good reason. SQL is powerful. SQL is well-maintained and well-supported. SQL is stable and guarantees data integrity. And SQL is fast, when you know how to use it.
Since transitioning to a new job, though, I found myself flung into a strange new world: the world of NoSQL databases (aka document-oriented databases). In particular, a database called MongoDB. More on that in a second.
When it comes to programming, I consider myself a person who is more interested in solving problems than strict academic purity. With that said, I do tend to be obnoxiously meticulous with my code, but I would much rather throw out the super-careful attention to detail and just get something done, and subsequently tidy up/refactor the code, if it really comes down to it. And I think it is this desire to get something done that has really increased my appreciation for Mongo. Although it is not without its pitfalls, and it is definitely not the right choice in a lot of situations. However, I've been discovering that it is the right choice in a few situations where SQL is not.
My initial reaction to using Mongo was one of disgust. Reading through the design patterns, it seemed like it encouraged huge duplication of data, restricted or impaired your access to the data, and didn't handle date-math very well at all. Unfortunately, all three of those initial impressions are true. But that doesn't mean you shouldn't use it. I won't get into the document-style approach to data because you can already find plenty of material on that yourself. It does involve duplicating data, which would otherwise just get joined in SQL. And that leads into the next point: there are no JOINs in Mongo. They do not exist; you can only query one table at a time. This is what I meant by restricting your access to data. But actually, as I've been learning, this really won't slow you down if you understand Big O notation and how to write Mongo queries effectively. And lastly, all dates in Mongo are stored in UTC time, no matter what. Mongo offers a way to group by dates, but no good way to transform that data into different timezones before grouping it. Which means your data is probably off by 5 hours.
With those downsides out of the way, let me explain why it's actually pretty cool. One of the applications I worked on at my last job was something called "Form Builder." It was actually a pretty slick application that let you create "forms." Forms could have multiple pages, and each page could have multiple questions. In other words, think Wufoo, but without all the pretty graphics. I created a few tables in SQL to build this: one for the forms themselves, another for the pages contained within those forms, a third for the questions on each page, and then one for receiving answers from users. Of course there are more tables involved than just these four, but you get the basic idea.
The "questions" table contained, among other things, the text of the question itself, an ENUM question type (represented in data as a TINYINT), and a number of different fields such as the maximum number of characters a user might enter. In the end, we came up with probably about 20 different question types, ranging from text input, to multiple choice as a drop down menu, multiple choice as radio buttons, multi-line text, asking the user to upload an image, asking the user to upload a file, a preformatted block of address fields, an email field... the list goes on. The point is, with all these disparate types of questions, each one is going to be configured slightly differently. It makes sense to limit the number of characters for text inputs. It doesn't make sense to limit the characters when it's a dropdown menu, however. It makes sense to configure an allowed list of file extensions when you're asking the user to upload a file, but not when they don't. It makes sense to configure a list of multiple choice options when you're asking a multiple choice question, but not otherwise.
What this means, when designing a database table in SQL, is that you really have two choices. Either you can have a bunch of columns that are sometimes relevant, depending on the particular row, or you can sort of abuse SQL by having some "general purpose" columns that get re-used for different purposes. These types of columns are usually either VARCHAR(8000) or TEXT, and they're considered bad practice by DBAs because you are essentially forfeiting the power of SQL by storing it in a different way. I ended up doing a mix of the two.
Here's a snapshot of the questions table:
As you can see, there's a character limit column on every field, even though not every field actually enforces a character limit. But then there's also this column called extra_info which looks like it contains a bunch of gobbledygook. That's really just a serialized PHP array, which is kind of like JSON, but not as elegant. (It predates JSON.) It's a way of storing any amount of arbitrary information in a quickly-usable-in-code format. Within SQL, storing data that way is a black box. There is no way to search it, but you store it that way because you don't need to search for it; you only need to access it. Lots of people do this, but it is considered an abuse of the database, because you're no longer really using the database.
What I didn't realize at the time, was that there was a different way of thinking about this problem entirely. This type of project was an absolutely perfect candidate for using Mongo, and I'll tell you why.
Mongo has different nomenclature than SQL. "Collections" are the equivalent of tables, "documents" are the equivalent of rows, and "fields" are the equivalent of columns. Every document in a collection is simply a JSON object. And a collection does not have any defined structure to it whatsoever. You read that correctly. It has no structure. There are no column definitions. One row in the table may have an entirely different set of columns than the next row. They're just arbitrary documents. It's all up to you. If you want one row to have one set of columns, fine. The documents are whatever you tell you them to be. They don't really care about each other.
With that said, you will end up having most of the same columns present on every document within a given collection, because otherwise there would be no point. But in Mongo, I could represent the same "questions" table above like this:
So you see here, one row doesn't enforce a character limit, so the field isn't present at all. Another row has an options array, whereas the rest don't have that column present in the first place. The difference between this and SQL is huge. In SQL, you can accomplish the same thing as I described above, but you cut off your own access to the data in the process. Mongo on the other hand is designed for this.
In Mongo, I could run a query like this, to find all the questions that have a character limit:
Or I could run a query like this, which will locate any questions that have an options array present, with one of those options being the word Red:
Mongo queries themselves are an off-shoot of JavaScript, which means you can use regular expressions too, which is awesome. (Although those are better suited for ad-hoc queries than actual production ones.)
From one SQL programmer to another, I would recommend giving Mongo a try. It has its annoyances, believe me. But after using it, you start to realize that SQL does too, and the realization that SQL isn't the only approach to anything and everything can be eye-opening. Having more tools in the toolbox helps make you a better programmer. Understanding when to use them, and when not to use them, is key as well. But I have to say, it's worth giving it a shot. I think you'll find there are some applications where it actually is better suited.
And by the way, if you're going to use MongoDB, you are going to need a client to access it as well. The one I've been using, the one that is pictured in the screenshot above, and the one that is hands down the best out there, is called Robomongo. The screenshot shows one of three possible ways of viewing the data (in tabular format). But it also lets you view documents in a hierarchical list, or as straight JSON text.
Page 1 of 6 
