Gordon Myers

Articles on Life, Truth, Love, Computers, and Music


Is God a person?

This is a question that many people have asked throughout the ages, and one that I've heard a lot myself. And as a Christian Scientist, it's sometimes been a point of contention with some of my Christian brothers and sisters, whether I mention anything or not. Recently, after an otherwise-excellent first date ended with the unprompted comment, "I'm sorry Gordon, your views on the Trinity are just too different," it's been something on my mind.

For those of you unfamiliar, the Trinity, or Godhead, is the popular Christian concept that God consists of three-persons-in-one: the Father, the Son, and the Holy Ghost. This same concept is represented more succinctly by the slogan, "Jesus is God." This is a commonly accepted belief in many Christian churches, now considered orthodox, but it was not always as widespread as it is today.

I've been reading a fascinating history book called When Jesus Became God by Richard Rubenstein. The book covers the history of the "Arian Controversy," which includes the famous Council of Nicaea, at the end of the 4th century. This was the point in history when various Christian Bishops came together to flesh out official church doctrine, and this was when they formally consented to the idea that God and Jesus are homoousios, a Greek word borrowed from pagan philosophy, which means "the same stuff." I find it fascinating that any meaningful sense of consensus was largely absent from this council. The Christian community was essentially split down the middle on this idea, and quite a lot of politics, human posturing, and personal amibition went into it. History also shows that this one simple decision quickly reverberated with a lot of unnecessary and ironically un-Christian violence.

Now I don't mean to reignite a centuries-old turf war. But regardless of what church councils "decided," or what's considered popular today, I'd like to examine the question of who God is, and how Jesus fits into that picture, from the perspective of what the Bible actually says and what the teachings of Christian Science reveal. As Isaiah says, Come now, and let us reason together.

Hear, O Israel: The Lord our God is one Lord.

The First Commandment of Judaism and Christianity states that there is one God. What does this mean? Clearly this is opposed to atheism, which says there is no God. And it's similarly opposed to pantheism, which says there are many gods. It's generally accepted, also, that God has the following four properties:

  • God is omnipotent, meaning He's all powerful / there's nothing He can't do
  • God is omniscient, meaning He knows everything
  • God is omnipresent, meaning He fills all space
  • God is omnibenevolent, which means He is all good

Those "omni-" words don't appear directly in the Bible, but are they supported by it? Does the Bible imply that God is omnipotent? Well, I can tell you that no less than four times, Jesus said, "all things are possible to God." Check! How about omniscient? In 1st John it says that "God is greater than our heart, and knoweth all things." Check! Omnipresent? Look no further than the poetry of the Psalms: #139 essentially says yes. Lastly, is God good? Well, if he wasn't, it'd be pretty scary praying to Him! But if you're looking for Biblical support, Jesus said, "Why do you call me 'good'? There is none good except God alone."

Now any freshman philosophy major can explain to you that there's no being who could possibly exist in our physical universe that truly has all four of those properties at the same time (and they'd be right), because of something called the problem of evil. But today's post is not to try and dissect that paradox -- that's a whole other can of worms! So for the sake of argument, we'll just take it on faith that God is indeed all-powerful, all-knowing, omnipresent good. Question: can an all-good, all-powerful, all-knowing, omnipresent being be a person?

Again, let me reiterate that lots and lots of people -- many of them smarter than me -- have already thought through this same question over the ages. In fact, one example of someone else who's thought about this question a little bit is Carl Sagan, the famous astrophysicist. Here's a quote from him:

The idea that God is an oversized white male with a flowing beard who sits in the sky and tallies the fall of every sparrow is ludicrous. But if by God one means the set of physical laws that govern the universe, then clearly there is such a God. This God is emotionally unsatisfying. It does not make much sense to pray to the law of gravity.
--Carl Sagan

We seem to be at a crossroads.

On the one hand, the orthodox Christian belief of Jesus as God provides a lot of comfort. To think that there's an infinitely powerful personal friend, ready to help you out of trouble and save you from despair in this physical life, is understandably appealing. But under the microscope of rational thought, inconsistencies start to appear. And if there's some additional qualifications that come with this (i.e. do/say the right thing or you're damned), one starts to question the "omnibenevolent" part. Dig deep enough and things might even seem "ludicrous," as Mr. Sagan put it.

On the other hand, what if God is simply the name for all the principles and laws governing the physical universe? Well then we can see that these are ever-present, and all-powerful by definition, but it just seems like such a cop-out. Those laws don't seem very intelligent, and certainly not very good. Instead it all seems chaotic, unknowable, and feels so cold. It certainly doesn't meet our criteria of the four "omni"s.

But what if I told you that there was a simple mistake with both of these approaches, and indeed, there's a third answer? On the one hand, if you start by saying, "I see, feel, hear, etc. with my material senses, and my material body is who I am," and then you remember reading in the Bible that "God created mankind in His image and likeness," you might naturally think, "well that means God must be just like me -- a material body. And hey, Jesus had one of those! Therefore Jesus must be God." Or on the other hand, if you start by saying, "I see, feel, hear, etc. with my material senses, and my material body is who I am," and then you study the scientific method, you might say, "everything is made out of matter just like me, and there certainly seem to be patterns to what I can observe in the physical world, maybe these patterns are God."

Both of these approaches reason out from the evidence of the material senses. Both start with the idea that I am material, therefore practically everything is material. You try to make God in the image and likeness of yourself. Matter is what we observe with our senses, but the Bible says God is Spirit, and so I think we need to start there. But that's hard, because spiritual things are un-quantifiable. What's the average mass of a mother's love? What's the velocity of it? Its existence is obvious, yet it extends far beyond chemicals and neurons. It can be difficult for people, at first, to reason about things spiritually, because we're not always used to it. But if you don't immediately accept the idea that you are made out of matter, and consider for a moment that instead you are spiritual, then the reasoning goes a little differently. Here's what Mary Baker Eddy, the Founder of Christian Science, had to say:

Human philosophy has made God manlike. Christian Science makes man Godlike. The first is error; the latter is truth. Metaphysics is above physics, and matter does not into metaphysical premises or conclusions.

Christian Science strongly emphasizes the thought that God is not corporeal, but incorporeal -- that is, bodiless. Mortals are corporeal, but God is incorporeal.

As the words person and personal are commonly and ignorantly employed, they often lead, when applied to Deity, to confused and erroneous conceptions of divinity and its dinstinctions from humanity. If the term personality, as applied to God, means infinite personality, then God is infinite Person, -- in the sense of inifinite personality, but not in the lower sense. An infinite Mind in a finite form is an absolute impossibility.

These are just some of the ideas that I'm going to be sharing during a short church meeting this Wednesday, August 24th, here in Madison. I'll be presenting further readings from the Bible as well as more of Eddy's exposition on it, on this same topic, "Is God a person?" If you're interested in finding out a little more, please join me at 7:30 this Wednesday evening.


My Pokemon Go Wishlist

Like many, I've been sucked into the smartphone craze that is Pokémon Go. The nostalgia of reliving a game many of us played 20 years ago, combined with the pride of collecting as many little critters as you can, combined with the inspiration of wandering around your city and discovering hidden gems makes for a potent formula for success. Still, despite the game's immense and immediate popularity, I can't help but noticing that it's not exactly feature-rich. It's definitely an underdeveloped app, which isn't helped by the server-side struggles they've had to keep up with demand. Today I will present my wishlist of the top 3 features I'd love to see rolled out in future iterations of Pokémon Go.

1. Attacking before Capturing

One of the fundamental game mechanics that you learn early on in every other Pokémon game (and there are quite a lot!) is the importance of weakening new prospects by attacking them first before trying to capture them in poké balls. That mechanic is sadly absent from Pokémon Go. It's somewhat made up for with the use of Razzberries, but only somewhat as they often prove ineffective. I'd love to see future iterations of the app allow your existing critters to attack the wild Pokémon before attempting to throw a poké ball at them, with some sort of HP gauge on screen. That would be more true to the original. They could even introduce different consequences depending on the (predetermined) disposition of the Pokémon: perhaps certain animals would stay and fight while others might immediately try to flee when attacked. There are lots of possibilities, but right now it feels like a big missed opportunity.

2. Leveling up at gyms

In the real world, when you go to the gym to workout, you leave feeling stronger. In Pokémon Go, gyms don't exactly work that way. Presently the only way to strengthen your Pokémon is outside of the gym, by feeding them candy. This message seems incongruent with reality. I think the longer a Pokémon remains in a gym the stronger it should be. Or at the least, upon every successful defense of a gym, it should gain some additional CP. Right now gyms are little more than spectacles of pride, but the game would be much improved if they were more, y'know, gym-like.

3. Logging in

One feature that they should consider including in future iterations of Pokémon Go is the ability to log in. I really feel it was a big miss on Niantic's part by not including this feature right off the bat. It can be very counterproductive when you want to play Pokémon Go but you're unsuccessful logging in and thereby are prevented from playing Pokémon Go. It's really a strange game mechanic in which you're not allowed to play the game in the first place, and seems like a pretty glaring oversight on part of the developers. Hopefully, in future releases of the app, there will actually be a functioning app to play. That would seem like a great feature to include.

So that's my wishlist for Pokémon Go. What do you think? Do you have ideas of your own for features you'd like to see? Leave a comment below!


It's the Star Wars movie we need right now, but not the Star Wars movie we deserve

This morning I saw Star Wars: The Force Awakens in theaters. It was everything everyone expected. It lived up to the hype. However I'm not writing this post to join the chorus of accolades; there will be enough of that and indeed there already is. This post assumes you enjoyed the film and are ready to move on to a healthy level of scrutiny. So keep in mind that as I talk about the flaws of the film, I do so earnestly from the perspective of someone who genuinely enjoyed the movie and will probably be back to the theaters to see it again. With that said, there were a couple of things about this film that bothered me.

Star Wars is something that a lot of people have very strong and passionate feelings about. For years it was carefully controlled and guided by George Lucas, another thing altogether that people have strong and passionate feelings about. Since Lucas sold the rights away and the mantle was picked up by J.J. Abrams, it's fair to say that people were nervous he might add too much lens flare, or otherwise not get things right. I'm simultaneously pleased and conflicted to say that J.J. did get it right. Conflicted only because he didn't actually make a new movie at all. The Force Awakens is a play-by-play, carbon copy of A New Hope, down to the smallest detail. Some have even gone so far as to call it plagiarism. I simply call it "playing it safe."

My roommate, who has not yet seen the film, just asked me as I was writing this if I thought the new film was better than Episode Four. That's a question I can't really answer, because it is Episode Four. This film had about as much new material as The Hangover 2 did. It's an exact replica of the original film, just with younger characters and bigger objects.

But that isn't really what bothers me. The new film was exactly what the masses needed most after the disaster that was the prequel trilogy. They had felt betrayed by their beloved director. How could the same man who gave the world Luke Skywalker, Han Solo, and Darth Vader be the one who made Anakin Skywalker, Jar Jar Binks, and midi-chlorians? Alas, as the years went by, Lucas spiraled further into delusion and denial like the hapless CG-junkie that's he's revealed himself to be. And the fans felt the sting of betrayal. The galaxy far, far away had been forever tainted. A Force Awakens calls back to the earlier days of greatness. It pays homage to its roots because it is its roots, with little more than a new face.

However, just as Kylo Ren couldn't shake his Dark Side heritage, the film couldn't shake some of the series' darker memories itself. I'm talking about Hayden Christensen's acting, newly manifest in Daisy Ridley. I know that people are going to praise her performance, and while I do believe she is leagues apart from Mr. Christensen, I still saw sparks of the dark side in her. And I don't mean the force.

In fact, I'm sure that both she and Abrams will be praised: Ridley for her acting, and J.J. for including a strong female lead. I don't buy it though. The first felt forced at times and the latter felt like affirmative action. If feminism in the 21st century has taught us anything, it's that a surefire formula for accolades in the film industry is to have a fiercly independent female lead. People think it's refreshing to have a damsel who's not in distress, who doesn't need a man to save her, and who proclaims her independence at every turn. I don't have a problem with a strong female lead, even though I think it's already become an overplayed trope years ago. But I do have a problem when the only motivation for that choice is pandering, and when the so-called strength of that lead wanders into overcompensation territory, as if it's apologizing for years of other films.

Case in point: Rey's insistence that she doesn't need Fin to hold her hand on Jakku. It was meant to highlight her independence, but it felt robotic, unrealistic, and needlessly rude. That's not how people interact in the real world. And her change of heart toward Fin, later in the film, comes too fast. It makes the character seem flaky and inconsistent. And when Fin is trying to do all the things a conventional male hero is supposed to do, she loves him for it. Now, granted, an argument could be made that Abrams was trying to reincarnate the same cocky, push-and-pull chemistry that Leia and Han perfected years ago into a new generation. But if the prequel trilogy taught us anything, it's that you can't force romance. No matter how many times Anakin said he loved Padmé, we never believed it, because tell-not-show storytelling is never believable.

There are other examples, too, that are a little more glaring, other glimpses where you see her channeling her inner Christensen. Watch Rey's expression as she becries Solo's death. The scene is parallel to when Luke is upset over Obi-wan's death, with only one little thing different: she's only known Han for a matter of hours, whereas Uncle Ben was family to Luke. Yes it sucks, but she's clearly over-acting.

But all of these minor quibbles are still dwarfed by my real complaint with the new film. My main complaint is not really about the film, not anything in it. I had this complaint long before ever seeing the film. My major gripe is about the fact that film exists in the first place. Because its mere existence invalidates the Prophecy of the Chosen One. No, invalidate is too soft a term. It utterly and completely destroys it. In layman's terms, it removes the need for any of the last six films and renders them all pointless.

The Prophecy of the Chosen One was an ancient Jedi prophecy that foretold the coming of a being who would forever bring balance to the force. This being was revealed (and confirmed) to be Anakin Skywalker, who "forever" destroyed the Sith and eliminated the influence of the Dark Side when he took out the evil Emperor Palpatine and sacrified himself as well. This prophecy was the entire point of the original three films, and further reinforced by their prequels.

The trouble is that if evil has been destroyed for good, and the prophey has been fulfilled, that doesn't leave room for any future villains or really any room for future films or literature. Peace is boring. Peace isn't dramatic. Star Wars is popular because the wars make things interesting. I'm ignoring the hoardes of expanded universe novels when I say that Return of the Jedi was meant to be the end. But I have yet to hear any satisfactory explanation as to why the Prophecy of the Chosen One isn't proved to be complete garbage by the re-introduction of the dark side after the fact.

Just to wade through the pedantry a bit, no, the phrase "bring balance to the force" does not simply mean to equalize the number of Jedi with the number of Sith. Lucas himself has confirmed that it meant the elimination of the Sith, and the dialogue in the prequel films supports this. The Jedi believe that the only truly "balanced" state of the Force is when the Dark Side is totally absent. The other main argument, or should I say rationalization, is that balance is inherently temporary. Well, what was the point, then? Why bother prophesying anything if it's all going to be meaningless 30 years later?

All these retconned delusions serve to do is spit in the face of the original story. The Force Awakens is more than just an innocuous reboot -- and it is a reboot in the truest sense of the word, not a sequel. With all its careful orchestration, tribute, and nostalgia, it repents of the the prequel trilogy, but only by selling the soul of the beloved original trilogy.

It had to do that, and I understand why. Although the heritage of the Jedi, explored thorougly in the Knights of the Old Republic, is rich and lasting, it isn't as marketable as Luke Skywalker or Han Solo. It had to be a sequel, not another prequel. What people needed was redemption from the Hollywood disasters that were the prequel films. And that's exactly what they got. But, unfortunately, underneath it all, no one seemed to notice that this redemption came with a price. The original trilogy was now all in vain, as we start over with fresh faces.

And the reason that's a problem is because, at its core, the original story (prequels included!) was good. If you don't believe me, watch any of the What if Star Wars Episode X was good? videos on YouTube. The heart of the story was always pure, even if it did end up with a fat body and an ugly face. And although this new story is great (how can it not be when it's a copy?), I am disappointed that The Force Awakens threw the baby out with the bathwater. The good story-telling that Lucas intended was the Star Wars that we deserved. But Star Wars: A New Hope, Mark 2 was what the people needed.


My Mario Maker Wishlist

I bought my copy of Mario Maker bundled with my brand new Wii U console the day the former came out. I have always been a fan of Mario, but an even bigger fan of puzzle games. And the fact that Mario Maker was a way of blending those two concepts made it too hard to resist. And there are some wonderfully creative levels out there. With that said, although Mario Maker gives level designers a fairly broad palette, overall the game still seems a little premature to me. So without further ado, I present my wishlist of the top three things I'd like to see added to Mario Maker.

#1. Better Level Searching

The interface for finding levels seems like little more than an after-thought in the mind of the developers. Understanding how looming deadlines and high demands for features probably pushed the level choosing interface to the back-burner, I can sympathize. But now that the game's out, I'd really like to see (preferably in the form of a free update) some kind of revision that will make searching for courses and course creators easier. This doesn't seem like a tall-order to me. Really, we just need to two things:

  1. Search for courses by name
  2. Search for authors by name

That would make a world of difference. Right now you need to know the obscure 16-digit course IDs in order to find anything, which means you really have to want it. Nintendo, please make this easier for all of us.

#2. Conditional Power-up Blocks

I'm really surprised that Nintendo chose to do power-ups the way that they did with Mario Maker, because it broke all past conventions. In every Mario game ever, when Mario's small, a power-up block would yield a mushroom. Once he's big, it might yield a different power-up like a fire flower or a feather. Or it might not. But in any case, being big was a prerequisite to getting the more "advanced" power-up. I'd like to see this element brought back because I think it adds some challenge to the game. Instant fire flowers cheapens things a bit.

#3. Vertical Levels

Nintendo gives you quite a lot of space to work with horizontally, but only two screens maximum of vertical space. I really wish they had included some sort of toggle for horizontal and vertical mode in much the same way that Microsoft Word lets you choose between portrait and landscape mode. Some folks have already tried to emulate this by using doors and warp pipes to make tessellated faux-vertical levels, but it's really not the same. Think auto-scrolling. How great would it be to have a level that makes you climb with the risk of being swallowed up by the auto-scrolling bottom?

Of course this raises a different concern, namely that people would want vertical levels that you could descend. And if you have two possible positions for the start and end, doesn't that mean you'd also need to allow for horizontal levels that move left? Well, yes! I think that would be a delightful side effect. Especially since some folks are already doing this in their sub-levels anyway.

What do you think about my wishlist? What things you would add? I purposely left off a lot of items that are, well, items, because I wouldn't be surprised to see some of that coming in the form of DLC. But let me know in the comments what you think.


Spirited Away

Today's blog post is going to be a movie review. This isn't a current movie by any means, either. In fact, it's 14 years old. It's also a film that I would expect many of you have already seen. But recently I felt an urge to rewatch it, and discovered a few things in the process. First, I realized that there are still many who haven't seen this film, or haven't even heard of it, and that's a shame. But more importantly, I realized some of the great lessons this film has to offer. I am talking of course about the film Spirited Away by Hayao Miyazaki.

For those not familiar, Miyazaki is a Japanese animator who makes kids' movies. And his films are kind of the gold standard in Japan. They're basically Japan's equivalent of Disney movies. In fact, if I'm not mistaken, Disney actually bought the English version rights to all of his films. Spirited Away in particular is one that I consider to be the crème de la crème of his work, and I'm not alone. It is the most successful Japanese film to date and won an Academy Award.

The story of the film is fundamentally one about a young girl overcoming her own fears and limitations in order to free her parents from the spell of an evil witch. It's also a story about friendship and love. And it's a story that draws heavily on Japanese mythology, featuring dragons, witches, talking frogs, giant babies, and everything from river to radish spirits. The artwork is spectacular, the story is unique, and the messages of the film are splendid. But as I was rewatching this film the other day for the upteenth time, one point in particular struck me as to what makes this film so incredible: there are no villains.

Right away people who have seen the film may dispute this fact. They'll remind me that the evil witch, Yubaba, is the story's central antagonist. And to that I'd say: no, she isn't really. Not really, anyway. At its core, the world of Spirited Away doesn't really have any villains. For me, that's what makes it truly so remarkable. But I need to explain and defend my position of why there really aren't any villains in this film. Please note that spoilers will follow, but this film is good enough that they won't detract at all from watching the film, so please keep reading.

I'll start by debunking the example already named: Yubaba is not a villain. What is it that makes her evil? Well, within the first 10 minutes of the film, she turns the main character's parents into giant pigs. Except... she doesn't actually do that; the parents do that to themselves. They overzealously decide to help themselves to plates full of food left out at what appears to be an abandoned theme park, but is in fact actually an enchanted spirit village. Yubaba isn't present or even aware of the humans at that point; the food is simply not meant for humans with the unfortunate side effects is that it automatically turns non-spirit diners into pigs. Yubaba didn't take any action at all here; it was the parents' own greed that resulted in the curse.

But Yubaba does decide to keep the parents as pigs so that she can raise them for slaughter. And this is the central plot point which the main character, Chihiro, is working to prevent. Yes, this seems evil, but I'm hesitant to really label it as being downright evil, and instead call it a callous business practice, not unlike many of the seemingly callous business practices that we see in the real world. It's not a malovelent, calculated plan; it's simply a matter of convenience. In the eyes of Yubaba, human beings are an entirely different species. They are like animals to her. And what meat company wouldn't freely take advantage of unclaimed livestock that wandered onto their premises? It's not always pleasant to talk about it, but it doesn't make her a fundamentally evil person.

So rather than examining Yubaba's inactions, let's look at her actions. Within the first 30 minutes of the movie, she hires Chihiro to work at the bath house. Chihiro asks her for a job, and she agrees to give her one. In fact, Yubaba has sworn an oath that if anyone, regardless of skill, circumstance, or ability, asks her for a job, she will always grant their request. And not merely on a probationary or provisional basis either; she offers permanent employment to anyone who asks. Does that sound evil to you? You'd be seriously hard-pressed to find that same kind of generosity from even some the best companies!

During Chihiro's employment, at several points, Yubaba compliments Chihiro when she does her job well. Moreover, Yubaba offers a path for Chihiro to follow that will ensure the safe release of her parents. And she keeps her promise. Again, what's evil about any of that? By the end of the film, Chihiro embraces her employer with a hug of gratitude, sending home the message that there aren't villains.

Another character to consider is a spirit called "No Face." Initially a meek, speechless spirit that is denied entrance to the bath house in which all the other spirits partake, this character takes notice of Chihiro and follows her in through a back door. It's shown just how lonely this character is, and how it will do anything to appease others and win their praise and affection. Ultimately it's revealed that this character's people-pleasing tactics are fundementally selfish, as it only acts out the behaviors it thinks others are desiring in order to fill a void. And when it still doesn't feel satisfied, it starts eating the other characters alive, which in turn triggers an increasingly insatiable hunger.

In short, No Face is a character that represents loneliness and lust. It starts out innocent, but due to its hunger for affection, it grows into a disgusting monster that consumes everything and everyone in its path. It does so by preying on the weaknesses of others, but only through illusion and manipulation. In fact, at the zenith of its lust, there is a scene where Chihiro confronts No Face and it is revealed just how desperate the creature is for Chihiro's validation. After having assaulted and consumed half the staff at the bath house, he nearly does the same to Chihiro. That's pretty evil, right?

But let's examine Chihiro's response to this situation: she sits patiently, quietly in front of him, completely unafraid of his condition. She has absolutely no fear. While every other character on screen is either frantically running away or trying to lock the doors, she enters the same room and calmly sits down beside him to chat. When he charges at her signaling he might try to eat her, her first instinct is to help him. She says, "before you eat me, eat this," and then hands him some magical medicine. The pill then triggers a violent reaction in which everything bad in his system is flushed out.

During this reaction, she runs away from him, which makes for a great chase scene. But as soon as that is over, No Face returns to his original form, and starts to follow Chihiro once more. At this point, she stops running away and invites him to sit next to her on the train as a friend. One of the supporting characters is initially shocked that she would let him anywhere near her, but Chihiro responds simply by saying, "I think the bath house makes him crazy. He needed to get out of there."

This, to me, is actually the most telling line of the whole film. This confirms why the message is so powerful, that there are no villains. Even after having been nearly assaulted by this character, she calmly recognizes that it is not the character, but the circumstance, that was the real problem. Chihiro not only helps him out of the unfortunate circumstance, but immediately befriends him and helps draw the best out of him, later helping him to get a job as a personal attendant of another character.

There are more examples, too. Zeniba could be seen as villanious, at least temporarily, for having nearly killed one of the other characters, Haku. But she quickly becomes a fast friend and supporter of both Chihiro and Haku. The giant baby could be seen as villanious, but after being forced out of his spoiled environment, he becomes an ardent defender of Chihiro as well. Even many of the spirit workers could be seen as evil, or at least as dangerous, for their xenophobia of humans, especially at the beginning of the film. But that too is overcome as they work beside Chihiro and begin to appreciate her.

The only villains in Spirited Away are intangible. Qualities like greed, lust, hatred, fear, envy, and theft are the villains of the film. These qualities are acted out, for a time, by various characters. Even Chihiro herself shows a lot of fear at the start of the film. But all of these qualities are overcome, either through the characters' individual growth, or through the help and support of friends. This is a truly powerful message. There are no evil characters; everyone can be redeemed.

Contrast this with even some of the most treasured Disney films. In Lion King there are clearly good characters and bad characters. There is no redemption for Scar. At the end of the film, he is eaten alive by his own hyenas. In Aladdin, there is no hope for Jafar. He is forever banished to a tiny prison. This is actually a pretty common message with most Western Disney movies: there are good people, and bad people, and the bad people should either be killed or exiled.

In Spirited Away, there are just people. (Well, sort of. If you count frogs and radishes as people.) These people sometimes act out bad qualities, and sometimes it can seem pretty scary, but even this can be redeemed by not giving into fear and persistently loving and supporting them. In the world of Spirited Away, gratitude is given even to those who were once seen as enemies. There are no villains. I think we can learn a lot from messages like these. And it sure would do some good to have more movies like that.


Purpose: Letting your brilliance shine through

Today I watched an hour-long video on YouTube. I know this seems like an impossible feat since the attention span of most of the YouTube audience (myself not excluded) is like that of a goldfish. But I was feeling hungry for something substantive tonight, and I don't just mean long.

For a few years now, there have been full, hour-long Christian Science lectures posted on YouTube. If you've never seen one of these before, I really recommend them. They cover a variety of topics and don't let the word "lecture" deter you or imply that they're dry. These are kind of like TED talks, but better. And there are some shorter ones posted, too, to give a sampling of what these are like without having to commit a full hour.

Tonight I watched one titled Purpose: Letting your brilliance shine through by Tom McElroy. The great thing about Christian Science lectures though (in my opinion) is that regardless of what the topic purports to be about, there is something there for you. There is some insight that only you will uniquely glean from listening to the lecture. For instance I wasn't particularly focused on finding my own sense of "purpose" this evening. I actually was stood up for a date so my focus was very different. Even so, I found some valuable, comforting ideas that spoke to me.

Most of all, I think Tom in particular gives a great introduction to Christian Science if you don't know much about it. So if you can spare an hour, I'd really recommend watcing this video. At least add it to your "Watch Later" list. Here's a short excerpt from the video:

To say that there's a science to it [to call it a Christian Science] is to say that if there was ever any truth in that love [if there was ever anything real, anything that actually happened there, anything that ever really took place in the healing, in the love, in the vision; if that was based on anything,] then whatever the truth was behind it [the science of it, the reality of it, the truth of it] still has to be true today. It's timeless. It has to be true for all people under all circumstances. [It has to apply to all people equally.] It's not something we earn or work our way up to and it's not something we can mess up.

Check it out for yourself! I promise you you'll learn something new.


Hey, I made a thing

I just published my very first Chrome Extension to the Google Play Store! It's actually not hard at all, and I'm going to come back here and beef up this post with more of the details when I have some more time. But in a nutshell, if you use Atlassian's Bitbucket service, you might be familiar with "pull requests," otherwise known as code reviews. These are immensely helpful, but sorely lacking one critical feature: there's no way to easily expand/collapse whole sections of a pull request. So, I added that feature! But I did so in a really lazy way. Rather than taking the time to figure out how to hook into the loaded events, I just a gigantic button on top that says "Active Toggle." Click that button once all the files have been loaded up, and it will then add individual "Toggle" buttons to each section. Enjoy!

https://chrome.google.com/webstore/detail/bitbucket-pull-request-to/hfebajohpclnfhfnlhgndbmcdnlchjjd


How to do Joins in MongoDB

If you've come here looking how to perform a JOIN operation on two or more collections in MongoDB, you've come to the wrong place. In fact, that is exactly what I'm about to show you how to do, but trust me, you shouldn't be doing it. If you're trying to do that, that's a clear indication that you have relational data. And if you have relational data, that means you've made the wrong choice for your database. Don't use Mongo; use a relational database instead. I know it's cool and shiny and new, but that is not a good rationale to use it. SQL is your best bet. I'd recommend you read Why You Should Never Use MongoDB by Sarah Mei. Now, with that disclaimer out of the way, I'll dive right into it.

We've been using MongoDB for some of our data storage at work. The choice to use Mongo was a decision made well before I arrived, and incidentally, we might be changing that out at some point in the future. But nevertheless, as it's the system currently in place, I've had to work within that system.

There were a number of pre-established Mongo queries in the codebase I've been working on, and I'm sorry to say many of them were really quite slow. So over the course of a weekend, I tried out a couple of ideas that seemed intuitive enough and managed to speed up some of these common queries by an order of magnitude. The queries I'm talking about grabbed data from multiple collections simultaneously, hence why they were initially so slow, and hence the title of this blog post. In this post I'm going to dissect a number of the techniques I used to speed up these Mongo queries, with plenty of code examples along the way.

Let's say you've got a collection in Mongo called Transactions. This table has a variety of fields on each row, including one field called userId, which is just the ObjectID (aka foreign key, for you SQL folks) of a document in the separate Users collection. You might want to retrieve a list of transactions in a given time period, and show some information on the screen, like the date, the total amount, and the first and last name of that user. But for this first part, let's hold off on any attempts at JOINs, and just look at accessing the Transactions collection alone.

I ran some benchmarks with the following code on my local machine, which was also running a local instance of MongoDB.

MongoClient conn = new MongoClient(new ServerAddress("127.0.0.1", 27017));
DB db = conn.getDB("MyDatabase");
DBCollection transactions = db.getCollection("Transactions");

SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);

Cursor cursor = transactions.find(match);
List<Map> rows = new ArrayList<>();

while ( cursor.hasNext() ) {
    Map<String, Object> row = new HashMap<>();
    DBObject dbObject = cursor.next();
    
    row.put("total", dbObject.get("total"));
    row.put("approved", dbObject.get("approved"));
    row.put("canceled", dbObject.get("total"));
    row.put("location", dbObject.get("canceled"));
    row.put("items", dbObject.get("items"));
    row.put("coupons", dbObject.get("coupons"));
    row.put("created", dbObject.get("created"));
    row.put("updated", dbObject.get("updated"));
    row.put("deleted", dbObject.get("deleted"));
    row.put("userId", dbObject.get("userId"));
    
    rows.add(row);
}
$conn = new Mongo('mongodb://localhost:27017');
$db = $conn->selectDB('MyDatabase');
$transactions = $db->selectCollection('Transactions');

$match = array(
    'created' => array('$gt' =>
        new MongoDate(strtotime('2014-06-01'))
    ));

$cursor = $transactions->find($match);
$rows = array();

while ( $cursor->hasNext() ) {
    $dbObject = $cursor->getNext();
    $rows[] = array(
        'total'    => $dbObject['total'],
        'approved' => $dbObject['approved'],
        'canceled' => $dbObject['canceled'],
        'location' => $dbObject['location'],
        'items'    => $dbObject['items'],
        'coupons'  => $dbObject['coupons'],
        'created'  => $dbObject['created'],
        'updated'  => $dbObject['updated'],
        'deleted'  => $dbObject['deleted'],
        'userId'   => $dbObject['userId']
    );
}

This code is obviously sanitized a bit here to highlight what I'm doing, but you might extend this to do any number of things. You might have some sort of POJO that corresponds to a single document in the collection, and instantiate a new one within each loop. Then you would call dbObject.get() to retrieve each of the properties of that row. I iterated this simple test hundreds of times on my local machine, and found, on average, it took 0.663 seconds to complete. And in case you're curious, the date range I've given here corresponds to roughly 30,000 documents. So that's not so bad.

But this pared-down example was not my use case, and my use case was performing poorly. So I thought, well, I don't need all those pieces of data in the dbObject. I really only needed two. So I formulated a hypothesis. My hypothesis was simple: if I only grab the data I need, and none of the data I don't, the query would perform better. This is akin to avoiding SELECT * in SQL (which is something you should always avoid). So to start, I made the most basic modification to the code possible, which you can see here:

SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);

Cursor cursor = transactions.find(match);
List<Map> rows = new ArrayList<>();

while ( cursor.hasNext() ) {
    Map<String, Object> row = new HashMap<>();
    DBObject dbObject = cursor.next();
    
    row.put("created", dbObject.get("created"));
    row.put("total", dbObject.get("total"));
    row.put("userId", dbObject.get("userId"));
    
    rows.add(row);
}
$match = array(
    'created' => array('$gt' =>
        new MongoDate(strtotime('2014-06-01'))
    ));

$cursor = $transactions->find($match);
$rows = array();

while ( $cursor->hasNext() ) {
    $dbObject = $cursor->getNext();
    $rows[] = array(
        'created'  => $dbObject['created'],
        'total'    => $dbObject['total'],
        'userId'   => $dbObject['userId']
    );
}

All I've done here is remove the .get() call on the properties I didn't need. In fact, at this point, the database driver is still returning all of those properties; I'm just not accessing them. I wanted to see if that alone would make any difference. And in fact, it did. Hundreds of iterations of this code averaged in at 0.405 seconds. That's a 63% speed improvement. Of course the percentage makes it seem more grandiose than it really is, since that's only a 0.25 second improvement, which is not that big of a gain. But it is still an improvement, and it was consistent. Accessing fewer properties from the cursor results in a speed improvement. But while this sped things up a tiny bit, I knew that we could do better by forcing the database driver to stop returning the extraneous data, a la a project clause:

SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);

DBObject project = new BasicDBObject("total", true);
project.put("userId", true);
project.put("created", true);

Cursor cursor = transactions.find(match, project);
List<Map> rows = new ArrayList<>();

while ( cursor.hasNext() ) {
    Map<String, Object> row = new HashMap<>();
    DBObject dbObject = cursor.next();
    
    row.put("created", dbObject.get("created"));
    row.put("total", dbObject.get("total"));
    row.put("userId", dbObject.get("userId"));
    
    rows.add(row);
}
$match = array(
    'created' => array('$gt' =>
        new MongoDate(strtotime('2014-06-01'))
    ));
$project = array(
    'created' => true,
    'total'   => true,
    'userId'  => true
);

$cursor = $transactions->find($match, $project);
$rows = array();

while ( $cursor->hasNext() ) {
    $dbObject = $cursor->getNext();
    $rows[] = array(
        'created'  => $dbObject['created'],
        'total'    => $dbObject['total'],
        'userId'   => $dbObject['userId']
    );
}

This isn't much different than the last example. I'm still selecting only the data points I care about, but now I've added a project clause to the find() call. This means the database driver is no longer returning all the extraneous properties in the first place. The results? On average, this call took 0.029 seconds. That's a 2,186% speed increase over our original query. And that is worth paying attention to. While my last example wasn't all that telling, this one, on the other hand, confirms my hypothesis. If you only select the data you need, and none of the data you don't need, your queries will perform better. (This is true on any database platform.) The consequence of this is that you can't really use a general-purpose POJO for your collection -- not if you want your code to perform well, that is. Instead, you might have any number of contextual POJOs that access different parts of the same collection. It's a trade-off that may prove worth it for the sheer speed.

And I had one more test, just because I was curious. Up until now I've been using the find() command to grab my data, but Mongo also has another way of retrieving data: the aggregate pipeline. I remember reading somewhere that the AP actually spun up multiple threads, whereas a simple find() call was restricted to one. (Don't ask me for a source on that, I'm vaguely remembering heresay.) So I wanted to see if simply switching out those method calls would have any added bonus. Here's what I tried:

SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);

DBObject project = new BasicDBObject("total", true);
project.put("userId", true);
project.put("created", true);

AggregationOptions aggregationOptions = AggregationOptions.builder()
     .batchSize(100)
     .outputMode(AggregationOptions.OutputMode.CURSOR)
     .allowDiskUse(true)
     .build();

List<DBObject> pipeline = Arrays.asList(match, project);
Cursor cursor = transactions.aggregate(pipeline, aggregationOptions);
List<Map> rows = new ArrayList<>();

while ( cursor.hasNext() ) {
    Map<String, Object> row = new HashMap<>();
    DBObject dbObject = cursor.next();
    
    row.put("created", dbObject.get("created"));
    row.put("total", dbObject.get("total"));
    row.put("userId", dbObject.get("userId"));
    
    rows.add(row);
}
$match = array(
    'created' => array('$gt' =>
        new MongoDate(strtotime('2014-06-01'))
    ));
$project = array(
    'created' => true,
    'total'   => true,
    'userId'  => true
);

$pipeline = array(array('$match' => $match), array('$project' => $project));
$cursor = $transactions->aggregateCursor($pipeline);
$rows = array();

while ( $cursor->hasNext() ) {
    $dbObject = $cursor->getNext();
    $rows[] = array(
        'created'  => $dbObject['created'],
        'total'    => $dbObject['total'],
        'userId'   => $dbObject['userId']
    );
}

That test ran, on average, in 0.035 seconds. That's still a 1,794% speed increase over our first test, but it's actually 0.006 seconds slower than then last one. Of course a number that small is a negligible difference. But the fact that there is no difference is worth noting. There is no tangible benefit to using the aggregate pipeline, without a $group clause, versus an ordinary call to find(). So we'd might as well stick with find(), especially considering we weren't aggregating anything, anyway.

But now comes the question of how we go about getting data from other collections. That userId is effectively a foreign key, so we need to do additional queries to get that information. (Side note: you could just duplicate the relevant information instead of, or along with, the foreign key, since that's kind of the Mongo way. But what happens when a person changes their name? This is the problem with non-relational databases.)

The code that I had originally set out to improve did something that I immediately recognized as bad: it looped over the cursor on Transactions, and for each value, ran another query to the Users collection. I refer to these kind of queries as "one-off" queries, since that's kind of what they are. Let me show you some code to better explain what I mean.

SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);

DBObject project = new BasicDBObject("total", true);
project.put("userId", true);
project.put("created", true);

Cursor cursor = transactions.find(match, project);
List<Map> rows = new ArrayList<>();

while ( cursor.hasNext() ) {
    Map<String, Object> row = new HashMap<>();
    DBObject dbObject = cursor.next();
    
    ObjectId userId = (ObjectId) dbObject.get("userId");
    row.put("created", dbObject.get("created"));
    row.put("total", dbObject.get("total"));
    row.put("userId", userId);
    
    // one-off query to the Users collection
    DBObject userProject = new BasicDBObject("firstName", true);
    userProject.put("lastName", true);
    DBObject user = users.findOne(userId, userProject);
    
    row.put("firstName", dbObject2.get("firstName"));
    row.put("lastName", dbObject2.get("lastName"));
    
    rows.add(row);
}
$match = array(
    'created' => array('$gt' =>
        new MongoDate(strtotime('2014-06-01'))
    ));
$project = array(
    'created' => true,
    'total'   => true,
    'userId'  => true
);

$cursor = $transactions->find($match, $project);
$rows = array();

while ( $cursor->hasNext() ) {
    $dbObject = $cursor->getNext();
    $userId = $cursor['userId'];
    
    // one-off query to the Users collection
    $userMatch = array('_id' => $userId);
    $userProject = array(
        'firstName' => true,
        'lastName'  => true
    );
    
    $user = $users->findOne($userMatch, $userProject);
    $rows[] = array(
        'created'   => $dbObject['created'],
        'total'     => $dbObject['total'],
        'userId'    => $dbObject['userId'],
        'firstName' => $user['firstName'],
        'lastName'  => $user['lastName']
    );
}

I can tell you that when I ran this code against my local instance of MongoDB, it took, on average, 0.319 seconds to run. That doesn't seem so bad at all, especially considering all it's doing. Again, this matches about 30,000 documents in the Transactions collection, and clearly we're making just as many calls to the Users collection. But while this seems fine on my local machine, that is not a realistic test. In real world circumstances, you would not have your database on the same server as your codebase. And even if you did, you won't forever. Inevitably you're going to need some code to run on a different server. So I re-ran the same tests using a remote instance of MongoDB. And that made a BIG difference. Suddenly this same little routine took 1709.182 seconds, on average. That's 28-and-a-half minutes. That is ridiculously bad. I will admit, though, that my wifi speed here at home is not the best. I re-ran the same test later, on a better network, and it performed at 829.917 seconds. That's still 14 minutes, which is dreadful.

Why would this simple operation take so long? And what can be done about it? Imagine this: let's say you went into the DMV office and needed to look up a bunch of records. You have a list of the records you need on a handy-dandy clipboard. So you stand in line, and once you're called to the desk, you ask the clerk, one by one, for the records on your clipboard. "Can you give me the details for Record A?" "Can you give me the details for Record B?" That's straight-forward enough, and will be efficient as it can be. But if the clerk you're talking to only has part of the data you need, and tells you you'll need to visit a different office to retrieve the other missing puzzle pieces, then it would be a bit slower.

If you're querying against your own local machine, it would go something like this:

  • Ask Cleark A for Record 1
  • Clerk A gives you everything they have about Record 1
    • Clerk A tells you to talk to Clerk B for the rest of the information
  • Leave Clerk A's line, and stand in Clerk B's line
  • Ask Clerk B for the rest of Record 1
  • Clerk B gives you the rest of Record 1
  • Leave Clerk B's line, and return to Clerk A's line
  • Repeat for Record 2...

That doesn't seem very efficient, does it? But that's the trouble with non-relational databases; disparate collections are in different places. And keep in mind that this analogy actually represents the best case scenario, where Clerk A and Clerk B are in the same room. But that isn't realistic. A more realistic illustration would involve Clerk A at the DMV office, and Clerk B located a mile or two down the road, at the Social Security office. So for each record on your clipboard, you drive back and forth from the DMV to the Social Security office. You can see why it'd be so slow. That "drive" back and forth is called network latency.

But we can do better than that. What if, instead of driving back and forth for each record, you simply asked the clerk at the DMV for all the data they had all at once, and then afterwards you compiled a comprehensive list of all the records you'd need from the Social Security office? That way, you'd only have to make the drive over there once, rather than making 30,000 drives back and forth. In Mongo, you'd only be doing two calls to the database: one bulk call to the Transactions collection, and then a subsequent bulk call to the Users collection. Here's some code to illustrate:

SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);

DBObject project = new BasicDBObject("total", true);
project.put("userId", true);
project.put("created", true);

MongoJoinCache join = new MongoJoinCache();
Cursor cursor = transactions.find(match, project);
List<Map> rows = new ArrayList<>();

int i = 0;
while ( cursor.hasNext() ) {
    Map<String, Object> row = new HashMap<>();
    DBObject dbObject = cursor.next();
    
    Object userId = (ObjectId) dbObject.get("userId");
    row.put("created", dbObject.get("created"));
    row.put("total", dbObject.get("total"));
    row.put("userId", userId);
    
    join.add(userId.toString(), i);
    rows.add(row);
    i++;
}

DBObject userMatch = join.resolveCache();
DBObject userProject = new BasicDBObject("firstName", true);
userProject.put("lastName", true);

cursor = users.find(userMatch, userProject);
while ( cursor.hasNext() ) {
    DBObject dbObject = cursor.next();
    Object userId = (ObjectId) dbObject.get("_id");
    Set<Integer> indexes = join.get(userId.toString());
    
    for (Integer index : indexes) {
        Map<String, Object> row = rows.get(index);
        row.put("firstName", dbObject.get("firstName"));
        row.put("lastName", dbObject.get("lastName"));
        rows.add(index, row);
    }
}

public class MongoJoinCache {
    private final Set<String> objectIds;
    private final Map<String, Set<Integer>> objectToIndexMapping;
    private int total = 0;
    
    public MongoJoinCache() {
        objectIds = new HashSet<>();
        objectToIndexMapping = new HashMap<>();
    }
    
    public void add(String objectId, Integer index) {
        objectIds.add(objectId);
        Set<Integer> indexes;
        if (objectToIndexMapping.containsKey(objectId)) {
            indexes = objectToIndexMapping.get(objectId);
        } else {
            indexes = new HashSet<>();
        }
        indexes.add(index);
        objectToIndexMapping.put(objectId, indexes);
        total++;
    }
    
    public Set<Integer> get(String objectId) {
        return objectToIndexMapping.get(objectId);
    }
    
    public Integer size() {
        return total;
    }
    
    public DBObject resolveCache() {
        if (size() == 0) {
            return null;
        }
        
        final BasicDBList ids = new BasicDBList();
        for (String id : objectIds) {
            ids.add(new ObjectId(id));
        }
        
        DBObject match = new BasicDBObject("_id", new BasicDBObject("$in", ids));
        return match;
    }
}
$match = array(
    'created' => array('$gt' =>
        new MongoDate(strtotime('2014-06-01'))
    ));
$project = array(
    'created' => true,
    'total'   => true,
    'userId'  => true
);

$userIds = array();
$cursor = $transactions->find($match, $project);
$rows = array();

$i = 0;
while ( $cursor->hasNext() ) {
    $dbObject = $cursor->getNext();
    $userId = strval($dbObject['userId']);
    $userIds[$userId][] = $i;
    
    $rows[] = array(
        'created'  => $dbObject['created'],
        'total'    => $dbObject['total'],
        'userId'   => $dbObject['userId']
    );
    $i++;
}

$userMatch = array('_id' => array('$in' => array()));
foreach ( $userIds as $userId => $indexes ) {
    $userMatch['_id']['$in'][] = $userId;
}

$userProject = array('firstName' => true, 'lastName' => true);
$cursor = $users->find($userMatch, $userProject);

while ( $cursor->hasNext() ) {
    $dbObject = $cursor->getNext();
    $userId = strval($dbObject['userId']);
    
    foreach ( $indexes as $userIds[$userId] ) {
        $rows[$index]['firstName'] = $dbObject['firstName'];
        $rows[$index]['lastName'] = $dbObject['lastName'];
    }
}

Phew. That's a lot more code! Well, on the Java side anyway. (PHP arrays are awesome; Java people don't even understand.) But the million dollar question is whether there is any tangible benefit to all that extra code I just threw at you. Here are the results: When running against my local machine, this ran for an average of 0.339 seconds. Against the previously-mentioned 0.319 seconds on my local machine, this is slightly slower. So why bother with all this extra code if it's slower? Because that's not really a fair test. The difference of 10 milliseconds is negligible, but more importantly, there is no universe in which you can reliably expect to have zero network latency. There will always be network latency in production environments. So a real test is against a remote instance.

So what happens when I ran this code against a remote MongoDB server? Remember that it took 28 ½ minutes before. With this code shown above, however, on average, it ran in 6.191 seconds. You read that correctly. That is an astronomical speed improvement of 27,508%. And even if your network latency isn't as bad as mine was (and it shouldn't be), you can still see nonetheless that this method will always be faster, by an order of magnitude.

If you think about it, it makes sense. We took a query which had been O(n²) in complexity and reduced it down to O(2n) -- at most. In fact, it's probably less than that in practice, since there are bound to be duplicate foreign keys. I'm guessing that the developers of relational databases do something exactly like this, under-the-hood in their database code. In order to join two tables, they probably first have to collect all the foreign keys from the first table into a set, and then grab the corresponding rows en masse from the second table. The big difference is that with SQL, you don't have to think about any of this. You just type INNER JOIN and bam, it's all magically done for you. But because Mongo doesn't include this as a concept, you have to write your own database wrapper code in order to be able to use it efficiently.

So what's the takeaway from all of this? First, you can write your own wrapper code to effectively perform a join operation in Mongo, and doing so is hugely advantageous in terms of speed, albeit slightly annoying. But it's vital to your success if you're going to use Mongo. Secondly, what I hope is the bigger takeaway is that you shouldn't have to. Like I've been saying all along, if you have to resort to writing what is essentially your own low-level database code, it's a sure sign that you're using the wrong database. If you've read this far, that means you should probably just use SQL.

TL;DR

To conclude, here are the more practical takeaways broken down:

  • Only select the data you absolutely need, and nothing more
    • Use a project clause to limit your data
    • Don't instantiate general purpose objects with every property
  • Don't do one-off queries inside of loops!
  • Minimize network latency by grabbing batches

And that is how you do a "Join" in Mongo.


MongoDB or: How I Learned to Stop Worrying and Love the Database

I come from a SQL background. For seven years, the primary database I worked with was Microsoft SQL Server. I did a little bit of work in MySQL occassionally as well, but in either case, I definitely came from the school of thought that SQL is the only way to store data. This is not without good reason. SQL is powerful. SQL is well-maintained and well-supported. SQL is stable and guarantees data integrity. And SQL is fast, when you know how to use it.

Since transitioning to a new job, though, I found myself flung into a strange new world: the world of NoSQL databases (aka document-oriented databases). In particular, a database called MongoDB. More on that in a second.

When it comes to programming, I consider myself a person who is more interested in solving problems than strict academic purity. With that said, I do tend to be obnoxiously meticulous with my code, but I would much rather throw out the super-careful attention to detail and just get something done, and subsequently tidy up/refactor the code, if it really comes down to it. And I think it is this desire to get something done that has really increased my appreciation for Mongo. Although it is not without its pitfalls, and it is definitely not the right choice in a lot of situations. However, I've been discovering that it is the right choice in a few situations where SQL is not.

My initial reaction to using Mongo was one of disgust. Reading through the design patterns, it seemed like it encouraged huge duplication of data, restricted or impaired  your access to the data, and didn't handle date-math very well at all. Unfortunately, all three of those initial impressions are true. But that doesn't mean you shouldn't use it. I won't get into the document-style approach to data because you can already find plenty of material on that yourself. It does involve duplicating data, which would otherwise just get joined in SQL. And that leads into the next point: there are no JOINs in Mongo. They do not exist; you can only query one table at a time. This is what I meant by restricting your access to data. But actually, as I've been learning, this really won't slow you down if you understand Big O notation and how to write Mongo queries effectively. And lastly, all dates in Mongo are stored in UTC time, no matter what. Mongo offers a way to group by dates, but no good way to transform that data into different timezones before grouping it. Which means your data is probably off by 5 hours.

With those downsides out of the way, let me explain why it's actually pretty cool. One of the applications I worked on at my last job was something called "Form Builder." It was actually a pretty slick application that let you create "forms." Forms could have multiple pages, and each page could have multiple questions. In other words, think Wufoo, but without all the pretty graphics. I created a few tables in SQL to build this: one for the forms themselves, another for the pages contained within those forms, a third for the questions on each page, and then one for receiving answers from users. Of course there are more tables involved than just these four, but you get the basic idea.

The "questions" table contained, among other things, the text of the question itself, an ENUM question type (represented in data as a TINYINT), and a number of different fields such as the maximum number of characters a user might enter. In the end, we came up with probably about 20 different question types, ranging from text input, to multiple choice as a drop down menu, multiple choice as radio buttons, multi-line text, asking the user to upload an image, asking the user to upload a file, a preformatted block of address fields, an email field... the list goes on. The point is, with all these disparate types of questions, each one is going to be configured slightly differently. It makes sense to limit the number of characters for text inputs. It doesn't make sense to limit the characters when it's a dropdown menu, however. It makes sense to configure an allowed list of file extensions when you're asking the user to upload a file, but not when they don't. It makes sense to configure a list of multiple choice options when you're asking a multiple choice question, but not otherwise.

What this means, when designing a database table in SQL, is that you really have two choices. Either you can have a bunch of columns that are sometimes relevant, depending on the particular row, or you can sort of abuse SQL by having some "general purpose" columns that get re-used for different purposes. These types of columns are usually either VARCHAR(8000) or TEXT, and they're considered bad practice by DBAs because you are essentially forfeiting the power of SQL by storing it in a different way. I ended up doing a mix of the two.

Here's a snapshot of the questions table:

As you can see, there's a character limit column on every field, even though not every field actually enforces a character limit. But then there's also this column called extra_info which looks like it contains a bunch of gobbledygook. That's really just a serialized PHP array, which is kind of like JSON, but not as elegant. (It predates JSON.) It's a way of storing any amount of arbitrary information in a quickly-usable-in-code format. Within SQL, storing data that way is a black box. There is no way to search it, but you store it that way because you don't need to search for it; you only need to access it. Lots of people do this, but it is considered an abuse of the database, because you're no longer really using the database.

What I didn't realize at the time, was that there was a different way of thinking about this problem entirely. This type of project was an absolutely perfect candidate for using Mongo, and I'll tell you why.

Mongo has different nomenclature than SQL. "Collections" are the equivalent of tables, "documents" are the equivalent of rows, and "fields" are the equivalent of columns. Every document in a collection is simply a JSON object. And a collection does not have any defined structure to it whatsoever. You read that correctly. It has no structure. There are no column definitions. One row in the table may have an entirely different set of columns than the next row. They're just arbitrary documents. It's all up to you. If you want one row to have one set of columns, fine. The documents are whatever you tell you them to be. They don't really care about each other.

With that said, you will end up having most of the same columns present on every document within a given collection, because otherwise there would be no point. But in Mongo, I could represent the same "questions" table above like this:

So you see here, one row doesn't enforce a character limit, so the field isn't present at all. Another row has an options array, whereas the rest don't have that column present in the first place. The difference between this and SQL is huge. In SQL, you can accomplish the same thing as I described above, but you cut off your own access to the data in the process. Mongo on the other hand is designed for this.

In Mongo, I could run a query like this, to find all the questions that have a character limit:

db.Questions.find({char_limit: {$exists: true}})

Or I could run a query like this, which will locate any questions that have an options array present, with one of those options being the word Red:

db.Questions.find({options: "Red"})

Mongo queries themselves are an off-shoot of JavaScript, which means you can use regular expressions too, which is awesome. (Although those are better suited for ad-hoc queries than actual production ones.)

From one SQL programmer to another, I would recommend giving Mongo a try. It has its annoyances, believe me. But after using it, you start to realize that SQL does too, and the realization that SQL isn't the only approach to anything and everything can be eye-opening. Having more tools in the toolbox helps make you a better programmer. Understanding when to use them, and when not to use them, is key as well. But I have to say, it's worth giving it a shot. I think you'll find there are some applications where it actually is better suited.

And by the way, if you're going to use MongoDB, you are going to need a client to access it as well. The one I've been using, the one that is pictured in the screenshot above, and the one that is hands down the best out there, is called Robomongo. The screenshot shows one of three possible ways of viewing the data (in tabular format). But it also lets you view documents in a hierarchical list, or as straight JSON text.


Why are they arguing?

The following post intersperses verses from the Bible account of Bartimaeus, from the 10th chapter of Mark, with my own interpretation of what might have been going through Bartimaeus' head at the time. Bible verses are italicized and presented in a serif font, with a summary at the end after the picture.


As he went out of Jericho with his disciples and a great number of people, blind Bartimaeus, the son of Timaeus, sat by the highway side begging.

I've been sitting here my whole life. What's going to happen today? The same thing that happens every day. People will walk by with their smug looks and feigned sympathy. I'm sick of this. Why did I have to be born this way?

And when he heard that it was Jesus of Nazareth, he began to cry out, and say, Jesus, thou son of David, have mercy on me.

Jesus... I've heard of him! This is my chance! They say he can heal the blind. They say he gives hope to widows. They say he can resurrect the dead. I need to get closer. Jesus, help!

And many charged him that he should hold his peace: but he cried the more a great deal, Thou son of David, have mercy on me.

Wait... what? Why are they arguing against me? These same people who walk by to the temple and look down on me and pretend to wish me well. Why don't they realize that this is what I need? Can they really not see how obtuse they're being? You know what, they have never done anything for me. I need to get closer to Jesus. I don't care what they say. This is my chance. Jesus, please hear me!

And Jesus stood still, and commanded him to be called. And they call the blind man, saying unto him, Be of good comfort, rise; he calleth thee.

He called me. He called me!

And he, casting away his garment, rose, and came to Jesus.

I am done with this life. I am done with this begging, this blindness, this exile within my own body. I am done with it. I am standing up to it and casting it aside once and for all.

And Jesus answered and said unto him, What wilt thou that I should do unto thee? The blind man said unto him, Lord, that I might receive my sight.

Wow. I've never felt a love like this before. That word doesn't even do it justice. I have never felt what I'm feeling now. But his question: what will you do to me? Isn't it obvious? Can't he see who I am? Can't he see what a wretch I am? I want you to heal that, Jesus!

And Jesus said unto him, Go thy way; thy faith hath made thee whole. And immediately he received his sight, and followed Jesus in the way.

Wait, he really couldn't see it. I'm not a wretch. I'm not hopeless. He couldn't see it because it's gone. I can see. This is who I've always been. Wow! I feel on fire! I can see! This is what I've always been looking for! I cannot contain or describe this joy I'm feeling! From now on I want to share this with everyone I meet. I have to share this with everyone I meet. He really does heal the blind. I can see! Praise God!

The part that always stands out to me the most in this story is the response by all the other bystanders when Bartimaeus first starts to reach out for Christ. They inexplicably resist his efforts. The question I'm always left with is the question I presented above: Why are they arguing against me? When someone is reaching out for geunine healing, what would you voice to them? Are you one of the voices in the crowd telling them "that's impossible" and essentially "get back in place"? The only explanation I have for why anyone would make these arguments is because they don't realize what they're saying. People get too self-absorbed to recognize what it is they're really communicating to others, otherwise they wouldn't say such things.

It also stems from the belief that people are made up of nothing but matter, and the consequent belief that either God, too, is somehow material, or that He doesn't exist at all. When we think of others as diagnoses, as matter bodies made up of certain proportions, proclivities, and of a certain age, those thoughts limit ourselves and others. And that way of thinking tends to discouarge others from breaking free of those limits. I think this statement by Mary Baker Eddy explains this well:

The belief that God lives in matter is pantheistic. The error, which says that Soul is in body, Mind is in matter, and good is in evil, must unsay it and cease from such utterances; else God will continue to be hidden from humanity, and mortals will sin without knowing that they are sinning, will lean on matter instead of Spirit, stumble with lameness, drop with drunkenness, consume with disease, -- all because of their blindness, their false sense concerning God and man.

The story above highlights Bartimaeus' encounter with the Christ. And it's good to realize that a genuine encounter with Christ is not something that can be faked. It's not something that can be induced by drugs or created through will-power. It's not dependent on ones attendance in church or having the right Sunday School teacher. An encounter with Christ is individual and unmistakeable. It requires both the receptivity of a humble beggar and the willingness to stand up for oneself, acknowledging that one is worthy and that there is hope.

You never know what the people around you may be going through. So today, I encourage you to watch what you say. Don't get caught up in the kind of subtle arguments that effectively tell Bartimaeus to sit back down. Don't think of people as material labels. Don't be a discouraging bystander with a narrow view of reality. See people as spiritual beings, never a mistake, never broken, never less-than. Acknowledge that each person you encounter has a unique purpose only they can fulfill. And that includes you! Be the voice of encouragement, and if you see someone reaching out for Christ, for heaven's sake, don't get in their way!

<< < Page 1 of 6 > >>