Gordon Myers

Articles on Life, Truth, Love, Computers, and Music


Offense, microaggressions, and the orb

Have you ever experienced a time when your brain messes up the order of words in sentences, and you don't realize it at first? I suppose dyslexic people struggle with this phenomenon on a daily basis, but normal people do, too, from time to time. Perhaps you've seen those posts on Facebook that say "Spot the problem" and then list some purported math formula, when in fact the equation is correct but there's a subtle grammatical error, like an extra word, or a missing letter, etc. I had one of those moments today, where my brain mixed up some words without me noticing. This moment quickly turned funny, and then got out of hand. More on that in a second.

First: some backstory. Over the weekend Donald Trump visited Saudi Arabia and found himself in a bizzare situation where he was literally touching his hand to a giant glowing orb, which you can see pictured above. Technically, it was a globe, but the sheer ridiculousness of the photo set folks on Twitter ablaze with hilarious commentary. I was reviewing some of the funny tweets while at work today, and found an article that curated some of the best. And there was one tweet, in particular, that really tickled my funny bone:

This was my dyslexia moment. Without even realizing it, my mind instantly jumbled up the word order as I read this. What I saw, at first glance, was "Orb is backwards bro." And so quite naturally, my first reaction was to examine the picture and try and see if the orb had somehow been turned around. Like... maybe it had been rotated 180° and there was something unique or funny that I had missed out on, obscured by a "dark side of the moon" effect.

But then it hit me. After longer than I would care to admit of trying to piece together what I was missing from the photo, I realized it. She wasn't talking about the physical orb at all. She was talking about the word orb. I checked again: I re-read the tweet to find that she had been saying that all along. And in parallel with this realization, it started to sink in just how funny her wordplay was. It was an intelligent commentary about the worship and awe of patriarchy. The fact that it took me so long to realize what should have been immediately obvious made it even funnier to me. In short, my own stupidity (that word's a little harsh, but just my failure to immediately recognize the joke) became kind of a meta-joke, in my own mind, enhancing the humor of the situation.

As I thought about it some more, I wanted to share. I thought that this journey from confusion to realization could more generally make the situation funnier for not just me, but anyone. So, I decided to send her a reply tweet. I wrote this:

Of course I don't know who the author of the original joke was, but I have to say I didn't really expect to hear back. My limited experience with Twitter has taught me that when you reply to strangers, especially when they're popular, your reply usually just ends up in the void. At best you get a quiet "heart" back and that's the end of it. I had no reason to expect anything different in this case. But the reaction that I got was, in fact, very different. Shortly after sending my reply (which was among dozens of others), I got a notification that the author of the original tweet had retweeted my response. But she added this snarky comment with it:

To be honest, that comment kind of set me aback. Like I said, it was not was I expecting. It was passive agressive, and kind of hostile. And it wasn't merely a reply to me; it was a retweet, which meant it was now exposed to all 14,000+ of her followers. Shortly after that, the mob descended. I received several other replies from other strangers suddenly on a crusade to expose my perceived awfulness through shame and derision.

Only... I never intended for any hostility. I offered, in earnest, what I thought was a genuinely funny commentary on an already funny statement. But context is woefully absent when you only have 140 characters. Suddenly more and more notifications starting spilling in, and they went from passive aggression to outright insults. Here's a shortlist of examples:

  • I can't find the word "actually" in this tweet
  • Did you really just #mansplain her own joke to her ?!? Really?! Really!!!!!
  • lol no thx diaper germ

To give a little context on that last one: my online handle is "soapergem," and I do happen to know from lots of experience that iPhones will constantly try to auto-correct "soaper" to "diaper." So I can only assume that the author in that case meant to call me "soaper germ" (which is kind of funny), but retained her phone's auto-correct suggestion, thinking that perhaps it's even funnier that way.

Since I hadn't expected any response whatsoever, let alone a torrent of mean tweets, it came as a little bit of a shock to the system. I was at work, and had taken only a few brief moments to share what I thought was a funny reflection on my experience. But all of a sudden I had a very hard time focusing on work, because it felt personal. I felt attacked, like my character had been misunderstood and maligned, and made a mockery of in front of 14,000 people. I was being held up as an example of male ego and condescension, and turned into a object of loathing. What helped me re-center myself were some writings from the original pastor of my church, the Rev. Mary Baker Eddy. This first quote is from an essay she wrote titled "Taking Offense."

We should go forth into life with the smallest expectations, but with the largest patience; with a keen relish for and appreciation of everything beautiful, great, and good, but with a temper so genial that the friction of the world shall not wear upon our sensibilities; with an equanimity so settled that no passing breath nor accidental disturbance shall agitate or ruffle it; with a charity broad enough to cover the whole world's evil, and sweet enough to neutralize what is bitter in it, -- determined not to be offended when no wrong is meant, nor even when it is, unless the offense be against God.

So, in short, I reminded myself that I should be determined not to be offended by this. I've been learning the importance of checking your own motives, so I examined mine. At best, I had hoped to start a short conversation. (Realistically, I was expecting no response whatsoever.) But I certainly had no intent of condescending to anyone. That totally misses the mark of my thought process. Knowing that my motives were good, I then recognized that I didn't need to take any of this personally. The mob mentality of the Internet ebbs and flows. If it happened to briefly turn against me for a day, so what?

This also got me thinking about microaggressions. This is a hot topic right now, and I began to recognize why. When someone says something careless, or misinterprets your motives, this can seem hurtful. It isn't world-shatteringly hurtful. But nonetheless, it's not very compassionate. I began to understand why schools have started to include microaggressions in their sensitivity training. Ideally, they don't want students to be treated with hostility. But generally speaking, the so-called solutions to microaggressions offered in these sessions don't seem very lasting. From what I've seen, the "solutions" revolve more around changing your privacy settings so that you can hide under a digital rock, or deleting tweets when they cause a stir.

This isn't an approach that ultimately encourages reconciliation. I would much rather have a conversation with the other person. Why have enemies, when you can have friends? I don't mean to be a broken record here, but in this particular instance I really had no intention of coming off as condescending. I'm also not sure how else I could have phrased it to fit within the character constraints and adequately convey what I was trying to say. It seems my message was perceived in a way that struck a chord with not only the author, but several of her followers as well. Condescension is just as bad as hostility in some ways, and it is unfortunate when anyone has to deal with being on the receiving end of that.

So to the original author of the joke: I would love to chat with you and hear your story. Perhaps you've had too many bad experiences with men who have "mansplained" things to you in the past, that it's become a bit of a sore spot that now makes you recoil. This post is already pretty long, so I have my doubts that she'll even see this. But I hope she can recognize that I would have used the exact same words that I used earlier today, when having a conversation with a close friend. Like I said, I was merely trying to relate a funny story on top of an already funny moment.

I'll close quickly with two other quotes that I have found inspiring, and relevant. The first is another one from the founding pastor of my church. This is from an essay she wrote, titled "Love Your Enemies."

Never return evil for evil; and, above all, do not fancy that you have been wronged when you have not been.

And this last quote is from award-winning filmmaker Cassie Jaye's recent presentation at the Institute of Noetic Sciences.

If I could give advice to anyone, and to our society at large: we have to stop expecting to be offended, and instead we have to start truly, openly, and sincerely listening. That will lead to a greater understanding of ourselves and others, having compassion for one another.


A tale of two Sauls

Prior to the election, I heard a number of conservative commentators and religious radio programs comparing candidate Donald Trump to Saul from the New Testament. For those of you not familiar, Saul was an infamous and merciless persecutor of Christians, who went through a dramatic transformation that eventually led to him become perhaps the most ardent supporter of the Christian faith, outside of Christ himself. God completely reformed Saul's character, so much so that he chose to change his name to Paul out of shame over his former self. And St. Paul went on to write nearly half of the books in the New Testament, and established lasting Christian churches where some of the other apostles had failed.

The point of making this comparison was to say that God can use anyone, no matter how morally-questionable they may be, to accomplish greatness. And as Donald Trump has appropriated the word "greatness" for himself, it's not surprising that his supporters eventually drew this comparison. However, as someone who's actually read the Bible, I have to say that this comparison is utter garbage, and anyone who makes it might just have scales on their eyes, blinding them to the truth.

The reasons why this comparison fails are manifold. First, even before Saul of the New Testament went through his remarkable transformation, he was already what you might call a Biblical scholar. Of course the Bible as we think of it hadn't yet been written, but there were the Jewish laws of the Torah, the Psalms, and the other Old Testament stories of King David, Elijah, and all the other prophets - all of which Saul knew like the back of his hand. Saul was a Pharisee, or what they called a "doctor of the law," so he had carefully studied and examined these texts since boyhood. His religion and intellect informed his every decision, and while immoral, nothing he did was irrational. In other words, he had tremendous intellectual prowess, and basically had his PhD in Hebrew and Old Testament Studies. On top of this, he was a powerful and persuasive speaker, motivated by the same youthful zeal that has led revolutions to success. Powerful and persuasive public speaker, historian and scholar, and highly respected religious authority... do any of these sound like monikers you would ascribe to Mr. Trump? The answer is obvious.

But okay, even though their backgrounds are very different, isn't the message of God calling the most unlikely sinner to greatness still a valid argument, with regard to Mr. Trump? There are two reasons why I say, resoundingly: no. First, to witness the reformation and transformation of a sinner into God's chosen requires that the sinner, y'know, actually reforms. Donald Trump has been utterly unrepentant, as a symptom of his own vanity. Baptist Pastor John Piper penned a piece titled How to Live Under an Unqualified President, in which he enumerates a list of reasons why Trump is immoral. (Side note: I was a little annoyed at how Pastor Piper quietly dismissed Hillary Clinton as equally unqualified without any real discussion, because while she definitely had issues with integrity, he is tacitly promoting a very large false equivalence there).

Similarly, Jesuit Priest James Martin has a piece titled I was a stranger and you did not welcome me in which he explains that the policy ideas which Trump has put forth are, in a very real sense, the anti-Christ. And Pope Francis famously declared that Trump's attitude demostrates that he is not a real Christian. Of course none of these criticisms would matter if they meant this was all leading up to a great transformation of character, that will help usher in God's kingdom. But that's not what the Trump supporters on the radio were talking about, and certainly not what Trump is interested in doing. The message of Paul's transformation has somehow been changed from genuine, heartfelt repentance... into brushing aside any lapse in moral judgment, so long as the offender is wearing the right team's colors. They voted for the immoral Saul, with no serious demand to ever see him changed into Paul.

Secondly, and even more importantly, is that Saul of the New Testament was not transformed by a vote of the people; he was transformed by the Holy Spirit. We don't get to dictate how/when the Holy Spirit works, or campaign for God's plan to occur only within our predetermined framework. "His ways are higher than our ways," as Isaiah says. Saul of the New Testament already had the approval of human institutions when he was authorized to round up and slaughter members of the Christian sect. It was not man's will, or the electoral college, that changed him. It was the genuine power of Christ. We cannot be so vain to imagine that a human popularity contest has the same power to compel a real change of heart, least of all from someone who's spent their career craving attention. While I do believe that God is changing and molding the characters of all of us, all the time, He does so on His schedule, not on our election cycle.

So to any of my Christian friends who may have been enticed, during the campaign, by this comparison of New Testament Saul to Trump, I ask you: can you imagine that God could have used someone like Hillary Clinton to do His will? "Have we not all one Father?"

Having categorically brushed aside this comparison between Trump and NT Saul, I will concede that there may still be a valid Biblical comparison to be made. If I had to pick between the two, I'd say Trump is a lot more like Saul of the Old Testament. Old Testament Saul, aka King Saul, is not nearly as well known as New Testament Saul. But his story is fascinating in its own right. It begins by explaining that OT Saul was the son of an incredibly wealthy and influential father - in other words, he came from money and grew up with a silver spoon. The story continues by explaining how he spent his youth essentially chasing tail (or "seeking asses," as the Bible puts it), with no strong sense of purpose in life. Then, through a series of unexpected events, Saul of the Old Testament becomes king of all Israel, to the complete surprise of everyone - including himself! And even after having accepted this prestigious appointment, he shows a reluctance to lead, and instead returns to continue to run his family business, tending to his field.

When King Saul eventually does take up his mantle, he displays utter disregard for even his own most-trusted advisors. And his brutal foreign policy, while at first appearing successful, eventually leads the kingdom into division and disarray, embroiling them in conflict with the Amalekites for generations. Moreover, in the face of an actual terrorist threat, OT Saul proved to be utterly unqualified and helpless. When the Philistines besieged them, Saul could do nothing, and it was his eventual-successor David who had to clean up the mess for him. David became a war hero, and while at first Saul congratulated him for this, he quickly grew jealous of David's renown. He went on to insult, betray, and persecute the war hero unjustly. And famously, King Saul struggled with irrational and erratic outbursts that became increasingly difficult to manage. He refused to let go of perceived slights and pursued his offenders relentlessly, blinding himself to reality.

The story of Saul of the Old Testament quickly starts to resemble watching a train run off the tracks, in slow motion. As his reign continued, Saul became increasingly unhinged. His mind started to unravel. And this happened at the alarm of his closest aides, who proved to be incompentent themselves and unable to deal with their troubled leader. Eventually, Saul did himself in, committing suicide when he was forced to come face-to-face with his own failures. I sincerely hope President Trump is never inspired to commit suicide. But outside of that final point, which of the two comparisons do you think is more fitting?

Am I being a little bit unfair by making this second comparison? Perhaps. (I certainly employed a few clever plays on words.) But just as I believe King Saul was doing his best, but ultimately was not the right man for the job, I can concede that Mr. Trump has what he believes are good intentions. But if the history of King Saul taught us anything, it's that unhinged leaders are unreliable and eventually and inevitably bring about their own demise.


Is God a person?

This is a question that many people have asked throughout the ages, and one that I've heard a lot myself. And as a Christian Scientist, it's sometimes been a point of contention with some of my Christian brothers and sisters, whether I mention anything or not. Recently, after an otherwise-excellent first date ended with the unprompted comment, "I'm sorry Gordon, your views on the Trinity are just too different," it's been something on my mind.

For those of you unfamiliar, the Trinity, or Godhead, is the popular Christian concept that God consists of three-persons-in-one: the Father, the Son, and the Holy Ghost. This same concept is represented more succinctly by the slogan, "Jesus is God." This is a commonly accepted belief in many Christian churches, now considered orthodox, but it was not always as widespread as it is today.

I've been reading a fascinating history book called When Jesus Became God by Richard Rubenstein. The book covers the history of the "Arian Controversy," which includes the famous Council of Nicaea, at the end of the 4th century. This was the point in history when various Christian Bishops came together to flesh out official church doctrine, and this was when they formally consented to the idea that God and Jesus are homoousios, a Greek word borrowed from pagan philosophy, which means "the same stuff." I find it fascinating that any meaningful sense of consensus was largely absent from this council. The Christian community was essentially split down the middle on this idea, and quite a lot of politics, human posturing, and personal amibition went into it. History also shows that this one simple decision quickly reverberated with a lot of unnecessary and ironically un-Christian violence.

Now I don't mean to reignite a centuries-old turf war. But regardless of what church councils "decided," or what's considered popular today, I'd like to examine the question of who God is, and how Jesus fits into that picture, from the perspective of what the Bible actually says and what the teachings of Christian Science reveal. As Isaiah says, Come now, and let us reason together.

Hear, O Israel: The Lord our God is one Lord.

The First Commandment of Judaism and Christianity states that there is one God. What does this mean? Clearly this is opposed to atheism, which says there is no God. And it's similarly opposed to pantheism, which says there are many gods. It's generally accepted, also, that God has the following four properties:

  • God is omnipotent, meaning He's all powerful / there's nothing He can't do
  • God is omniscient, meaning He knows everything
  • God is omnipresent, meaning He fills all space
  • God is omnibenevolent, which means He is all good

Those "omni-" words don't appear directly in the Bible, but are they supported by it? Does the Bible imply that God is omnipotent? Well, I can tell you that no less than four times, Jesus said, "all things are possible to God." Check! How about omniscient? In 1st John it says that "God is greater than our heart, and knoweth all things." Check! Omnipresent? Look no further than the poetry of the Psalms: #139 essentially says yes. Lastly, is God good? Well, if he wasn't, it'd be pretty scary praying to Him! But if you're looking for Biblical support, Jesus said, "Why do you call me 'good'? There is none good except God alone."

Now any freshman philosophy major can explain to you that there's no being who could possibly exist in our physical universe that truly has all four of those properties at the same time (and they'd be right), because of something called the problem of evil. But today's post is not to try and dissect that paradox -- that's a whole other can of worms! So for the sake of argument, we'll just take it on faith that God is indeed all-powerful, all-knowing, omnipresent good. Question: can an all-good, all-powerful, all-knowing, omnipresent being be a person?

Again, let me reiterate that lots and lots of people -- many of them smarter than me -- have already thought through this same question over the ages. In fact, one example of someone else who's thought about this question a little bit is Carl Sagan, the famous astrophysicist. Here's a quote from him:

The idea that God is an oversized white male with a flowing beard who sits in the sky and tallies the fall of every sparrow is ludicrous. But if by God one means the set of physical laws that govern the universe, then clearly there is such a God. This God is emotionally unsatisfying. It does not make much sense to pray to the law of gravity.
--Carl Sagan

We seem to be at a crossroads.

On the one hand, the orthodox Christian belief of Jesus as God provides a lot of comfort. To think that there's an infinitely powerful personal friend, ready to help you out of trouble and save you from despair in this physical life, is understandably appealing. But under the microscope of rational thought, inconsistencies start to appear. And if there's some additional qualifications that come with this (i.e. do/say the right thing or you're damned), one starts to question the "omnibenevolent" part. Dig deep enough and things might even seem "ludicrous," as Mr. Sagan put it.

On the other hand, what if God is simply the name for all the principles and laws governing the physical universe? Well then we can see that these are ever-present, and all-powerful by definition, but it just seems like such a cop-out. Those laws don't seem very intelligent, and certainly not very good. Instead it all seems chaotic, unknowable, and feels so cold. It certainly doesn't meet our criteria of the four "omni"s.

But what if I told you that there was a simple mistake with both of these approaches, and indeed, there's a third answer? On the one hand, if you start by saying, "I see, feel, hear, etc. with my material senses, and my material body is who I am," and then you remember reading in the Bible that "God created mankind in His image and likeness," you might naturally think, "well that means God must be just like me -- a material body. And hey, Jesus had one of those! Therefore Jesus must be God." Or on the other hand, if you start by saying, "I see, feel, hear, etc. with my material senses, and my material body is who I am," and then you study the scientific method, you might say, "everything is made out of matter just like me, and there certainly seem to be patterns to what I can observe in the physical world, maybe these patterns are God."

Both of these approaches reason out from the evidence of the material senses. Both start with the idea that I am material, therefore practically everything is material. You try to make God in the image and likeness of yourself. Matter is what we observe with our senses, but the Bible says God is Spirit, and so I think we need to start there. But that's hard, because spiritual things are un-quantifiable. What's the average mass of a mother's love? What's the velocity of it? Its existence is obvious, yet it extends far beyond chemicals and neurons. It can be difficult for people, at first, to reason about things spiritually, because we're not always used to it. But if you don't immediately accept the idea that you are made out of matter, and consider for a moment that instead you are spiritual, then the reasoning goes a little differently. Here's what Mary Baker Eddy, the Founder of Christian Science, had to say:

Human philosophy has made God manlike. Christian Science makes man Godlike. The first is error; the latter is truth. Metaphysics is above physics, and matter does not into metaphysical premises or conclusions.

Christian Science strongly emphasizes the thought that God is not corporeal, but incorporeal -- that is, bodiless. Mortals are corporeal, but God is incorporeal.

As the words person and personal are commonly and ignorantly employed, they often lead, when applied to Deity, to confused and erroneous conceptions of divinity and its dinstinctions from humanity. If the term personality, as applied to God, means infinite personality, then God is infinite Person, -- in the sense of inifinite personality, but not in the lower sense. An infinite Mind in a finite form is an absolute impossibility.

These are just some of the ideas that I'm going to be sharing during a short church meeting this Wednesday, August 24th, here in Madison. I'll be presenting further readings from the Bible as well as more of Eddy's exposition on it, on this same topic, "Is God a person?" If you're interested in finding out a little more, please join me at 7:30 this Wednesday evening.


My Pokemon Go Wishlist

Like many, I've been sucked into the smartphone craze that is Pokémon Go. The nostalgia of reliving a game many of us played 20 years ago, combined with the pride of collecting as many little critters as you can, combined with the inspiration of wandering around your city and discovering hidden gems makes for a potent formula for success. Still, despite the game's immense and immediate popularity, I can't help but noticing that it's not exactly feature-rich. It's definitely an underdeveloped app, which isn't helped by the server-side struggles they've had to keep up with demand. Today I will present my wishlist of the top 3 features I'd love to see rolled out in future iterations of Pokémon Go.

1. Attacking before Capturing

One of the fundamental game mechanics that you learn early on in every other Pokémon game (and there are quite a lot!) is the importance of weakening new prospects by attacking them first before trying to capture them in poké balls. That mechanic is sadly absent from Pokémon Go. It's somewhat made up for with the use of Razzberries, but only somewhat as they often prove ineffective. I'd love to see future iterations of the app allow your existing critters to attack the wild Pokémon before attempting to throw a poké ball at them, with some sort of HP gauge on screen. That would be more true to the original. They could even introduce different consequences depending on the (predetermined) disposition of the Pokémon: perhaps certain animals would stay and fight while others might immediately try to flee when attacked. There are lots of possibilities, but right now it feels like a big missed opportunity.

2. Leveling up at gyms

In the real world, when you go to the gym to workout, you leave feeling stronger. In Pokémon Go, gyms don't exactly work that way. Presently the only way to strengthen your Pokémon is outside of the gym, by feeding them candy. This message seems incongruent with reality. I think the longer a Pokémon remains in a gym the stronger it should be. Or at the least, upon every successful defense of a gym, it should gain some additional CP. Right now gyms are little more than spectacles of pride, but the game would be much improved if they were more, y'know, gym-like.

3. Logging in

One feature that they should consider including in future iterations of Pokémon Go is the ability to log in. I really feel it was a big miss on Niantic's part by not including this feature right off the bat. It can be very counterproductive when you want to play Pokémon Go but you're unsuccessful logging in and thereby are prevented from playing Pokémon Go. It's really a strange game mechanic in which you're not allowed to play the game in the first place, and seems like a pretty glaring oversight on part of the developers. Hopefully, in future releases of the app, there will actually be a functioning app to play. That would seem like a great feature to include.

So that's my wishlist for Pokémon Go. What do you think? Do you have ideas of your own for features you'd like to see? Leave a comment below!


It's the Star Wars movie we need right now, but not the Star Wars movie we deserve

This morning I saw Star Wars: The Force Awakens in theaters. It was everything everyone expected. It lived up to the hype. However I'm not writing this post to join the chorus of accolades; there will be enough of that and indeed there already is. This post assumes you enjoyed the film and are ready to move on to a healthy level of scrutiny. So keep in mind that as I talk about the flaws of the film, I do so earnestly from the perspective of someone who genuinely enjoyed the movie and will probably be back to the theaters to see it again. With that said, there were a couple of things about this film that bothered me.

Star Wars is something that a lot of people have very strong and passionate feelings about. For years it was carefully controlled and guided by George Lucas, another thing altogether that people have strong and passionate feelings about. Since Lucas sold the rights away and the mantle was picked up by J.J. Abrams, it's fair to say that people were nervous he might add too much lens flare, or otherwise not get things right. I'm simultaneously pleased and conflicted to say that J.J. did get it right. Conflicted only because he didn't actually make a new movie at all. The Force Awakens is a play-by-play, carbon copy of A New Hope, down to the smallest detail. Some have even gone so far as to call it plagiarism. I simply call it "playing it safe."

My roommate, who has not yet seen the film, just asked me as I was writing this if I thought the new film was better than Episode Four. That's a question I can't really answer, because it is Episode Four. This film had about as much new material as The Hangover 2 did. It's an exact replica of the original film, just with younger characters and bigger objects.

But that isn't really what bothers me. The new film was exactly what the masses needed most after the disaster that was the prequel trilogy. They had felt betrayed by their beloved director. How could the same man who gave the world Luke Skywalker, Han Solo, and Darth Vader be the one who made Anakin Skywalker, Jar Jar Binks, and midi-chlorians? Alas, as the years went by, Lucas spiraled further into delusion and denial like the hapless CG-junkie that's he's revealed himself to be. And the fans felt the sting of betrayal. The galaxy far, far away had been forever tainted. A Force Awakens calls back to the earlier days of greatness. It pays homage to its roots because it is its roots, with little more than a new face.

However, just as Kylo Ren couldn't shake his Dark Side heritage, the film couldn't shake some of the series' darker memories itself. I'm talking about Hayden Christensen's acting, newly manifest in Daisy Ridley. I know that people are going to praise her performance, and while I do believe she is leagues apart from Mr. Christensen, I still saw sparks of the dark side in her. And I don't mean the force.

In fact, I'm sure that both she and Abrams will be praised: Ridley for her acting, and J.J. for including a strong female lead. I don't buy it though. The first felt forced at times and the latter felt like affirmative action. If feminism in the 21st century has taught us anything, it's that a surefire formula for accolades in the film industry is to have a fiercly independent female lead. People think it's refreshing to have a damsel who's not in distress, who doesn't need a man to save her, and who proclaims her independence at every turn. I don't have a problem with a strong female lead, even though I think it's already become an overplayed trope years ago. But I do have a problem when the only motivation for that choice is pandering, and when the so-called strength of that lead wanders into overcompensation territory, as if it's apologizing for years of other films.

Case in point: Rey's insistence that she doesn't need Fin to hold her hand on Jakku. It was meant to highlight her independence, but it felt robotic, unrealistic, and needlessly rude. That's not how people interact in the real world. And her change of heart toward Fin, later in the film, comes too fast. It makes the character seem flaky and inconsistent. And when Fin is trying to do all the things a conventional male hero is supposed to do, she loves him for it. Now, granted, an argument could be made that Abrams was trying to reincarnate the same cocky, push-and-pull chemistry that Leia and Han perfected years ago into a new generation. But if the prequel trilogy taught us anything, it's that you can't force romance. No matter how many times Anakin said he loved Padmé, we never believed it, because tell-not-show storytelling is never believable.

There are other examples, too, that are a little more glaring, other glimpses where you see her channeling her inner Christensen. Watch Rey's expression as she becries Solo's death. The scene is parallel to when Luke is upset over Obi-wan's death, with only one little thing different: she's only known Han for a matter of hours, whereas Uncle Ben was family to Luke. Yes it sucks, but she's clearly over-acting.

But all of these minor quibbles are still dwarfed by my real complaint with the new film. My main complaint is not really about the film, not anything in it. I had this complaint long before ever seeing the film. My major gripe is about the fact that film exists in the first place. Because its mere existence invalidates the Prophecy of the Chosen One. No, invalidate is too soft a term. It utterly and completely destroys it. In layman's terms, it removes the need for any of the last six films and renders them all pointless.

The Prophecy of the Chosen One was an ancient Jedi prophecy that foretold the coming of a being who would forever bring balance to the force. This being was revealed (and confirmed) to be Anakin Skywalker, who "forever" destroyed the Sith and eliminated the influence of the Dark Side when he took out the evil Emperor Palpatine and sacrified himself as well. This prophecy was the entire point of the original three films, and further reinforced by their prequels.

The trouble is that if evil has been destroyed for good, and the prophey has been fulfilled, that doesn't leave room for any future villains or really any room for future films or literature. Peace is boring. Peace isn't dramatic. Star Wars is popular because the wars make things interesting. I'm ignoring the hoardes of expanded universe novels when I say that Return of the Jedi was meant to be the end. But I have yet to hear any satisfactory explanation as to why the Prophecy of the Chosen One isn't proved to be complete garbage by the re-introduction of the dark side after the fact.

Just to wade through the pedantry a bit, no, the phrase "bring balance to the force" does not simply mean to equalize the number of Jedi with the number of Sith. Lucas himself has confirmed that it meant the elimination of the Sith, and the dialogue in the prequel films supports this. The Jedi believe that the only truly "balanced" state of the Force is when the Dark Side is totally absent. The other main argument, or should I say rationalization, is that balance is inherently temporary. Well, what was the point, then? Why bother prophesying anything if it's all going to be meaningless 30 years later?

All these retconned delusions serve to do is spit in the face of the original story. The Force Awakens is more than just an innocuous reboot -- and it is a reboot in the truest sense of the word, not a sequel. With all its careful orchestration, tribute, and nostalgia, it repents of the the prequel trilogy, but only by selling the soul of the beloved original trilogy.

It had to do that, and I understand why. Although the heritage of the Jedi, explored thorougly in the Knights of the Old Republic, is rich and lasting, it isn't as marketable as Luke Skywalker or Han Solo. It had to be a sequel, not another prequel. What people needed was redemption from the Hollywood disasters that were the prequel films. And that's exactly what they got. But, unfortunately, underneath it all, no one seemed to notice that this redemption came with a price. The original trilogy was now all in vain, as we start over with fresh faces.

And the reason that's a problem is because, at its core, the original story (prequels included!) was good. If you don't believe me, watch any of the What if Star Wars Episode X was good? videos on YouTube. The heart of the story was always pure, even if it did end up with a fat body and an ugly face. And although this new story is great (how can it not be when it's a copy?), I am disappointed that The Force Awakens threw the baby out with the bathwater. The good story-telling that Lucas intended was the Star Wars that we deserved. But Star Wars: A New Hope, Mark 2 was what the people needed.


My Mario Maker Wishlist

I bought my copy of Mario Maker bundled with my brand new Wii U console the day the former came out. I have always been a fan of Mario, but an even bigger fan of puzzle games. And the fact that Mario Maker was a way of blending those two concepts made it too hard to resist. And there are some wonderfully creative levels out there. With that said, although Mario Maker gives level designers a fairly broad palette, overall the game still seems a little premature to me. So without further ado, I present my wishlist of the top three things I'd like to see added to Mario Maker.

#1. Better Level Searching

The interface for finding levels seems like little more than an after-thought in the mind of the developers. Understanding how looming deadlines and high demands for features probably pushed the level choosing interface to the back-burner, I can sympathize. But now that the game's out, I'd really like to see (preferably in the form of a free update) some kind of revision that will make searching for courses and course creators easier. This doesn't seem like a tall-order to me. Really, we just need to two things:

  1. Search for courses by name
  2. Search for authors by name

That would make a world of difference. Right now you need to know the obscure 16-digit course IDs in order to find anything, which means you really have to want it. Nintendo, please make this easier for all of us.

#2. Conditional Power-up Blocks

I'm really surprised that Nintendo chose to do power-ups the way that they did with Mario Maker, because it broke all past conventions. In every Mario game ever, when Mario's small, a power-up block would yield a mushroom. Once he's big, it might yield a different power-up like a fire flower or a feather. Or it might not. But in any case, being big was a prerequisite to getting the more "advanced" power-up. I'd like to see this element brought back because I think it adds some challenge to the game. Instant fire flowers cheapens things a bit.

#3. Vertical Levels

Nintendo gives you quite a lot of space to work with horizontally, but only two screens maximum of vertical space. I really wish they had included some sort of toggle for horizontal and vertical mode in much the same way that Microsoft Word lets you choose between portrait and landscape mode. Some folks have already tried to emulate this by using doors and warp pipes to make tessellated faux-vertical levels, but it's really not the same. Think auto-scrolling. How great would it be to have a level that makes you climb with the risk of being swallowed up by the auto-scrolling bottom?

Of course this raises a different concern, namely that people would want vertical levels that you could descend. And if you have two possible positions for the start and end, doesn't that mean you'd also need to allow for horizontal levels that move left? Well, yes! I think that would be a delightful side effect. Especially since some folks are already doing this in their sub-levels anyway.

What do you think about my wishlist? What things you would add? I purposely left off a lot of items that are, well, items, because I wouldn't be surprised to see some of that coming in the form of DLC. But let me know in the comments what you think.


Spirited Away

Today's blog post is going to be a movie review. This isn't a current movie by any means, either. In fact, it's 14 years old. It's also a film that I would expect many of you have already seen. But recently I felt an urge to rewatch it, and discovered a few things in the process. First, I realized that there are still many who haven't seen this film, or haven't even heard of it, and that's a shame. But more importantly, I realized some of the great lessons this film has to offer. I am talking of course about the film Spirited Away by Hayao Miyazaki.

For those not familiar, Miyazaki is a Japanese animator who makes kids' movies. And his films are kind of the gold standard in Japan. They're basically Japan's equivalent of Disney movies. In fact, if I'm not mistaken, Disney actually bought the English version rights to all of his films. Spirited Away in particular is one that I consider to be the crème de la crème of his work, and I'm not alone. It is the most successful Japanese film to date and won an Academy Award.

The story of the film is fundamentally one about a young girl overcoming her own fears and limitations in order to free her parents from the spell of an evil witch. It's also a story about friendship and love. And it's a story that draws heavily on Japanese mythology, featuring dragons, witches, talking frogs, giant babies, and everything from river to radish spirits. The artwork is spectacular, the story is unique, and the messages of the film are splendid. But as I was rewatching this film the other day for the upteenth time, one point in particular struck me as to what makes this film so incredible: there are no villains.

Right away people who have seen the film may dispute this fact. They'll remind me that the evil witch, Yubaba, is the story's central antagonist. And to that I'd say: no, she isn't really. Not really, anyway. At its core, the world of Spirited Away doesn't really have any villains. For me, that's what makes it truly so remarkable. But I need to explain and defend my position of why there really aren't any villains in this film. Please note that spoilers will follow, but this film is good enough that they won't detract at all from watching the film, so please keep reading.

I'll start by debunking the example already named: Yubaba is not a villain. What is it that makes her evil? Well, within the first 10 minutes of the film, she turns the main character's parents into giant pigs. Except... she doesn't actually do that; the parents do that to themselves. They overzealously decide to help themselves to plates full of food left out at what appears to be an abandoned theme park, but is in fact actually an enchanted spirit village. Yubaba isn't present or even aware of the humans at that point; the food is simply not meant for humans with the unfortunate side effects is that it automatically turns non-spirit diners into pigs. Yubaba didn't take any action at all here; it was the parents' own greed that resulted in the curse.

But Yubaba does decide to keep the parents as pigs so that she can raise them for slaughter. And this is the central plot point which the main character, Chihiro, is working to prevent. Yes, this seems evil, but I'm hesitant to really label it as being downright evil, and instead call it a callous business practice, not unlike many of the seemingly callous business practices that we see in the real world. It's not a malovelent, calculated plan; it's simply a matter of convenience. In the eyes of Yubaba, human beings are an entirely different species. They are like animals to her. And what meat company wouldn't freely take advantage of unclaimed livestock that wandered onto their premises? It's not always pleasant to talk about it, but it doesn't make her a fundamentally evil person.

So rather than examining Yubaba's inactions, let's look at her actions. Within the first 30 minutes of the movie, she hires Chihiro to work at the bath house. Chihiro asks her for a job, and she agrees to give her one. In fact, Yubaba has sworn an oath that if anyone, regardless of skill, circumstance, or ability, asks her for a job, she will always grant their request. And not merely on a probationary or provisional basis either; she offers permanent employment to anyone who asks. Does that sound evil to you? You'd be seriously hard-pressed to find that same kind of generosity from even some the best companies!

During Chihiro's employment, at several points, Yubaba compliments Chihiro when she does her job well. Moreover, Yubaba offers a path for Chihiro to follow that will ensure the safe release of her parents. And she keeps her promise. Again, what's evil about any of that? By the end of the film, Chihiro embraces her employer with a hug of gratitude, sending home the message that there aren't villains.

Another character to consider is a spirit called "No Face." Initially a meek, speechless spirit that is denied entrance to the bath house in which all the other spirits partake, this character takes notice of Chihiro and follows her in through a back door. It's shown just how lonely this character is, and how it will do anything to appease others and win their praise and affection. Ultimately it's revealed that this character's people-pleasing tactics are fundementally selfish, as it only acts out the behaviors it thinks others are desiring in order to fill a void. And when it still doesn't feel satisfied, it starts eating the other characters alive, which in turn triggers an increasingly insatiable hunger.

In short, No Face is a character that represents loneliness and lust. It starts out innocent, but due to its hunger for affection, it grows into a disgusting monster that consumes everything and everyone in its path. It does so by preying on the weaknesses of others, but only through illusion and manipulation. In fact, at the zenith of its lust, there is a scene where Chihiro confronts No Face and it is revealed just how desperate the creature is for Chihiro's validation. After having assaulted and consumed half the staff at the bath house, he nearly does the same to Chihiro. That's pretty evil, right?

But let's examine Chihiro's response to this situation: she sits patiently, quietly in front of him, completely unafraid of his condition. She has absolutely no fear. While every other character on screen is either frantically running away or trying to lock the doors, she enters the same room and calmly sits down beside him to chat. When he charges at her signaling he might try to eat her, her first instinct is to help him. She says, "before you eat me, eat this," and then hands him some magical medicine. The pill then triggers a violent reaction in which everything bad in his system is flushed out.

During this reaction, she runs away from him, which makes for a great chase scene. But as soon as that is over, No Face returns to his original form, and starts to follow Chihiro once more. At this point, she stops running away and invites him to sit next to her on the train as a friend. One of the supporting characters is initially shocked that she would let him anywhere near her, but Chihiro responds simply by saying, "I think the bath house makes him crazy. He needed to get out of there."

This, to me, is actually the most telling line of the whole film. This confirms why the message is so powerful, that there are no villains. Even after having been nearly assaulted by this character, she calmly recognizes that it is not the character, but the circumstance, that was the real problem. Chihiro not only helps him out of the unfortunate circumstance, but immediately befriends him and helps draw the best out of him, later helping him to get a job as a personal attendant of another character.

There are more examples, too. Zeniba could be seen as villanious, at least temporarily, for having nearly killed one of the other characters, Haku. But she quickly becomes a fast friend and supporter of both Chihiro and Haku. The giant baby could be seen as villanious, but after being forced out of his spoiled environment, he becomes an ardent defender of Chihiro as well. Even many of the spirit workers could be seen as evil, or at least as dangerous, for their xenophobia of humans, especially at the beginning of the film. But that too is overcome as they work beside Chihiro and begin to appreciate her.

The only villains in Spirited Away are intangible. Qualities like greed, lust, hatred, fear, envy, and theft are the villains of the film. These qualities are acted out, for a time, by various characters. Even Chihiro herself shows a lot of fear at the start of the film. But all of these qualities are overcome, either through the characters' individual growth, or through the help and support of friends. This is a truly powerful message. There are no evil characters; everyone can be redeemed.

Contrast this with even some of the most treasured Disney films. In Lion King there are clearly good characters and bad characters. There is no redemption for Scar. At the end of the film, he is eaten alive by his own hyenas. In Aladdin, there is no hope for Jafar. He is forever banished to a tiny prison. This is actually a pretty common message with most Western Disney movies: there are good people, and bad people, and the bad people should either be killed or exiled.

In Spirited Away, there are just people. (Well, sort of. If you count frogs and radishes as people.) These people sometimes act out bad qualities, and sometimes it can seem pretty scary, but even this can be redeemed by not giving into fear and persistently loving and supporting them. In the world of Spirited Away, gratitude is given even to those who were once seen as enemies. There are no villains. I think we can learn a lot from messages like these. And it sure would do some good to have more movies like that.


Purpose: Letting your brilliance shine through

Today I watched an hour-long video on YouTube. I know this seems like an impossible feat since the attention span of most of the YouTube audience (myself not excluded) is like that of a goldfish. But I was feeling hungry for something substantive tonight, and I don't just mean long.

For a few years now, there have been full, hour-long Christian Science lectures posted on YouTube. If you've never seen one of these before, I really recommend them. They cover a variety of topics and don't let the word "lecture" deter you or imply that they're dry. These are kind of like TED talks, but better. And there are some shorter ones posted, too, to give a sampling of what these are like without having to commit a full hour.

Tonight I watched one titled Purpose: Letting your brilliance shine through by Tom McElroy. The great thing about Christian Science lectures though (in my opinion) is that regardless of what the topic purports to be about, there is something there for you. There is some insight that only you will uniquely glean from listening to the lecture. For instance I wasn't particularly focused on finding my own sense of "purpose" this evening. I actually was stood up for a date so my focus was very different. Even so, I found some valuable, comforting ideas that spoke to me.

Most of all, I think Tom in particular gives a great introduction to Christian Science if you don't know much about it. So if you can spare an hour, I'd really recommend watcing this video. At least add it to your "Watch Later" list. Here's a short excerpt from the video:

To say that there's a science to it [to call it a Christian Science] is to say that if there was ever any truth in that love [if there was ever anything real, anything that actually happened there, anything that ever really took place in the healing, in the love, in the vision; if that was based on anything,] then whatever the truth was behind it [the science of it, the reality of it, the truth of it] still has to be true today. It's timeless. It has to be true for all people under all circumstances. [It has to apply to all people equally.] It's not something we earn or work our way up to and it's not something we can mess up.

Check it out for yourself! I promise you you'll learn something new.


Hey, I made a thing

I just published my very first Chrome Extension to the Google Play Store! It's actually not hard at all, and I'm going to come back here and beef up this post with more of the details when I have some more time. But in a nutshell, if you use Atlassian's Bitbucket service, you might be familiar with "pull requests," otherwise known as code reviews. These are immensely helpful, but sorely lacking one critical feature: there's no way to easily expand/collapse whole sections of a pull request. So, I added that feature! But I did so in a really lazy way. Rather than taking the time to figure out how to hook into the loaded events, I just a gigantic button on top that says "Active Toggle." Click that button once all the files have been loaded up, and it will then add individual "Toggle" buttons to each section. Enjoy!

https://chrome.google.com/webstore/detail/bitbucket-pull-request-to/hfebajohpclnfhfnlhgndbmcdnlchjjd


How to do Joins in MongoDB

If you've come here looking how to perform a JOIN operation on two or more collections in MongoDB, you've come to the wrong place. In fact, that is exactly what I'm about to show you how to do, but trust me, you shouldn't be doing it. If you're trying to do that, that's a clear indication that you have relational data. And if you have relational data, that means you've made the wrong choice for your database. Don't use Mongo; use a relational database instead. I know it's cool and shiny and new, but that is not a good rationale to use it. SQL is your best bet. I'd recommend you read Why You Should Never Use MongoDB by Sarah Mei. Now, with that disclaimer out of the way, I'll dive right into it.

We've been using MongoDB for some of our data storage at work. The choice to use Mongo was a decision made well before I arrived, and incidentally, we might be changing that out at some point in the future. But nevertheless, as it's the system currently in place, I've had to work within that system.

There were a number of pre-established Mongo queries in the codebase I've been working on, and I'm sorry to say many of them were really quite slow. So over the course of a weekend, I tried out a couple of ideas that seemed intuitive enough and managed to speed up some of these common queries by an order of magnitude. The queries I'm talking about grabbed data from multiple collections simultaneously, hence why they were initially so slow, and hence the title of this blog post. In this post I'm going to dissect a number of the techniques I used to speed up these Mongo queries, with plenty of code examples along the way.

Let's say you've got a collection in Mongo called Transactions. This table has a variety of fields on each row, including one field called userId, which is just the ObjectID (aka foreign key, for you SQL folks) of a document in the separate Users collection. You might want to retrieve a list of transactions in a given time period, and show some information on the screen, like the date, the total amount, and the first and last name of that user. But for this first part, let's hold off on any attempts at JOINs, and just look at accessing the Transactions collection alone.

I ran some benchmarks with the following code on my local machine, which was also running a local instance of MongoDB.

MongoClient conn = new MongoClient(new ServerAddress("127.0.0.1", 27017));
DB db = conn.getDB("MyDatabase");
DBCollection transactions = db.getCollection("Transactions");

SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);

Cursor cursor = transactions.find(match);
List<Map> rows = new ArrayList<>();

while ( cursor.hasNext() ) {
    Map<String, Object> row = new HashMap<>();
    DBObject dbObject = cursor.next();
    
    row.put("total", dbObject.get("total"));
    row.put("approved", dbObject.get("approved"));
    row.put("canceled", dbObject.get("total"));
    row.put("location", dbObject.get("canceled"));
    row.put("items", dbObject.get("items"));
    row.put("coupons", dbObject.get("coupons"));
    row.put("created", dbObject.get("created"));
    row.put("updated", dbObject.get("updated"));
    row.put("deleted", dbObject.get("deleted"));
    row.put("userId", dbObject.get("userId"));
    
    rows.add(row);
}
$conn = new Mongo('mongodb://localhost:27017');
$db = $conn->selectDB('MyDatabase');
$transactions = $db->selectCollection('Transactions');

$match = array(
    'created' => array('$gt' =>
        new MongoDate(strtotime('2014-06-01'))
    ));

$cursor = $transactions->find($match);
$rows = array();

while ( $cursor->hasNext() ) {
    $dbObject = $cursor->getNext();
    $rows[] = array(
        'total'    => $dbObject['total'],
        'approved' => $dbObject['approved'],
        'canceled' => $dbObject['canceled'],
        'location' => $dbObject['location'],
        'items'    => $dbObject['items'],
        'coupons'  => $dbObject['coupons'],
        'created'  => $dbObject['created'],
        'updated'  => $dbObject['updated'],
        'deleted'  => $dbObject['deleted'],
        'userId'   => $dbObject['userId']
    );
}

This code is obviously sanitized a bit here to highlight what I'm doing, but you might extend this to do any number of things. You might have some sort of POJO that corresponds to a single document in the collection, and instantiate a new one within each loop. Then you would call dbObject.get() to retrieve each of the properties of that row. I iterated this simple test hundreds of times on my local machine, and found, on average, it took 0.663 seconds to complete. And in case you're curious, the date range I've given here corresponds to roughly 30,000 documents. So that's not so bad.

But this pared-down example was not my use case, and my use case was performing poorly. So I thought, well, I don't need all those pieces of data in the dbObject. I really only needed two. So I formulated a hypothesis. My hypothesis was simple: if I only grab the data I need, and none of the data I don't, the query would perform better. This is akin to avoiding SELECT * in SQL (which is something you should always avoid). So to start, I made the most basic modification to the code possible, which you can see here:

SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);

Cursor cursor = transactions.find(match);
List<Map> rows = new ArrayList<>();

while ( cursor.hasNext() ) {
    Map<String, Object> row = new HashMap<>();
    DBObject dbObject = cursor.next();
    
    row.put("created", dbObject.get("created"));
    row.put("total", dbObject.get("total"));
    row.put("userId", dbObject.get("userId"));
    
    rows.add(row);
}
$match = array(
    'created' => array('$gt' =>
        new MongoDate(strtotime('2014-06-01'))
    ));

$cursor = $transactions->find($match);
$rows = array();

while ( $cursor->hasNext() ) {
    $dbObject = $cursor->getNext();
    $rows[] = array(
        'created'  => $dbObject['created'],
        'total'    => $dbObject['total'],
        'userId'   => $dbObject['userId']
    );
}

All I've done here is remove the .get() call on the properties I didn't need. In fact, at this point, the database driver is still returning all of those properties; I'm just not accessing them. I wanted to see if that alone would make any difference. And in fact, it did. Hundreds of iterations of this code averaged in at 0.405 seconds. That's a 63% speed improvement. Of course the percentage makes it seem more grandiose than it really is, since that's only a 0.25 second improvement, which is not that big of a gain. But it is still an improvement, and it was consistent. Accessing fewer properties from the cursor results in a speed improvement. But while this sped things up a tiny bit, I knew that we could do better by forcing the database driver to stop returning the extraneous data, a la a project clause:

SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);

DBObject project = new BasicDBObject("total", true);
project.put("userId", true);
project.put("created", true);

Cursor cursor = transactions.find(match, project);
List<Map> rows = new ArrayList<>();

while ( cursor.hasNext() ) {
    Map<String, Object> row = new HashMap<>();
    DBObject dbObject = cursor.next();
    
    row.put("created", dbObject.get("created"));
    row.put("total", dbObject.get("total"));
    row.put("userId", dbObject.get("userId"));
    
    rows.add(row);
}
$match = array(
    'created' => array('$gt' =>
        new MongoDate(strtotime('2014-06-01'))
    ));
$project = array(
    'created' => true,
    'total'   => true,
    'userId'  => true
);

$cursor = $transactions->find($match, $project);
$rows = array();

while ( $cursor->hasNext() ) {
    $dbObject = $cursor->getNext();
    $rows[] = array(
        'created'  => $dbObject['created'],
        'total'    => $dbObject['total'],
        'userId'   => $dbObject['userId']
    );
}

This isn't much different than the last example. I'm still selecting only the data points I care about, but now I've added a project clause to the find() call. This means the database driver is no longer returning all the extraneous properties in the first place. The results? On average, this call took 0.029 seconds. That's a 2,186% speed increase over our original query. And that is worth paying attention to. While my last example wasn't all that telling, this one, on the other hand, confirms my hypothesis. If you only select the data you need, and none of the data you don't need, your queries will perform better. (This is true on any database platform.) The consequence of this is that you can't really use a general-purpose POJO for your collection -- not if you want your code to perform well, that is. Instead, you might have any number of contextual POJOs that access different parts of the same collection. It's a trade-off that may prove worth it for the sheer speed.

And I had one more test, just because I was curious. Up until now I've been using the find() command to grab my data, but Mongo also has another way of retrieving data: the aggregate pipeline. I remember reading somewhere that the AP actually spun up multiple threads, whereas a simple find() call was restricted to one. (Don't ask me for a source on that, I'm vaguely remembering heresay.) So I wanted to see if simply switching out those method calls would have any added bonus. Here's what I tried:

SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);

DBObject project = new BasicDBObject("total", true);
project.put("userId", true);
project.put("created", true);

AggregationOptions aggregationOptions = AggregationOptions.builder()
     .batchSize(100)
     .outputMode(AggregationOptions.OutputMode.CURSOR)
     .allowDiskUse(true)
     .build();

List<DBObject> pipeline = Arrays.asList(match, project);
Cursor cursor = transactions.aggregate(pipeline, aggregationOptions);
List<Map> rows = new ArrayList<>();

while ( cursor.hasNext() ) {
    Map<String, Object> row = new HashMap<>();
    DBObject dbObject = cursor.next();
    
    row.put("created", dbObject.get("created"));
    row.put("total", dbObject.get("total"));
    row.put("userId", dbObject.get("userId"));
    
    rows.add(row);
}
$match = array(
    'created' => array('$gt' =>
        new MongoDate(strtotime('2014-06-01'))
    ));
$project = array(
    'created' => true,
    'total'   => true,
    'userId'  => true
);

$pipeline = array(array('$match' => $match), array('$project' => $project));
$cursor = $transactions->aggregateCursor($pipeline);
$rows = array();

while ( $cursor->hasNext() ) {
    $dbObject = $cursor->getNext();
    $rows[] = array(
        'created'  => $dbObject['created'],
        'total'    => $dbObject['total'],
        'userId'   => $dbObject['userId']
    );
}

That test ran, on average, in 0.035 seconds. That's still a 1,794% speed increase over our first test, but it's actually 0.006 seconds slower than then last one. Of course a number that small is a negligible difference. But the fact that there is no difference is worth noting. There is no tangible benefit to using the aggregate pipeline, without a $group clause, versus an ordinary call to find(). So we'd might as well stick with find(), especially considering we weren't aggregating anything, anyway.

But now comes the question of how we go about getting data from other collections. That userId is effectively a foreign key, so we need to do additional queries to get that information. (Side note: you could just duplicate the relevant information instead of, or along with, the foreign key, since that's kind of the Mongo way. But what happens when a person changes their name? This is the problem with non-relational databases.)

The code that I had originally set out to improve did something that I immediately recognized as bad: it looped over the cursor on Transactions, and for each value, ran another query to the Users collection. I refer to these kind of queries as "one-off" queries, since that's kind of what they are. Let me show you some code to better explain what I mean.

SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);

DBObject project = new BasicDBObject("total", true);
project.put("userId", true);
project.put("created", true);

Cursor cursor = transactions.find(match, project);
List<Map> rows = new ArrayList<>();

while ( cursor.hasNext() ) {
    Map<String, Object> row = new HashMap<>();
    DBObject dbObject = cursor.next();
    
    ObjectId userId = (ObjectId) dbObject.get("userId");
    row.put("created", dbObject.get("created"));
    row.put("total", dbObject.get("total"));
    row.put("userId", userId);
    
    // one-off query to the Users collection
    DBObject userProject = new BasicDBObject("firstName", true);
    userProject.put("lastName", true);
    DBObject user = users.findOne(userId, userProject);
    
    row.put("firstName", dbObject2.get("firstName"));
    row.put("lastName", dbObject2.get("lastName"));
    
    rows.add(row);
}
$match = array(
    'created' => array('$gt' =>
        new MongoDate(strtotime('2014-06-01'))
    ));
$project = array(
    'created' => true,
    'total'   => true,
    'userId'  => true
);

$cursor = $transactions->find($match, $project);
$rows = array();

while ( $cursor->hasNext() ) {
    $dbObject = $cursor->getNext();
    $userId = $cursor['userId'];
    
    // one-off query to the Users collection
    $userMatch = array('_id' => $userId);
    $userProject = array(
        'firstName' => true,
        'lastName'  => true
    );
    
    $user = $users->findOne($userMatch, $userProject);
    $rows[] = array(
        'created'   => $dbObject['created'],
        'total'     => $dbObject['total'],
        'userId'    => $dbObject['userId'],
        'firstName' => $user['firstName'],
        'lastName'  => $user['lastName']
    );
}

I can tell you that when I ran this code against my local instance of MongoDB, it took, on average, 0.319 seconds to run. That doesn't seem so bad at all, especially considering all it's doing. Again, this matches about 30,000 documents in the Transactions collection, and clearly we're making just as many calls to the Users collection. But while this seems fine on my local machine, that is not a realistic test. In real world circumstances, you would not have your database on the same server as your codebase. And even if you did, you won't forever. Inevitably you're going to need some code to run on a different server. So I re-ran the same tests using a remote instance of MongoDB. And that made a BIG difference. Suddenly this same little routine took 1709.182 seconds, on average. That's 28-and-a-half minutes. That is ridiculously bad. I will admit, though, that my wifi speed here at home is not the best. I re-ran the same test later, on a better network, and it performed at 829.917 seconds. That's still 14 minutes, which is dreadful.

Why would this simple operation take so long? And what can be done about it? Imagine this: let's say you went into the DMV office and needed to look up a bunch of records. You have a list of the records you need on a handy-dandy clipboard. So you stand in line, and once you're called to the desk, you ask the clerk, one by one, for the records on your clipboard. "Can you give me the details for Record A?" "Can you give me the details for Record B?" That's straight-forward enough, and will be efficient as it can be. But if the clerk you're talking to only has part of the data you need, and tells you you'll need to visit a different office to retrieve the other missing puzzle pieces, then it would be a bit slower.

If you're querying against your own local machine, it would go something like this:

  • Ask Cleark A for Record 1
  • Clerk A gives you everything they have about Record 1
    • Clerk A tells you to talk to Clerk B for the rest of the information
  • Leave Clerk A's line, and stand in Clerk B's line
  • Ask Clerk B for the rest of Record 1
  • Clerk B gives you the rest of Record 1
  • Leave Clerk B's line, and return to Clerk A's line
  • Repeat for Record 2...

That doesn't seem very efficient, does it? But that's the trouble with non-relational databases; disparate collections are in different places. And keep in mind that this analogy actually represents the best case scenario, where Clerk A and Clerk B are in the same room. But that isn't realistic. A more realistic illustration would involve Clerk A at the DMV office, and Clerk B located a mile or two down the road, at the Social Security office. So for each record on your clipboard, you drive back and forth from the DMV to the Social Security office. You can see why it'd be so slow. That "drive" back and forth is called network latency.

But we can do better than that. What if, instead of driving back and forth for each record, you simply asked the clerk at the DMV for all the data they had all at once, and then afterwards you compiled a comprehensive list of all the records you'd need from the Social Security office? That way, you'd only have to make the drive over there once, rather than making 30,000 drives back and forth. In Mongo, you'd only be doing two calls to the database: one bulk call to the Transactions collection, and then a subsequent bulk call to the Users collection. Here's some code to illustrate:

SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
Date startDate = f.parse("2014-06-01");
DBObject gt = new BasicDBObject("$gt", startDate);
DBObject match = new BasicDBObject("created", gt);

DBObject project = new BasicDBObject("total", true);
project.put("userId", true);
project.put("created", true);

MongoJoinCache join = new MongoJoinCache();
Cursor cursor = transactions.find(match, project);
List<Map> rows = new ArrayList<>();

int i = 0;
while ( cursor.hasNext() ) {
    Map<String, Object> row = new HashMap<>();
    DBObject dbObject = cursor.next();
    
    Object userId = (ObjectId) dbObject.get("userId");
    row.put("created", dbObject.get("created"));
    row.put("total", dbObject.get("total"));
    row.put("userId", userId);
    
    join.add(userId.toString(), i);
    rows.add(row);
    i++;
}

DBObject userMatch = join.resolveCache();
DBObject userProject = new BasicDBObject("firstName", true);
userProject.put("lastName", true);

cursor = users.find(userMatch, userProject);
while ( cursor.hasNext() ) {
    DBObject dbObject = cursor.next();
    Object userId = (ObjectId) dbObject.get("_id");
    Set<Integer> indexes = join.get(userId.toString());
    
    for (Integer index : indexes) {
        Map<String, Object> row = rows.get(index);
        row.put("firstName", dbObject.get("firstName"));
        row.put("lastName", dbObject.get("lastName"));
        rows.add(index, row);
    }
}

public class MongoJoinCache {
    private final Set<String> objectIds;
    private final Map<String, Set<Integer>> objectToIndexMapping;
    private int total = 0;
    
    public MongoJoinCache() {
        objectIds = new HashSet<>();
        objectToIndexMapping = new HashMap<>();
    }
    
    public void add(String objectId, Integer index) {
        objectIds.add(objectId);
        Set<Integer> indexes;
        if (objectToIndexMapping.containsKey(objectId)) {
            indexes = objectToIndexMapping.get(objectId);
        } else {
            indexes = new HashSet<>();
        }
        indexes.add(index);
        objectToIndexMapping.put(objectId, indexes);
        total++;
    }
    
    public Set<Integer> get(String objectId) {
        return objectToIndexMapping.get(objectId);
    }
    
    public Integer size() {
        return total;
    }
    
    public DBObject resolveCache() {
        if (size() == 0) {
            return null;
        }
        
        final BasicDBList ids = new BasicDBList();
        for (String id : objectIds) {
            ids.add(new ObjectId(id));
        }
        
        DBObject match = new BasicDBObject("_id", new BasicDBObject("$in", ids));
        return match;
    }
}
$match = array(
    'created' => array('$gt' =>
        new MongoDate(strtotime('2014-06-01'))
    ));
$project = array(
    'created' => true,
    'total'   => true,
    'userId'  => true
);

$userIds = array();
$cursor = $transactions->find($match, $project);
$rows = array();

$i = 0;
while ( $cursor->hasNext() ) {
    $dbObject = $cursor->getNext();
    $userId = strval($dbObject['userId']);
    $userIds[$userId][] = $i;
    
    $rows[] = array(
        'created'  => $dbObject['created'],
        'total'    => $dbObject['total'],
        'userId'   => $dbObject['userId']
    );
    $i++;
}

$userMatch = array('_id' => array('$in' => array()));
foreach ( $userIds as $userId => $indexes ) {
    $userMatch['_id']['$in'][] = $userId;
}

$userProject = array('firstName' => true, 'lastName' => true);
$cursor = $users->find($userMatch, $userProject);

while ( $cursor->hasNext() ) {
    $dbObject = $cursor->getNext();
    $userId = strval($dbObject['userId']);
    
    foreach ( $indexes as $userIds[$userId] ) {
        $rows[$index]['firstName'] = $dbObject['firstName'];
        $rows[$index]['lastName'] = $dbObject['lastName'];
    }
}

Phew. That's a lot more code! Well, on the Java side anyway. (PHP arrays are awesome; Java people don't even understand.) But the million dollar question is whether there is any tangible benefit to all that extra code I just threw at you. Here are the results: When running against my local machine, this ran for an average of 0.339 seconds. Against the previously-mentioned 0.319 seconds on my local machine, this is slightly slower. So why bother with all this extra code if it's slower? Because that's not really a fair test. The difference of 10 milliseconds is negligible, but more importantly, there is no universe in which you can reliably expect to have zero network latency. There will always be network latency in production environments. So a real test is against a remote instance.

So what happens when I ran this code against a remote MongoDB server? Remember that it took 28 ½ minutes before. With this code shown above, however, on average, it ran in 6.191 seconds. You read that correctly. That is an astronomical speed improvement of 27,508%. And even if your network latency isn't as bad as mine was (and it shouldn't be), you can still see nonetheless that this method will always be faster, by an order of magnitude.

If you think about it, it makes sense. We took a query which had been O(n²) in complexity and reduced it down to O(2n) -- at most. In fact, it's probably less than that in practice, since there are bound to be duplicate foreign keys. I'm guessing that the developers of relational databases do something exactly like this, under-the-hood in their database code. In order to join two tables, they probably first have to collect all the foreign keys from the first table into a set, and then grab the corresponding rows en masse from the second table. The big difference is that with SQL, you don't have to think about any of this. You just type INNER JOIN and bam, it's all magically done for you. But because Mongo doesn't include this as a concept, you have to write your own database wrapper code in order to be able to use it efficiently.

So what's the takeaway from all of this? First, you can write your own wrapper code to effectively perform a join operation in Mongo, and doing so is hugely advantageous in terms of speed, albeit slightly annoying. But it's vital to your success if you're going to use Mongo. Secondly, what I hope is the bigger takeaway is that you shouldn't have to. Like I've been saying all along, if you have to resort to writing what is essentially your own low-level database code, it's a sure sign that you're using the wrong database. If you've read this far, that means you should probably just use SQL.

TL;DR

To conclude, here are the more practical takeaways broken down:

  • Only select the data you absolutely need, and nothing more
    • Use a project clause to limit your data
    • Don't instantiate general purpose objects with every property
  • Don't do one-off queries inside of loops!
  • Minimize network latency by grabbing batches

And that is how you do a "Join" in Mongo.

<< < Page 1 of 6 > >>