The other day ChatGPT saved my post-graduate academic career.
Okay, that’s dramatic. But it did save me from completely bombing my final data engineering course exam. Between unsatisfactory course instruction, some unfortunately timed missed classes, and difficult exam subject matter, I was coming into this test like Pedro Pascal in this meme.
Side note: Pedro Pascal is my favorite actor right now. I recently watched The Last of Us and The Unbearable Weight of Massive Talent and he was incredible in both. Did you know one of Pedro Pascal’s first ever acting roles was as a possible romantic interest in Buffy the Vampire Slayer who gets killed by vampires within two minutes of getting introduced on screen. Look it up. Okay back to ChatGPT
However, I, like everyone else in the world with access to the internet, have been hearing a lot about ChatGPT lately. I decided to give it a shot at salvaging my chances of living to fight another day. For the record, our exams are all open book so this isn’t academic dishonesty. We’re actually encouraged to use AI tools because that’s very much in the realm of data analysis.
I had high expectations for ChatGPT based on what I had been hearing but I was still absolutely blown away. I learned more taking the exam using ChatGPT as a tool than I had in four weeks of listening to lectures. It was like having a tutor sit next to me and thoroughly explain questions. My typical exam strategy is to google a general question and sift through pages and pages of code examples until I find one that may be applicable to the problem that I’m working on and try to reverse engineer it in a way that would allow me to build an answer. Interacting with ChatGPT was more like having a conversation with someone who already had all the answers and was willing to explain to me why they worked. Let’s examine.
My exam was primarily quizing our ability to analyze a data set using PySpark. Since I had missed most of the hands-on courses my understanding of PySpark was limited to essentially “a tool that analyzes data in Python”. So I had ChatGPT start there with me.
This alone is a better description than anything provided to me by my course instructor. Let’s continue. Some of the following questions required working knowledge on the advantages of different Big Data processing systems and I hit a wall of understanding.
After some multiple choice questions the exam moved into the hands-on portion, which, as I mentioned, went over material that we covered on days I had completely missed. For coding, I work best with examples to get me started and then that allows me to branch off and solve problems from a solid base. So I had ChatGPT do just that for me. We were given a data set of around 15,000 car accidents that had occurred in Wayne County in the past 10 years along with dozens of columns of data about each of the accidents. I asked ChatGPT to help me get started with one the questions.
Not only did ChatGPT provide me with a basis for the code I needed to do my own analysis, but it explained the uses of each of the functions for me in a way that was much easier to understand than my course instructor.
A few questions later I ran into a need to filter my results based on criteria and I didn’t know the syntax for doing so. So I asked ChatGPT.
From these two questions I was able to get a handle on the syntax of PySpark and complete the rest of the questions on my own and with my course materials. But the ChatGPT responses were invaluable.
Of course, this is somewhat of a gray area. In a way, ChatGPT did solve these problems for me almost completely and I didn’t need to truly know the syntax of PySpark to pass this exam. But on the flip side of that, maybe that isn’t as important. Maybe its more important to understand the core principles of using a coding language and less important to memorize specific commands and syntax to run that language. ChatGPT essentially allowed me to speak PySpark at a high level without the hours of studying required to know the formatting. It’d be like having an interpreter standing next to you in a foreign country and being able to essentially communicate even though you don’t know all of the words to the language.
This seems to be the direction we can move with this technology. In the same way Google reduced the importance of memorizing specific facts, ChatGPT will reduce the importance of memorizing structure and increase the emphasis on high-level creative thinking.
It’s truly incredible. If you haven’t used it yet I strongly recommend giving it a try.
After my test I continued playing with it with different prompts and was able to even generate a list of fantasy audiobooks I can explore after I finish the one I’m currently listening to.
This is going to be a huge part of our future and I for one am excited about it.