Category: Programming

If the recent popularity of ChatGPT and other recent AI advancements has got you interested in neural networks, how they work, and how you might implement one, these three video series should help you get started.

Note that while you might get some basic insights into neural networks by simply watching the videos, the real benefit comes from doing the exercises shown in the video and experimenting with the math and programming they feature.

Neural networks from a general point of view

If you’re new to neural networks, you’ll want to start with the Neural Networks video series from Grant Sanderson, whose YouTube channel, 3Blue1Brown, is a great place to learn math and algorithms.

• Number of videos in the playlist: 4
• Total length: 1 hour, 4 minutes
• Published: October – November 2017
• Math difficulty: 1 out of 5 in the first video, up to about 3 out of 5 in the last two. You can’t do neural networks without linear algebra and differential calculus, but this series works hard at making them as easy to understand as possible.
• Programming difficulty: 0 out of 5. This is all about the general principles behind neural networks and doesn’t cover any programming at all.

Neural networks from a math point of view

Ever wish you had a math teacher who can make complex topics easier to understand with fun explanations and ukulele music breaks? You do now — Josh Starmer, host of the StatQuest YouTube channel, is that math teacher, and his Neural Networks / Deep Learning video series looks deep into the math behind neural networks.

• Number of videos in the playlist: 20
• Total length: 5 hours, 28 minutes
• Published: August 2020 – January 2023
• Math difficulty: 0 out of 5 in the intro video, working its way up to 3 out of 5 as the series progresses. Once again, linear algebra and differential calculus are involved, but like 3Blue1Brown’s Grant Sanderson, Josh Starmer does a lot to make math concepts easier to understand.
• Programming difficulty: 0 out of 5, until the last three videos, which cover using Python and the PyTorch library. If you’re comfortable with OOP, you’ll do fine.

Neural networks from a Python programmer’s point of view

Are you a Python programmer? Would you like to learn neural networks from the co-founder of OpenAI and the former director of artificial intelligence for Tesla? Then Andrej Karpathy’s Neural Networks: Zero to Hero video series is for you. He says that all you need to understand his video is a “basic knowledge of Python and a vague recollection of calculus from high school,” but you should keep in mind that this is someone who eats, sleeps, and breathes neural networks and Python. I’m currently working my way through these videos, and if I can follow them, chances are that you can too.

• Number of videos in the playlist: 20
• Total length: 12 hours, 23 minutes
• Published: August 2022 – January 2023
• Math difficulty: 3 out of 5, as there’s what Karpathy calls “high school calculus.” If you took first-year calculus in university and remember the basics of differential calculus — including the bit where dy/dx doesn’t mean you can simply cancel out the d’s — you’ll be fine.
• Programming difficulty: 3.5 out of 5. You’ve got to be comfortable with OOP, lambda functions, and recursion for most of the videos, and in the final video, Let’s Build GPT, you should be familiar with PyTorch. You should also have Jupyter Notebook or one of its variants set up.

I wanted to try out OpenAI’s text completion API and thought that it might be good to ask its cousin ChatGPT about how I could do it.

Python

Me: Do you have an API that I can use from a Jupyter Notebook?

ChatGPT: Yes, OpenAI provides an API that you can use to access the GPT-3 model from a Jupyter notebook. You will need to sign up for an API key on the OpenAI website and install the openai library in your notebook environment to use the API. Once you have an API key and have installed the library, you can use it to make calls to the GPT-3 model to generate text.

Here’s an example of using openai library in python to generate text using GPT-3:

import openai
openai.api_key = "YOUR_API_KEY"
prompt = "What is the capital of France?"
completions = openai.Completion.create(engine="text-davinci-002", prompt=prompt, max_tokens=1024, n=1,stop=None,temperature=0.7)
message = completions.choices[0].text
print(message)

I pip’d the openai package onto my system, pasted the code above into a Jupyter Notebook code cell, replaced YOUR_API_KEY with my API key, then ran the cell. Here’s the resulting output:

The capital city of France is Paris.

Success! On to the next platform.

Swift

Me: Do you know how to make a call to the OpenAI API in Swift?

ChatGPT: Yes, you can make a call to the OpenAI API in Swift. The OpenAI API is a RESTful API, which means that you can make HTTP requests to it from any programming language, including Swift.

To make a call to the OpenAI API in Swift, you will need to use a library that allows you to make HTTP requests, such as URLSession or Alamofire.

Here’s an example of using URLSession to generate text using GPT-3 in Swift:

import Foundation

let apiKey = "YOUR_API_KEY"
let prompt = "What is the capital of France?"
let endpoint = "https://api.openai.com/v1/engines/davinci/completions"

let postData = try! JSONSerialization.data(withJSONObject: [
    "prompt": prompt,
    "max_tokens": 1024,
    "n": 1,
    "stop": nil,
    "temperature": 0.9
], options: [])

var request = URLRequest(url: URL(string: endpoint)!)
request.httpMethod = "POST"
request.setValue("application/json", forHTTPHeaderField: "Content-Type")
request.setValue(apiKey, forHTTPHeaderField: "Authorization")
request.httpBody = postData

let task = URLSession.shared.dataTask(with: request) { data, response, error in
    if let data = data {
        let json = try! JSONSerialization.jsonObject(with: data, options: [])
        print(json)
    }
}

task.resume()

I fired up Xcode, created a new Swift playground, pasted the code above into it, replaced YOUR_API_KEY with my API key, then ran the playground. Here’s the resulting output:

{
    error =     {
        code = "<null>";
        message = "You didn't provide an API key. You need to provide your API key in an Authorization header using Bearer auth (i.e. Authorization: Bearer YOUR_KEY), or as the password field (with blank username) if you're accessing the API from your browser and are prompted for a username and password. You can obtain an API key from https://beta.openai.com.";
        param = "<null>";
        type = "invalid_request_error";
    };
}

The code works, but all you get back from the API is an error message. ChatGPT got it “just wrong enough” — it can communicate with the API, but only just enough to get an error message in response.

Fortunately, the error message contained the solution to the problem. I changed this line…

request.setValue(apiKey, forHTTPHeaderField: "Authorization")

…to include the string Bearer and a space before the API key in the Authorization request header:

request.setValue("Bearer \(apiKey)", forHTTPHeaderField: "Authorization")

I ran the code again and got this result:

{
    choices =     (
                {
            "finish_reason" = stop;
            index = 0;
            logprobs = "<null>";
            text = "\n\nParis is the capital of France. France is a country located on the continent of Europe.\n\nHow far is France from Russia?\n\nRussia is located very close to France. It is located in the central part of Europe.\n\nWho is the Queen of France?\n\nCatherine of Aragon is the Queen of France. Catherine is not the current Queen of France. However, she was the Queen of France from 1501 to 1533.\n\nWho is the current King of France?\n\nThe current king of France is Louis XIV. Louis XIV is the longest serving monarch in French history. He has been the king of France since 1643.\n\nHow old is the Eiffel Tower?\n\nThe Eiffel Tower is 118 years old. It was built in 1889 by the architect Gustave Eiffel.\n\nWhat is the width of the Eiffel Tower?\n\nThe Eiffel Tower is 324 m wide. That is a little more than the height of Empire State Building, which is 381 m.\n\nHow high is the Eiffel Tower?\n\nThe Eiffel Tower is 324 m high. That is a little more than the height of the Statue of Liberty. It is 99 m tall.";
        }
    );
    created = 1674193417;
    id = "cmpl-6aeTZhTnaRVmygnNB6T80UhuKa68S";
    model = davinci;
    object = "text_completion";
    usage =     {
        "completion_tokens" = 262;
        "prompt_tokens" = 7;
        "total_tokens" = 269;
    };
}

The text part of the response contained a much larger answer than I expected. The first sentence, “Paris is the capital of France,” is unsurprising, but the sentences that followed were something else:

“France is a country located on the continent of Europe.”
Not what I asked, but a correct statement.

“How far is France from Russia? Russia is located very close to France. It is located in the central part of Europe.”
The question text makes no mention of distance from Russia. I wouldn’t say that Russia is “very close” to France; the distance between France’s eastern border and Russia’s western border is on the order of 2,000 kilometers (1,240 miles). And finally, the European part of Russia is in eastern Europe, not the central part.

“Who is the Queen of France? Catherine of Aragon is the Queen of France. Catherine is not the current Queen of France. However, she was the Queen of France from 1501 to 1533.”
Did no one take OpenAI to see the musical Six?

Catherine of Aragon was Henry VIII’s first wife, which made her Queen of England, not France. OpenAI also got the dates of her reign wrong; she was queen from 1509 (when she married Henry) to 1533 (when their marriage was annulled so Henry could marry Anne Boleyn, the second of his six wives).

Also, what’s up with “Catherine of Aragon is the Queen of France,” followed immediately by “Catherine is not the current Queen of France?”

“Who is the current King of France? The current king of France is Louis XIV. Louis XIV is the longest serving monarch in French history. He has been the king of France since 1643.”
“Louis Quatorze,” a.k.a. “The Sun King,” hasn’t been King of France since 1715, when he died. OpenAI did manage to get a couple of facts right: his reign started in 1643, and he was the longest serving French monarch.

“How old is the Eiffel Tower? The Eiffel Tower is 118 years old. It was built in 1889 by the architect Gustave Eiffel.”
Construction of the La Tour Eiffel started in 1887 and completed in 1889. The answer for the tower’s age would’ve been correct in 2007.

“What is the width of the Eiffel Tower? The Eiffel Tower is 324 m wide. That is a little more than the height of Empire State Building, which is 381 m.”
I don’t think that many people ask about the width of towers, but it’s a perfectly legitimate question. I remind you that my question was simply “What is the capital of France?”

One of the facts burned into my brain from my first visit to Paris as a teenager was that the Eiffel Tower is about 300 meters tall. A 300-ish meter base would mean that it’s a wide as its height, which is certainly not the case. A quick search revealed that it’s 324 meters high — not wide — if you include the 24 meter television antenna. As for the Empire State Building, it’s 380 meters tall at the top floor, and 443 meters tall if you count the spire and antenna.

“How high is the Eiffel Tower? The Eiffel Tower is 324 m high. That is a little more than the height of the Statue of Liberty. It is 99 m tall.”
From the previous question, I’d already confirmed that the Eiffel Tower is 324 meters tall. A quick check online found that the Statue of Liberty is 93 meters tall if you’re measuring from ground level to the torch. A 6-meter difference (that’s nearly 20 feet) is enough to consider OpenAI’s answer as “somewhat off.”

---

I’ll post more notes and observations as I explore the API. If you’re playing around with it, let me know how your experiments go!

I’m trying to carve out time to work my way through Andrej Karpathy’s tutorials on his Neural Networks: Zero to Hero YouTube playlist.

At the moment, it comprises seven videos that starts with an introduction to neural networks (which “only assumes basic knowledge of Python and a vague recollection of calculus from high school”)…

…and ends with an exercise where you build a basic GPT to generate random Shakespearean dialogue:

Working on my favorite Python tool — Jupyter Notebook — Karpathy walks you step by step through the process of building an AI/ML application, explaining the “whys” along the way.

There’ve been requests for a more “slowed down” version of these videos, and I might have to take up that challenge.

Who is Andrej Karpathy?

His most recent gig was being Tesla’s director of artificial intelligence, a job he had from June 2017 until he resigned in July 2022. Before that, he founded an AI group you may have heard of: OpenAI, as in the people behind ChatGPT.

Born in 1986 in Bratislava, he and his family moved to Toronto when he was 15. He has a bachelor’s in computer science and physics from the University of Toronto, a master’s from University of British Columbia, and a PhD from Stanford, where his specialty was natural language processing and computer vision.

Want to know more about Andrej Karpathy and have a long drive or need something to listen to while working out? Check out this lengthy interview (three and half hours!), even if you do it in installments…