Algorithm Antics: AI Models Take on Physics Problems

In the ever-evolving landscape of AI, it’s easy to get swept up in the hype surrounding the latest and greatest models. Recently, New Zealand Physics teacher Benny Pan shared an intriguing LinkedIn post where he put GPT4-o1, Mistral, and GPT-4o mini through their paces on a complex NCEA Level 3 physics problem involving the rotational inertia of a satellite. Inspired by Benny’s experiment, I decided to conduct my own test, this time focusing on a complex 2D collision calculation.

Armed with Microsoft Co-pilot, Google Gemini, GPT-o1, and GPT-4o mini, I set out to see how these models would handle the challenge. Just like Benny, I used quality data and Chain-of-Thought (COT) prompt engineering techniques to ensure a fair and rigorous evaluation.

So, fellow educators, buckle up and join me on this enlightening journey through the world of AI in education. Let’s explore how these powerful tools can enhance our teaching and inspire our students to reach new heights in their understanding of physics.

The Problem

The problem I have used comes from the 2022 Level 3 NCEA Physics Mechanical Systems paper which involved a question where comets and asteroids, known for their loosely formed structures, collide and break apart easily. In this particular case, we have a 1.00 × 10² kg object hurtling through space at a velocity of 5.00 × 10² m/s. This object collides with a stationary 3.00 × 10² kg object. The collision, occurring without any external forces, results in the formation of two new ‘rocks’, A and B, each with a mass of 2.00 × 10² kg. These rocks move away from each other at angles of 20° on either side of the initial direction of the impacting object as shown in the diagrams below.

The task of the AI models is to demonstrate that the momentum of rocks A and B are equal in magnitude, despite their different directions. To do this, we will use either vector diagrams and calculations or a combination of both. The prompt will start by determining the total momentum before the collision and then proceed to calculate the speeds of rocks A and B. This exercise will not only illustrate the principles of momentum conservation but also provide a deeper understanding of collision dynamics in space.

Developing the Prompt

Chain-of-Thought (CoT) prompting is a technique in artificial intelligence that involves guiding the AI through a series of logical steps to arrive at a solution. This method helps the AI model break down complex problems into manageable parts, ensuring a more accurate and coherent response. In the context of our collision problem, CoT prompting was applied by first calculating the total momentum before the collision. This initial step set the foundation for understanding the system’s dynamics. Next, we used vector diagrams and/or calculations to show the momentum distribution post-collision, ensuring that the directions and magnitudes were accurately represented. Finally, we calculated the speeds of rocks A and B, verifying that their momenta were equal in size despite their different directions. By following this structured approach, we ensured that each step logically followed from the previous one, leading (hopefully) a clear and precise solution.

So, with this in mind the CoT prompt was created:

Identify the given data:

Mass of the first object: (m1 = 1.00 x 10²) kg

Velocity of the first object: (v1 = 5.00 x 10²) m/s

Mass of the second object: (m2 = 3.00 x 10²) kg (stationary, so v2 = 0 m/s)

Mass of rocks A and B after collision: mA = mB = 2.00 x 10²) kg

Angles of movement for rocks A and B: (θA = 20°) and (θB = -20°)

Calculate the total momentum before the collision:

Use the formula for momentum: (p = mv)

Calculate the momentum of the first object

Since the second object is stationary, its momentum is zero.

Determine the total initial momentum by summing the momenta of both objects.

Momentum is conserved in both the x and y directions.

Write the equations for the conservation of momentum in the x-direction.

Write the equations for the conservation of momentum in the y-direction.

Express the final momenta of rocks A and B:

Use vector components to express the momenta of rocks A and B.

Break down the momentum vectors into x and y components.

Show that the magnitudes of the momenta of rocks A and B are equal:

Use the conservation equations to demonstrate that the magnitudes of the momenta are the same.

Verify that the x and y components satisfy the conservation laws.

Use the momentum components to find the velocities of rocks A and B.

Apply the Pythagorean theorem to determine the speeds from the x and y components.

Solve the problem.

Applying the Prompt to Each Model

I then applied that same prompt to each model and reviewed the results.

Microsoft Copilot

Given Data:

Mass of the first object: $m1 = 1.00 x 10$ ² kg

$Velocity of the first object: v1 = 5.00 x 10$ ² m/s

$Mass of the second object: m2 = 3.00 x 10$ ² kg (stationary, so v2 = 0 m/s)

$Mass of rocks A and B after collision: mA = mB = 2.00 x 10$ ² kg

$Angles of movement for rocks A and B: θ A = 20 ° and θ B = -20 °$

Calculate the Total Momentum Before the Collision:

The formula for momentum is p = mv

Momentum of the first object:

p1 = m1 x v1 = (1.00 x 10² kg) x (5.00 x 10² m/s) = 5.00 x 10^4 kg.m/s

Momentum of the second object: Since the second object is stationary:

p2 = m2 x v2 = (3.00 x 10² kg x 0 m/s = 0 kg.m/s

Total initial momentum:

p(total initial) = p1 + p2 = 5.00 x 10^4 \kg.m/s + 0 = 5.00 x 10^4 kg.m/s

Conservation of Momentum Equations:

Momentum is conserved in both the x and y directions.

Conservation of momentum in the x-direction:

Conservation of momentum in the y-direction:

m1v1 = mAvA sin(θA) + mBvB sin(-θB)

0 = mAvA sin(θA) - mBvB sin(θB)

Express the Final Momenta of Rocks A and B:

Using vector components to express the momenta of rocks A and B:

For rock A:

pAy = mAvA sin(θA)

For rock B:

Show that the Magnitudes of the Momenta of Rocks A and B are Equal:

Using the conservation equations:

From the x-direction equation:

From the y-direction equation:

Since sin(-20°) = -sin(20°)

we get: 0 = mAvAsin(20°) - mBvBsin(20°)

Given that mA = mB, it follows that: vA = vB

Calculate the Speeds of Rocks A and B:

Using the momentum components to find the velocities of rocks A and B:

From the x-direction equation:

m1v1 = mAvAcos(20°) + mBvBcos(20°)

5.00 x 10^4 = 2.00 x 10² vAcos(20°) + 2.00 x 10²vBcos(20°)

5.00 x 10^4 = 2.00 x 10² vAcos(20°) + 2.00 x 10² vAcos(20°)

5.00 x 10^4 = 4.00 x 10² vAcos(20°)

vA = 5.00 x 10^4/4.00 x 10²cos(20°)

vA - 133 m/s

Since vA = vB, the speed of rock B is also approximately 133 m/s.

Google Gemini

Analyzing the Given Data

Mass of object 1: m1 = 1.00 x 10² kg

Velocity of object 1: v1 = 5.00 x 10² m/s

Mass of object 2: m2 = 3.00 x 10² kg

Velocity of object 2: v2 = 0 m/s

Mass of rocks A and B after collision: mA = mB = 2.00 x 10² kg

Angles of movement for rocks A and B: θA = 20°, θB = -20°

Calculating Total Momentum Before the Collision

p1 = m1 * v1 = (1.00 x 10² kg) * (5.00 x 10² m/s) = 5.00 x 10^4 kg*m/s

p2 = m2 * v2 = 0 kg*m/s (since v2 = 0)

p(initial) = p1 + p2 = 5.00 x 10^4 kg*m/s

Conservation of Momentum Equations

p(initialx) = p(finalx)

m1 * v1 = mA * vAx + mB * vBx

p(initialy = p(finaly)

0 = mA * vAy + mB * vBy

Expressing Final Momenta of Rocks A and B

Rock A:

vAx = vA * cos(θA)

vAy = vA * sin(θA)

Rock B:

vBx = vB * cos(θB)

vBy = vB * sin(θB)

Showing Equal Magnitudes of Momenta for Rocks A and B

mA * vAy + mB * vBy = 0

Since mA = mB and θA = -θB, we can substitute:

mA * vA * sin(θA) + mA * vB * sin(-θA) = 0

Simplifying:

mA * vA * sin(θA) - mA * vB * sin(θA) = 0

mA * (vA - vB) * sin(θA) = 0

Since mA ≠ 0 and sin(θA) ≠ 0, we must have:

vA - vB = 0

Therefore, vA = vB, meaning the magnitudes of the momenta of rocks A and B are equal.

Calculating Speeds of Rocks A and B

m1 * v1 = mA * vA * cos(θA) + mA * vB * cos(θB)

Since vA = vB and θA = -θB:

m1 * v1 = 2 * mA * vA * cos(θA)

Solving for vA:

vA = (m1 * v1) / (2 * mA * cos(θA))

Plugging in the given values:

vA = (1.00 x 10^2 kg * 5.00 x 10^2 m/s) / (2 * 2.00 x 10^2 kg * cos(20°))

vA ≈ 1.25 x 10^2 m/s

Using the x-direction conservation equation:

Since vA = vB, vB ≈ 1.25 x 10^2 m/s as well.

GPT-o1

Given Data:

- Mass of the first object: (m1 = 1.00 x 10² kg)

- Velocity of the first object: (v1 = 5.00 x 10² m/s)

- Mass of the second object (stationary): (m2 = 3.00 x 10² kg) so (v2 = 0 m/s)

- Masses after collision:

- (mA = 2.00 x 10² kg)

- (mB = 2.00 x 10² kg)

- Angles of movement:

- (θA = +20°)

- (θB = -20°)

Step 1: Calculate the Total Initial Momentum

Since the second object is stationary, its momentum is zero. The total initial momentum is due only to the first object:

p(initial) = m1v1 = (1.00 x 10² kg) (5.00 x 10² m/s) = 50,000 kg.m/s

Step 2: Set Up Conservation of Momentum Equations

In the (x)-direction:

p(initial)x = p(final)x

50,000 kg.m/s = mA vAcosθA + mBvBcosθB

In the (y)-direction:

Since there's no initial (y)-momentum:

0 = mAvAsinθA + mBvBsinθB

Step 3: Show that the Magnitudes of the Momenta of Rocks A and B are Equal

From the (y)-direction conservation:

0 = mAvAsinθA + mBvB(-θA)

0 = mAvAsinθA - mBvBsinθA

mAvA = mBvB

Since (mA = mB):

vA = vB

Step 4: Solve for the Speed (v) of Rocks A and B

Now, the (x)-direction conservation equation becomes:

50,000 = mAvcosθA + mBvcosθB

50,000 = mAvcos20° + mBvcos (-20°)

Since (cos(-θ) =cosθ): 50,000 = v(mA + mB)cos20°

50,000 = v (2.00 x 10² kg + 2.00 x 10² kg) cos 20°

50,000 = v (4.00 x 10² kg) cos 20°

Calculate (cos 20°): cos 20° = 0.9397

Now solve for (v):

v = 50,000/ (4.00 x 10² kg) (0.9397)

v = 50,000/375.88

v = 133.0 m/s

Step 5: Verify Conservation in the (y)-Direction

Using (vA = vB = 133.0 m/s):

0 = mAvA sin 20° + mBvB sin (-20°)

0 = 2.00 x 10² kg) (133.0m/s) (0.3420) - (2.00 x 10² kg) (133.0 m/s) (0.3420)

0 = 90,996 - 90,996

0 = 0

Answer:

The speeds of rocks A and B after the collision are both approximately 133 m/s.

ChatGPT -4o mini

1. Identify the Given Data:

- Mass of the first object: (m1 = 1.00 x 10² kg)

- Velocity of the first object: (v1 = 5.00 x 10² m/s)

- Mass of the second object (stationary): (m2 = 3.00 x 10² kg), so (v2 = 0 m/s)

- Mass of rocks A and B after collision: (mA = mB = 2.00 x 10² kg)

- Angles of movement for rocks A and B: (θA = 20°), (θB = -20°)

2. Calculate the Total Momentum Before the Collision:

Use the momentum formula:

p = mv

- Momentum of the first object:

p1 = m1 x v1 = 1.00 x 10² kg x 5.00 x 10² m/s = 50000 kg.m/s

- Momentum of the second object (stationary):

p2 = m2 x v2 = 3.00 x 10² kg x 0 m/s = 0 kg.m/s

- Total initial momentum:

p(total initial) = p1 + p2 = 50000 kg.m/s

3. Set up the Conservation of Momentum Equations:

Momentum is conserved in both the x and y directions.

Conservation of Momentum in the x-direction:

Let pAx and pBx be the x-components of the momentum for rocks A and B, respectively. The total initial momentum in the x-direction must equal the total final momentum in the x-direction.

p(initial) = pAx + pBx

Conservation of Momentum in the y-direction:

The total momentum in the y-direction before the collision is zero (because both objects move only in the x-direction). After the collision, the y-components of the momentum of rocks A and B must cancel each other out.

0 = pAy + pBy

4. Express the Final Momenta of Rocks A and B:

Use vector components to express the momenta of rocks A and B. The x and y components of momentum for each rock are calculated using their angles of motion and their total momentum.

- For rock A:

pAx = mA.vA.cos(θA)

pAy = mA.vA.sin(θA)

- For rock B:

pBx = mB.t vB.cos(θB)

pBy = mB.vB.sin(θB)

5. Show that the Magnitudes of the Momenta of Rocks A and B Are Equal:

The magnitude of the momenta can be expressed as:

- For rock A:

pA = √ (pAx² + pAy²)

- For rock B:

pB = √ (pBx² + pBy²)

Using the conservation equations and the angles given, we can demonstrate that the magnitudes of the momenta of rocks A and B are equal.

6. Verify the x and y Components Satisfy the Conservation Laws:

Check that:

p(initial) = pAx + pBx and 0 = pAy + pBy

7. Calculate the Speeds of Rocks A and B:

Using the momentum components, we can calculate the velocities (speeds) of rocks A and B by rearranging the formula (p = mv \) to solve for (v):

vA = √ ((pAx² + pAy²)/mA)

vB = √ ((pBx² + pBy²)/mB)

Now applying the Pythagorean theorem to determine the speeds from the x and y components:

vA = √ ((pAx/mA)² + (pAy/mA²))

vB = √ ((pBx/mB)² + (pBy/mB²))

These steps show that the speeds of rocks A and B after the collision are the same. From the results:

- Speed of Rock A: vA = 93.87 m/s

- Speed of Rock B: vB = 93.87 m/s

Both rocks have the same speed after the collision, approximately 93.87 m/s

Conclusion

When comparing the responses of different AI models to the 2D momentum question, it’s clear that each model has its strengths and weaknesses. Microsoft Copilot and ChatGPT-o1 both arrived at the correct answer of 133 m/s, demonstrating a solid understanding of the principles of momentum conservation and vector addition so passed with excellence. In contrast, Gemini and ChatGPT-4 Mini provided answers of 125 m/s and 93.87 m/s, respectively, so Gemini gets an achieved and ChatGPT-4 mini gets a Merit. These discrepancies stemmed from differences in how each model handles trigonometric calculations and the conservation of momentum in both dimensions. The chain of thought approach to prompt engineering appears to be more effective in Copilot and ChatGPT-o1, as they consistently applied the correct physics principles and calculations. For teachers, this highlights the importance of cross-verifying AI-generated answers to ensure the underlying physics principles are correctly applied. Despite the differences, all models offer valuable insights and can be useful tools in the classroom, provided their outputs are carefully evaluated. This comparison underscores the need for critical thinking and verification when using AI in educational settings and how different prompt engineering approaches can affect responses in different models.