The Trouble with Practice Exams

by John Cunningham, W1AI

Learn not memorize

As volunteer examiners we see people getting their ham radio license by memorizing the answers using practice exams, learning little in the process, then dropping out of ham radio because they don’t know where to begin.  If you’re going to invest all those hours studying for the exam, most people want to learn something about ham radio in the process.

Practice exams do serve a useful purpose, proving your readiness for the exam after you have completed a course of study.  However, they should not replace education.

This article explores some of the issues with getting a ham license using only practice exams.  We begin with an examination of how the actual exams are structured.

 

How Exams are Generated

The National Conference of Volunteer Examiner Coordinators (NCVEC) maintains the question pools used for all U.S. amateur radio examinations.  The FCC regulations require at least ten times as many questions in the pool as appear on the actual exam, but the NCVEC generally exceeds that requirement by a considerable margin.  As required by the FCC, the NCVEC publishes the question pools to the general public well in advance of their effective date.

Each U.S. question pool is divided into 10 topics, each topic into between 1 and 8 groups, and each group contains between 10 and 24 questions.  While it is not required by the FCC, in all cases the number of groups is exactly the same as the number of questions on the exam.

Question pool	Effective dates	Topics	Groups	Questions on exam	Questions pool size
Technician	7/01/2022 - 6/30/2026	10	35	35	411
General	7/01/2019 - 6/30/2023	10	35	35	454
Extra	7/01/2020 - 6/30/2024	10	50	50	621

 

The rules for generating exams only require that every exam have the same number of questions from each topic as there are groups in that topic, but the actual exams are always generated by taking one question, at random, from each group.

This creates an interesting phenomenon: questions in larger groups are underrepresented on the exams.  Consider the following two groups:

• Group G5C (Resistors, capacitors, and inductors in series and parallel; transformers) contains 18 questions, so each question in the group has a 1 chance in 18 of appearing on your General exam.
• Group G5A (Reactance; inductance; capacitance; impedance; impedance matching) contains 11 questions, so each question in the group has 1 chance in 11 of appearing on your General class exam.

While you will always get exactly one question from group G5C and one from G5A on the General class exam, any one specific question in group G5A is almost twice as likely to appear than any one specific question in group G5C.

Since the free online practice exams are generated in the same fashion as the actual exams, this phenomenon increases the number of practice exams you must take to have a high probability of seeing all the questions in the pool.

 

How many random exams?

Assuming you decide to memorize the answers using random exams, just how many do you need to take?  The answer depends on how good your memory is and how confident you want to be in passing.  Some people are happy to squeak by on the second or third attempt, while others aren’t happy unless they score 100%.

Let’s assume your memory is pretty good, and you’re aiming for a score around 85%.  After all, you’re looking for the cheapest way to get your license, so you don’t want to risk paying the $15 exam fee (varies by VEC) again.

So you decide that you want to see 90% of the questions at least two times.  With a fairly good memory, as long as you cram most of your study into a few days immediately prior to the exam, you should be able to remember most of them.  Your long-term retention will be poor, but that doesn’t matter, as you’re just memorizing answers, anyway.

Because of the way the exams are generated, the easiest way to determine the solution was to write a computer program that generates many random exams using precisely the same rules used to generate the actual exams.  I averaged the results of 500 trials to eliminate the effects of random fluctuations.   For the mathematicians in the crowd, the result is a binomial distribution.

This table shows how many practice exams you need, and estimates how long it will take, assuming you spend 30 seconds per question:

Question pool	Number of exams	Questions per exam	Time per exam	Total "study" time
Technician	43	35	18 minutes	13 hours
General	56	35	18 minutes	16 hours
Extra	60	50	25 minutes	25 hours

 

In all cases, when you reach the point where 90% of the questions have been seen at least twice, there are still approximately 2% of the questions which have been seen zero times and 8% which have been seen just once.  Unfortunately, at least 1/3 of the questions have been seen six or more times!

For those aiming for a higher score, practice exams are frustrating.  The closer you get to 100%, the more it repeats questions you’ve already learned.  To understand this, consider the case where you’ve memorized 90% of the answers.  Since practice exams pick questions at random, that should mean that 90% of the questions it picks are ones you’ve already memorized.

The exam groups exacerbate the problem, especially since the largest groups tend to have the hardest questions.  You take a 35-question practice exam, and you might only see 2 questions you haven’t yet memorized.  On average, you might see one new question every 5-10 minutes.

While these statistics are useful predictors of total effort, you don’t need to count the number of practice exams you take.  You’ll know you’re ready when you consistently pass them.

 

How many to see them all?

Some people aren’t happy unless they’re sure they will get 100% on the exam, so they ask us how many exams to be guaranteed to see all the questions in the pool at least twice.  Unfortunately, there is no number which absolutely guarantees it.   It’s a matter of probabilities.

It’s like flipping a coin many times.  Flip it once, and there’s a 50% chance you’ve only seen tails.  Twice, it’s a 25% chance.  Three times, it’s 12.5%.  It keeps going like that forever, never quite reaching zero.   After 100 flips, there’s still a .00000000000000000000000000008% chance you’ve only seen tails.  There is no finite number of coin flips that absolutely guarantees you will see at least one heads.

However, it is possible to determine the number of random exams you have to take to have a 90% probability of having seen every question at least twice:

Question pool	Number of exams	Questions per exam	Time per exam	Total "study" time
Technician	118	35	18 minutes	34 hours
General	189	35	18 minutes	55 hours
Extra	199	50	25 minutes	83 hours

 

Not all created equal

Alert: As of this writing some websites are offering practice exams using expired question pools!  Those are not the questions you will see on the exam!

If you are going to use practice exams, we think HamTestOnline™ has the best. Here’s why:

• Ease of use – You’re going to be answering several thousand questions, so you want it to be easy. You don’t want to have to maneuver the mouse to click a tiny circle every time.
   
• Immediate feedback – While not knowing your score until the end more closely simulates the real exam, immediate feedback is better for achieving your memorization goal.   After you make a guess, you should see the correct answer while the question is still fresh in your mind.
   
• Save session – If you don't have time to finish the exam, you can sign in later, even from a different computer, and continue where you left off.
   
• Tracks history – It’s handy to be able to look back and review the questions you missed on prior exams.
   
• Integrated question pool display – A great way to accelerate memorization is to review the questions that give you trouble.

 

Here’s how the most popular practice exam sites stack up.  (Green is good, red is bad.)

Website	Ease of use	Immediate feedback	Save session	Tracks history	Integrated question pool
HamTestOnline™	Click anywhere on the answer.	Yes	Yes	Yes	Yes
QRZ.com	Have to click a small circle	Yes, but doesn’t show correct answer when you get it wrong	No	No	No
eHam.net	Have to click a tiny circle.	No, have to wait until the end.	No	No	No
AA9PW.com	Have to click a tiny circle.	No, have to wait until the end.	No	No	No
W8MHB.com	Have to click a tiny circle, then click another button to advance.	No, forces you to answer (not skip) all questions first.	Yes	Yes	No
KB0MGA.net	Have to click a tiny circle, then click another button to advance.	Yes	No	No	No

 

There’s one additional feature we at HamTestOnline™ like the most (because it’s how we stay in business).  At any time you can change your mind and switch to our subscription-based online courses.  And, since we've been tracking your question history all along, you don't have to start from scratch.  We know to focus on those topics and questions where you have trouble.

Hey, you might make our dream come true and actually learn something about ham radio after all!

 

Why our practice exams are the best

Let's get started!

 

(This article is partly based on a paper I published in the Journal of the Association of Computing Machinery (JACM): "Introducing Computer Simulation through Graphics," J. Cunningham, G. Summers, T. Honneycutt, 1976.)