rev 0.2 09/27 Improved coherency using brain rules throughout development process.  End deliverable is an Online Course.

CAT Storyboard.docx  rcw Storyboard including Schedule, Topic Outline, User Interface Prototype, Deliverable Structure

rev 0.1 initial. mapped subject topics to brain rules in a 1:1 fashion.

Aligning Adaptive Testing Methods with Medina's Brain Rules

I am not sure what direction you are headed for this project.  Are you going to develop a paper that researches all aspects of Adaptive testing?  You had mentioned interning with a company - but I don't know if that is possible. If not, doing as much research as possible on the topic would be the best way to do.  You might also investigate how to actually develop the questions.  I have worked with Educational Testing Service to develop questions and it is a grueling process. I would guess that developing questions for adaptive testing is even more important because you are determining a range or skillset level (not sure that is what it is called) instead of a pass or fail. I do know I kinda hate those adaptive tests because it seems like I don't know any of the answers - and then the test is over.

So - what is your plan for the semester?  How are you going to go about doing this project?  And what have you learned thus far about learning?  Not about the topic - but about the process of developing this project?

JWB

Attention: Making the topic of Adaptive Testing interesting

In adaptive testing, a user’s ability is estimated at the beginning of the test.  The supplied estimate is an irrelevant starting point and no knowledge of the user’s actual ability is needed before assessment begins.  The assessment process, called computer adaptive testing or CAT, is performed in real-time and is generally completed more quickly than a standardized test.

During the CAT process, a new estimate of the user’s ability is generated after each question is completed.  Throughout the process, the estimate converges upon the user’s actual ability, by evaluating current and prior answers.  Testing is completed when the error, or difference between the user’s actual and estimated ability, is within the limits set by the administrator.

CAT is effective at grouping incoming student’s or workers into ability categories so that training or learning experiences occur at an optimally efficient pace.  No two tests are identical, which reduces the chances of cheating.  Students' whose ability deviates from the median group ability are evaluated more accurately than in static testing.

Computer Adaptive Testing (CAT) can be expensive to implement, so it lends itself well to a large use application, such as an entire school district, an incoming class or a group of new soldiers. CAT works by using a large number of questions from which a few are administered.  These questions are called items and they are cumulatively stored in a table called an item bank.  Upon starting a test, the administrator “guesses” what the user’s ability in the topic is, usually 50% being a reasonable starting point.  After the user is administered the first question, their response is immediately analyzed by an adaptive algorithm that is based on Item Response Theory (IRT).

IRT evaluates the statistical probability of the user getting the question correct, then looks at the cumulative response to all prior questions and selects another question from the item bank.  After each response, the estimate of the user’s ability is updated and with a reliable CAT, this estimate starts converging upon the user’s actual ability.  In this manner, if a person stumbles on a few questions or gets lucky on a few by guessing right, Item Response Theory (IRT) eliminates these response bumps in a couple of iterations and continues to converge upon the user’s actual ability.

Adaptive testing generally requires fewer questions and less time than a static test to determine ability.  The test is finished when the error or the difference between estimated and actual ability reaches the administrator’s desired limit.  The test may also arbitrarily commence at a fixed number of questions at which point the administrator will still know the range of the user’s ability.

The creation and validation of the item bank requires several specialists.  An SME is needed to generate all of the test items.  A psychometrics expert is used to validate the response reliability.  An individual or company experienced in IRT is needed to setup the flow of the test questions.  An isolated test group is required to confirm test dependability.  Finally, summative evaluation, is used to correct the test until it reaches the design objective prior to release for actual use.

Vision trumps

Reliable CAT cannot be implemented without a psychometrics expert and a specialist in Item Relation Theory (IRT).  We will visually look at this complex undertaking and demonstrate how it works in a fundamental manner.

Short-term memory

There is a bit of unique subject terminology that must be understood to engage in understanding Adaptive Testing.  Terms, equations and acronyms will be explored to build your understanding.

Long-term memory

All of the terminology will be viewed from several different angles.  Although repetitive, these views will help install a building framework and understanding of Computer Adaptive Testing.

Use more senses

To date, most work in CAT has been done using text and multiple-choice assessments.  We will briefly examine the benefits and practicality of researching and developing assessments using additional senses.

Sleep

 Whose sleeping and whose getting their sleep in this business.  We will provide an overview of some major and lesser known players in this field.

No stress

What are the open source and proprietary tools used in the business?  How difficult is it to transform a developed assessment into a reliable adaptive assessment?

Male not equal female

T.B.D.

Brain wired differently

Are any of the styles of CAT used in the military, the workforce, higher education and K-12 more effective than others for each particular group.  Or does the classical Rasch model work equally well for everyone?

Explore

What is in current development in this field?  Where is the future going?  Which trends are being dropped?  We will survey ongoing commercial and academic work in this energetic and promising field.