Copyright © 2003 jsd

How to Deal with Ill-Posed Questions
John Denker

Whenever you get a question or any kind of assignment, the first step should be to ask yourself how badly ill-posed it is. Make sure all ill-posed problems are recognized as such.

1  Introduction: Types of Ill-Posed Problems

Students often complain about “story problems” (in contrast to problems where you just plug given numbers into given formulas and turn the crank, i.e. “plug-and-chug” problems).

There are lots of reasons why story problems are harder than plug-and-chug problems:

  1. Sometimes the statement of the assignment contains extraneous information, so you have to figure out what to ignore.
  2. Sometimes the statement of the assignment omits some of the crucial information, so you have to look elsewhere to find it.
  3. Sometimes it isn’t obvious what formula(s) to apply.
  4. Sometimes the assignment is ambiguous (also called underspecified, underdetermined, or underconstrained). There are multiple possible answers, depending on what assumptions you make.
  5. Sometimes the assignment is intractable. That is, an answer exists in principle, but there’s no reasonable way of finding it.
  6. Sometimes the assignment is inconsistent (also called overspecified, overdetermined, or overconstrained). No solution exists even in principle, because of requirements that are inconsistent with other requirements, or inconsistent with the laws of nature.
  7. Sometimes the assignment is not ill-posed in the technical sense, but still it is wrong, because what the customers are asking for is not what they really want.

There is also one hugely important way in which ill-posed story problems are nicer than other problems:

If you are wondering whether you have to do story problems, the answer is yes. Get used to it.

Presumably you are not planning to spend the rest of your life in school. In the real world (i.e. outside of school) you will never see a plug-and-chug problem. It’s very nearly 100% story problems, and more often than not, ill-posed story problems.

Suppose you are at work, and the boss or the customers come to you with a question. Do you really think they are going to ask you to plug given numbers into a given formula? The customers aren’t idiots; if they had the numbers and formulas they would have turned the crank themselves. They are coming to you because they want you to do something they could not immediately do themselves. They want you to use your judgment. They want you to track down the missing information and figure out what formulas to apply. If you won’t do the job, you’ll be fired and replaced by somebody who will.

An extremely underspecified task is sometimes called “a message to Garcia” – as explained in reference 1.

The most trouble comes from inconsistencies (item #6 on the list above). Presumably you’ve been taught that it’s important to follow instructions. However, in the real world you can’t always follow the instructions, because the instructions are wrong.

Dealing with ill-posed problems requires high-level thinking. This is often called critical thinking to distinguish it from lower-level forms of thought. See reference 2 for more about thinking skills and how to sharpen them.

Specific recommendations for how to deal with an ill-posed problem are presented in section 8. But first, let’s discuss some examples.

*   Contents

2  Some Simple Examples

2.1  Square Root

Suppose somebody tells me that

x · x = 9

and asks me what the numerical value of x is. Alas I cannot say what x “is”. I can say what x might be, not what it “is”. It might be 3 or it might be −3. In the absence of further information, we cannot say that either value is better than the other.

The fundamental problem here is that asking what x “is” is not a good way to frame the discussion. It would be much cleverer to say “solve for x”. Then, assuming x is a scalar, the solution is a solution-set with two elements. Let’s be clear: the solution is not a numerical value; it is a set. Specifically, x is an element of the set {−3, 3}.

If we look more closely, we see that the original question is ill-posed in yet another way. Equation 1 is underspecified, since it uses a dot product, and doesn’t specify that x is a scalar. It could be that x is a vector of length 3 in some unspecified direction. In this case the solution-set has infinitely many members. In the absence of further information, we cannot say any of them is better than the others.

Figure 1: Various Vectors of Length 3

2.2  Packing

I have been known to pose the following question as homework:

"I buy milk in ordinary standard one-gallon plastic jugs. I have a picnic cooler with inside dimensions 12"x12"x12". How many such milk jugs can I fit in the cooler?"

All too often, some of the students are incensed that I would ask such a question without telling them how big a milk jug is. Obviously you can’t answer the question without that information. They complain that I “didn’t cover that in class this week”.

I realize that you were not born knowing how big a gallon jug is. I realize the topic has not been discussed in class. It is not covered in the textbook either. But you must not whine about it. From now on, we are playing by new rules: Figure it out! FIGURE IT OUT! Your mission, should you decide to accept it, is to figure out how big a milk jug is. Maybe you don’t buy milk in one-gallon jugs. Maybe you don’t buy milk at all. But even so, it wouldn’t kill you to go to the store and measure one as it sits there on the shelf.

Figure it out.

If I had asked you explicitly to figure out how big a milk jug is, you would have been able to do it. The task here isn’t really any harder; you just need to know the rules. The rule is: If solving the question that was asked requires knowing the size of a milk jug, the question is implicitly asking you to figure that out.

To say the same thing the other way: Do not assume that everything explicitly mentioned in the question is relevant, or that everything not explicitly mentioned is irrelevant.

2.3  Oxides of Nitrogen

Here’s an example that can be understood at the level of high-school chemistry. We are given the following reaction equation:

3 O2 + 2 N2  x NO + y NO2 + z NO3

and we are asked to find x, y and z so as to balance the equation.

High-school students should be able to take one look at that and realize that it is grossly ill-posed.1

At the high-school level, the best one can do is to say that the solution-set is a one-dimensional continuum. Specifically, it is a straight line-segment embedded in a 3D space. The endpoints are:


Anything on the straight line between those two points is a solution. You can verify this by plugging into equation 2. Actually finding the solution from scratch is slightly more complicated, but only slightly.2

At the level of an undergraduate p-chem course you could perhaps find specific values for x, y, and z as a function of temperature and pressure. However, insofar as the question we were given did not specify the temperature and pressure, it remains definitely ill-posed, i.e. underdetermined. Furthermore, insofar as it doesn’t consider other possible products, it is also overdetermined. That is, the way the question is framed embodies assumptions that may not correspond to reality.

2.4  Mississippi River

Here’s an example that can be discussed without taking time to dig up additional information (such as the size of a milk jug). The assignment is to estimate how much water flows down the Mississippi river; specifically, how much water flows past New Orleans in one year.

This question works best in a one-on-one setting. It can also be used in a classroom setting, but beware that different students will solve it at wildly different rates, and it is sub-optimal if one or two students “run away” with the problem while the others are still struggling with it.

Most people, the first time they hear this question, will probably say they have no idea. They are intimidated, if not outright panicked. However, that’s no excuse. Anybody who grew up in the United States probably knows enough facts to be able to figure it out, to an order-of-magnitude approximation, without looking anything up. It requires marshaling quite a few odd bits of information, but it’s quite doable. Practically everybody concedes in retrospect that they “should” have been able to do it.

On the other hand, the fact remains that most people have a hard time figuring this out ... and unlike the milk-jug example in section 2.2, simply asking people to figure it out isn’t particularly helpful in the short run. Motivating them to try harder isn’t particularly helpful in the short run.

Solving this problem requires sifting your memory to call up several facts. Memory and recall are largely subconscious, imperfectly understood processes, involving billions of neurons all acting at the same time. This is very unlike the straightforward, single-minded, conscious process of measuring a milk jug.

You can’t get an idea out of your memory in a good way if you didn’t put it in to your memory in a good way. Being able to use your memory effectively is super important, but it is beyond the scope of the present discussion. Suffice it to say that this is not something you can learn in a day. See reference 2.

When you have answered the Mississippi flow question, you should congratulate yourself and celebrate for a moment.

Then, later, you should come back to this question, because there is an even more interesting exercise waiting for you: Set aside your original solution and find another inequivalent solution. (I know of two good mutually-independent solutions; there may be more.)

Many people find it much harder to find the second solution. The second solution is not inherently trickier; the only problem is that the first solution somehow impedes the search for additional solutions. But keep going. Remember the feeling involved. This is a good exercise for out-of-the-box thinking.

Note that the numerical answer is vastly less interesting than the methods of solution. That’s my point: methods can be discussed. Methods can be taught.

2.5  Following Instructions: Closed for Business

Once upon a time I was working with my brother in his law office. It was a Saturday, and he had his kids with him. At about 2:00 his 2-year-old daughter announces "I think it’s high time for lunch."

My brother says, "Yes, sweetie, it’s high time for lunch. Where would you like to go?"

She says, "Eegee’s. They have really good hot dogs."

So we go trundling off to Eegee’s. But when we get there, the parking lot is empty, and there’s a big sign in the door that says "CLOSED. Please come again later."

We were about to drive off, but I said "Wait a minnit. It doesn’t make sense for a joint like this to be closed at 2:00 on a Saturday. Let’s go take a look." Sure enough, the place was open. Six or eight staff were lounging behind the counter, wondering why there were so few customers.

I told the staff "It’s a good thing we didn’t follow the instructions, or you’d have no customers at all. If you turn that sign around, I’ll bet business will pick up considerably."

Over lunch we had a good discussion of the importance of not believing every sign and not following every instruction.

Of course it could have gone the other way: I can imagine walking up to the restaurant and finding it closed. “Of course it’s closed” says the bystander; “can’t you read?”

2.6  Following Instructions: Hand-Held Football

Here’s an example that was a turning point in my life, much more important than milk jugs or hot dogs.

Back when I was a junior in college I designed a hand-held electronic “Auto Race” game, as a consultant for Mattel. Then they came back and asked me to design a follow-on, namely a “Football” game.

They provided a detailed specification. The Football game had to use exactly the same hardware as the Auto Race game (to minimize production costs). The software had to fit into the same amount of memory. They specified that it would beep three times when certain game events occurred, and beep two times when other game events occurred. They even specified the pitch and the duration of the beeps. And many, many other details.

They forgot to specify that the game was supposed to be fun.

I took their specifications document and put it up on a high shelf, out of sight, and proceeded to design a game. It turns out that you can’t play Football using a steering wheel and a gearshift, so I changed the hardware. And beeping two times or three times isn’t very interesting, so I wrote a subroutine that would play music, and another one that made a tweet-tweet noise to represent the referee’s whistle. The software grew much bigger than the Auto Race software.

In short, I violated virtually every element of the specification.

In retrospect, you might think that each such decision was easy and obvious, but they weren’t easy or obvious at the time. It wasn’t obvious that it was even possible for such a low-power processor to play music. Indeed it wasn’t the least bit obvious that it would be possible to make a decent Football game. If you had asked me in advance whether it was possible to play Football using 27 little red dots and a few pushbuttons, I would have said no, it’s not possible.

After spending months working on this, I was getting quite nervous, because if they didn’t like the game, I would be in serious financial trouble, because they would refuse to pay me. I had, after all, not come anywhere close to fulfilling the contract as written. So with some trepidation I arranged for a meeting at Mattel headquarters, so their engineers and executives could see the demo for the first time.

It worked out. They liked it. After a while I noticed that they weren’t even pretending to evaluate it – they just wanted to play the game. The higher executives were elbowing the lower executives out of the way. They very quickly forgot about all those details in the specification.

I’m a firm believer in giving the customers what they want. But that means giving them what they really want, not necessarily what they say they want.

2.7  Coffee + Cream

Suppose you have some hot coffee and some cold cream. The objective is to make the coffee as cool as possible, within some time limit (say three minutes). The question is, should you add the cream to the coffee immediately, or should you wait until the last moment to add the cream?

Hint: Before mixing, while the coffee is cooling down, the cream is warming up. The relative rates are not well determined by the statement of the problem.

2.8  Snapping the Thread

Consider the system shown in figure 2. There is a massive object hanging by a thin thread, with another piece of thread (of the same type) extending below the object. Farther down is a slab of wood, so that the object, if/when it falls, does not make a dent in your desk. It may help to wrap the loose end of the thread around a stick, to serve as a handle, so it doesn’t cut into your hand. Arrange that the weight of the object is about half the breaking strength of the thread.

Figure 2: Object + Threads

To use this demo in a classroom situation, it helps to have a dummy. A crash-test dummy with Secchi-disk fids on his head works fine; see reference 3. A dunce-cap is optional. Name him Simplicio if you can’t think of a better name. If you’re not a ventriloquist, you can keep on your computer various pre-recorded utterances that Simplicio can say when needed. The point of using a dummy (rather than calling on a student) is that you do not want the class members to identify with him.

So, ask Simplicio what will happen if you pull down on the thread, as shown in the figure. Will the thread break above or below the object? Simplicio says that obviously the thread will break above the object, because the tension in the thread is higher there. The force from your hand gets added to the weight of the object.

Thereupon you put some slack in the lower thread, and then snap it downwards with a mighty jerk, breaking the lower thread against the inertia of the object. The class realizes you have just made a fool of the dummy.

Next you re-tie the thread, and ask Simplicio the same question: Will the thread break above of below the object? Simplicio, having learned his lesson, predicts it will break below the object, just like last time.

Thereupon you pull down on the lower thread, ever so gradually, until the upper thread breaks.

The moral of the story is, don’t be a dummy. Not every yes/no question has a yes/no answer.

Note: When introducing the idea of ill-posed questions, be sure to call on the dummy ... rather than asking for a show of hands or (worse) calling on a particular student. It is OK to make a fool of the dummy, but not OK to make a fool of real persons. You especially don’t want to do anything that penalizes somebody who volunteers to participate. (You would think this would be obvious, but I’ve seen it done wrong.)

2.9  Yo-Yo

Suppose we have a spool (such as a Yo-Yo) set up approximately as shown in figure 3. The outer rim of the spool rolls without slipping on the table-top.

Ask Simplicio Question #1, namely: When I pull on the string, does the spool move toward me, or away from me?

This is a seemingly-simple yes/no question, rather like the question in section 2.8. Simplicio gets it wrong, twice. Again the moral of the story is: Don’t be a dummy. Some questions are ill-posed. Learn how to deal with ill-posed questions.

Figure 3: Which Way Does The Yo-Yo Go?

One good way to deal with this situation is to re-frame the question. Specifically, it is much more interesting to ask Question #2: Under what conditions will the yo-yo move left, and under what conditions will it move right?

After you have answered Question #2, we can move on to an even more interesting question, namely Question #3: How many different ways can you find for analyzing question #2?

As discussed in reference 2, being able to analyze problems in more than one way is an important high-level thinking skill.

2.10  Barge + Trench Plates + Bridge

The following exercise comes from Chuck Britton: Suppose you are sitting in your office, and you get an urgent message from the commander of a river barge. The barge is approaching a bridge. The bridge is quite low, and the water level is unexpectedly high. On deck is a tall stack of trench plates, i.e. large flat steel plates about one inch thick. There is concern that the stack of trench plates might hit the bridge.

Figure 4: Barge + Trench Plates + Bridge

The crew has the option of throwing a few trench plates overboard. The question is, will this improve the clearance? Or will it just make things worse?

Remark: The description of the problem, as given above, is seriously underspecified. If you were aboard the barge, you could quickly obtain additional information, but sitting in your office you are missing some crucial details. This is quite typical of real-world problems: Often the problem in its original context was well-posed, but after a few stages of communication some details get lost, and the problem becomes ill-posed.

This is why standardized multiple-choice questions are so unrealistic. Not all simple yes/no questions have simple yes/no answers. In the real world, you would handle this question either by asking for more information, or by giving a nuanced answer: Under such-and-such conditions “yes”, otherwise “no”.

2.11  South – East – North

You start out at point A. You travel strictly south for one mile. You then make a 90 degree turn and travel strictly east for one mile. You then make another 90 degree turn and travel strictly north for one mile. You find that you have returned to point A.

For simplicity, we approximate the earth’s surface as spherical. You are constrained to traveling on the surface.

Part 1: Where is point A?

Part 2: How do you know? How sure are you? Can you prove it?

2.12  Sum of Two Rectangles

In figure 5, the gray region represents some raw data. We have lots and lots of data points, with very high precision. We know a priori that the gray region can be represented as the sum of two rectangles – a red rectangle and a blue rectangle. All we need to do is a simple “curve fit”, to determine the height, width, and center of the two rectangles. Two equally-perfect fits are shown in the diagram. Alas, this leaves us with tremendous uncertainty about the area, height, and center of the red and blue rectangles.

Figure 5: Fitting Two Rectangles

2.13  Light/Switch Puzzle

Here’s a puzzle:

You start out in the attic. There are three switches. It is known that there are three light bulbs in the basement. The switches control the lights, but you don’t know which goes with which. You get to make one trip to the basement to figure this out. How?

Whenever you get a question like this, the first step is to check whether it is well posed. Spoiler: This one is grossly ill-posed.

New question: Prove scientifically that it is ill-posed, i.e. that there cannot possibly be any reliable way of achieving the stated goal unless additional assumptions are made.

Now for the creative part: Come up with an interesting set of assumptions that make the goal achievable.

Example: Here’s a solution not involving much physics: Assume I have a cell phone, and I have a friend with a cell phone. I send the friend to the basement while I stay upstairs and flip switches. This is not a joke; I’ve done exactly this on many many occasions, when trying to find which circuit breaker controls which circuit.

So ... can you find some other set(s) of assumptions that lead to a solution?

Huge hint: There are some that involve a bit of physics.

2.14  Why Do Forces Always Occur in Pairs?

This question (“Why do forces always occur in pairs?”) appeared in an introductory textbook. It’s a disaster on many levels; see reference 4 for details. For one thing, it appears to be asking about causation, which is very much the wrong question; see section 3.1 for more on this.

2.15  The Barometer Story

There is a legendary story about using a barometer to measure the height of a building; see reference 5. (There seems to be a cottage industry devoted to misattributing this story, but that’s a separate issue.) The point of the story is that a question can have multiple answers. Sometimes they range from obvious to creative to eccentric to perverse. The questioner should not assume that his favorite answer, or his favorite method of obtaining the answer, is the only correct one.

3  Infamous Categories

In addition to the specific examples enumerated in section 2, it is worth paying extra attention to entire categories of questions that generally do not mean what they say.

3.1  Causation, or Not

Any question that begins with the word «Why» should raise red flags.

Very often there is no scientific answer to such a question. As discussed in reference 6, science is supposed to tell us what happens, and may or may not explain how it happens ... but the basic laws rarely if ever say «why» it happens. This is what separates physics from metaphysics and speculative philosophy.

When you see the question «Why X?», it is often necessary to transmute it into a different question:

For more on this, see reference 6.

The steps itemized here are a subset of the general procedure as discussed in section 8.

3.2  OTBE: Other Things Being Equal

Whenever you hear the phrase “other things being equal” you should suspect that the problem is ill-posed. That’s because you don’t generally know which other things are supposed to be held equal. Some examples of this, including a simple mechanical demonstration, with application to ideal gas laws, are discussed in reference 7.

3.3  Deception

Questions that are intentionally deceptive should generally be avoided at the introductory level. However, at some point it is worth spiraling back to cover this topic. Examples include:

These questions are not sincere. The questioner already knows the answers, and designed the questions to be deceptive. In a classroom situation, you never want the teacher to be the bad guy, so it is important to cast such question as coming from a third party.

Most – but not all – real-world questions are sincere. The questioner really wants to know the answer. The question, even if flawed, is the best he can do.   However, one must learn to deal with politicians, lawyers, and merchants who understand ill-posed questions all too well. Some of them specialize in tricking people into making bad decisions.

Normally, on a blueprint or circuit diagram, every component is there for a reason, and the system is trying to do what it’s supposed to do. This often helps you understand the design.   When conducting a security review, you must consider the possibility that this-or-that feature is there for all the wrong reasons, to subtly subvert the system.

Deciding when to teach students about deception requires a delicate judgment call. Oddly enough, people learn more about this in English class than they do in physics, logic, or teacher-training classes. Topics include satire, metaphor, roman-à-clef, allusion, and allegory, where the words say one thing but mean another.

4  Regularizers

In figure 5, it may be that both solutions are perfectly reasonable, plausible solutions.

On the other hand, depending on as-yet-unmentioned details, it may be that we have some reason to prefer one solution over the other, as we now discuss.

It may be that we have additional information that we can bring to bear. For example, suppose that we know a priori that the aspect ratio of each rectangle should be close to unity. Then when we do the curve fit, we can construct an objective function that contains not only the usual terms that penalize residuals that stray too far from zero, but also an additional term, namely a regularizer that penalizes aspect ratios that stray too far from unity. We find that the left-hand solution in figure 5 does a better job of minimizing this new objective function, much better than the right-hand solution. For the next level of detail, see reference 8.

Regularizers are a fairly general, fairly powerful way of making an underspecified problem less underspecified. They are a way of constructing an objective function that incorporates the knowledge that we have. This is a huge improvement over an objective function that only knows about the residuals.

If you give too much weight to the regularizer term, it can distort the fit. This is an example of a bias-variance tradeoff. Details are beyond the scope of the present discussion.

5  Approximations

Often an intractable problem can be made tractable by introducing approximations. Document whatever approximations you make.

What separates the professionals from the bush leagues is the idea of a controlled approximation.

6  Brittle Laws versus Robust Laws

In the classroom, as in life generally, there will always be tradeoffs between simplicity and accuracy. In an intro-level course, simplicity will be favored. There will never be an exact right way to make the tradeoff ... by which I mean the optimum will be partly determined by taste, and will vary from class to class and even from day to day.

  1. For starters, approximations are necessary in order to keep the students from getting brain-blisters. They can’t learn everything at once, but they have to start somewhere.
  2. Furthermore, approximations are necessary to keep the cognitive load on the teachers within reasonable bounds. It is not practical to require every physics teacher to have domain expertise in fluid dynamics and every other field to which physics ideas can be applied.

One might be tempted to just announce "Everything I will ever say is an idealization and an approximation" ... but even that isn’t quite right. Some topics in introductory physics (e.g. conservation of momentum) really are exact, so far as we know, and other parts are close enough for all practical purposes.

This creates a genuine dilemma when formulating story problems. We apply an exact law to a real-world scenario and come up with an inexact conclusion. That’s just plain tricky. Einstein (1921) wrote:

Insofern sich die Sätze der Mathematik auf die Wirklichkeit beziehen, sind sie nicht sicher, und insofern sie sicher sind, beziehen sie sich nicht auf die Wirklichkeit.

Insofar as theories of mathematics speak about reality, they are not certain, and insofar as they are certain, they do not speak about reality.

In other words:

This raises some interesting metaphysical questions about why we think the law is exact, even when all real-world applications are inexact.

This is particularly hard for students who are taking (or have just taken) high-school geometry, where it is emphasized that there is no notion of proximity or continuity when stating theorems. If you misstate the premises of a theorem even slightly, the conclusion does not hold at all. If you apply a theorem to a case where the premises don’t exactly hold, the conclusion does not hold at all.

As another manifestation of the same idea: If you remember half of your password, you won’t be able to half-way log in.

So why is it that we can approximately apply the laws of physics? We derive laws that apply to massless strings and frictionless pulleys ... but then we apply them to real-world strings and real-world pulleys, and get approximately-correct answers. This is genuinely tricky.

Part of the answer is that often there is a notion of continuity in physical situations. For example, it the thrust is misaligned from the horizontal by some small angle theta, the horizontal component of thrust will be approximately equal to the total thrust. Its magnitude will be off by an amount that is second order in theta.

This leads directly to the notion of controlled approximations as mentioned in section 5. Consider the contrast:

  a) Assume thrust is horizontal.
  b) Assume thrust is nearly horizontal.
  c) Ascertain that being slightly non-horizontal has effects that are second order in small quantities, and therefore unimportant for present purposes.

Another example:

  a) Assume massless strings and frictionless pulleys.
  b) Assume nearly-massless strings and nearly-frictionless pulleys.
  c) Ascertain that if your mass is off by 1% (due to strings) and your force is off by 1% (due to pulleys), your predicted acceleration will be off by about 2%, maybe less, because of the nice continuity properties of the F=ma law.

I really want to emphasize that this sort of continuity cannot be taken for granted. If you randomly mis-type 1% of the characters when writing a long C++ program, your program will not work right, not even approximately.

On the other hand, computers – spreadsheets in particular – are useful for teaching about continuity of physics laws. You can change the input numbers by 1% in each direction, and see how that affects the output numbers.

Some recommendations for teachers:

 1) When writing story problems, don't demand exact answers.
  1a) Bad: asking about "equal" versus "greater than".
  1b) Better: asking about "approximately equal" versus
   "substantially greater than".
 2) Similarly, don't define things that don't need to be
  2a) Bad: Define thrust to be horizontal.
  2b) Better: Say "in this idealized situation, the thrust is
   horizontal, close enough for present purposes."
 3) More generally: be clear, even at the most introductory
  levels, about
  3a) exact statements, versus
  3b) inexact statements.
 4) Be clear, even at the most introductory levels, about
  4a) "brittle" laws, i.e. laws that hold only when their
   premises hold exactly, versus
  4b) "robust" laws, i.e. laws have nice continuity properties,
   so that they can be applied to slightly messy situations.

Remember: You know which laws are brittle and which are robust, but the students weren’t born knowing that ... and this issue isn’t well covered in typical textbooks.

7  Remarks

1) If you want an interesting job, try being a manager in a research lab, “managing” a bunch of people who don’t follow instructions unless they feel like it. It’s like herding cats. These people are very, very successful, because they have good judgment about which instructions to follow and which to ignore.

2) Solving underspecified problems involves taking initiative and taking personal responsibility. Presumably you go to school because you want to learn something. Learning is your responsibility. As the instructor, my job is to help you meet that responsibility. I cannot meet it for you. Possibly some stuff you read in the textbook (and elsewhere) is erroneous. Possibly some of it is irrelevant. Certainly it is incomplete, in the sense that what you learn in school is at most 10% of what you need to function in the real world. Learning is your responsibility; all I can do is help you get started in the right direction.

You cannot escape responsibility for answering a question by finding fault with the question. Life is not a made-for-TV courtroom drama where the objective is to get off on a technicality.

3) I suspect that one of the many reasons why the Harry Potter books are so popular is their portrayal of rule-breaking, with all the negative and positive consequences it brings.

4) The idea of meditating on a kōan that has no right answer, or a multitude of equally good answers, goes back 1000 years that I know of, and may be as old as Buddhism itself.

More recently, the topic has been investigated intensively and formally. The idea of well-posed versus ill-posed can be traced back to Hadamard (reference 9) if not earlier, in the context of differential equations. A wide class of ill-posed questions can be handled using Tikhonov regularization (reference 10).

8  Recommendations

8.1  In General

  1. Whenever you get a question or any kind of assignment, the first step should be to ask yourself how badly ill-posed it is. Make sure all ill-posed problems are recognized as such.
  2. If the question omits some important information, do whatever is necessary to obtain more information. In some cases, the person who posed the question has additional information that you can easily obtain just by asking. In other cases, you will have to look elsewhere. Use your initiative.

    In some cases the question becomes an ESP exam: You have to read the mind of the questioner to figure out what was intended. Sometimes the wording of the question is unclear, and sometimes it is outright deceptive.

  3. If the question as posed permits more than one answer, see if one of the answers is “optimal” in some sense. Document what notion of optimality you are assuming.
  4. If the question as posed permits more than one answer, and there’s no clear way to choose among them, try to describe the entire solution set. At the very least, describe a few elements of the solution set, and note that other solutions exist.
  5. If the question as posed is inconsistent, impossible, perverse, or simply misguided, do your best to deflect the question. Suggest a better question. Negotiate if necessary, so that you can answer the question that should have been asked. Some examples of this sort of thing can be found in section 3.1.

    Very often you need to negotiate with the customer to make sure all the right questions are being asked. Sometimes the negotiation is very simple (“Would you like fries with that?”) ... and sometimes it can be much more complicated and sophisticated.

  6. If you need to use approximate data and/or approximate formulas, document what approximations you are using. As the questioner how much tolerance is allowed on the answer. To the extent possible, use controlled approximations. Report an estimate of the uncertainty in the final answer – unless the answer is so clearly within tolerances that worrying about the uncertainty isn’t important.
  7. In the real world there are politicians, lawyers, and merchants who understand ill-posed questions all too clearly. They specialize in tricking people into making bad decisions.
  8. Never allow yourself to be placed in a situation where you have to give a cut-and-dried answer to an ill-posed question.

8.2  Teaching About Ill-Posed Questions

  1. First introduce the idea of ill-posed questions, then give students experience dealing with such things.

    There is an elaborate mathematical formalism for dealing with ill-posed questions. Most of it is overkill for present purposes, but even so, everybody should know the basic ideas such as well-posed, solution set, underconstrained, and overconstrained.

  2. In real life, whenever encountering a new problem, the first step should be to ask how badly ill-posed the question is. Make this point explicitly. Reinforce it by giving students a steady diet of ill-posed questions.
  3. Beware that unlike real life, the end-of-chapter exercises in typical textbook are almost never ill-posed.

    Among other things, students should not assume that every variable mentioned in the question is relevant, nor assume that every variable omitted from the question is irrelevant. You can easily improve the textbook questions modifying them, by adding or removing information. For example:

    If the solution requires knowing the speed of sound in air, and information is provided in the question, you can remove it. Students can look it up, or figure it out.   Conversely, if the speed of sound is irrelevant, you can mention it in the question, and then marvel at how many students feel obliged to factor it into the solution.

  4. Please don’t tell me it is important to strip the exercises down to the bare essentials, so that students can focus on the basic physics ideas. I’m not buying it. Learning to think is infinitely more important than learning any particular physics factoid.

    I am quite serious about this: If the students cannot handle a question with extraneous information, forget about the physics and deal with the fundamental reasoning issue first. It’s by far the higher priority.

  5. Do not constrain the method of solution, except in the rarest of situations. The point is, even when the solution is unique, the method of obtaining the solution is not unique. Sometimes people behave as if there were an unwritten rule that says “Solve using the methods of this chapter” — but that is a very unwise rule, because (a) it encourages mindless equation-hunting, and (b) it penalizes creativity and insight.

    For example, suppose we have just finished a chapter on Gauss-Jordan elimination, and the assignment is to find the inverse of the following matrix. Show your work.

    0     0    0.8     0.6
    0     0    −0.6     0.8
    0     −1    0     0
    −1     0    0     0


    Now suppose a student writes down the correct answer, with no calculations at all. The “show your work” section consists of a single word.

    I would gladly accept that solution, if it’s the right word. (There are multiple acceptable words, so even this part of the exercise is underdetermined.) I would have more respect for that student than for all the others who did the gory calculations.

    Along the same lines: If students can answer the question by equation-hunting, there is nothing wrong with the students and nothing wrong with the equations. There’s probably something wrong with the question, insofar as it isn’t making the point you wanted to make, but that’s your problem, not the students’ problem. See reference 2 for more on this.

  6. Students tend to treat every question as an ESP exam. They try to read the mind of the person who posed the question, to figure out what was intended, because this is the only way to get enough information to permit a unique solution to the problem. This is not a sensible way to solve problems. In the real world, you would resolve the ambiguities by asking questions, but that is not allowed on tests, and often not allowed on homework either. And the questions might reveal that the solution is inherently non-unique and/or that the questioner is making bad assumptions. See section 2 for examples of this.

    At the very least, encourage students to document whatever assumptions they make on the homework. For example: “You said triangle, but I assume you meant right triangle, because otherwise the problem is so ill-posed as to be meaningless.”

  7. At some point, teaching about deception is important.
  8. Please don’t tell me it is important to strip the exercises down in order to «compel» the students to solve things a particular way, as e.g. Heller&Heller would have you do. I’m not buying it. Having multiple ways of solving the problem makes it possible to check the work. This is central to any notion of critical reasoning. Exercises that can be solved in only one way are very poor preparation for life in the real world. Futhermore, real science and real engineering are highly creative enterprises. Exercises that are stripped down to the point where they can only be solved one way rob the subject of all creativity. It’s as if art and music classes had been reduced to painting-by-number and turning the crank on a barrel organ.

9  For Further Reading

Elbert Hubbard,
“A Message to Garcia”
John Denker,
“Learning, Remembering, and Thinking”
John Denker,
“Secchi Disk”
John Denker,
“Review of Hewitt, Conceptual Physics
Wikipedia article: Barometer question
John Denker,
“Cause and Effect”
John Denker,
“Gas Laws”
Wikipeda article: Tikhonov regularization
Jacques Hadamard
“Sur les problèmes aux dérivées partielles et leur signification physique” (1902)
Andrey Nikolayevich Tikhonov,
“On the stability of inverse problems” (1943)

Hint: Two equations in three unknowns.
Hint: Treat z as a known, and solve for x and y as a function of z. Constrain the solution-set to non-negative values.
Copyright © 2003 jsd