The OP mentions "less than a kazayis" as well as "crumbs". It is proper to note that the two terms are not automatically the same in Halachah. A question can be asked as to when does a small piece of chametz achieve crumb status?
There are two concerns of chametz on Pesach related to the OP(besides the prohibition of deriving benefit):
Not to own chametz.
Not to eat chametz.
The amount of chametz required to violate either prohibition is the "kazayis" (olive sized; approx. 1 ounce.) Once chametz is a kazayis in one piece/gathering, it is prohibited unless it cannot be eaten by a dog.
The Gemara in Pesachim 6a tells us that crumbs do not count on Pesach. They do not even need a "bitul" declaration.
However, the Gemara in Pesachim on 45a tells us that "Batzek b'sidkei areyvah" (pieces of dough found in the cracks of a mixing bowl) should be removed even if they are less than a kezayis.
Are these two Gemaras contradictory?
Also, the Gemara in Yoma 74a, tells us that "Chatzi shiur assur" (less than the prohibited amount is also forbidden). R' Yochanan holds this is Torah Law, but Reish Lakish says it is Rabbinic Law. Therefore, (either way) eating a crumb of chametz on Pesach would be forbidden even if it did not measure a kazayis. (the difference being that the one who eats less than a kezayis is not subject to punishment)
The question is however, if you could violate the concept of "chatzi shiur" by owning less than a kazayis of chametz? Does the prohibition of "chatzi shiur" apply to owning chametz like it applies to eating chametz? According to many authorities, this depends on how you learn the Gemara on 45a (removing dough in cracks).
The Meiri holds that you must remove pieces less than a kazayis only if they could become a kazayis if added all together (in the mixing bowl/same area). Otherwise, you need not remove them. The Shaarei Teshuvah on S.A. 442 brings the opinion of the Chacham Tzvi (which matches the Shaagas Aryeh end of Siman 81) that the idea of "chatzi shiur" does not apply to the prohibition of owning chametz (only eating chametz). Therefore, we need not remove larger bits of half cookie or pretzel which are less than a kazayis. The Toras Chesed (siman 44) goes so far as to say that less than kazayis pieces are in fact like crumbs.
But the Rif, (unlike the Meiri) on the other hand, holds that you must remove even the less than kazayis pieces because of the prohibition on owning chametz (chatzi shiur).
The Mechaber (Bes Yosef) S.A. 442:8 and 11, is himself lenient concerning removal of less than kazayis pieces. However the Magen Avraham brings the strict opinion. The Mishnah Berurah and Shulchan Aruch HaRav bring this M"A and hold that in fact we do have an obligation to remove these less than kazayis pieces.
Although these last mentioned poskim are strict, they do agree that once such less than kazayis pieces become a bit dirty, they no longer need to be removed. Apparently, such ruining grants them "crumb status". Certainly, no one suspects they will come to be willingly eaten. Therefore, as crumbs, they have no importance.
Now we can see that there is no contradiction between the Gemaras on 6a and 45a. 6a is not worried about "crumbs". But it does mention that a "gluska yaffa" (a nice piece of chametz; apparently even less than a kazayis) is a problem. 45a similarly concerns itself with viable pieces less than a kazayis that have not been demoted to crumb status yet. They are also not on the floor, and because they exist in the kitchen/mixing bowl, they could end up in food.
So now we can summarize and answer the OP:
A nice individually wrapped chametz candy, or a small pack of croutons measuring less than a kazayis would still need to be removed (if you found them). Otherwise, (if you don't find them) you rely on the "bitul" declaration.
Broken pretzel pieces and half cookies measuring less than a kezayis that are still edible and clean in good condition, should be removed. (of course the bitul helps as in #1 above.)
In #1 and #2 above, a large amount of these things in one area would be viewed as a full kazayis if added up together; even if they are individually smaller than a kazayis.
These things in #1 and #2 above may violate the concept of "chatzi shiur" on owning them (according to the strict authorities) and certainly might come to be eaten. Therefore they are a problem.
Dirty or slightly disgusting, albeit significant, pieces of half pretzel or cookie, less than a kazayis, need not be removed unless they could fall into food. (I am not sure if bitul is even needed here but it certainly works).
Crumbs need not be removed. Crumbs need no bitul. They are automatically nullified. They only need to be removed if they can fall into food.
From my own understanding of the above sugias, and what I received from my Rebbeim, even if these crumbs in #3 and #4 above were in one area and could be gathered together to form a kazayis, it still would not count as a kazayis of chametz.
Therefore, crumbs at the bottom of a drawer or on the floor in a corner, need no removal even if they are visible and easily accessed.
Some people go out of their way to remove it anyway because of chumras Pesach. They feel that such crumbs may somehow get into food, and/or just don't feel good looking at it and want to serve Hashem above the letter of the law. This is acceptable since Pesach is a time to serve Hashem with extra chumras and chametz represents the yetzer hara being removed.