Enzymatic degradation of abundant renewable polysaccharides such as cellulose and starch is a field that has the attention of both the industrial and scientific community. Most of the polysaccharide degrading enzymes are classified into several glycoside hydrolase families. They are often organized in a modular manner which includes a catalytic domain connected to one or more carbohydrate-binding modules. The carbohydrate-binding modules (CBM) have been shown to increase the proximity of the enzyme to its substrate, especially for insoluble substrates. Therefore, these modules are considered to enhance enzymatic hydrolysis. These properties have played an important role in many biotechnological applications with the aim to improve the efficiency of polysaccharide degradation. The domain organization of glycoside hydrolases (GHs) equipped with one or more CBM does vary within organisms. This review comprehensively highlights the presence of CBM as ancillary modules and explores the diversity of GHs carrying one or more of these modules that actively act either on cellulose or starch. Special emphasis is given to the cellulase and amylase distribution within the filamentous microorganisms from the genera of Streptomyces and Aspergillus that are well known to have a great capacity for secreting a wide range of these polysaccharide degrading enzyme. The potential of the CBM and other ancillary domains for the design of improved polysaccharide decomposing enzymes is discussed.